chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,369 @@
|
||||
# Adaptive Crawling: Building Dynamic Knowledge That Grows on Demand
|
||||
|
||||
*Published on January 29, 2025 • 8 min read*
|
||||
|
||||
*By [unclecode](https://x.com/unclecode) • Follow me on [X/Twitter](https://x.com/unclecode) for more web scraping insights*
|
||||
|
||||
---
|
||||
|
||||
## The Knowledge Capacitor
|
||||
|
||||
Imagine a capacitor that stores energy, releasing it precisely when needed. Now imagine that for information. That's Adaptive Crawling—a term I coined to describe a fundamentally different approach to web crawling. Instead of the brute force of traditional deep crawling, we build knowledge dynamically, growing it based on queries and circumstances, like a living organism responding to its environment.
|
||||
|
||||
This isn't just another crawling optimization. It's a paradigm shift from "crawl everything, hope for the best" to "crawl intelligently, know when to stop."
|
||||
|
||||
## Why I Built This
|
||||
|
||||
I've watched too many startups burn through resources with a dangerous misconception: that LLMs make everything efficient. They don't. They make things *possible*, not necessarily *smart*. When you combine brute-force crawling with LLM processing, you're not just wasting time—you're hemorrhaging money on tokens, compute, and opportunity cost.
|
||||
|
||||
Consider this reality:
|
||||
- **Traditional deep crawling**: 500 pages → 50 useful → $15 in LLM tokens → 2 hours wasted
|
||||
- **Adaptive crawling**: 15 pages → 14 useful → $2 in tokens → 10 minutes → **7.5x cost reduction**
|
||||
|
||||
But it's not about crawling less. It's about crawling *right*.
|
||||
|
||||
## The Information Theory Foundation
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
### 🧮 **Pure Statistics, No Magic**
|
||||
|
||||
My first principle was crucial: start with classic statistical approaches. No embeddings. No LLMs. Just pure information theory:
|
||||
|
||||
```python
|
||||
# Information gain calculation - the heart of adaptive crawling
|
||||
def calculate_information_gain(new_page, knowledge_base):
|
||||
new_terms = extract_terms(new_page) - existing_terms(knowledge_base)
|
||||
overlap = calculate_overlap(new_page, knowledge_base)
|
||||
|
||||
# High gain = many new terms + low overlap
|
||||
gain = len(new_terms) / (1 + overlap)
|
||||
return gain
|
||||
```
|
||||
|
||||
This isn't regression to older methods—it's recognition that we've forgotten powerful, efficient solutions in our rush to apply LLMs everywhere.
|
||||
|
||||
</div>
|
||||
|
||||
## The A* of Web Crawling
|
||||
|
||||
Adaptive crawling implements what I call "information scenting"—like A* pathfinding but for knowledge acquisition. Each link is evaluated not randomly, but by its probability of contributing meaningful information toward answering current and future queries.
|
||||
|
||||
<div style="display: flex; align-items: center; background-color: #3f3f44; padding: 20px; margin: 20px 0; border-left: 4px solid #09b5a5;">
|
||||
<div style="font-size: 48px; margin-right: 20px;">🎯</div>
|
||||
<div>
|
||||
<strong>The Scenting Algorithm:</strong><br>
|
||||
From available links, we select those with highest information gain. It's not about following every path—it's about following the <em>right</em> paths. Like a bloodhound following the strongest scent to its target.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
## The Three Pillars of Intelligence
|
||||
|
||||
### 1. Coverage: The Breadth Sensor
|
||||
Measures how well your knowledge spans the query space. Not just "do we have pages?" but "do we have the RIGHT pages?"
|
||||
|
||||
### 2. Consistency: The Coherence Detector
|
||||
Information from multiple sources should align. When pages agree, confidence rises. When they conflict, we need more data.
|
||||
|
||||
### 3. Saturation: The Efficiency Guardian
|
||||
The most crucial metric. When new pages stop adding information, we stop crawling. Simple. Powerful. Ignored by everyone else.
|
||||
|
||||
## Real Impact: Time, Money, and Sanity
|
||||
|
||||
Let me show you what this means for your bottom line:
|
||||
|
||||
### Building a Customer Support Knowledge Base
|
||||
|
||||
**Traditional Approach:**
|
||||
```python
|
||||
# Crawl entire documentation site
|
||||
results = await crawler.crawl_bfs("https://docs.company.com", max_depth=5)
|
||||
# Result: 1,200 pages, 18 hours, $150 in API costs
|
||||
# Useful content: ~100 pages scattered throughout
|
||||
```
|
||||
|
||||
**Adaptive Approach:**
|
||||
```python
|
||||
# Grow knowledge based on actual support queries
|
||||
knowledge = await adaptive.digest(
|
||||
start_url="https://docs.company.com",
|
||||
query="payment processing errors refund policies"
|
||||
)
|
||||
# Result: 45 pages, 12 minutes, $8 in API costs
|
||||
# Useful content: 42 pages, all relevant
|
||||
```
|
||||
|
||||
**Savings: 93% time reduction, 95% cost reduction, 100% more sanity**
|
||||
|
||||
## The Dynamic Growth Pattern
|
||||
|
||||
<div style="text-align: center; padding: 40px; background-color: #1a1a1c; border: 1px dashed #3f3f44; margin: 30px 0;">
|
||||
<div style="font-size: 24px; color: #09b5a5; margin-bottom: 10px;">
|
||||
Knowledge grows like crystals in a supersaturated solution
|
||||
</div>
|
||||
<div style="color: #a3abba;">
|
||||
Add a query (seed), and relevant information crystallizes around it.<br>
|
||||
Change the query, and the knowledge structure adapts.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
This is the beauty of adaptive crawling: your knowledge base becomes a living entity that grows based on actual needs, not hypothetical completeness.
|
||||
|
||||
## Why "Adaptive"?
|
||||
|
||||
I specifically chose "Adaptive" because it captures the essence: the system adapts to what it finds. Dense technical documentation might need 20 pages for confidence. A simple FAQ might need just 5. The crawler doesn't follow a recipe—it reads the room and adjusts.
|
||||
|
||||
This is my term, my concept, and I have extensive plans for its evolution.
|
||||
|
||||
## The Progressive Roadmap
|
||||
|
||||
This is just the beginning. My roadmap for Adaptive Crawling:
|
||||
|
||||
### Phase 1 (Current): Statistical Foundation
|
||||
- Pure information theory approach
|
||||
- No dependencies on expensive models
|
||||
- Proven efficiency gains
|
||||
|
||||
### Phase 2 (Now Available): Embedding Enhancement
|
||||
- Semantic understanding layered onto statistical base
|
||||
- Still efficient, now even smarter
|
||||
- Optional, not required
|
||||
|
||||
### Phase 3 (Future): LLM Integration
|
||||
- LLMs for complex reasoning tasks only
|
||||
- Used surgically, not wastefully
|
||||
- Always with statistical foundation underneath
|
||||
|
||||
## The Efficiency Revolution
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
### 💰 **The Economics of Intelligence**
|
||||
|
||||
For a typical SaaS documentation crawl:
|
||||
|
||||
**Traditional Deep Crawling:**
|
||||
- Pages crawled: 1,000
|
||||
- Useful pages: 80
|
||||
- Time spent: 3 hours
|
||||
- LLM tokens used: 2.5M
|
||||
- Cost: $75
|
||||
- Efficiency: 8%
|
||||
|
||||
**Adaptive Crawling:**
|
||||
- Pages crawled: 95
|
||||
- Useful pages: 88
|
||||
- Time spent: 15 minutes
|
||||
- LLM tokens used: 200K
|
||||
- Cost: $6
|
||||
- Efficiency: 93%
|
||||
|
||||
**That's not optimization. That's transformation.**
|
||||
|
||||
</div>
|
||||
|
||||
## Missing the Forest for the Trees
|
||||
|
||||
The startup world has a dangerous blind spot. We're so enamored with LLMs that we forget: just because you CAN process everything with an LLM doesn't mean you SHOULD.
|
||||
|
||||
Classic NLP and statistical methods can:
|
||||
- Filter irrelevant content before it reaches LLMs
|
||||
- Identify patterns without expensive inference
|
||||
- Make intelligent decisions in microseconds
|
||||
- Scale without breaking the bank
|
||||
|
||||
Adaptive crawling proves this. It uses battle-tested information theory to make smart decisions BEFORE expensive processing.
|
||||
|
||||
## Your Knowledge, On Demand
|
||||
|
||||
```python
|
||||
# Monday: Customer asks about authentication
|
||||
auth_knowledge = await adaptive.digest(
|
||||
"https://docs.api.com",
|
||||
"oauth jwt authentication"
|
||||
)
|
||||
|
||||
# Tuesday: They ask about rate limiting
|
||||
# The crawler adapts, builds on existing knowledge
|
||||
rate_limit_knowledge = await adaptive.digest(
|
||||
"https://docs.api.com",
|
||||
"rate limiting throttling quotas"
|
||||
)
|
||||
|
||||
# Your knowledge base grows intelligently, not indiscriminately
|
||||
```
|
||||
|
||||
## The Competitive Edge
|
||||
|
||||
Companies using adaptive crawling will have:
|
||||
- **90% lower crawling costs**
|
||||
- **Knowledge bases that actually answer questions**
|
||||
- **Update cycles in minutes, not days**
|
||||
- **Happy customers who find answers fast**
|
||||
- **Engineers who sleep at night**
|
||||
|
||||
Those still using brute force? They'll wonder why their infrastructure costs keep rising while their customers keep complaining.
|
||||
|
||||
## The Embedding Evolution (Now Available!)
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
### 🧠 **Semantic Understanding Without the Cost**
|
||||
|
||||
The embedding strategy brings semantic intelligence while maintaining efficiency:
|
||||
|
||||
```python
|
||||
# Statistical strategy - great for exact terms
|
||||
config_statistical = AdaptiveConfig(
|
||||
strategy="statistical" # Default
|
||||
)
|
||||
|
||||
# Embedding strategy - understands concepts
|
||||
config_embedding = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
embedding_model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
n_query_variations=10
|
||||
)
|
||||
```
|
||||
|
||||
**The magic**: It automatically expands your query into semantic variations, maps the coverage space, and identifies gaps to fill intelligently.
|
||||
|
||||
</div>
|
||||
|
||||
### Real-World Comparison
|
||||
|
||||
<div style="display: flex; gap: 20px; margin: 20px 0;">
|
||||
<div style="flex: 1; background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px;">
|
||||
|
||||
**Query**: "authentication oauth"
|
||||
|
||||
**Statistical Strategy**:
|
||||
- Searches for exact terms
|
||||
- 12 pages crawled
|
||||
- 78% confidence
|
||||
- Fast but literal
|
||||
|
||||
</div>
|
||||
<div style="flex: 1; background-color: #1a1a1c; border: 1px solid #09b5a5; padding: 20px;">
|
||||
|
||||
**Embedding Strategy**:
|
||||
- Understands "auth", "login", "SSO"
|
||||
- 8 pages crawled
|
||||
- 92% confidence
|
||||
- Semantic comprehension
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### Detecting Irrelevance
|
||||
|
||||
One killer feature: the embedding strategy knows when to give up:
|
||||
|
||||
```python
|
||||
# Crawling Python docs with a cooking query
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/",
|
||||
query="how to make spaghetti carbonara"
|
||||
)
|
||||
|
||||
# System detects irrelevance and stops
|
||||
# Confidence: 5% (below threshold)
|
||||
# Pages crawled: 2
|
||||
# Stopped reason: "below_minimum_relevance_threshold"
|
||||
```
|
||||
|
||||
No more crawling hundreds of pages hoping to find something that doesn't exist!
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Choose your strategy
|
||||
config = AdaptiveConfig(
|
||||
strategy="embedding", # or "statistical"
|
||||
embedding_min_confidence_threshold=0.1 # Stop if irrelevant
|
||||
)
|
||||
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Watch intelligence at work
|
||||
result = await adaptive.digest(
|
||||
start_url="https://your-docs.com",
|
||||
query="your users' actual questions"
|
||||
)
|
||||
|
||||
# See the efficiency
|
||||
adaptive.print_stats()
|
||||
print(f"Found {adaptive.confidence:.0%} of needed information")
|
||||
print(f"In just {len(result.crawled_urls)} pages")
|
||||
print(f"Saving you {1000 - len(result.crawled_urls)} unnecessary crawls")
|
||||
```
|
||||
|
||||
## A Personal Note
|
||||
|
||||
I created Adaptive Crawling because I was tired of watching smart people make inefficient choices. We have incredibly powerful statistical tools that we've forgotten in our rush toward LLMs. This is my attempt to bring balance back to the Force.
|
||||
|
||||
This is not just a feature. It's a philosophy: **Grow knowledge on demand. Stop when you have enough. Save time, money, and computational resources for what really matters.**
|
||||
|
||||
## The Future is Adaptive
|
||||
|
||||
<div style="text-align: center; padding: 40px; background-color: #1a1a1c; border: 1px dashed #3f3f44; margin: 30px 0;">
|
||||
<div style="font-size: 24px; color: #09b5a5; margin-bottom: 10px;">
|
||||
Traditional Crawling: Drinking from a firehose<br>
|
||||
Adaptive Crawling: Sipping exactly what you need
|
||||
</div>
|
||||
<div style="color: #a3abba;">
|
||||
The future of web crawling isn't about processing more data.<br>
|
||||
It's about processing the <em>right</em> data.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Join me in making web crawling intelligent, efficient, and actually useful. Because in the age of information overload, the winners won't be those who collect the most data—they'll be those who collect the *right* data.
|
||||
|
||||
---
|
||||
|
||||
*Adaptive Crawling is now part of Crawl4AI. [Get started with the documentation](/core/adaptive-crawling/) or [dive into the mathematical framework](https://github.com/unclecode/crawl4ai/blob/main/PROGRESSIVE_CRAWLING.md). For updates on my work in information theory and efficient AI, follow me on [X/Twitter](https://x.com/unclecode).*
|
||||
|
||||
<style>
|
||||
/* Custom styles for this article */
|
||||
.markdown-body pre {
|
||||
background-color: #1e1e1e !important;
|
||||
border: 1px solid #3f3f44;
|
||||
}
|
||||
|
||||
.markdown-body code {
|
||||
background-color: #3f3f44;
|
||||
color: #50ffff;
|
||||
padding: 2px 6px;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
.markdown-body pre code {
|
||||
background-color: transparent;
|
||||
color: #e8e9ed;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.markdown-body blockquote {
|
||||
border-left: 4px solid #09b5a5;
|
||||
background-color: #1a1a1c;
|
||||
padding: 15px 20px;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.markdown-body h2 {
|
||||
color: #50ffff;
|
||||
border-bottom: 1px dashed #3f3f44;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
.markdown-body h3 {
|
||||
color: #09b5a5;
|
||||
}
|
||||
|
||||
.markdown-body strong {
|
||||
color: #50ffff;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,46 @@
|
||||
## Introducing Event Streams and Interactive Hooks in Crawl4AI
|
||||
|
||||
|
||||
|
||||
In the near future, I’m planning to enhance Crawl4AI’s capabilities by introducing an event stream mechanism that will give clients deeper, real-time insights into the crawling process. Today, hooks are a powerful feature at the code level—they let developers define custom logic at key points in the crawl. However, when using Crawl4AI as a service (e.g., through a Dockerized API), there isn’t an easy way to interact with these hooks at runtime.
|
||||
|
||||
**What’s Changing?**
|
||||
|
||||
I’m working on a solution that will allow the crawler to emit a continuous stream of events, updating clients on the current crawling stage, encountered pages, and any decision points. This event stream could be exposed over a standardized protocol like Server-Sent Events (SSE) or WebSockets, enabling clients to “subscribe” and listen as the crawler works.
|
||||
|
||||
**Interactivity Through Process IDs**
|
||||
|
||||
A key part of this new design is the concept of a unique process ID for each crawl session. Imagine you’re listening to an event stream that informs you:
|
||||
- The crawler just hit a certain page
|
||||
- It triggered a hook and is now pausing for instructions
|
||||
|
||||
With the event stream in place, you can send a follow-up request back to the server—referencing the unique process ID—to provide extra data, instructions, or parameters. This might include selecting which links to follow next, adjusting extraction strategies, or providing authentication tokens for a protected API. Once the crawler receives these instructions, it resumes execution with the updated context.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Server
|
||||
participant Crawler
|
||||
|
||||
Client->>Server: Start crawl request
|
||||
Server->>Crawler: Initiate crawl with Process ID
|
||||
Crawler-->>Server: Event: Page hit
|
||||
Server-->>Client: Stream: Page hit event
|
||||
Client->>Server: Instruction for Process ID
|
||||
Server->>Crawler: Update crawl with new instructions
|
||||
Crawler-->>Server: Event: Crawl completed
|
||||
Server-->>Client: Stream: Crawl completed
|
||||
```
|
||||
|
||||
**Benefits for Developers and Users**
|
||||
|
||||
1. **Fine-Grained Control**: Instead of predefining all logic upfront, you can dynamically guide the crawler in response to actual data and conditions encountered mid-crawl.
|
||||
2. **Real-Time Insights**: Monitor progress, errors, or network bottlenecks as they happen, without waiting for the entire crawl to finish.
|
||||
3. **Enhanced Collaboration**: Different team members or automated systems can watch the same crawl events and provide input, making the crawling process more adaptive and intelligent.
|
||||
|
||||
**Next Steps**
|
||||
|
||||
I’m currently exploring the best APIs, technologies, and patterns to make this vision a reality. My goal is to deliver a seamless developer experience—one that integrates with existing Crawl4AI workflows while offering new flexibility and power.
|
||||
|
||||
Stay tuned for more updates as I continue building this feature out. In the meantime, I’d love to hear any feedback or suggestions you might have to help shape this interactive, event-driven future of web crawling with Crawl4AI.
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
# The LLM Context Protocol: Why Your AI Assistant Needs Memory, Reasoning, and Examples
|
||||
|
||||
*Published on January 24, 2025 • 8 min read*
|
||||
|
||||
---
|
||||
|
||||
## The Problem with Teaching Robots to Code
|
||||
|
||||
Picture this: You hand someone a dictionary and ask them to write poetry. They know every word, its spelling, its definition—but do they know how words dance together? How certain combinations evoke emotion while others fall flat? This is exactly what we're doing when we throw API documentation at our AI assistants and expect magic.
|
||||
|
||||
I've spent countless hours watching my AI coding assistant struggle with my own library, Crawl4AI. Despite feeding it comprehensive documentation, it would generate code that was *technically* correct but practically useless. Like a tourist speaking from a phrasebook—grammatically sound, culturally tone-deaf.
|
||||
|
||||
## Enter the Three-Dimensional Context Protocol
|
||||
|
||||
What if, instead of dumping information, we provided *wisdom*? Not just the "what," but the "how" and "why"? This led me to develop what I call the **LLM Context Protocol**—a structured approach that mirrors how humans actually master libraries.
|
||||
|
||||
Think of it as HTTP for AI context. Just as HTTP doesn't dictate your website's content but provides a reliable structure for communication, this protocol doesn't prescribe *how* you write your documentation—it provides a framework for *what* your AI needs to truly understand your code.
|
||||
|
||||
### The Three Pillars of Library Wisdom
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
#### 🧠 **Memory: The Foundation**
|
||||
```markdown
|
||||
# AsyncWebCrawler.arun() - Memory Context
|
||||
|
||||
## Signature
|
||||
async def arun(
|
||||
url: str,
|
||||
config: CrawlerConfig = None,
|
||||
session_id: str = None,
|
||||
**kwargs
|
||||
) -> CrawlResult
|
||||
|
||||
## Parameters
|
||||
- url: Target URL to crawl
|
||||
- config: Optional configuration object
|
||||
- session_id: Optional session identifier for caching
|
||||
...
|
||||
```
|
||||
|
||||
This is your API reference—the facts, the parameters, the return types. It's the easiest part to generate and, ironically, the least useful in isolation. It's like memorizing a dictionary without understanding grammar.
|
||||
|
||||
</div>
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
#### 🎯 **Reasoning: The Soul**
|
||||
```markdown
|
||||
# AsyncWebCrawler Design Philosophy - Reasoning Context
|
||||
|
||||
## Why Async-First Architecture?
|
||||
|
||||
Crawl4AI uses AsyncWebCrawler as its primary interface because modern web
|
||||
scraping demands concurrency. Here's the thinking:
|
||||
|
||||
1. **Network I/O is slow**: Waiting synchronously wastes 90% of execution time
|
||||
2. **Modern sites are complex**: Multiple resources load in parallel
|
||||
3. **Scale matters**: You're rarely crawling just one page
|
||||
|
||||
## When to Use Session Management
|
||||
|
||||
Session management isn't just about performance—it's about appearing human:
|
||||
- Use sessions when crawling multiple pages from the same domain
|
||||
- Reuse browser contexts to maintain cookies and local storage
|
||||
- But don't overdo it: too long sessions look suspicious
|
||||
|
||||
## The Cache Strategy Decision Tree
|
||||
if static_content and infrequent_updates:
|
||||
use_cache_mode('read_write')
|
||||
elif dynamic_content and real_time_needed:
|
||||
use_cache_mode('bypass')
|
||||
else:
|
||||
use_cache_mode('read_only') # Safe default
|
||||
```
|
||||
|
||||
This is where the library creator's philosophy lives. It's not just *what* the library does, but *why* it does it that way. This is the hardest part to write because it requires genuine understanding—and it's a red flag when a library lacks it.
|
||||
|
||||
</div>
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
#### 💻 **Examples: The Practice**
|
||||
```python
|
||||
# Crawling with JavaScript execution
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="css:.lazy-loaded-content"
|
||||
)
|
||||
|
||||
# Extracting structured data with CSS selectors
|
||||
result = await crawler.arun(
|
||||
url="https://shop.example.com",
|
||||
extraction_strategy=CSSExtractionStrategy({
|
||||
"prices": "span.price::text",
|
||||
"titles": "h2.product-title::text"
|
||||
})
|
||||
)
|
||||
|
||||
# Session-based crawling with custom headers
|
||||
async with crawler:
|
||||
result1 = await crawler.arun(url1, session_id="product_scan")
|
||||
result2 = await crawler.arun(url2, session_id="product_scan")
|
||||
```
|
||||
|
||||
Pure code. No fluff. Just patterns in action. Because sometimes, you just need to see how it's done.
|
||||
|
||||
</div>
|
||||
|
||||
## Why This Matters (Especially for Smaller LLMs)
|
||||
|
||||
Here's the thing about AI assistants: the smaller ones can't think their way out of a paper bag. They're like eager interns—full of potential but needing clear guidance. When you rely on a large language model to "figure it out" from raw API docs, you're asking it to reinvent your library's philosophy from scratch. Every. Single. Time.
|
||||
|
||||
By providing structured context across these three dimensions, we're not just documenting—we're teaching. We're transferring not just knowledge, but wisdom.
|
||||
|
||||
## The Cultural DNA of Your Library
|
||||
|
||||
<div style="display: flex; align-items: center; background-color: #3f3f44; padding: 20px; margin: 20px 0; border-left: 4px solid #09b5a5;">
|
||||
<div style="font-size: 48px; margin-right: 20px;">🧬</div>
|
||||
<div>
|
||||
<strong>Your library's reasoning is its cultural DNA.</strong><br>
|
||||
It reflects your taste, your architectural decisions, your opinions about how things should be done. A library without reasoning is like a recipe without techniques—sure, you have the ingredients, but good luck making something edible.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Think about it: When you learn a new library, what are you really after? You want mastery. And mastery comes from understanding:
|
||||
- **Memory** tells you what's possible
|
||||
- **Reasoning** tells you what's sensible
|
||||
- **Examples** show you what's practical
|
||||
|
||||
Together, they create wisdom.
|
||||
|
||||
## Beyond Manual Documentation
|
||||
|
||||
Now, here's where it gets interesting. I didn't hand-craft thousands of lines of structured documentation for Crawl4AI. Who has that kind of time? Instead, I built a tool that:
|
||||
|
||||
1. Analyzes your codebase
|
||||
2. Extracts API signatures and structures (Memory)
|
||||
3. Identifies patterns and architectural decisions (Reasoning)
|
||||
4. Collects real-world usage from tests and examples (Examples)
|
||||
5. Generates structured LLM context files
|
||||
|
||||
The beauty? This tool is becoming part of Crawl4AI itself. Because if we're going to revolutionize how AI understands our code, we might as well automate it.
|
||||
|
||||
## The Protocol, Not the Prescription
|
||||
|
||||
Remember: this is a protocol, not a prescription. Just as HTTP doesn't tell you what website to build, the LLM Context Protocol doesn't dictate your documentation style. It simply says:
|
||||
|
||||
> "If you want an AI to truly understand your library, it needs three things: facts, philosophy, and patterns."
|
||||
|
||||
How you deliver those is up to you. The protocol just ensures nothing important gets lost in translation.
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
Curious about implementing this for your own library? The context generation tool will be open-sourced as part of Crawl4AI. If you're interested in early access or want to discuss the approach, drop me a DM on X [@unclecode](https://twitter.com/unclecode).
|
||||
|
||||
Because let's face it: if we're going to live in a world where AI writes half our code, we might as well teach it properly.
|
||||
|
||||
---
|
||||
|
||||
## A Final Thought
|
||||
|
||||
<div style="text-align: center; padding: 40px; background-color: #1a1a1c; border: 1px dashed #3f3f44; margin: 30px 0;">
|
||||
<div style="font-size: 24px; color: #09b5a5; margin-bottom: 10px;">
|
||||
Memory + Reasoning + Examples = Wisdom
|
||||
</div>
|
||||
<div style="color: #a3abba;">
|
||||
And wisdom, not information, is what makes great developers—human or artificial.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
*Want to see this in action? Check out the [Crawl4AI LLM Context Builder](/core/llmtxt/) and experience the difference structured context makes.*
|
||||
|
||||
<style>
|
||||
/* Custom styles for this article */
|
||||
.markdown-body pre {
|
||||
background-color: #1e1e1e !important;
|
||||
border: 1px solid #3f3f44;
|
||||
}
|
||||
|
||||
.markdown-body code {
|
||||
background-color: #3f3f44;
|
||||
color: #50ffff;
|
||||
padding: 2px 6px;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
.markdown-body pre code {
|
||||
background-color: transparent;
|
||||
color: #e8e9ed;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.markdown-body blockquote {
|
||||
border-left: 4px solid #09b5a5;
|
||||
background-color: #1a1a1c;
|
||||
padding: 15px 20px;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.markdown-body h2 {
|
||||
color: #50ffff;
|
||||
border-bottom: 1px dashed #3f3f44;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
.markdown-body h3 {
|
||||
color: #09b5a5;
|
||||
}
|
||||
|
||||
.markdown-body strong {
|
||||
color: #50ffff;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,355 @@
|
||||
# Solving the Virtual Scroll Puzzle: How Crawl4AI Captures What Others Miss
|
||||
|
||||
*Published on June 29, 2025 • 10 min read*
|
||||
|
||||
*By [unclecode](https://x.com/unclecode) • Follow me on [X/Twitter](https://x.com/unclecode) for more web scraping insights*
|
||||
|
||||
---
|
||||
|
||||
## The Invisible Content Crisis
|
||||
|
||||
You know that feeling when you're scrolling through Twitter, and suddenly realize you can't scroll back to that brilliant tweet from an hour ago? It's not your browser being quirky—it's virtual scrolling at work. And if this frustrates you as a user, imagine being a web scraper trying to capture all those tweets.
|
||||
|
||||
Here's the dirty secret of modern web development: **most of the content you see doesn't actually exist**.
|
||||
|
||||
Let me explain. Open Twitter right now and scroll for a bit. Now inspect the DOM. You'll find maybe 20-30 tweet elements, yet you just scrolled past hundreds. Where did they go? They were never really there—just temporary ghosts passing through a revolving door of DOM elements.
|
||||
|
||||
This is virtual scrolling, and it's everywhere: Twitter, Instagram, LinkedIn, Reddit, data tables, analytics dashboards. It's brilliant for performance but catastrophic for traditional web scraping.
|
||||
|
||||
## The Great DOM Disappearing Act
|
||||
|
||||
Let's visualize what's happening:
|
||||
|
||||
```
|
||||
Traditional Infinite Scroll: Virtual Scroll:
|
||||
┌─────────────┐ ┌─────────────┐
|
||||
│ Item 1 │ │ Item 11 │ ← Items 1-10? Gone.
|
||||
│ Item 2 │ │ Item 12 │ ← Only what's visible
|
||||
│ ... │ │ Item 13 │ exists in the DOM
|
||||
│ Item 10 │ │ Item 14 │
|
||||
│ Item 11 NEW │ │ Item 15 │
|
||||
│ Item 12 NEW │ └─────────────┘
|
||||
└─────────────┘
|
||||
DOM: 12 items & growing DOM: Always ~5 items
|
||||
```
|
||||
|
||||
Traditional scrapers see this and capture... 5 items. Out of thousands. It's like trying to photograph a train by taking a picture of one window.
|
||||
|
||||
## Why Virtual Scroll Broke Everything
|
||||
|
||||
When I first encountered this with Crawl4AI, I thought it was a bug. My scraper would perfectly capture the initial tweets, but scrolling did... nothing. The DOM element count stayed constant. The HTML size barely changed. Yet visually, new content kept appearing.
|
||||
|
||||
It took me embarrassingly long to realize: **the website was gaslighting my scraper**.
|
||||
|
||||
Virtual scroll is deceptively simple:
|
||||
1. Keep only visible items in DOM (usually 10-30 elements)
|
||||
2. As user scrolls down, remove top items, add bottom items
|
||||
3. As user scrolls up, remove bottom items, add top items
|
||||
4. Maintain the illusion of a continuous list
|
||||
|
||||
For users, it's seamless. For scrapers, it's a nightmare. Traditional approaches fail because:
|
||||
- `document.scrollingElement.scrollHeight` lies to you
|
||||
- Waiting for new elements is futile—they replace, not append
|
||||
- Screenshots only capture the current viewport
|
||||
- Even browser automation tools get fooled
|
||||
|
||||
## The Three-State Solution
|
||||
|
||||
After much experimentation (and several cups of coffee), I realized we needed to think differently. Instead of fighting virtual scroll, we needed to understand it. This led to identifying three distinct scrolling behaviors:
|
||||
|
||||
### State 1: No Change (The Stubborn Page)
|
||||
```javascript
|
||||
scroll() → same content → continue trying
|
||||
```
|
||||
The page doesn't react to scrolling. Either we've hit the end, or it's not a scrollable container.
|
||||
|
||||
### State 2: Appending (The Traditional Friend)
|
||||
```javascript
|
||||
scroll() → old content + new content → all good!
|
||||
```
|
||||
Classic infinite scroll. New content appends to existing content. Our traditional tools work fine here.
|
||||
|
||||
### State 3: Replacing (The Trickster)
|
||||
```javascript
|
||||
scroll() → completely different content → capture everything!
|
||||
```
|
||||
Virtual scroll detected! Content is being replaced. This is where our new magic happens.
|
||||
|
||||
## Introducing VirtualScrollConfig
|
||||
|
||||
Here's how Crawl4AI solves this puzzle:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, VirtualScrollConfig, CrawlerRunConfig
|
||||
|
||||
# Configure virtual scroll handling
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="#timeline", # What to scroll
|
||||
scroll_count=30, # How many times
|
||||
scroll_by="container_height", # How much each time
|
||||
wait_after_scroll=0.5 # Pause for content to load
|
||||
)
|
||||
|
||||
# Use it in your crawl
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://twitter.com/search?q=AI",
|
||||
config=config
|
||||
)
|
||||
# result.html now contains ALL tweets, not just visible ones!
|
||||
```
|
||||
|
||||
But here's where it gets clever...
|
||||
|
||||
## The Magic Behind the Scenes
|
||||
|
||||
When Crawl4AI encounters a virtual scroll container, it:
|
||||
|
||||
1. **Takes a snapshot** of the initial HTML
|
||||
2. **Scrolls** by the configured amount
|
||||
3. **Waits** for the DOM to update
|
||||
4. **Compares** the new HTML with the previous
|
||||
5. **Detects** which of our three states we're in
|
||||
6. **For State 3** (virtual scroll), stores the HTML chunk
|
||||
7. **Repeats** until done
|
||||
8. **Merges** all chunks intelligently
|
||||
|
||||
The merging is crucial. We can't just concatenate HTML—we'd get duplicates. Instead, we:
|
||||
- Parse each chunk into elements
|
||||
- Create fingerprints using normalized text
|
||||
- Keep only unique elements
|
||||
- Maintain the original order
|
||||
- Return clean, complete HTML
|
||||
|
||||
## Real-World Example: Capturing Twitter Threads
|
||||
|
||||
Let's see this in action with a real Twitter thread:
|
||||
|
||||
```python
|
||||
async def capture_twitter_thread():
|
||||
# Configure for Twitter's specific behavior
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="[data-testid='primaryColumn']",
|
||||
scroll_count=50, # Enough for long threads
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=1.0 # Twitter needs time to load
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config,
|
||||
# Also extract structured data
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o-mini",
|
||||
schema={
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"author": {"type": "string"},
|
||||
"content": {"type": "string"},
|
||||
"timestamp": {"type": "string"},
|
||||
"replies": {"type": "integer"},
|
||||
"retweets": {"type": "integer"},
|
||||
"likes": {"type": "integer"}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://twitter.com/elonmusk/status/...",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Parse the extracted tweets
|
||||
import json
|
||||
tweets = json.loads(result.extracted_content)
|
||||
|
||||
print(f"Captured {len(tweets)} tweets from the thread")
|
||||
for tweet in tweets[:5]:
|
||||
print(f"@{tweet['author']}: {tweet['content'][:100]}...")
|
||||
```
|
||||
|
||||
## Performance Insights
|
||||
|
||||
During testing, we achieved remarkable results:
|
||||
|
||||
| Site | Without Virtual Scroll | With Virtual Scroll | Improvement |
|
||||
|------|------------------------|---------------------|-------------|
|
||||
| Twitter Timeline | 10 tweets | 490 tweets | **49x** |
|
||||
| Instagram Grid | 12 posts | 999 posts | **83x** |
|
||||
| LinkedIn Feed | 5 posts | 200 posts | **40x** |
|
||||
| Reddit Comments | 25 comments | 500 comments | **20x** |
|
||||
|
||||
The best part? It's automatic. If the page doesn't use virtual scroll, Crawl4AI handles it normally. No configuration changes needed.
|
||||
|
||||
## When to Use Virtual Scroll
|
||||
|
||||
Use `VirtualScrollConfig` when:
|
||||
- ✅ Scrolling seems to "eat" previous content
|
||||
- ✅ DOM element count stays suspiciously constant
|
||||
- ✅ You're scraping Twitter, Instagram, LinkedIn, Reddit
|
||||
- ✅ Working with modern data tables or dashboards
|
||||
- ✅ Traditional scrolling captures only a fraction of content
|
||||
|
||||
Don't use it when:
|
||||
- ❌ Content accumulates normally (use `scan_full_page` instead)
|
||||
- ❌ Page has no scrollable containers
|
||||
- ❌ You only need the initially visible content
|
||||
- ❌ Working with static or traditionally paginated sites
|
||||
|
||||
## Advanced Techniques
|
||||
|
||||
### Handling Mixed Content
|
||||
|
||||
Some sites mix approaches—featured content stays while regular content virtualizes:
|
||||
|
||||
```python
|
||||
# News site with pinned articles + virtual scroll feed
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector=".main-feed", # Only the feed scrolls virtually
|
||||
scroll_count=30,
|
||||
scroll_by="container_height"
|
||||
)
|
||||
|
||||
# Featured articles remain throughout the crawl
|
||||
# Regular articles are captured via virtual scroll
|
||||
```
|
||||
|
||||
### Optimizing Performance
|
||||
|
||||
```python
|
||||
# Fast scrolling for simple content
|
||||
fast_config = VirtualScrollConfig(
|
||||
container_selector="#feed",
|
||||
scroll_count=100,
|
||||
scroll_by=500, # Fixed pixels for speed
|
||||
wait_after_scroll=0.1 # Minimal wait
|
||||
)
|
||||
|
||||
# Careful scrolling for complex content
|
||||
careful_config = VirtualScrollConfig(
|
||||
container_selector=".timeline",
|
||||
scroll_count=50,
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=1.5 # More time for lazy loading
|
||||
)
|
||||
```
|
||||
|
||||
### Debugging Virtual Scroll
|
||||
|
||||
Want to see it in action? Set `headless=False`:
|
||||
|
||||
```python
|
||||
browser_config = BrowserConfig(headless=False)
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Watch the magic happen!
|
||||
result = await crawler.arun(url="...", config=config)
|
||||
```
|
||||
|
||||
## The Technical Deep Dive
|
||||
|
||||
For the curious, here's how our deduplication works:
|
||||
|
||||
```javascript
|
||||
// Simplified version of our deduplication logic
|
||||
function createFingerprint(element) {
|
||||
const text = element.innerText
|
||||
.toLowerCase()
|
||||
.replace(/[\s\W]/g, ''); // Remove spaces and symbols
|
||||
return text;
|
||||
}
|
||||
|
||||
function mergeChunks(chunks) {
|
||||
const seen = new Set();
|
||||
const unique = [];
|
||||
|
||||
for (const chunk of chunks) {
|
||||
const elements = parseHTML(chunk);
|
||||
for (const element of elements) {
|
||||
const fingerprint = createFingerprint(element);
|
||||
if (!seen.has(fingerprint)) {
|
||||
seen.add(fingerprint);
|
||||
unique.push(element);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return unique;
|
||||
}
|
||||
```
|
||||
|
||||
Simple, but effective. We normalize text to catch duplicates even with slight HTML differences.
|
||||
|
||||
## What This Means for Web Scraping
|
||||
|
||||
Virtual scroll support in Crawl4AI represents a paradigm shift. We're no longer limited to what's immediately visible or what traditional scrolling reveals. We can now capture the full content of virtually any modern website.
|
||||
|
||||
This opens new possibilities:
|
||||
- **Complete social media analysis**: Every tweet, every comment, every reaction
|
||||
- **Comprehensive data extraction**: Full tables, complete lists, entire feeds
|
||||
- **Historical research**: Capture entire timelines, not just recent posts
|
||||
- **Competitive intelligence**: See everything your competitors are showing their users
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
Ready to capture what others miss? Here's a complete example to get you started:
|
||||
|
||||
```python
|
||||
# Save this as virtual_scroll_demo.py
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig
|
||||
|
||||
async def main():
|
||||
# Configure virtual scroll
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="#main-content", # Adjust for your target
|
||||
scroll_count=20,
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=0.5
|
||||
)
|
||||
|
||||
# Set up the crawler
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config,
|
||||
verbose=True # See what's happening
|
||||
)
|
||||
|
||||
# Crawl and capture everything
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/feed", # Your target URL
|
||||
config=config
|
||||
)
|
||||
|
||||
print(f"Captured {len(result.html)} characters of content")
|
||||
print(f"Found {result.html.count('article')} articles") # Adjust selector
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Conclusion: The Future is Already Here
|
||||
|
||||
Virtual scrolling was supposed to be the end of comprehensive web scraping. Instead, it became the catalyst for smarter, more sophisticated tools. With Crawl4AI's virtual scroll support, we're not just keeping up with modern web development—we're staying ahead of it.
|
||||
|
||||
The web is evolving, becoming more dynamic, more efficient, and yes, more challenging to scrape. But with the right tools and understanding, every challenge becomes an opportunity.
|
||||
|
||||
Welcome to the future of web scraping. Welcome to a world where virtual scroll is no longer a barrier, but just another feature we handle seamlessly.
|
||||
|
||||
---
|
||||
|
||||
## Learn More
|
||||
|
||||
- 📖 [Virtual Scroll Documentation](https://docs.crawl4ai.com/advanced/virtual-scroll) - Complete API reference and configuration options
|
||||
- 💻 [Interactive Examples](https://docs.crawl4ai.com/examples/virtual_scroll_example.py) - Try it yourself with our test server
|
||||
- 🚀 [Get Started with Crawl4AI](https://docs.crawl4ai.com/core/quickstart) - Full installation and setup guide
|
||||
- 🤝 [Join our Community](https://github.com/unclecode/crawl4ai) - Share your experiences and get help
|
||||
|
||||
*Have you encountered virtual scroll challenges? How did you solve them? Share your story in our [GitHub discussions](https://github.com/unclecode/crawl4ai/discussions)!*
|
||||
@@ -0,0 +1,90 @@
|
||||
# Crawl4AI Blog
|
||||
|
||||
Welcome to the Crawl4AI blog! Here you'll find detailed release notes, technical insights, and updates about the project. Whether you're looking for the latest improvements or want to dive deep into web crawling techniques, this is the place.
|
||||
|
||||
## Featured Articles
|
||||
|
||||
### [When to Stop Crawling: The Art of Knowing "Enough"](articles/adaptive-crawling-revolution.md)
|
||||
*January 29, 2025*
|
||||
|
||||
Traditional crawlers are like tourists with unlimited time—they'll visit every street, every alley, every dead end. But what if your crawler could think like a researcher with a deadline? Discover how Adaptive Crawling revolutionizes web scraping by knowing when to stop. Learn about the three-layer intelligence system that evaluates coverage, consistency, and saturation to build focused knowledge bases instead of endless page collections.
|
||||
|
||||
[Read the full article →](articles/adaptive-crawling-revolution.md)
|
||||
|
||||
### [The LLM Context Protocol: Why Your AI Assistant Needs Memory, Reasoning, and Examples](articles/llm-context-revolution.md)
|
||||
*January 24, 2025*
|
||||
|
||||
Ever wondered why your AI coding assistant struggles with your library despite comprehensive documentation? This article introduces the three-dimensional context protocol that transforms how AI understands code. Learn why memory, reasoning, and examples together create wisdom—not just information.
|
||||
|
||||
[Read the full article →](articles/llm-context-revolution.md)
|
||||
|
||||
## Latest Release
|
||||
|
||||
### [Crawl4AI v0.9.1 – Bug Fixes & PruningContentFilter Whitelist](../blog/release-v0.9.1.md)
|
||||
*July 2026*
|
||||
|
||||
Crawl4AI v0.9.1 is a patch release with 12 bug fixes and a new `preserve_classes`/`preserve_tags` whitelist for PruningContentFilter.
|
||||
|
||||
Key highlights:
|
||||
- **🏷️ PruningContentFilter Whitelist**: Protect specific CSS classes or HTML tags from density-based pruning
|
||||
- **🐛 12 Bug Fixes**: Docker auth gate UI, Windows browser crash, HTTP timeout unit mismatch, and more
|
||||
- **📦 Dependency**: lxml ceiling widened to allow 6.x
|
||||
|
||||
[Read full release notes →](../blog/release-v0.9.1.md)
|
||||
|
||||
## Recent Releases
|
||||
|
||||
### [Crawl4AI v0.8.5 – Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes](../blog/release-v0.8.5.md)
|
||||
*March 2026*
|
||||
|
||||
Crawl4AI v0.8.5 is the biggest release since v0.8.0, bringing automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.
|
||||
|
||||
[Read full release notes →](../blog/release-v0.8.5.md)
|
||||
|
||||
### [Crawl4AI v0.8.0 – Crash Recovery & Prefetch Mode](../blog/release-v0.8.0.md)
|
||||
*January 2026*
|
||||
|
||||
Crawl4AI v0.8.0 introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.
|
||||
|
||||
Key highlights:
|
||||
- **🔄 Deep Crawl Crash Recovery**: `on_state_change` callback for real-time state persistence, `resume_state` to continue from checkpoints
|
||||
- **⚡ Prefetch Mode**: `prefetch=True` for 5-10x faster URL discovery, perfect for two-phase crawling patterns
|
||||
- **🔒 Security Fixes**: Hooks disabled by default, `file://` URLs blocked on Docker API, `__import__` removed from sandbox
|
||||
|
||||
[Read full release notes →](../blog/release-v0.8.0.md)
|
||||
|
||||
### [Crawl4AI v0.7.8 – Stability & Bug Fix Release](../blog/release-v0.7.8.md)
|
||||
*December 2025*
|
||||
|
||||
Crawl4AI v0.7.8 is a focused stability release addressing 11 bugs reported by the community. Fixes for Docker deployments, LLM extraction, URL handling, and dependency compatibility.
|
||||
|
||||
Key highlights:
|
||||
- **🐳 Docker API Fixes**: ContentRelevanceFilter deserialization, ProxyConfig serialization, cache folder permissions
|
||||
- **🤖 LLM Improvements**: Configurable rate limiter backoff, HTML input format support
|
||||
- **📦 Dependencies**: Replaced deprecated PyPDF2 with pypdf, Pydantic v2 ConfigDict compatibility
|
||||
|
||||
[Read full release notes →](../blog/release-v0.7.8.md)
|
||||
|
||||
---
|
||||
|
||||
## Older Releases
|
||||
|
||||
| Version | Date | Highlights |
|
||||
|---------|------|------------|
|
||||
| [v0.7.7](../blog/release-v0.7.7.md) | November 2025 | Self-hosting platform, real-time monitoring, smart browser pool |
|
||||
| [v0.7.6](../blog/release-v0.7.6.md) | October 2025 | Webhook infrastructure, reliable delivery, custom auth |
|
||||
| [v0.7.5](../blog/release-v0.7.5.md) | September 2025 | Docker Hooks System, enhanced LLM integration, HTTPS preservation |
|
||||
| [v0.7.4](../blog/release-v0.7.4.md) | August 2025 | LLM-powered table extraction, performance improvements |
|
||||
| [v0.7.3](../blog/release-v0.7.3.md) | July 2025 | Undetected browser, multi-URL config, memory monitoring |
|
||||
| [v0.7.1](../blog/release-v0.7.1.md) | June 2025 | Bug fixes and stability improvements |
|
||||
| [v0.7.0](../blog/release-v0.7.0.md) | May 2025 | Adaptive crawling, virtual scroll, link analysis |
|
||||
|
||||
## Project History
|
||||
|
||||
Curious about how Crawl4AI has evolved? Check out our [complete changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for a detailed history of all versions and updates.
|
||||
|
||||
## Stay Updated
|
||||
|
||||
- Star us on [GitHub](https://github.com/unclecode/crawl4ai)
|
||||
- Follow [@unclecode](https://twitter.com/unclecode) on Twitter
|
||||
- Join our community discussions on GitHub
|
||||
@@ -0,0 +1,144 @@
|
||||
# Crawl4AI Blog
|
||||
|
||||
Welcome to the Crawl4AI blog! Here you'll find detailed release notes, technical insights, and updates about the project. Whether you're looking for the latest improvements or want to dive deep into web crawling techniques, this is the place.
|
||||
|
||||
## Featured Articles
|
||||
|
||||
### [When to Stop Crawling: The Art of Knowing "Enough"](articles/adaptive-crawling-revolution.md)
|
||||
*January 29, 2025*
|
||||
|
||||
Traditional crawlers are like tourists with unlimited time—they'll visit every street, every alley, every dead end. But what if your crawler could think like a researcher with a deadline? Discover how Adaptive Crawling revolutionizes web scraping by knowing when to stop. Learn about the three-layer intelligence system that evaluates coverage, consistency, and saturation to build focused knowledge bases instead of endless page collections.
|
||||
|
||||
[Read the full article →](articles/adaptive-crawling-revolution.md)
|
||||
|
||||
### [The LLM Context Protocol: Why Your AI Assistant Needs Memory, Reasoning, and Examples](articles/llm-context-revolution.md)
|
||||
*January 24, 2025*
|
||||
|
||||
Ever wondered why your AI coding assistant struggles with your library despite comprehensive documentation? This article introduces the three-dimensional context protocol that transforms how AI understands code. Learn why memory, reasoning, and examples together create wisdom—not just information.
|
||||
|
||||
[Read the full article →](articles/llm-context-revolution.md)
|
||||
|
||||
## Latest Release
|
||||
|
||||
Here’s the blog index entry for **v0.6.0**, written to match the exact tone and structure of your previous entries:
|
||||
|
||||
---
|
||||
|
||||
### [Crawl4AI v0.6.0 – World-Aware Crawling, Pre-Warmed Browsers, and the MCP API](releases/0.6.0.md)
|
||||
*April 23, 2025*
|
||||
|
||||
Crawl4AI v0.6.0 is our most powerful release yet. This update brings major architectural upgrades including world-aware crawling (set geolocation, locale, and timezone), real-time traffic capture, and a memory-efficient crawler pool with pre-warmed pages.
|
||||
|
||||
The Docker server now exposes a full-featured MCP socket + SSE interface, supports streaming, and comes with a new Playground UI. Plus, table extraction is now native, and the new stress-test framework supports crawling 1,000+ URLs.
|
||||
|
||||
Other key changes:
|
||||
|
||||
* Native support for `result.media["tables"]` to export DataFrames
|
||||
* Full network + console logs and MHTML snapshot per crawl
|
||||
* Browser pooling and pre-warming for faster cold starts
|
||||
* New streaming endpoints via MCP API and Playground
|
||||
* Robots.txt support, proxy rotation, and improved session handling
|
||||
* Deprecated old markdown names, legacy modules cleaned up
|
||||
* Massive repo cleanup: ~36K insertions, ~5K deletions across 121 files
|
||||
|
||||
[Read full release notes →](releases/0.6.0.md)
|
||||
|
||||
---
|
||||
|
||||
Let me know if you want me to auto-update the actual file or just paste this into the markdown.
|
||||
|
||||
### [Crawl4AI v0.5.0: Deep Crawling, Scalability, and a New CLI!](releases/0.5.0.md)
|
||||
|
||||
My dear friends and crawlers, there you go, this is the release of Crawl4AI v0.5.0! This release brings a wealth of new features, performance improvements, and a more streamlined developer experience. Here's a breakdown of what's new:
|
||||
|
||||
**Major New Features:**
|
||||
|
||||
* **Deep Crawling:** Explore entire websites with configurable strategies (BFS, DFS, Best-First). Define custom filters and URL scoring for targeted crawls.
|
||||
* **Memory-Adaptive Dispatcher:** Handle large-scale crawls with ease! Our new dispatcher dynamically adjusts concurrency based on available memory and includes built-in rate limiting.
|
||||
* **Multiple Crawler Strategies:** Choose between the full-featured Playwright browser-based crawler or a new, *much* faster HTTP-only crawler for simpler tasks.
|
||||
* **Docker Deployment:** Deploy Crawl4AI as a scalable, self-contained service with built-in API endpoints and optional JWT authentication.
|
||||
* **Command-Line Interface (CLI):** Interact with Crawl4AI directly from your terminal. Crawl, configure, and extract data with simple commands.
|
||||
* **LLM Configuration (`LLMConfig`):** A new, unified way to configure LLM providers (OpenAI, Anthropic, Ollama, etc.) for extraction, filtering, and schema generation. Simplifies API key management and switching between models.
|
||||
|
||||
**Minor Updates & Improvements:**
|
||||
|
||||
* **LXML Scraping Mode:** Faster HTML parsing with `LXMLWebScrapingStrategy`.
|
||||
* **Proxy Rotation:** Added `ProxyRotationStrategy` with a `RoundRobinProxyStrategy` implementation.
|
||||
* **PDF Processing:** Extract text, images, and metadata from PDF files.
|
||||
* **URL Redirection Tracking:** Automatically follows and records redirects.
|
||||
* **Robots.txt Compliance:** Optionally respect website crawling rules.
|
||||
* **LLM-Powered Schema Generation:** Automatically create extraction schemas using an LLM.
|
||||
* **`LLMContentFilter`:** Generate high-quality, focused markdown using an LLM.
|
||||
* **Improved Error Handling & Stability:** Numerous bug fixes and performance enhancements.
|
||||
* **Enhanced Documentation:** Updated guides and examples.
|
||||
|
||||
**Breaking Changes & Migration:**
|
||||
|
||||
This release includes several breaking changes to improve the library's structure and consistency. Here's what you need to know:
|
||||
|
||||
* **`arun_many()` Behavior:** Now uses the `MemoryAdaptiveDispatcher` by default. The return type depends on the `stream` parameter in `CrawlerRunConfig`. Adjust code that relied on unbounded concurrency.
|
||||
* **`max_depth` Location:** Moved to `CrawlerRunConfig` and now controls *crawl depth*.
|
||||
* **Deep Crawling Imports:** Import `DeepCrawlStrategy` and related classes from `crawl4ai.deep_crawling`.
|
||||
* **`BrowserContext` API:** Updated; the old `get_context` method is deprecated.
|
||||
* **Optional Model Fields:** Many data model fields are now optional. Handle potential `None` values.
|
||||
* **`ScrapingMode` Enum:** Replaced with strategy pattern (`WebScrapingStrategy`, `LXMLWebScrapingStrategy`).
|
||||
* **`content_filter` Parameter:** Removed from `CrawlerRunConfig`. Use extraction strategies or markdown generators with filters.
|
||||
* **Removed Functionality:** The synchronous `WebCrawler`, the old CLI, and docs management tools have been removed.
|
||||
* **Docker:** Significant changes to deployment. See the [Docker documentation](../deploy/docker/README.md).
|
||||
* **`ssl_certificate.json`:** This file has been removed.
|
||||
* **Config**: FastFilterChain has been replaced with FilterChain
|
||||
* **Deep-Crawl**: DeepCrawlStrategy.arun now returns Union[CrawlResultT, List[CrawlResultT], AsyncGenerator[CrawlResultT, None]]
|
||||
* **Proxy**: Removed synchronous WebCrawler support and related rate limiting configurations
|
||||
* **LLM Parameters:** Use the new `LLMConfig` object instead of passing `provider`, `api_token`, `base_url`, and `api_base` directly to `LLMExtractionStrategy` and `LLMContentFilter`.
|
||||
|
||||
**In short:** Update imports, adjust `arun_many()` usage, check for optional fields, and review the Docker deployment guide.
|
||||
|
||||
## License Change
|
||||
|
||||
Crawl4AI v0.5.0 updates the license to Apache 2.0 *with a required attribution clause*. This means you are free to use, modify, and distribute Crawl4AI (even commercially), but you *must* clearly attribute the project in any public use or distribution. See the updated `LICENSE` file for the full legal text and specific requirements.
|
||||
|
||||
**Get Started:**
|
||||
|
||||
* **Installation:** `pip install "crawl4ai[all]"` (or use the Docker image)
|
||||
* **Documentation:** [https://docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
* **GitHub:** [https://github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
|
||||
I'm very excited to see what you build with Crawl4AI v0.5.0!
|
||||
|
||||
---
|
||||
|
||||
### [0.4.2 - Configurable Crawlers, Session Management, and Smarter Screenshots](releases/0.4.2.md)
|
||||
*December 12, 2024*
|
||||
|
||||
The 0.4.2 update brings massive improvements to configuration, making crawlers and browsers easier to manage with dedicated objects. You can now import/export local storage for seamless session management. Plus, long-page screenshots are faster and cleaner, and full-page PDF exports are now possible. Check out all the new features to make your crawling experience even smoother.
|
||||
|
||||
[Read full release notes →](releases/0.4.2.md)
|
||||
|
||||
---
|
||||
|
||||
### [0.4.1 - Smarter Crawling with Lazy-Load Handling, Text-Only Mode, and More](releases/0.4.1.md)
|
||||
*December 8, 2024*
|
||||
|
||||
This release brings major improvements to handling lazy-loaded images, a blazing-fast Text-Only Mode, full-page scanning for infinite scrolls, dynamic viewport adjustments, and session reuse for efficient crawling. If you're looking to improve speed, reliability, or handle dynamic content with ease, this update has you covered.
|
||||
|
||||
[Read full release notes →](releases/0.4.1.md)
|
||||
|
||||
---
|
||||
|
||||
### [0.4.0 - Major Content Filtering Update](releases/0.4.0.md)
|
||||
*December 1, 2024*
|
||||
|
||||
Introduced significant improvements to content filtering, multi-threaded environment handling, and user-agent generation. This release features the new PruningContentFilter, enhanced thread safety, and improved test coverage.
|
||||
|
||||
[Read full release notes →](releases/0.4.0.md)
|
||||
|
||||
## Project History
|
||||
|
||||
Curious about how Crawl4AI has evolved? Check out our [complete changelog](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md) for a detailed history of all versions and updates.
|
||||
|
||||
## Stay Updated
|
||||
|
||||
- Star us on [GitHub](https://github.com/unclecode/crawl4ai)
|
||||
- Follow [@unclecode](https://twitter.com/unclecode) on Twitter
|
||||
- Join our community discussions on GitHub
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Release Summary for Version 0.4.0 (December 1, 2024)
|
||||
|
||||
## Overview
|
||||
The 0.4.0 release introduces significant improvements to content filtering, multi-threaded environment handling, user-agent generation, and test coverage. Key highlights include the introduction of the PruningContentFilter, designed to automatically identify and extract the most valuable parts of an HTML document, as well as enhancements to the BM25ContentFilter to extend its versatility and effectiveness.
|
||||
|
||||
## Major Features and Enhancements
|
||||
|
||||
### 1. PruningContentFilter
|
||||
- Introduced a new unsupervised content filtering strategy that scores and prunes less relevant nodes in an HTML document based on metrics like text and link density.
|
||||
- Focuses on retaining the most valuable parts of the content, making it highly effective for extracting relevant information from complex web pages.
|
||||
- Fully documented with updated README and expanded user guides.
|
||||
|
||||
### 2. User-Agent Generator
|
||||
- Added a user-agent generator utility that resolves compatibility issues and supports customizable user-agent strings.
|
||||
- By default, the generator randomizes user agents for each request, adding diversity, but users can customize it for tailored scenarios.
|
||||
|
||||
### 3. Enhanced Thread Safety
|
||||
- Improved handling of multi-threaded environments by adding better thread locks for parallel processing, ensuring consistency and stability when running multiple threads.
|
||||
|
||||
### 4. Extended Content Filtering Strategies
|
||||
- Users now have access to both the PruningContentFilter for unsupervised extraction and the BM25ContentFilter for supervised filtering based on user queries.
|
||||
- Enhanced BM25ContentFilter with improved capabilities to process page titles, meta tags, and descriptions, allowing for more effective classification and clustering of text chunks.
|
||||
|
||||
### 5. Documentation Updates
|
||||
- Updated examples and tutorials to promote the use of the PruningContentFilter alongside the BM25ContentFilter, providing clear instructions for selecting the appropriate filter for each use case.
|
||||
|
||||
### 6. Unit Test Enhancements
|
||||
- Added unit tests for PruningContentFilter to ensure accuracy and reliability.
|
||||
- Enhanced BM25ContentFilter tests to cover additional edge cases and performance metrics, particularly for malformed HTML inputs.
|
||||
|
||||
## Revised Change Logs for Version 0.4.0
|
||||
|
||||
### PruningContentFilter (Dec 01, 2024)
|
||||
- Introduced the PruningContentFilter to optimize content extraction by pruning less relevant HTML nodes.
|
||||
- **Affected Files:**
|
||||
- **crawl4ai/content_filter_strategy.py**: Added a scoring-based pruning algorithm.
|
||||
- **README.md**: Updated to include PruningContentFilter usage.
|
||||
- **docs/md_v2/basic/content_filtering.md**: Expanded user documentation, detailing the use and benefits of PruningContentFilter.
|
||||
|
||||
### Unit Tests for PruningContentFilter (Dec 01, 2024)
|
||||
- Added comprehensive unit tests for PruningContentFilter to ensure correctness and efficiency.
|
||||
- **Affected Files:**
|
||||
- **tests/async/test_content_filter_prune.py**: Created tests covering different pruning scenarios to ensure stability and correctness.
|
||||
|
||||
### Enhanced BM25ContentFilter Tests (Dec 01, 2024)
|
||||
- Expanded tests to cover additional extraction scenarios and performance metrics, improving robustness.
|
||||
- **Affected Files:**
|
||||
- **tests/async/test_content_filter_bm25.py**: Added tests for edge cases, including malformed HTML inputs.
|
||||
|
||||
### Documentation and Example Updates (Dec 01, 2024)
|
||||
- Revised examples to illustrate the use of PruningContentFilter alongside existing content filtering methods.
|
||||
- **Affected Files:**
|
||||
- **docs/examples/quickstart_async.py**: Enhanced example clarity and usability for new users.
|
||||
|
||||
## Experimental Features
|
||||
- The PruningContentFilter is still under experimental development, and we continue to gather feedback for further refinements.
|
||||
|
||||
## Conclusion
|
||||
This release significantly enhances the content extraction capabilities of Crawl4ai with the introduction of the PruningContentFilter, improved supervised filtering with BM25ContentFilter, and robust multi-threaded handling. Additionally, the user-agent generator provides much-needed versatility, resolving compatibility issues faced by many users.
|
||||
|
||||
Users are encouraged to experiment with the new content filtering methods to determine which best suits their needs.
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Release Summary for Version 0.4.1 (December 8, 2024): Major Efficiency Boosts with New Features!
|
||||
|
||||
_This post was generated with the help of ChatGPT, take everything with a grain of salt. 🧂_
|
||||
|
||||
Hi everyone,
|
||||
|
||||
I just finished putting together version 0.4.1 of Crawl4AI, and there are a few changes in here that I think you’ll find really helpful. I’ll explain what’s new, why it matters, and exactly how you can use these features (with the code to back it up). Let’s get into it.
|
||||
|
||||
---
|
||||
|
||||
### Handling Lazy Loading Better (Images Included)
|
||||
|
||||
One thing that always bugged me with crawlers is how often they miss lazy-loaded content, especially images. In this version, I made sure Crawl4AI **waits for all images to load** before moving forward. This is useful because many modern websites only load images when they’re in the viewport or after some JavaScript executes.
|
||||
|
||||
Here’s how to enable it:
|
||||
|
||||
```python
|
||||
await crawler.crawl(
|
||||
url="https://example.com",
|
||||
wait_for_images=True # Add this argument to ensure images are fully loaded
|
||||
)
|
||||
```
|
||||
|
||||
What this does is:
|
||||
1. Waits for the page to reach a "network idle" state.
|
||||
2. Ensures all images on the page have been completely loaded.
|
||||
|
||||
This single change handles the majority of lazy-loading cases you’re likely to encounter.
|
||||
|
||||
---
|
||||
|
||||
### Text-Only Mode (Fast, Lightweight Crawling)
|
||||
|
||||
Sometimes, you don’t need to download images or process JavaScript at all. For example, if you’re crawling to extract text data, you can enable **text-only mode** to speed things up. By disabling images, JavaScript, and other heavy resources, this mode makes crawling **3-4 times faster** in most cases.
|
||||
|
||||
Here’s how to turn it on:
|
||||
|
||||
```python
|
||||
crawler = AsyncPlaywrightCrawlerStrategy(
|
||||
text_mode=True # Set this to True to enable text-only crawling
|
||||
)
|
||||
```
|
||||
|
||||
When `text_mode=True`, the crawler automatically:
|
||||
- Disables GPU processing.
|
||||
- Blocks image and JavaScript resources.
|
||||
- Reduces the viewport size to 800x600 (you can override this with `viewport_width` and `viewport_height`).
|
||||
|
||||
If you need to crawl thousands of pages where you only care about text, this mode will save you a ton of time and resources.
|
||||
|
||||
---
|
||||
|
||||
### Adjusting the Viewport Dynamically
|
||||
|
||||
Another useful addition is the ability to **dynamically adjust the viewport size** to match the content on the page. This is particularly helpful when you’re working with responsive layouts or want to ensure all parts of the page load properly.
|
||||
|
||||
Here’s how it works:
|
||||
1. The crawler calculates the page’s width and height after it loads.
|
||||
2. It adjusts the viewport to fit the content dimensions.
|
||||
3. (Optional) It uses Chrome DevTools Protocol (CDP) to simulate zooming out so everything fits in the viewport.
|
||||
|
||||
To enable this, use:
|
||||
|
||||
```python
|
||||
await crawler.crawl(
|
||||
url="https://example.com",
|
||||
adjust_viewport_to_content=True # Dynamically adjusts the viewport
|
||||
)
|
||||
```
|
||||
|
||||
This approach makes sure the entire page gets loaded into the viewport, especially for layouts that load content based on visibility.
|
||||
|
||||
---
|
||||
|
||||
### Simulating Full-Page Scrolling
|
||||
|
||||
Some websites load data dynamically as you scroll down the page. To handle these cases, I added support for **full-page scanning**. It simulates scrolling to the bottom of the page, checking for new content, and capturing it all.
|
||||
|
||||
Here’s an example:
|
||||
|
||||
```python
|
||||
await crawler.crawl(
|
||||
url="https://example.com",
|
||||
scan_full_page=True, # Enables scrolling
|
||||
scroll_delay=0.2 # Waits 200ms between scrolls (optional)
|
||||
)
|
||||
```
|
||||
|
||||
What happens here:
|
||||
1. The crawler scrolls down in increments, waiting for content to load after each scroll.
|
||||
2. It stops when no new content appears (i.e., dynamic elements stop loading).
|
||||
3. It scrolls back to the top before finishing (if necessary).
|
||||
|
||||
If you’ve ever had to deal with infinite scroll pages, this is going to save you a lot of headaches.
|
||||
|
||||
---
|
||||
|
||||
### Reusing Browser Sessions (Save Time on Setup)
|
||||
|
||||
By default, every time you crawl a page, a new browser context (or tab) is created. That’s fine for small crawls, but if you’re working on a large dataset, it’s more efficient to reuse the same session.
|
||||
|
||||
I added a method called `create_session` for this:
|
||||
|
||||
```python
|
||||
session_id = await crawler.create_session()
|
||||
|
||||
# Use the same session for multiple crawls
|
||||
await crawler.crawl(
|
||||
url="https://example.com/page1",
|
||||
session_id=session_id # Reuse the session
|
||||
)
|
||||
await crawler.crawl(
|
||||
url="https://example.com/page2",
|
||||
session_id=session_id
|
||||
)
|
||||
```
|
||||
|
||||
This avoids creating a new tab for every page, speeding up the crawl and reducing memory usage.
|
||||
|
||||
---
|
||||
|
||||
### Other Updates
|
||||
|
||||
Here are a few smaller updates I’ve made:
|
||||
- **Light Mode**: Use `light_mode=True` to disable background processes, extensions, and other unnecessary features, making the browser more efficient.
|
||||
- **Logging**: Improved logs to make debugging easier.
|
||||
- **Defaults**: Added sensible defaults for things like `delay_before_return_html` (now set to 0.1 seconds).
|
||||
|
||||
---
|
||||
|
||||
### How to Get the Update
|
||||
|
||||
You can install or upgrade to version `0.4.1` like this:
|
||||
|
||||
```bash
|
||||
pip install crawl4ai --upgrade
|
||||
```
|
||||
|
||||
As always, I’d love to hear your thoughts. If there’s something you think could be improved or if you have suggestions for future versions, let me know!
|
||||
|
||||
Enjoy the new features, and happy crawling! 🕷️
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
## 🚀 Crawl4AI 0.4.2 Update: Smarter Crawling Just Got Easier (Dec 12, 2024)
|
||||
|
||||
### Hey Developers,
|
||||
|
||||
I’m excited to share Crawl4AI 0.4.2—a major upgrade that makes crawling smarter, faster, and a whole lot more intuitive. I’ve packed in a bunch of new features to simplify your workflows and improve your experience. Let’s cut to the chase!
|
||||
|
||||
---
|
||||
|
||||
### 🔧 **Configurable Browser and Crawler Behavior**
|
||||
|
||||
You’ve asked for better control over how browsers and crawlers are configured, and now you’ve got it. With the new `BrowserConfig` and `CrawlerRunConfig` objects, you can set up your browser and crawling behavior exactly how you want. No more cluttering `arun` with a dozen arguments—just pass in your configs and go.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
from crawl4ai import BrowserConfig, CrawlerRunConfig, AsyncWebCrawler
|
||||
|
||||
browser_config = BrowserConfig(headless=True, viewport_width=1920, viewport_height=1080)
|
||||
crawler_config = CrawlerRunConfig(cache_mode="BYPASS")
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=crawler_config)
|
||||
print(result.markdown[:500])
|
||||
```
|
||||
|
||||
This setup is a game-changer for scalability, keeping your code clean and flexible as we add more parameters in the future.
|
||||
|
||||
Remember: If you like to use the old way, you can still pass arguments directly to `arun` as before, no worries!
|
||||
|
||||
---
|
||||
|
||||
### 🔐 **Streamlined Session Management**
|
||||
|
||||
Here’s the big one: You can now pass local storage and cookies directly. Whether it’s setting values programmatically or importing a saved JSON state, managing sessions has never been easier. This is a must-have for authenticated crawls—just export your storage state once and reuse it effortlessly across runs.
|
||||
|
||||
**Example:**
|
||||
1. Open a browser, log in manually, and export the storage state.
|
||||
2. Import the JSON file for seamless authenticated crawling:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/protected",
|
||||
storage_state="my_storage_state.json"
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 🔢 **Handling Large Pages: Supercharged Screenshots and PDF Conversion**
|
||||
|
||||
Two big upgrades here:
|
||||
|
||||
- **Blazing-fast long-page screenshots**: Turn extremely long web pages into clean, high-quality screenshots—without breaking a sweat. It’s optimized to handle large content without lag.
|
||||
|
||||
- **Full-page PDF exports**: Now, you can also convert any page into a PDF with all the details intact. Perfect for archiving or sharing complex layouts.
|
||||
|
||||
---
|
||||
|
||||
### 🔧 **Other Cool Stuff**
|
||||
|
||||
- **Anti-bot enhancements**: Magic mode now handles overlays, user simulation, and anti-detection features like a pro.
|
||||
- **JavaScript execution**: Execute custom JS snippets to handle dynamic content. No more wrestling with endless page interactions.
|
||||
|
||||
---
|
||||
|
||||
### 📊 **Performance Boosts and Dev-friendly Updates**
|
||||
|
||||
- Faster rendering and viewport adjustments for better performance.
|
||||
- Improved cookie and local storage handling for seamless authentication.
|
||||
- Better debugging with detailed logs and actionable error messages.
|
||||
|
||||
---
|
||||
|
||||
### 🔠 **Use Cases You’ll Love**
|
||||
|
||||
1. **Authenticated Crawls**: Login once, export your storage state, and reuse it across multiple requests without the headache.
|
||||
2. **Long-page Screenshots**: Perfect for blogs, e-commerce pages, or any endless-scroll website.
|
||||
3. **PDF Export**: Create professional-looking page PDFs in seconds.
|
||||
|
||||
---
|
||||
|
||||
### Let’s Get Crawling
|
||||
|
||||
Crawl4AI 0.4.2 is ready for you to download and try. I’m always looking for ways to improve, so don’t hold back—share your thoughts and feedback.
|
||||
|
||||
Happy Crawling! 🚀
|
||||
|
||||
@@ -0,0 +1,524 @@
|
||||
# Crawl4AI v0.5.0 Release Notes
|
||||
|
||||
**Release Theme: Power, Flexibility, and Scalability**
|
||||
|
||||
Crawl4AI v0.5.0 is a major release focused on significantly enhancing the
|
||||
library's power, flexibility, and scalability. Key improvements include a new
|
||||
**deep crawling** system, a **memory-adaptive dispatcher** for handling
|
||||
large-scale crawls, **multiple crawling strategies** (including a fast HTTP-only
|
||||
crawler), **Docker** deployment options, and a powerful **command-line interface
|
||||
(CLI)**. This release also includes numerous bug fixes, performance
|
||||
optimizations, and documentation updates.
|
||||
|
||||
**Important Note:** This release contains several **breaking changes**. Please
|
||||
review the "Breaking Changes" section carefully and update your code
|
||||
accordingly.
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Deep Crawling
|
||||
|
||||
Crawl4AI now supports deep crawling, allowing you to explore websites beyond the
|
||||
initial URLs. This is controlled by the `deep_crawl_strategy` parameter in
|
||||
`CrawlerRunConfig`. Several strategies are available:
|
||||
|
||||
- **`BFSDeepCrawlStrategy` (Breadth-First Search):** Explores the website level
|
||||
by level. (Default)
|
||||
- **`DFSDeepCrawlStrategy` (Depth-First Search):** Explores each branch as
|
||||
deeply as possible before backtracking.
|
||||
- **`BestFirstCrawlingStrategy`:** Uses a scoring function to prioritize which
|
||||
URLs to crawl next.
|
||||
|
||||
```python
|
||||
import time
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BFSDeepCrawlStrategy
|
||||
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
|
||||
from crawl4ai.deep_crawling import DomainFilter, ContentTypeFilter, FilterChain, URLPatternFilter, KeywordRelevanceScorer, BestFirstCrawlingStrategy
|
||||
import asyncio
|
||||
|
||||
# Create a filter chain to filter urls based on patterns, domains and content type
|
||||
filter_chain = FilterChain(
|
||||
[
|
||||
DomainFilter(
|
||||
allowed_domains=["docs.crawl4ai.com"],
|
||||
blocked_domains=["old.docs.crawl4ai.com"],
|
||||
),
|
||||
URLPatternFilter(patterns=["*core*", "*advanced*"],),
|
||||
ContentTypeFilter(allowed_types=["text/html"]),
|
||||
]
|
||||
)
|
||||
|
||||
# Create a keyword scorer that prioritises the pages with certain keywords first
|
||||
keyword_scorer = KeywordRelevanceScorer(
|
||||
keywords=["crawl", "example", "async", "configuration"], weight=0.7
|
||||
)
|
||||
|
||||
# Set up the configuration
|
||||
deep_crawl_config = CrawlerRunConfig(
|
||||
deep_crawl_strategy=BestFirstCrawlingStrategy(
|
||||
max_depth=2,
|
||||
include_external=False,
|
||||
filter_chain=filter_chain,
|
||||
url_scorer=keyword_scorer,
|
||||
),
|
||||
scraping_strategy=LXMLWebScrapingStrategy(),
|
||||
stream=True,
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
start_time = time.perf_counter()
|
||||
results = []
|
||||
async for result in await crawler.arun(url="https://docs.crawl4ai.com", config=deep_crawl_config):
|
||||
print(f"Crawled: {result.url} (Depth: {result.metadata['depth']}), score: {result.metadata['score']:.2f}")
|
||||
results.append(result)
|
||||
duration = time.perf_counter() - start_time
|
||||
print(f"\n✅ Crawled {len(results)} high-value pages in {duration:.2f} seconds")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Breaking Change:** The `max_depth` parameter is now part of `CrawlerRunConfig`
|
||||
and controls the _depth_ of the crawl, not the number of concurrent crawls. The
|
||||
`arun()` and `arun_many()` methods are now decorated to handle deep crawling
|
||||
strategies. Imports for deep crawling strategies have changed. See the
|
||||
[Deep Crawling documentation](../../core/deep-crawling.md) for more details.
|
||||
|
||||
### 2. Memory-Adaptive Dispatcher
|
||||
|
||||
The new `MemoryAdaptiveDispatcher` dynamically adjusts concurrency based on
|
||||
available system memory and includes built-in rate limiting. This prevents
|
||||
out-of-memory errors and avoids overwhelming target websites.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MemoryAdaptiveDispatcher
|
||||
import asyncio
|
||||
|
||||
# Configure the dispatcher (optional, defaults are used if not provided)
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=80.0, # Pause if memory usage exceeds 80%
|
||||
check_interval=0.5, # Check memory every 0.5 seconds
|
||||
)
|
||||
|
||||
async def batch_mode():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=["https://docs.crawl4ai.com", "https://github.com/unclecode/crawl4ai"],
|
||||
config=CrawlerRunConfig(stream=False), # Batch mode
|
||||
dispatcher=dispatcher,
|
||||
)
|
||||
for result in results:
|
||||
print(f"Crawled: {result.url} with status code: {result.status_code}")
|
||||
|
||||
async def stream_mode():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# OR, for streaming:
|
||||
async for result in await crawler.arun_many(
|
||||
urls=["https://docs.crawl4ai.com", "https://github.com/unclecode/crawl4ai"],
|
||||
config=CrawlerRunConfig(stream=True),
|
||||
dispatcher=dispatcher,
|
||||
):
|
||||
print(f"Crawled: {result.url} with status code: {result.status_code}")
|
||||
|
||||
print("Dispatcher in batch mode:")
|
||||
asyncio.run(batch_mode())
|
||||
print("-" * 50)
|
||||
print("Dispatcher in stream mode:")
|
||||
asyncio.run(stream_mode())
|
||||
```
|
||||
|
||||
**Breaking Change:** `AsyncWebCrawler.arun_many()` now uses
|
||||
`MemoryAdaptiveDispatcher` by default. Existing code that relied on unbounded
|
||||
concurrency may require adjustments.
|
||||
|
||||
### 3. Multiple Crawling Strategies (Playwright and HTTP)
|
||||
|
||||
Crawl4AI now offers two crawling strategies:
|
||||
|
||||
- **`AsyncPlaywrightCrawlerStrategy` (Default):** Uses Playwright for
|
||||
browser-based crawling, supporting JavaScript rendering and complex
|
||||
interactions.
|
||||
- **`AsyncHTTPCrawlerStrategy`:** A lightweight, fast, and memory-efficient
|
||||
HTTP-only crawler. Ideal for simple scraping tasks where browser rendering is
|
||||
unnecessary.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, HTTPCrawlerConfig
|
||||
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
|
||||
import asyncio
|
||||
|
||||
# Use the HTTP crawler strategy
|
||||
http_crawler_config = HTTPCrawlerConfig(
|
||||
method="GET",
|
||||
headers={"User-Agent": "MyCustomBot/1.0"},
|
||||
follow_redirects=True,
|
||||
verify_ssl=True
|
||||
)
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy(browser_config =http_crawler_config)) as crawler:
|
||||
result = await crawler.arun("https://example.com")
|
||||
print(f"Status code: {result.status_code}")
|
||||
print(f"Content length: {len(result.html)}")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### 4. Docker Deployment
|
||||
|
||||
Crawl4AI can now be easily deployed as a Docker container, providing a
|
||||
consistent and isolated environment. The Docker image includes a FastAPI server
|
||||
with both streaming and non-streaming endpoints.
|
||||
|
||||
```bash
|
||||
# Build the image (from the project root)
|
||||
docker build -t crawl4ai .
|
||||
|
||||
# Run the container
|
||||
docker run -d -p 8000:8000 --name crawl4ai crawl4ai
|
||||
```
|
||||
|
||||
**API Endpoints:**
|
||||
|
||||
- `/crawl` (POST): Non-streaming crawl.
|
||||
- `/crawl/stream` (POST): Streaming crawl (NDJSON).
|
||||
- `/health` (GET): Health check.
|
||||
- `/schema` (GET): Returns configuration schemas.
|
||||
- `/md/{url}` (GET): Returns markdown content of the URL.
|
||||
- `/llm/{url}` (GET): Returns LLM extracted content.
|
||||
- `/token` (POST): Get JWT token
|
||||
|
||||
**Breaking Changes:**
|
||||
|
||||
- Docker deployment now requires a `.llm.env` file for API keys.
|
||||
- Docker deployment now requires Redis and a new `config.yml` structure.
|
||||
- Server startup now uses `supervisord` instead of direct process management.
|
||||
- Docker server now requires authentication by default (JWT tokens).
|
||||
|
||||
See the [Docker deployment documentation](../../core/docker-deployment.md) for
|
||||
detailed instructions.
|
||||
|
||||
### 5. Command-Line Interface (CLI)
|
||||
|
||||
A new CLI (`crwl`) provides convenient access to Crawl4AI's functionality from
|
||||
the terminal.
|
||||
|
||||
```bash
|
||||
# Basic crawl
|
||||
crwl https://example.com
|
||||
|
||||
# Get markdown output
|
||||
crwl https://example.com -o markdown
|
||||
|
||||
# Use a configuration file
|
||||
crwl https://example.com -B browser.yml -C crawler.yml
|
||||
|
||||
# Use LLM-based extraction
|
||||
crwl https://example.com -e extract.yml -s schema.json
|
||||
|
||||
# Ask a question about the crawled content
|
||||
crwl https://example.com -q "What is the main topic?"
|
||||
|
||||
# See usage examples
|
||||
crwl --example
|
||||
```
|
||||
|
||||
See the [CLI documentation](../docs/md_v2/core/cli.md) for more details.
|
||||
|
||||
### 6. LXML Scraping Mode
|
||||
|
||||
Added `LXMLWebScrapingStrategy` for faster HTML parsing using the `lxml`
|
||||
library. This can significantly improve scraping performance, especially for
|
||||
large or complex pages. Set `scraping_strategy=LXMLWebScrapingStrategy()` in
|
||||
your `CrawlerRunConfig`.
|
||||
|
||||
**Breaking Change:** The `ScrapingMode` enum has been replaced with a strategy
|
||||
pattern. Use `WebScrapingStrategy` (default) or `LXMLWebScrapingStrategy`.
|
||||
|
||||
### 7. Proxy Rotation
|
||||
|
||||
Added `ProxyRotationStrategy` abstract base class with `RoundRobinProxyStrategy`
|
||||
concrete implementation.
|
||||
|
||||
```python
|
||||
import re
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
BrowserConfig,
|
||||
CrawlerRunConfig,
|
||||
CacheMode,
|
||||
RoundRobinProxyStrategy,
|
||||
)
|
||||
import asyncio
|
||||
from crawl4ai import ProxyConfig
|
||||
async def main():
|
||||
# Load proxies and create rotation strategy
|
||||
proxies = ProxyConfig.from_env()
|
||||
#eg: export PROXIES="ip1:port1:username1:password1,ip2:port2:username2:password2"
|
||||
if not proxies:
|
||||
print("No proxies found in environment. Set PROXIES env variable!")
|
||||
return
|
||||
|
||||
proxy_strategy = RoundRobinProxyStrategy(proxies)
|
||||
|
||||
# Create configs
|
||||
browser_config = BrowserConfig(headless=True, verbose=False)
|
||||
run_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
proxy_rotation_strategy=proxy_strategy
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
urls = ["https://httpbin.org/ip"] * (len(proxies) * 2) # Test each proxy twice
|
||||
|
||||
print("\n📈 Initializing crawler with proxy rotation...")
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
print("\n🚀 Starting batch crawl with proxy rotation...")
|
||||
results = await crawler.arun_many(
|
||||
urls=urls,
|
||||
config=run_config
|
||||
)
|
||||
for result in results:
|
||||
if result.success:
|
||||
ip_match = re.search(r'(?:[0-9]{1,3}\.){3}[0-9]{1,3}', result.html)
|
||||
current_proxy = run_config.proxy_config if run_config.proxy_config else None
|
||||
|
||||
if current_proxy and ip_match:
|
||||
print(f"URL {result.url}")
|
||||
print(f"Proxy {current_proxy.server} -> Response IP: {ip_match.group(0)}")
|
||||
verified = ip_match.group(0) == current_proxy.ip
|
||||
if verified:
|
||||
print(f"✅ Proxy working! IP matches: {current_proxy.ip}")
|
||||
else:
|
||||
print("❌ Proxy failed or IP mismatch!")
|
||||
print("---")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Other Changes and Improvements
|
||||
|
||||
- **Added: `LLMContentFilter` for intelligent markdown generation.** This new
|
||||
filter uses an LLM to create more focused and relevant markdown output.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, DefaultMarkdownGenerator
|
||||
from crawl4ai.content_filter_strategy import LLMContentFilter
|
||||
from crawl4ai import LLMConfig
|
||||
import asyncio
|
||||
|
||||
llm_config = LLMConfig(provider="gemini/gemini-1.5-pro", api_token="env:GEMINI_API_KEY")
|
||||
|
||||
markdown_generator = DefaultMarkdownGenerator(
|
||||
content_filter=LLMContentFilter(llm_config=llm_config, instruction="Extract key concepts and summaries")
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(markdown_generator=markdown_generator)
|
||||
async def main():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://docs.crawl4ai.com", config=config)
|
||||
print(result.markdown.fit_markdown)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
- **Added: URL redirection tracking.** The crawler now automatically follows
|
||||
HTTP redirects (301, 302, 307, 308) and records the final URL in the
|
||||
`redirected_url` field of the `CrawlResult` object. No code changes are
|
||||
required to enable this; it's automatic.
|
||||
|
||||
- **Added: LLM-powered schema generation utility.** A new `generate_schema`
|
||||
method has been added to `JsonCssExtractionStrategy` and
|
||||
`JsonXPathExtractionStrategy`. This greatly simplifies creating extraction
|
||||
schemas.
|
||||
|
||||
```python
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai import LLMConfig
|
||||
|
||||
llm_config = LLMConfig(provider="gemini/gemini-1.5-pro", api_token="env:GEMINI_API_KEY")
|
||||
|
||||
schema = JsonCssExtractionStrategy.generate_schema(
|
||||
html="<div class='product'><h2>Product Name</h2><span class='price'>$99</span></div>",
|
||||
llm_config = llm_config,
|
||||
query="Extract product name and price"
|
||||
)
|
||||
print(schema)
|
||||
```
|
||||
|
||||
Expected Output (may vary slightly due to LLM)
|
||||
```JSON
|
||||
{
|
||||
"name": "ProductExtractor",
|
||||
"baseSelector": "div.product",
|
||||
"fields": [
|
||||
{"name": "name", "selector": "h2", "type": "text"},
|
||||
{"name": "price", "selector": ".price", "type": "text"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **Added: robots.txt compliance support.** The crawler can now respect
|
||||
`robots.txt` rules. Enable this by setting `check_robots_txt=True` in
|
||||
`CrawlerRunConfig`.
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(check_robots_txt=True)
|
||||
```
|
||||
|
||||
- **Added: PDF processing capabilities.** Crawl4AI can now extract text, images,
|
||||
and metadata from PDF files (both local and remote). This uses a new
|
||||
`PDFCrawlerStrategy` and `PDFContentScrapingStrategy`.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(crawler_strategy=PDFCrawlerStrategy()) as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://arxiv.org/pdf/2310.06825.pdf",
|
||||
config=CrawlerRunConfig(
|
||||
scraping_strategy=PDFContentScrapingStrategy()
|
||||
)
|
||||
)
|
||||
print(result.markdown) # Access extracted text
|
||||
print(result.metadata) # Access PDF metadata (title, author, etc.)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
- **Added: Support for frozenset serialization.** Improves configuration
|
||||
serialization, especially for sets of allowed/blocked domains. No code changes
|
||||
required.
|
||||
|
||||
- **Added: New `LLMConfig` parameter.** This new parameter can be passed for
|
||||
extraction, filtering, and schema generation tasks. It simplifies passing
|
||||
provider strings, API tokens, and base URLs across all sections where LLM
|
||||
configuration is necessary. It also enables reuse and allows for quick
|
||||
experimentation between different LLM configurations.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMConfig
|
||||
from crawl4ai import LLMExtractionStrategy
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
# Example of using LLMConfig with LLMExtractionStrategy
|
||||
llm_config = LLMConfig(provider="openai/gpt-4o", api_token="YOUR_API_KEY")
|
||||
strategy = LLMExtractionStrategy(llm_config=llm_config, schema=...)
|
||||
|
||||
# Example usage within a crawler
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=CrawlerRunConfig(extraction_strategy=strategy)
|
||||
)
|
||||
```
|
||||
**Breaking Change:** Removed old parameters like `provider`, `api_token`,
|
||||
`base_url`, and `api_base` from `LLMExtractionStrategy` and
|
||||
`LLMContentFilter`. Users should migrate to using the `LLMConfig` object.
|
||||
|
||||
- **Changed: Improved browser context management and added shared data support.
|
||||
(Breaking Change:** `BrowserContext` API updated). Browser contexts are now
|
||||
managed more efficiently, reducing resource usage. A new `shared_data`
|
||||
dictionary is available in the `BrowserContext` to allow passing data between
|
||||
different stages of the crawling process. **Breaking Change:** The
|
||||
`BrowserContext` API has changed, and the old `get_context` method is
|
||||
deprecated.
|
||||
|
||||
- **Changed:** Renamed `final_url` to `redirected_url` in `CrawledURL`. This
|
||||
improves consistency and clarity. Update any code referencing the old field
|
||||
name.
|
||||
|
||||
- **Changed:** Improved type hints and removed unused files. This is an internal
|
||||
improvement and should not require code changes.
|
||||
|
||||
- **Changed:** Reorganized deep crawling functionality into dedicated module.
|
||||
(**Breaking Change:** Import paths for `DeepCrawlStrategy` and related classes
|
||||
have changed). This improves code organization. Update imports to use the new
|
||||
`crawl4ai.deep_crawling` module.
|
||||
|
||||
- **Changed:** Improved HTML handling and cleanup codebase. (**Breaking
|
||||
Change:** Removed `ssl_certificate.json` file). This removes an unused file.
|
||||
If you were relying on this file for custom certificate validation, you'll
|
||||
need to implement an alternative approach.
|
||||
|
||||
- **Changed:** Enhanced serialization and config handling. (**Breaking Change:**
|
||||
`FastFilterChain` has been replaced with `FilterChain`). This change
|
||||
simplifies config and improves the serialization.
|
||||
|
||||
- **Added:** Modified the license to Apache 2.0 _with a required attribution
|
||||
clause_. See the `LICENSE` file for details. All users must now clearly
|
||||
attribute the Crawl4AI project when using, distributing, or creating
|
||||
derivative works.
|
||||
|
||||
- **Fixed:** Prevent memory leaks by ensuring proper closure of Playwright
|
||||
pages. No code changes required.
|
||||
|
||||
- **Fixed:** Make model fields optional with default values (**Breaking
|
||||
Change:** Code relying on all fields being present may need adjustment).
|
||||
Fields in data models (like `CrawledURL`) are now optional, with default
|
||||
values (usually `None`). Update code to handle potential `None` values.
|
||||
|
||||
- **Fixed:** Adjust memory threshold and fix dispatcher initialization. This is
|
||||
an internal bug fix; no code changes are required.
|
||||
|
||||
- **Fixed:** Ensure proper exit after running doctor command. No code changes
|
||||
are required.
|
||||
- **Fixed:** JsonCss selector and crawler improvements.
|
||||
- **Fixed:** Not working long page screenshot (#403)
|
||||
- **Documentation:** Updated documentation URLs to the new domain.
|
||||
- **Documentation:** Added SERP API project example.
|
||||
- **Documentation:** Added clarifying comments for CSS selector behavior.
|
||||
- **Documentation:** Add Code of Conduct for the project (#410)
|
||||
|
||||
## Breaking Changes Summary
|
||||
|
||||
- **Dispatcher:** The `MemoryAdaptiveDispatcher` is now the default for
|
||||
`arun_many()`, changing concurrency behavior. The return type of `arun_many`
|
||||
depends on the `stream` parameter.
|
||||
- **Deep Crawling:** `max_depth` is now part of `CrawlerRunConfig` and controls
|
||||
crawl depth. Import paths for deep crawling strategies have changed.
|
||||
- **Browser Context:** The `BrowserContext` API has been updated.
|
||||
- **Models:** Many fields in data models are now optional, with default values.
|
||||
- **Scraping Mode:** `ScrapingMode` enum replaced by strategy pattern
|
||||
(`WebScrapingStrategy`, `LXMLWebScrapingStrategy`).
|
||||
- **Content Filter:** Removed `content_filter` parameter from
|
||||
`CrawlerRunConfig`. Use extraction strategies or markdown generators with
|
||||
filters instead.
|
||||
- **Removed:** Synchronous `WebCrawler`, CLI, and docs management functionality.
|
||||
- **Docker:** Significant changes to Docker deployment, including new
|
||||
requirements and configuration.
|
||||
- **File Removed**: Removed ssl_certificate.json file which might affect
|
||||
existing certificate validations
|
||||
- **Renamed**: final_url to redirected_url for consistency
|
||||
- **Config**: FastFilterChain has been replaced with FilterChain
|
||||
- **Deep-Crawl**: DeepCrawlStrategy.arun now returns Union[CrawlResultT,
|
||||
List[CrawlResultT], AsyncGenerator[CrawlResultT, None]]
|
||||
- **Proxy**: Removed synchronous WebCrawler support and related rate limiting
|
||||
configurations
|
||||
|
||||
## Migration Guide
|
||||
|
||||
1. **Update Imports:** Adjust imports for `DeepCrawlStrategy`,
|
||||
`BreadthFirstSearchStrategy`, and related classes due to the new
|
||||
`deep_crawling` module structure.
|
||||
2. **`CrawlerRunConfig`:** Move `max_depth` to `CrawlerRunConfig`. If using
|
||||
`content_filter`, migrate to an extraction strategy or a markdown generator
|
||||
with a filter.
|
||||
3. **`arun_many()`:** Adapt code to the new `MemoryAdaptiveDispatcher` behavior
|
||||
and the return type.
|
||||
4. **`BrowserContext`:** Update code using the `BrowserContext` API.
|
||||
5. **Models:** Handle potential `None` values for optional fields in data
|
||||
models.
|
||||
6. **Scraping:** Replace `ScrapingMode` enum with `WebScrapingStrategy` or
|
||||
`LXMLWebScrapingStrategy`.
|
||||
7. **Docker:** Review the updated Docker documentation and adjust your
|
||||
deployment accordingly.
|
||||
8. **CLI:** Migrate to the new `crwl` command and update any scripts using the
|
||||
old CLI.
|
||||
9. **Proxy:**: Removed synchronous WebCrawler support and related rate limiting
|
||||
configurations.
|
||||
10. **Config:**: Replace FastFilterChain to FilterChain
|
||||
@@ -0,0 +1,143 @@
|
||||
# Crawl4AI v0.6.0 Release Notes
|
||||
|
||||
We're excited to announce the release of **Crawl4AI v0.6.0**, our biggest and most feature-rich update yet. This version introduces major architectural upgrades, brand-new capabilities for geo-aware crawling, high-efficiency scraping, and real-time streaming support for scalable deployments.
|
||||
|
||||
---
|
||||
|
||||
## Highlights
|
||||
|
||||
### 1. **World-Aware Crawlers**
|
||||
Crawl as if you’re anywhere in the world. With v0.6.0, each crawl can simulate:
|
||||
- Specific GPS coordinates
|
||||
- Browser locale
|
||||
- Timezone
|
||||
|
||||
Example:
|
||||
```python
|
||||
CrawlerRunConfig(
|
||||
url="https://browserleaks.com/geo",
|
||||
locale="en-US",
|
||||
timezone_id="America/Los_Angeles",
|
||||
geolocation=GeolocationConfig(
|
||||
latitude=34.0522,
|
||||
longitude=-118.2437,
|
||||
accuracy=10.0
|
||||
)
|
||||
)
|
||||
```
|
||||
Great for accessing region-specific content or testing global behavior.
|
||||
|
||||
---
|
||||
|
||||
### 2. **Native Table Extraction**
|
||||
Extract HTML tables directly into usable formats like Pandas DataFrames or CSV with zero parsing hassle. All table data is available under `result.media["tables"]`.
|
||||
|
||||
Example:
|
||||
```python
|
||||
raw_df = pd.DataFrame(
|
||||
result.media["tables"][0]["rows"],
|
||||
columns=result.media["tables"][0]["headers"]
|
||||
)
|
||||
```
|
||||
This makes it ideal for scraping financial data, pricing pages, or anything tabular.
|
||||
|
||||
---
|
||||
|
||||
### 3. **Browser Pooling & Pre-Warming**
|
||||
We've overhauled browser management. Now, multiple browser instances can be pooled and pages pre-warmed for ultra-fast launches:
|
||||
- Reduces cold-start latency
|
||||
- Lowers memory spikes
|
||||
- Enhances parallel crawling stability
|
||||
|
||||
This powers the new **Docker Playground** experience and streamlines heavy-load crawling.
|
||||
|
||||
---
|
||||
|
||||
### 4. **Traffic & Snapshot Capture**
|
||||
Need full visibility? You can now capture:
|
||||
- Full network traffic logs
|
||||
- Console output
|
||||
- MHTML page snapshots for post-crawl audits and debugging
|
||||
|
||||
No more guesswork on what happened during your crawl.
|
||||
|
||||
---
|
||||
|
||||
### 5. **MCP API and Streaming Support**
|
||||
We’re exposing **MCP socket and SSE endpoints**, allowing:
|
||||
- Live streaming of crawl results
|
||||
- Real-time integration with agents or frontends
|
||||
- A new Playground UI for interactive crawling
|
||||
|
||||
This is a major step towards making Crawl4AI real-time ready.
|
||||
|
||||
---
|
||||
|
||||
### 6. **Stress-Test Framework**
|
||||
Want to test performance under heavy load? v0.6.0 includes a new memory stress-test suite that supports 1,000+ URL workloads. Ideal for:
|
||||
- Load testing
|
||||
- Performance benchmarking
|
||||
- Validating memory efficiency
|
||||
|
||||
---
|
||||
|
||||
## Core Improvements
|
||||
- Robots.txt compliance
|
||||
- Proxy rotation support
|
||||
- Improved URL normalization and session reuse
|
||||
- Shared data across crawler hooks
|
||||
- New page routing logic
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes & Deprecations
|
||||
- Legacy `crawl4ai/browser/*` modules are removed. Update imports accordingly.
|
||||
- `AsyncPlaywrightCrawlerStrategy.get_page` now uses a new function signature.
|
||||
- Deprecated markdown generator aliases now point to `DefaultMarkdownGenerator` with warning.
|
||||
|
||||
---
|
||||
|
||||
## Miscellaneous Updates
|
||||
- FastAPI validators replaced custom validation logic
|
||||
- Docker build now based on a Chromium layer
|
||||
- Repo-wide cleanup: ~36,000 insertions, ~5,000 deletions
|
||||
|
||||
---
|
||||
|
||||
## New Examples Included
|
||||
- Geo-location crawling
|
||||
- Network + console log capture
|
||||
- Docker MCP API usage
|
||||
- Markdown selector usage
|
||||
- Crypto project data extraction
|
||||
|
||||
---
|
||||
|
||||
## Watch the Release Video
|
||||
Want a visual walkthrough of all these updates? Watch the video:
|
||||
🔗 https://youtu.be/9x7nVcjOZks
|
||||
|
||||
If you're new to Crawl4AI, start here:
|
||||
🔗 https://www.youtube.com/watch?v=xo3qK6Hg9AA&t=15s
|
||||
|
||||
---
|
||||
|
||||
## Join the Community
|
||||
We’ve just opened up our **Discord** for the public. Join us to:
|
||||
- Ask questions
|
||||
- Share your projects
|
||||
- Get help or contribute
|
||||
|
||||
💬 https://discord.gg/wpYFACrHR4
|
||||
|
||||
---
|
||||
|
||||
## Install or Upgrade
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Live long and import crawl4ai. 🖖
|
||||
|
||||
@@ -0,0 +1,343 @@
|
||||
# 🚀 Crawl4AI v0.7.0: The Adaptive Intelligence Update
|
||||
|
||||
*January 28, 2025 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.0—the Adaptive Intelligence Update. This release introduces fundamental improvements in how Crawl4AI handles modern web complexity through adaptive learning, intelligent content discovery, and advanced extraction capabilities.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Adaptive Crawling**: Your crawler now learns and adapts to website patterns
|
||||
- **Virtual Scroll Support**: Complete content extraction from infinite scroll pages
|
||||
- **Link Preview with Intelligent Scoring**: Intelligent link analysis and prioritization
|
||||
- **Async URL Seeder**: Discover thousands of URLs in seconds with intelligent filtering
|
||||
- **Performance Optimizations**: Significant speed and memory improvements
|
||||
|
||||
## 🧠 Adaptive Crawling: Intelligence Through Pattern Learning
|
||||
|
||||
**The Problem:** Websites change. Class names shift. IDs disappear. Your carefully crafted selectors break at 3 AM, and you wake up to empty datasets and angry stakeholders.
|
||||
|
||||
**My Solution:** I implemented an adaptive learning system that observes patterns, builds confidence scores, and adjusts extraction strategies on the fly. It's like having a junior developer who gets better at their job with every page they scrape.
|
||||
|
||||
### Technical Deep-Dive
|
||||
|
||||
The Adaptive Crawler maintains a persistent state for each domain, tracking:
|
||||
- Pattern success rates
|
||||
- Selector stability over time
|
||||
- Content structure variations
|
||||
- Extraction confidence scores
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
|
||||
# Configure adaptive crawler
|
||||
config = AdaptiveConfig(
|
||||
strategy="statistical", # or "embedding" for semantic understanding
|
||||
max_pages=10,
|
||||
confidence_threshold=0.7, # Stop at 70% confidence
|
||||
top_k_links=3, # Follow top 3 links per page
|
||||
min_gain_threshold=0.05 # Need 5% information gain to continue
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
print("Starting adaptive crawl about Python decorators...")
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/glossary.html",
|
||||
query="python decorators functions wrapping"
|
||||
)
|
||||
|
||||
print(f"\n✅ Crawling Complete!")
|
||||
print(f"• Confidence Level: {adaptive.confidence:.0%}")
|
||||
print(f"• Pages Crawled: {len(result.crawled_urls)}")
|
||||
print(f"• Knowledge Base: {len(adaptive.state.knowledge_base)} documents")
|
||||
|
||||
# Get most relevant content
|
||||
relevant = adaptive.get_relevant_content(top_k=3)
|
||||
print(f"\nMost Relevant Pages:")
|
||||
for i, page in enumerate(relevant, 1):
|
||||
print(f"{i}. {page['url']} (relevance: {page['score']:.2%})")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **News Aggregation**: Maintain 95%+ extraction accuracy even as news sites update their templates
|
||||
- **E-commerce Monitoring**: Track product changes across hundreds of stores without constant maintenance
|
||||
- **Research Data Collection**: Build robust academic datasets that survive website redesigns
|
||||
- **Reduced Maintenance**: Cut selector update time by 80% for frequently-changing sites
|
||||
|
||||
## 🌊 Virtual Scroll: Complete Content Capture
|
||||
|
||||
**The Problem:** Modern web apps only render what's visible. Scroll down, new content appears, old content vanishes into the void. Traditional crawlers capture that first viewport and miss 90% of the content. It's like reading only the first page of every book.
|
||||
|
||||
**My Solution:** I built Virtual Scroll support that mimics human browsing behavior, capturing content as it loads and preserving it before the browser's garbage collector strikes.
|
||||
|
||||
### Implementation Details
|
||||
|
||||
```python
|
||||
from crawl4ai import VirtualScrollConfig
|
||||
|
||||
# For social media feeds (Twitter/X style)
|
||||
twitter_config = VirtualScrollConfig(
|
||||
container_selector="[data-testid='primaryColumn']",
|
||||
scroll_count=20, # Number of scrolls
|
||||
scroll_by="container_height", # Smart scrolling by container size
|
||||
wait_after_scroll=1.0 # Let content load
|
||||
)
|
||||
|
||||
# For e-commerce product grids (Instagram style)
|
||||
grid_config = VirtualScrollConfig(
|
||||
container_selector="main .product-grid",
|
||||
scroll_count=30,
|
||||
scroll_by=800, # Fixed pixel scrolling
|
||||
wait_after_scroll=1.5 # Images need time
|
||||
)
|
||||
|
||||
# For news feeds with lazy loading
|
||||
news_config = VirtualScrollConfig(
|
||||
container_selector=".article-feed",
|
||||
scroll_count=50,
|
||||
scroll_by="page_height", # Viewport-based scrolling
|
||||
wait_after_scroll=0.5 # Wait for content to load
|
||||
)
|
||||
|
||||
# Use it in your crawl
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://twitter.com/trending",
|
||||
config=CrawlerRunConfig(
|
||||
virtual_scroll_config=twitter_config,
|
||||
# Combine with other features
|
||||
extraction_strategy=JsonCssExtractionStrategy({
|
||||
"tweets": {
|
||||
"selector": "[data-testid='tweet']",
|
||||
"fields": {
|
||||
"text": {"selector": "[data-testid='tweetText']", "type": "text"},
|
||||
"likes": {"selector": "[data-testid='like']", "type": "text"}
|
||||
}
|
||||
}
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
print(f"Captured {len(result.extracted_content['tweets'])} tweets")
|
||||
```
|
||||
|
||||
**Key Capabilities:**
|
||||
- **DOM Recycling Awareness**: Detects and handles virtual DOM element recycling
|
||||
- **Smart Scroll Physics**: Three modes - container height, page height, or fixed pixels
|
||||
- **Content Preservation**: Captures content before it's destroyed
|
||||
- **Intelligent Stopping**: Stops when no new content appears
|
||||
- **Memory Efficient**: Streams content instead of holding everything in memory
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Social Media Analysis**: Capture entire Twitter threads with hundreds of replies, not just top 10
|
||||
- **E-commerce Scraping**: Extract 500+ products from infinite scroll catalogs vs. 20-50 with traditional methods
|
||||
- **News Aggregation**: Get all articles from modern news sites, not just above-the-fold content
|
||||
- **Research Applications**: Complete data extraction from academic databases using virtual pagination
|
||||
|
||||
## 🔗 Link Preview: Intelligent Link Analysis and Scoring
|
||||
|
||||
**The Problem:** You crawl a page and get 200 links. Which ones matter? Which lead to the content you actually want? Traditional crawlers force you to follow everything or build complex filters.
|
||||
|
||||
**My Solution:** I implemented a three-layer scoring system that analyzes links like a human would—considering their position, context, and relevance to your goals.
|
||||
|
||||
### Intelligent Link Analysis and Scoring
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import CrawlerRunConfig, CacheMode, AsyncWebCrawler
|
||||
from crawl4ai.adaptive_crawler import LinkPreviewConfig
|
||||
|
||||
async def main():
|
||||
# Configure intelligent link analysis
|
||||
link_config = LinkPreviewConfig(
|
||||
include_internal=True,
|
||||
include_external=False,
|
||||
max_links=10,
|
||||
concurrency=5,
|
||||
query="python tutorial", # For contextual scoring
|
||||
score_threshold=0.3,
|
||||
verbose=True
|
||||
)
|
||||
# Use in your crawl
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://www.geeksforgeeks.org/",
|
||||
config=CrawlerRunConfig(
|
||||
link_preview_config=link_config,
|
||||
score_links=True, # Enable intrinsic scoring
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
)
|
||||
|
||||
# Access scored and sorted links
|
||||
if result.success and result.links:
|
||||
for link in result.links.get("internal", []):
|
||||
text = link.get('text', 'No text')[:40]
|
||||
print(
|
||||
text,
|
||||
f"{link.get('intrinsic_score', 0):.1f}/10" if link.get('intrinsic_score') is not None else "0.0/10",
|
||||
f"{link.get('contextual_score', 0):.2f}/1" if link.get('contextual_score') is not None else "0.00/1",
|
||||
f"{link.get('total_score', 0):.3f}" if link.get('total_score') is not None else "0.000"
|
||||
)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Scoring Components:**
|
||||
|
||||
1. **Intrinsic Score**: Based on link quality indicators
|
||||
- Position on page (navigation, content, footer)
|
||||
- Link attributes (rel, title, class names)
|
||||
- Anchor text quality and length
|
||||
- URL structure and depth
|
||||
|
||||
2. **Contextual Score**: Relevance to your query using BM25 algorithm
|
||||
- Keyword matching in link text and title
|
||||
- Meta description analysis
|
||||
- Content preview scoring
|
||||
|
||||
3. **Total Score**: Combined score for final ranking
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Research Efficiency**: Find relevant papers 10x faster by following only high-score links
|
||||
- **Competitive Analysis**: Automatically identify important pages on competitor sites
|
||||
- **Content Discovery**: Build topic-focused crawlers that stay on track
|
||||
- **SEO Audits**: Identify and prioritize high-value internal linking opportunities
|
||||
|
||||
## 🎣 Async URL Seeder: Automated URL Discovery at Scale
|
||||
|
||||
**The Problem:** You want to crawl an entire domain but only have the homepage. Or worse, you want specific content types across thousands of pages. Manual URL discovery? That's a job for machines, not humans.
|
||||
|
||||
**My Solution:** I built Async URL Seeder—a turbocharged URL discovery engine that combines multiple sources with intelligent filtering and relevance scoring.
|
||||
|
||||
### Technical Architecture
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncUrlSeeder, SeedingConfig
|
||||
|
||||
async def main():
|
||||
async with AsyncUrlSeeder() as seeder:
|
||||
# Discover Python tutorial URLs
|
||||
config = SeedingConfig(
|
||||
source="sitemap", # Use sitemap
|
||||
pattern="*python*", # URL pattern filter
|
||||
extract_head=True, # Get metadata
|
||||
query="python tutorial", # For relevance scoring
|
||||
scoring_method="bm25",
|
||||
score_threshold=0.2,
|
||||
max_urls=10
|
||||
)
|
||||
|
||||
print("Discovering Python async tutorial URLs...")
|
||||
urls = await seeder.urls("https://www.geeksforgeeks.org/", config)
|
||||
|
||||
print(f"\n✅ Found {len(urls)} relevant URLs:")
|
||||
for i, url_info in enumerate(urls[:5], 1):
|
||||
print(f"\n{i}. {url_info['url']}")
|
||||
if url_info.get('relevance_score'):
|
||||
print(f" Relevance: {url_info['relevance_score']:.3f}")
|
||||
if url_info.get('head_data', {}).get('title'):
|
||||
print(f" Title: {url_info['head_data']['title'][:60]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Discovery Methods:**
|
||||
- **Sitemap Mining**: Parses robots.txt and all linked sitemaps
|
||||
- **Common Crawl**: Queries the Common Crawl index for historical URLs
|
||||
- **Intelligent Crawling**: Follows links with smart depth control
|
||||
- **Pattern Analysis**: Learns URL structures and generates variations
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Migration Projects**: Discover 10,000+ URLs from legacy sites in under 60 seconds
|
||||
- **Market Research**: Map entire competitor ecosystems automatically
|
||||
- **Academic Research**: Build comprehensive datasets without manual URL collection
|
||||
- **SEO Audits**: Find every indexable page with content scoring
|
||||
- **Content Archival**: Ensure no content is left behind during site migrations
|
||||
|
||||
## ⚡ Performance Optimizations
|
||||
|
||||
This release includes significant performance improvements through optimized resource handling, better concurrency management, and reduced memory footprint.
|
||||
|
||||
### What We Optimized
|
||||
|
||||
```python
|
||||
# Optimized crawling with v0.7.0 improvements
|
||||
results = []
|
||||
for url in urls:
|
||||
result = await crawler.arun(
|
||||
url,
|
||||
config=CrawlerRunConfig(
|
||||
# Performance optimizations
|
||||
wait_until="domcontentloaded", # Faster than networkidle
|
||||
cache_mode=CacheMode.ENABLED # Enable caching
|
||||
)
|
||||
)
|
||||
results.append(result)
|
||||
```
|
||||
|
||||
**Performance Gains:**
|
||||
- **Startup Time**: 70% faster browser initialization
|
||||
- **Page Loading**: 40% reduction with smart resource blocking
|
||||
- **Extraction**: 3x faster with compiled CSS selectors
|
||||
- **Memory Usage**: 60% reduction with streaming processing
|
||||
- **Concurrent Crawls**: Handle 5x more parallel requests
|
||||
|
||||
|
||||
## 🔧 Important Changes
|
||||
|
||||
### Breaking Changes
|
||||
- `link_extractor` renamed to `link_preview` (better reflects functionality)
|
||||
- Minimum Python version now 3.9
|
||||
- `CrawlerConfig` split into `CrawlerRunConfig` and `BrowserConfig`
|
||||
|
||||
### Migration Guide
|
||||
```python
|
||||
# Old (v0.6.x)
|
||||
from crawl4ai import CrawlerConfig
|
||||
config = CrawlerConfig(timeout=30000)
|
||||
|
||||
# New (v0.7.0)
|
||||
from crawl4ai import CrawlerRunConfig, BrowserConfig
|
||||
browser_config = BrowserConfig(timeout=30000)
|
||||
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
|
||||
```
|
||||
|
||||
## 🤖 Coming Soon: Intelligent Web Automation
|
||||
|
||||
I'm currently working on bringing advanced automation capabilities to Crawl4AI. This includes:
|
||||
|
||||
- **Crawl Agents**: Autonomous crawlers that understand your goals and adapt their strategies
|
||||
- **Auto JS Generation**: Automatic JavaScript code generation for complex interactions
|
||||
- **Smart Form Handling**: Intelligent form detection and filling
|
||||
- **Context-Aware Actions**: Crawlers that understand page context and make decisions
|
||||
|
||||
These features are under active development and will revolutionize how we approach web automation. Stay tuned!
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.0
|
||||
```
|
||||
|
||||
Check out the [updated documentation](https://docs.crawl4ai.com).
|
||||
|
||||
Questions? Issues? I'm always listening:
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
|
||||
---
|
||||
|
||||
*P.S. If you're using Crawl4AI in production, I'd love to hear about it. Your use cases inspire the next features.*
|
||||
@@ -0,0 +1,43 @@
|
||||
# 🛠️ Crawl4AI v0.7.1: Minor Cleanup Update
|
||||
|
||||
*July 17, 2025 • 2 min read*
|
||||
|
||||
---
|
||||
|
||||
A small maintenance release that removes unused code and improves documentation.
|
||||
|
||||
## 🎯 What's Changed
|
||||
|
||||
- **Removed unused StealthConfig** from `crawl4ai/browser_manager.py`
|
||||
- **Updated documentation** with better examples and parameter explanations
|
||||
- **Fixed virtual scroll configuration** examples in docs
|
||||
|
||||
## 🧹 Code Cleanup
|
||||
|
||||
Removed unused `StealthConfig` import and configuration that wasn't being used anywhere in the codebase. The project uses its own custom stealth implementation through JavaScript injection instead.
|
||||
|
||||
```python
|
||||
# Removed unused code:
|
||||
from playwright_stealth import StealthConfig
|
||||
stealth_config = StealthConfig(...) # This was never used
|
||||
```
|
||||
|
||||
## 📖 Documentation Updates
|
||||
|
||||
- Fixed adaptive crawling parameter examples
|
||||
- Updated session management documentation
|
||||
- Corrected virtual scroll configuration examples
|
||||
|
||||
## 🚀 Installation
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.1
|
||||
```
|
||||
|
||||
No breaking changes - upgrade directly from v0.7.0.
|
||||
|
||||
---
|
||||
|
||||
Questions? Issues?
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
@@ -0,0 +1,98 @@
|
||||
# 🚀 Crawl4AI v0.7.2: CI/CD & Dependency Optimization Update
|
||||
|
||||
*July 25, 2025 • 3 min read*
|
||||
|
||||
---
|
||||
|
||||
This release introduces automated CI/CD pipelines for seamless releases and optimizes dependencies for a lighter, more efficient package.
|
||||
|
||||
## 🎯 What's New
|
||||
|
||||
### 🔄 Automated Release Pipeline
|
||||
- **GitHub Actions CI/CD**: Automated PyPI and Docker Hub releases on tag push
|
||||
- **Multi-platform Docker images**: Support for both AMD64 and ARM64 architectures
|
||||
- **Version consistency checks**: Ensures tag, package, and Docker versions align
|
||||
- **Automated release notes**: GitHub releases created automatically
|
||||
|
||||
### 📦 Dependency Optimization
|
||||
- **Moved sentence-transformers to optional dependencies**: Significantly reduces default installation size
|
||||
- **Lighter Docker images**: Optimized Dockerfile for faster builds and smaller images
|
||||
- **Better dependency management**: Core vs. optional dependencies clearly separated
|
||||
|
||||
## 🏗️ CI/CD Pipeline
|
||||
|
||||
The new automated release process ensures consistent, reliable releases:
|
||||
|
||||
```yaml
|
||||
# Trigger releases with a simple tag
|
||||
git tag v0.7.2
|
||||
git push origin v0.7.2
|
||||
|
||||
# Automatically:
|
||||
# ✅ Validates version consistency
|
||||
# ✅ Builds and publishes to PyPI
|
||||
# ✅ Builds multi-platform Docker images
|
||||
# ✅ Pushes to Docker Hub with proper tags
|
||||
# ✅ Creates GitHub release
|
||||
```
|
||||
|
||||
## 💾 Lighter Installation
|
||||
|
||||
Default installation is now significantly smaller:
|
||||
|
||||
```bash
|
||||
# Core installation (smaller, faster)
|
||||
pip install crawl4ai==0.7.2
|
||||
|
||||
# With ML features (includes sentence-transformers)
|
||||
pip install crawl4ai[transformer]==0.7.2
|
||||
|
||||
# Full installation
|
||||
pip install crawl4ai[all]==0.7.2
|
||||
```
|
||||
|
||||
## 🐳 Docker Improvements
|
||||
|
||||
Enhanced Docker support with multi-platform images:
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.2
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Available tags:
|
||||
# - unclecode/crawl4ai:0.7.2 (specific version)
|
||||
# - unclecode/crawl4ai:0.7 (minor version)
|
||||
# - unclecode/crawl4ai:0 (major version)
|
||||
# - unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
## 🔧 Technical Details
|
||||
|
||||
### Dependency Changes
|
||||
- `sentence-transformers` moved from required to optional dependencies
|
||||
- Reduces default installation by ~500MB
|
||||
- No impact on functionality when transformer features aren't needed
|
||||
|
||||
### CI/CD Configuration
|
||||
- GitHub Actions workflows for automated releases
|
||||
- Version validation before publishing
|
||||
- Parallel PyPI and Docker Hub deployments
|
||||
- Automatic tagging strategy for Docker images
|
||||
|
||||
## 🚀 Installation
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.2
|
||||
```
|
||||
|
||||
No breaking changes - direct upgrade from v0.7.0 or v0.7.1.
|
||||
|
||||
---
|
||||
|
||||
Questions? Issues?
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
*P.S. The new CI/CD pipeline will make future releases faster and more reliable. Thanks for your patience as we improve our release process!*
|
||||
@@ -0,0 +1,170 @@
|
||||
# 🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update
|
||||
|
||||
*August 6, 2025 • 5 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.3—the Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
|
||||
- **Flexible Docker LLM Providers**: Configure LLM providers via environment variables
|
||||
- **Bug Fixes**: Resolved several critical issues for better stability
|
||||
- **Documentation Updates**: Clearer examples and improved API documentation
|
||||
|
||||
## 🎨 Multi-URL Configurations: One Size Doesn't Fit All
|
||||
|
||||
**The Problem:** You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handling—caching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.
|
||||
|
||||
**My Solution:** I implemented URL-specific configurations that let you define different strategies for different URL patterns in a single crawl batch. First match wins, with optional fallback support.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
|
||||
|
||||
# Define specialized configs for different content types
|
||||
configs = [
|
||||
# Documentation sites - aggressive caching, include links
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*docs*", "*documentation*"],
|
||||
cache_mode="write",
|
||||
markdown_generator_options={"include_links": True}
|
||||
),
|
||||
|
||||
# News/blog sites - fresh content, scroll for lazy loading
|
||||
CrawlerRunConfig(
|
||||
url_matcher=lambda url: 'blog' in url or 'news' in url,
|
||||
cache_mode="bypass",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight/2);"
|
||||
),
|
||||
|
||||
# API endpoints - structured extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*.json", "*api*"],
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o-mini",
|
||||
extraction_type="structured"
|
||||
)
|
||||
),
|
||||
|
||||
# Default fallback for everything else
|
||||
CrawlerRunConfig() # No url_matcher = matches everything
|
||||
]
|
||||
|
||||
# Crawl multiple URLs with appropriate configs
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=[
|
||||
"https://docs.python.org/3/", # → Uses documentation config
|
||||
"https://blog.python.org/", # → Uses blog config
|
||||
"https://api.github.com/users", # → Uses API config
|
||||
"https://example.com/" # → Uses default config
|
||||
],
|
||||
config=configs
|
||||
)
|
||||
```
|
||||
|
||||
**Matching Capabilities:**
|
||||
- **String Patterns**: Wildcards like `"*.pdf"`, `"*/blog/*"`
|
||||
- **Function Matchers**: Lambda functions for complex logic
|
||||
- **Mixed Matchers**: Combine strings and functions with AND/OR logic
|
||||
- **Fallback Support**: Default config when nothing matches
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Mixed Content Sites**: Handle blogs, docs, and downloads in one crawl
|
||||
- **Multi-Domain Crawling**: Different strategies per domain without separate runs
|
||||
- **Reduced Complexity**: No more if/else forests in your extraction code
|
||||
- **Better Performance**: Each URL gets exactly the processing it needs
|
||||
|
||||
## 🐳 Docker: Flexible LLM Provider Configuration
|
||||
|
||||
**The Problem:** Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.
|
||||
|
||||
**My Solution:** Configure LLM providers via environment variables. Switch providers without touching code or rebuilding images.
|
||||
|
||||
### Deployment Flexibility
|
||||
|
||||
```bash
|
||||
# Option 1: Direct environment variables
|
||||
docker run -d \
|
||||
-e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
|
||||
-e GROQ_API_KEY="your-key" \
|
||||
-p 11235:11235 \
|
||||
unclecode/crawl4ai:latest
|
||||
|
||||
# Option 2: Using .llm.env file (recommended for production)
|
||||
# Create .llm.env file:
|
||||
# LLM_PROVIDER=openai/gpt-4o-mini
|
||||
# OPENAI_API_KEY=your-openai-key
|
||||
# GROQ_API_KEY=your-groq-key
|
||||
|
||||
docker run -d \
|
||||
--env-file .llm.env \
|
||||
-p 11235:11235 \
|
||||
unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
Override per request when needed:
|
||||
```python
|
||||
# Use default provider from .llm.env
|
||||
response = requests.post("http://localhost:11235/crawl", json={
|
||||
"url": "https://example.com",
|
||||
"extraction_strategy": {"type": "llm"}
|
||||
})
|
||||
|
||||
# Override to use different provider for this specific request
|
||||
response = requests.post("http://localhost:11235/crawl", json={
|
||||
"url": "https://complex-page.com",
|
||||
"extraction_strategy": {
|
||||
"type": "llm",
|
||||
"provider": "openai/gpt-4" # Override default
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Cost Optimization**: Use cheaper models for simple tasks, premium for complex
|
||||
- **A/B Testing**: Compare provider performance without deployment changes
|
||||
- **Fallback Strategies**: Switch providers on-the-fly during outages
|
||||
- **Development Flexibility**: Test locally with one provider, deploy with another
|
||||
- **Secure Configuration**: Keep API keys in `.llm.env` file, not in commands
|
||||
|
||||
## 🔧 Bug Fixes & Improvements
|
||||
|
||||
This release includes several important bug fixes that improve stability and reliability:
|
||||
|
||||
- **URL Matcher Fallback**: Fixed edge cases in URL pattern matching logic
|
||||
- **Memory Management**: Resolved memory leaks in long-running crawl sessions
|
||||
- **Sitemap Processing**: Fixed redirect handling in sitemap fetching
|
||||
- **Table Extraction**: Improved table detection and extraction accuracy
|
||||
- **Error Handling**: Better error messages and recovery from network failures
|
||||
|
||||
## 📚 Documentation Enhancements
|
||||
|
||||
Based on community feedback, we've updated:
|
||||
- Clearer examples for multi-URL configuration
|
||||
- Improved CrawlResult documentation with all available fields
|
||||
- Fixed typos and inconsistencies across documentation
|
||||
- Added real-world URLs in examples for better understanding
|
||||
- New comprehensive demo showcasing all v0.7.3 features
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thanks to our contributors and the entire community for feedback and bug reports.
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [Full Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
- [Feature Demo](https://github.com/unclecode/crawl4ai/blob/main/docs/releases_review/demo_v0.7.3.py)
|
||||
|
||||
---
|
||||
|
||||
*Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deployment—they're game changers!*
|
||||
|
||||
**Happy Crawling! 🕷️**
|
||||
|
||||
*- The Crawl4AI Team*
|
||||
@@ -0,0 +1,314 @@
|
||||
# Crawl4AI v0.7.6 Release Notes
|
||||
|
||||
*Release Date: October 22, 2025*
|
||||
|
||||
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
|
||||
|
||||
## 🎯 What's New
|
||||
|
||||
### Webhook Support for Docker Job Queue API
|
||||
|
||||
The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!
|
||||
|
||||
**Key Capabilities:**
|
||||
|
||||
- ✅ **Universal Webhook Support**: Both `/crawl/job` and `/llm/job` endpoints now support webhooks
|
||||
- ✅ **Flexible Delivery Modes**: Choose notification-only or include full data in the webhook payload
|
||||
- ✅ **Reliable Delivery**: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
|
||||
- ✅ **Custom Authentication**: Add custom headers for webhook authentication
|
||||
- ✅ **Global Configuration**: Set default webhook URL in `config.yml` for all jobs
|
||||
- ✅ **Task Type Identification**: Distinguish between `crawl` and `llm_extraction` tasks
|
||||
|
||||
### How It Works
|
||||
|
||||
Instead of constantly checking job status:
|
||||
|
||||
**OLD WAY (Polling):**
|
||||
```python
|
||||
# Submit job
|
||||
response = requests.post("http://localhost:11235/crawl/job", json=payload)
|
||||
task_id = response.json()['task_id']
|
||||
|
||||
# Poll until complete
|
||||
while True:
|
||||
status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
|
||||
if status.json()['status'] == 'completed':
|
||||
break
|
||||
time.sleep(5) # Wait and try again
|
||||
```
|
||||
|
||||
**NEW WAY (Webhooks):**
|
||||
```python
|
||||
# Submit job with webhook
|
||||
payload = {
|
||||
"urls": ["https://example.com"],
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhook",
|
||||
"webhook_data_in_payload": True
|
||||
}
|
||||
}
|
||||
response = requests.post("http://localhost:11235/crawl/job", json=payload)
|
||||
|
||||
# Done! Webhook will notify you when complete
|
||||
# Your webhook handler receives the results automatically
|
||||
```
|
||||
|
||||
### Crawl Job Webhooks
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:11235/crawl/job \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"urls": ["https://example.com"],
|
||||
"browser_config": {"headless": true},
|
||||
"crawler_config": {"cache_mode": "bypass"},
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhooks/crawl-complete",
|
||||
"webhook_data_in_payload": false,
|
||||
"webhook_headers": {
|
||||
"X-Webhook-Secret": "your-secret-token"
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### LLM Extraction Job Webhooks (NEW!)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:11235/llm/job \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"url": "https://example.com/article",
|
||||
"q": "Extract the article title, author, and publication date",
|
||||
"schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
|
||||
"provider": "openai/gpt-4o-mini",
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhooks/llm-complete",
|
||||
"webhook_data_in_payload": true
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Webhook Payload Structure
|
||||
|
||||
**Success (with data):**
|
||||
```json
|
||||
{
|
||||
"task_id": "llm_1698765432",
|
||||
"task_type": "llm_extraction",
|
||||
"status": "completed",
|
||||
"timestamp": "2025-10-22T10:30:00.000000+00:00",
|
||||
"urls": ["https://example.com/article"],
|
||||
"data": {
|
||||
"extracted_content": {
|
||||
"title": "Understanding Web Scraping",
|
||||
"author": "John Doe",
|
||||
"date": "2025-10-22"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Failure:**
|
||||
```json
|
||||
{
|
||||
"task_id": "crawl_abc123",
|
||||
"task_type": "crawl",
|
||||
"status": "failed",
|
||||
"timestamp": "2025-10-22T10:30:00.000000+00:00",
|
||||
"urls": ["https://example.com"],
|
||||
"error": "Connection timeout after 30s"
|
||||
}
|
||||
```
|
||||
|
||||
### Simple Webhook Handler Example
|
||||
|
||||
```python
|
||||
from flask import Flask, request, jsonify
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
@app.route('/webhook', methods=['POST'])
|
||||
def handle_webhook():
|
||||
payload = request.json
|
||||
|
||||
task_id = payload['task_id']
|
||||
task_type = payload['task_type']
|
||||
status = payload['status']
|
||||
|
||||
if status == 'completed':
|
||||
if 'data' in payload:
|
||||
# Process data directly
|
||||
data = payload['data']
|
||||
else:
|
||||
# Fetch from API
|
||||
endpoint = 'crawl' if task_type == 'crawl' else 'llm'
|
||||
response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
|
||||
data = response.json()
|
||||
|
||||
# Your business logic here
|
||||
print(f"Job {task_id} completed!")
|
||||
|
||||
elif status == 'failed':
|
||||
error = payload.get('error', 'Unknown error')
|
||||
print(f"Job {task_id} failed: {error}")
|
||||
|
||||
return jsonify({"status": "received"}), 200
|
||||
|
||||
app.run(port=8080)
|
||||
```
|
||||
|
||||
## 📊 Performance Improvements
|
||||
|
||||
- **Reduced Server Load**: Eliminates constant polling requests
|
||||
- **Lower Latency**: Instant notification vs. polling interval delay
|
||||
- **Better Resource Usage**: Frees up client connections while jobs run in background
|
||||
- **Scalable Architecture**: Handles high-volume crawling workflows efficiently
|
||||
|
||||
## 🐛 Bug Fixes
|
||||
|
||||
- Fixed webhook configuration serialization for Pydantic HttpUrl fields
|
||||
- Improved error handling in webhook delivery service
|
||||
- Enhanced Redis task storage for webhook config persistence
|
||||
|
||||
## 🌍 Expected Real-World Impact
|
||||
|
||||
### For Web Scraping Workflows
|
||||
- **Reduced Costs**: Less API calls = lower bandwidth and server costs
|
||||
- **Better UX**: Instant notifications improve user experience
|
||||
- **Scalability**: Handle 100s of concurrent jobs without polling overhead
|
||||
|
||||
### For LLM Extraction Pipelines
|
||||
- **Async Processing**: Submit LLM extraction jobs and move on
|
||||
- **Batch Processing**: Queue multiple extractions, get notified as they complete
|
||||
- **Integration**: Easy integration with workflow automation tools (Zapier, n8n, etc.)
|
||||
|
||||
### For Microservices
|
||||
- **Event-Driven**: Perfect for event-driven microservice architectures
|
||||
- **Decoupling**: Decouple job submission from result processing
|
||||
- **Reliability**: Automatic retries ensure webhooks are delivered
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- Webhook configuration is optional
|
||||
- Existing code continues to work without modification
|
||||
- Polling is still supported for jobs without webhook config
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### New Documentation
|
||||
- **[WEBHOOK_EXAMPLES.md](../deploy/docker/WEBHOOK_EXAMPLES.md)** - Comprehensive webhook usage guide
|
||||
- **[docker_webhook_example.py](../docs/examples/docker_webhook_example.py)** - Working code examples
|
||||
|
||||
### Updated Documentation
|
||||
- **[Docker README](../deploy/docker/README.md)** - Added webhook sections
|
||||
- API documentation with webhook examples
|
||||
|
||||
## 🛠️ Migration Guide
|
||||
|
||||
No migration needed! Webhooks are opt-in:
|
||||
|
||||
1. **To use webhooks**: Add `webhook_config` to your job payload
|
||||
2. **To keep polling**: Continue using your existing code
|
||||
|
||||
### Quick Start
|
||||
|
||||
```python
|
||||
# Just add webhook_config to your existing payload
|
||||
payload = {
|
||||
# Your existing configuration
|
||||
"urls": ["https://example.com"],
|
||||
"browser_config": {...},
|
||||
"crawler_config": {...},
|
||||
|
||||
# NEW: Add webhook configuration
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhook",
|
||||
"webhook_data_in_payload": True
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Global Webhook Configuration (config.yml)
|
||||
|
||||
```yaml
|
||||
webhooks:
|
||||
enabled: true
|
||||
default_url: "https://myapp.com/webhooks/default" # Optional
|
||||
data_in_payload: false
|
||||
retry:
|
||||
max_attempts: 5
|
||||
initial_delay_ms: 1000
|
||||
max_delay_ms: 32000
|
||||
timeout_ms: 30000
|
||||
headers:
|
||||
User-Agent: "Crawl4AI-Webhook/1.0"
|
||||
```
|
||||
|
||||
## 🚀 Upgrade Instructions
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest image
|
||||
docker pull unclecode/crawl4ai:0.7.6
|
||||
|
||||
# Or use latest tag
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run with webhook support
|
||||
docker run -d \
|
||||
-p 11235:11235 \
|
||||
--env-file .llm.env \
|
||||
--name crawl4ai \
|
||||
unclecode/crawl4ai:0.7.6
|
||||
```
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Use notification-only mode** for large results - fetch data separately to avoid large webhook payloads
|
||||
2. **Set custom headers** for webhook authentication and request tracking
|
||||
3. **Configure global default webhook** for consistent handling across all jobs
|
||||
4. **Implement idempotent webhook handlers** - same webhook may be delivered multiple times on retry
|
||||
5. **Use structured schemas** with LLM extraction for predictable webhook data
|
||||
|
||||
## 🎬 Demo
|
||||
|
||||
Try the release demo:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.6.py
|
||||
```
|
||||
|
||||
This comprehensive demo showcases:
|
||||
- Crawl job webhooks (notification-only and with data)
|
||||
- LLM extraction webhooks (with JSON schema support)
|
||||
- Custom headers for authentication
|
||||
- Webhook retry mechanism
|
||||
- Real-time webhook receiver
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
|
||||
|
||||
## 📞 Support
|
||||
|
||||
- **Documentation**: https://docs.crawl4ai.com
|
||||
- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
|
||||
- **Discord**: https://discord.gg/crawl4ai
|
||||
|
||||
---
|
||||
|
||||
**Happy crawling with webhooks!** 🕷️🪝
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,138 @@
|
||||
# Crawl4AI 0.4.3: Major Performance Boost & LLM Integration
|
||||
|
||||
We're excited to announce Crawl4AI 0.4.3, focusing on three key areas: Speed & Efficiency, LLM Integration, and Core Platform Improvements. This release significantly improves crawling performance while adding powerful new LLM-powered features.
|
||||
|
||||
## ⚡ Speed & Efficiency Improvements
|
||||
|
||||
### 1. Memory-Adaptive Dispatcher System
|
||||
The new dispatcher system provides intelligent resource management and real-time monitoring:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, DisplayMode
|
||||
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher, CrawlerMonitor
|
||||
|
||||
async def main():
|
||||
urls = ["https://example1.com", "https://example2.com"] * 50
|
||||
|
||||
# Configure memory-aware dispatch
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=80.0, # Auto-throttle at 80% memory
|
||||
check_interval=0.5, # Check every 0.5 seconds
|
||||
max_session_permit=20, # Max concurrent sessions
|
||||
monitor=CrawlerMonitor( # Real-time monitoring
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await dispatcher.run_urls(
|
||||
urls=urls,
|
||||
crawler=crawler,
|
||||
config=CrawlerRunConfig()
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Streaming Support
|
||||
Process crawled URLs in real-time instead of waiting for all results:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(stream=True)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
async for result in await crawler.arun_many(urls, config=config):
|
||||
print(f"Got result for {result.url}")
|
||||
# Process each result immediately
|
||||
```
|
||||
|
||||
### 3. LXML-Based Scraping
|
||||
New LXML scraping strategy offering up to 20x faster parsing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
scraping_strategy=LXMLWebScrapingStrategy(),
|
||||
cache_mode=CacheMode.ENABLED
|
||||
)
|
||||
```
|
||||
|
||||
## 🤖 LLM Integration
|
||||
|
||||
### 1. LLM-Powered Markdown Generation
|
||||
Smart content filtering and organization using LLMs:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
markdown_generator=DefaultMarkdownGenerator(
|
||||
content_filter=LLMContentFilter(
|
||||
provider="openai/gpt-4o",
|
||||
instruction="Extract technical documentation and code examples"
|
||||
)
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Automatic Schema Generation
|
||||
Generate extraction schemas instantly using LLMs instead of manual CSS/XPath writing:
|
||||
|
||||
```python
|
||||
schema = JsonCssExtractionStrategy.generate_schema(
|
||||
html_content,
|
||||
schema_type="CSS",
|
||||
query="Extract product name, price, and description"
|
||||
)
|
||||
```
|
||||
|
||||
## 🔧 Core Improvements
|
||||
|
||||
### 1. Proxy Support & Rotation
|
||||
Integrated proxy support with automatic rotation and verification:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Robots.txt Compliance
|
||||
Built-in robots.txt support with SQLite caching:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(check_robots_txt=True)
|
||||
result = await crawler.arun(url, config=config)
|
||||
if result.status_code == 403:
|
||||
print("Access blocked by robots.txt")
|
||||
```
|
||||
|
||||
### 3. URL Redirection Tracking
|
||||
Track final URLs after redirects:
|
||||
|
||||
```python
|
||||
result = await crawler.arun(url)
|
||||
print(f"Initial URL: {url}")
|
||||
print(f"Final URL: {result.redirected_url}")
|
||||
```
|
||||
|
||||
## Performance Impact
|
||||
|
||||
- Memory usage reduced by up to 40% with adaptive dispatcher
|
||||
- Parsing speed increased up to 20x with LXML strategy
|
||||
- Streaming reduces memory footprint for large crawls by ~60%
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
```
|
||||
|
||||
For complete examples, check our [demo repository](https://github.com/unclecode/crawl4ai/examples).
|
||||
|
||||
## Stay Connected
|
||||
|
||||
- Star us on [GitHub](https://github.com/unclecode/crawl4ai)
|
||||
- Follow [@unclecode](https://twitter.com/unclecode)
|
||||
- Join our [Discord](https://discord.gg/crawl4ai)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
@@ -0,0 +1,318 @@
|
||||
# 🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update
|
||||
|
||||
*September 29, 2025 • 8 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.5—focused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Docker Hooks System**: Custom Python functions at key pipeline points with function-based API
|
||||
- **Function-Based Hooks**: New `hooks_to_string()` utility with Docker client auto-conversion
|
||||
- **Enhanced LLM Integration**: Custom providers with temperature control
|
||||
- **HTTPS Preservation**: Secure internal link handling
|
||||
- **Bug Fixes**: Resolved multiple community-reported issues
|
||||
- **Improved Docker Error Handling**: Better debugging and reliability
|
||||
|
||||
## 🔧 Docker Hooks System: Pipeline Customization
|
||||
|
||||
Every scraping project needs custom logic—authentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.
|
||||
|
||||
### Real Example: Authentication & Performance
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
# Real working hooks for httpbin.org
|
||||
hooks_config = {
|
||||
"on_page_context_created": """
|
||||
async def hook(page, context, **kwargs):
|
||||
print("Hook: Setting up page context")
|
||||
# Block images to speed up crawling
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
print("Hook: Images blocked")
|
||||
return page
|
||||
""",
|
||||
|
||||
"before_retrieve_html": """
|
||||
async def hook(page, context, **kwargs):
|
||||
print("Hook: Before retrieving HTML")
|
||||
# Scroll to bottom to load lazy content
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(1000)
|
||||
print("Hook: Scrolled to bottom")
|
||||
return page
|
||||
""",
|
||||
|
||||
"before_goto": """
|
||||
async def hook(page, context, url, **kwargs):
|
||||
print(f"Hook: About to navigate to {url}")
|
||||
# Add custom headers
|
||||
await page.set_extra_http_headers({
|
||||
'X-Test-Header': 'crawl4ai-hooks-test'
|
||||
})
|
||||
return page
|
||||
"""
|
||||
}
|
||||
|
||||
# Test with Docker API
|
||||
payload = {
|
||||
"urls": ["https://httpbin.org/html"],
|
||||
"hooks": {
|
||||
"code": hooks_config,
|
||||
"timeout": 30
|
||||
}
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/crawl", json=payload)
|
||||
result = response.json()
|
||||
|
||||
if result.get('success'):
|
||||
print("✅ Hooks executed successfully!")
|
||||
print(f"Content length: {len(result.get('markdown', ''))} characters")
|
||||
```
|
||||
|
||||
**Available Hook Points:**
|
||||
- `on_browser_created`: Browser setup
|
||||
- `on_page_context_created`: Page context configuration
|
||||
- `before_goto`: Pre-navigation setup
|
||||
- `after_goto`: Post-navigation processing
|
||||
- `on_user_agent_updated`: User agent changes
|
||||
- `on_execution_started`: Crawl initialization
|
||||
- `before_retrieve_html`: Pre-extraction processing
|
||||
- `before_return_html`: Final HTML processing
|
||||
|
||||
### Function-Based Hooks API
|
||||
|
||||
Writing hooks as strings works, but lacks IDE support and type checking. v0.7.5 introduces a function-based approach with automatic conversion!
|
||||
|
||||
**Option 1: Using the `hooks_to_string()` Utility**
|
||||
|
||||
```python
|
||||
from crawl4ai import hooks_to_string
|
||||
import requests
|
||||
|
||||
# Define hooks as regular Python functions (with full IDE support!)
|
||||
async def on_page_context_created(page, context, **kwargs):
|
||||
"""Block images to speed up crawling"""
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
await page.set_viewport_size({"width": 1920, "height": 1080})
|
||||
return page
|
||||
|
||||
async def before_goto(page, context, url, **kwargs):
|
||||
"""Add custom headers"""
|
||||
await page.set_extra_http_headers({
|
||||
'X-Crawl4AI': 'v0.7.5',
|
||||
'X-Custom-Header': 'my-value'
|
||||
})
|
||||
return page
|
||||
|
||||
# Convert functions to strings
|
||||
hooks_code = hooks_to_string({
|
||||
"on_page_context_created": on_page_context_created,
|
||||
"before_goto": before_goto
|
||||
})
|
||||
|
||||
# Use with REST API
|
||||
payload = {
|
||||
"urls": ["https://httpbin.org/html"],
|
||||
"hooks": {"code": hooks_code, "timeout": 30}
|
||||
}
|
||||
response = requests.post("http://localhost:11235/crawl", json=payload)
|
||||
```
|
||||
|
||||
**Option 2: Docker Client with Automatic Conversion (Recommended!)**
|
||||
|
||||
```python
|
||||
from crawl4ai.docker_client import Crawl4aiDockerClient
|
||||
|
||||
# Define hooks as functions (same as above)
|
||||
async def on_page_context_created(page, context, **kwargs):
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
return page
|
||||
|
||||
async def before_retrieve_html(page, context, **kwargs):
|
||||
# Scroll to load lazy content
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(1000)
|
||||
return page
|
||||
|
||||
# Use Docker client - conversion happens automatically!
|
||||
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
|
||||
|
||||
results = await client.crawl(
|
||||
urls=["https://httpbin.org/html"],
|
||||
hooks={
|
||||
"on_page_context_created": on_page_context_created,
|
||||
"before_retrieve_html": before_retrieve_html
|
||||
},
|
||||
hooks_timeout=30
|
||||
)
|
||||
|
||||
if results and results.success:
|
||||
print(f"✅ Hooks executed! HTML length: {len(results.html)}")
|
||||
```
|
||||
|
||||
**Benefits of Function-Based Hooks:**
|
||||
- ✅ Full IDE support (autocomplete, syntax highlighting)
|
||||
- ✅ Type checking and linting
|
||||
- ✅ Easier to test and debug
|
||||
- ✅ Reusable across projects
|
||||
- ✅ Automatic conversion in Docker client
|
||||
- ✅ No breaking changes - string hooks still work!
|
||||
|
||||
## 🤖 Enhanced LLM Integration
|
||||
|
||||
Enhanced LLM integration with custom providers, temperature control, and base URL configuration.
|
||||
|
||||
### Multi-Provider Support
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
# Test with different providers
|
||||
async def test_llm_providers():
|
||||
# OpenAI with custom temperature
|
||||
openai_strategy = LLMExtractionStrategy(
|
||||
provider="gemini/gemini-2.5-flash-lite",
|
||||
api_token="your-api-token",
|
||||
temperature=0.7, # New in v0.7.5
|
||||
instruction="Summarize this page in one sentence"
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://example.com",
|
||||
config=CrawlerRunConfig(extraction_strategy=openai_strategy)
|
||||
)
|
||||
|
||||
if result.success:
|
||||
print("✅ LLM extraction completed")
|
||||
print(result.extracted_content)
|
||||
|
||||
# Docker API with enhanced LLM config
|
||||
llm_payload = {
|
||||
"url": "https://example.com",
|
||||
"f": "llm",
|
||||
"q": "Summarize this page in one sentence.",
|
||||
"provider": "gemini/gemini-2.5-flash-lite",
|
||||
"temperature": 0.7
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/md", json=llm_payload)
|
||||
```
|
||||
|
||||
**New Features:**
|
||||
- Custom `temperature` parameter for creativity control
|
||||
- `base_url` for custom API endpoints
|
||||
- Multi-provider environment variable support
|
||||
- Docker API integration
|
||||
|
||||
## 🔒 HTTPS Preservation
|
||||
|
||||
**The Problem:** Modern web apps require HTTPS everywhere. When crawlers downgrade internal links from HTTPS to HTTP, authentication breaks and security warnings appear.
|
||||
|
||||
**Solution:** HTTPS preservation maintains secure protocols throughout crawling.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy
|
||||
|
||||
async def test_https_preservation():
|
||||
# Enable HTTPS preservation
|
||||
url_filter = URLPatternFilter(
|
||||
patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
exclude_external_links=True,
|
||||
preserve_https_for_internal_links=True, # New in v0.7.5
|
||||
deep_crawl_strategy=BFSDeepCrawlStrategy(
|
||||
max_depth=2,
|
||||
max_pages=5,
|
||||
filter_chain=FilterChain([url_filter])
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
async for result in await crawler.arun(
|
||||
url="https://quotes.toscrape.com",
|
||||
config=config
|
||||
):
|
||||
# All internal links maintain HTTPS
|
||||
internal_links = [link['href'] for link in result.links['internal']]
|
||||
https_links = [link for link in internal_links if link.startswith('https://')]
|
||||
|
||||
print(f"HTTPS links preserved: {len(https_links)}/{len(internal_links)}")
|
||||
for link in https_links[:3]:
|
||||
print(f" → {link}")
|
||||
```
|
||||
|
||||
## 🛠️ Bug Fixes and Improvements
|
||||
|
||||
### Major Fixes
|
||||
- **URL Processing**: Fixed '+' sign preservation in query parameters (#1332)
|
||||
- **Proxy Configuration**: Enhanced proxy string parsing (old `proxy` parameter deprecated)
|
||||
- **Docker Error Handling**: Comprehensive error messages with status codes
|
||||
- **Memory Management**: Fixed leaks in long-running sessions
|
||||
- **JWT Authentication**: Fixed Docker JWT validation issues (#1442)
|
||||
- **Playwright Stealth**: Fixed stealth features for Playwright integration (#1481)
|
||||
- **API Configuration**: Fixed config handling to prevent overriding user-provided settings (#1505)
|
||||
- **Docker Filter Serialization**: Resolved JSON encoding errors in deep crawl strategy (#1419)
|
||||
- **LLM Provider Support**: Fixed custom LLM provider integration for adaptive crawler (#1291)
|
||||
- **Performance Issues**: Resolved backoff strategy failures and timeout handling (#989)
|
||||
|
||||
### Community-Reported Issues Fixed
|
||||
This release addresses multiple issues reported by the community through GitHub issues and Discord discussions:
|
||||
- Fixed browser configuration reference errors
|
||||
- Resolved dependency conflicts with cssselect
|
||||
- Improved error messaging for failed authentications
|
||||
- Enhanced compatibility with various proxy configurations
|
||||
- Fixed edge cases in URL normalization
|
||||
|
||||
### Configuration Updates
|
||||
```python
|
||||
# Old proxy config (deprecated)
|
||||
# browser_config = BrowserConfig(proxy="http://proxy:8080")
|
||||
|
||||
# New enhanced proxy config
|
||||
browser_config = BrowserConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy:8080",
|
||||
"username": "optional-user",
|
||||
"password": "optional-pass"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
1. **Python 3.10+ Required**: Upgrade from Python 3.9
|
||||
2. **Proxy Parameter Deprecated**: Use new `proxy_config` structure
|
||||
3. **New Dependency**: Added `cssselect` for better CSS handling
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
```bash
|
||||
# Install latest version
|
||||
pip install crawl4ai==0.7.5
|
||||
|
||||
# Docker deployment
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
**Try the Demo:**
|
||||
```bash
|
||||
# Run working examples
|
||||
python docs/releases_review/demo_v0.7.5.py
|
||||
```
|
||||
|
||||
**Resources:**
|
||||
- 📖 Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- 🐙 GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- 💬 Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- 🐦 Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
@@ -0,0 +1,626 @@
|
||||
# 🚀 Crawl4AI v0.7.7: The Self-Hosting & Monitoring Update
|
||||
|
||||
*November 14, 2025 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.7—the Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **📊 Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool status
|
||||
- **🔌 Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data
|
||||
- **⚡ WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards
|
||||
- **🎮 Control Actions**: Manual browser management (kill, restart, cleanup)
|
||||
- **🔥 Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion
|
||||
- **🧹 Janitor Cleanup System**: Automatic resource management with event logging
|
||||
- **📈 Production Metrics**: 6 critical metrics for operational excellence
|
||||
- **🏭 Integration Ready**: Prometheus, alerting, and log aggregation examples
|
||||
- **🐛 Critical Bug Fixes**: Async LLM extraction, DFS crawling, viewport config, and more
|
||||
|
||||
## 📊 Real-time Monitoring Dashboard: Complete Visibility
|
||||
|
||||
**The Problem:** Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the container—memory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.
|
||||
|
||||
**My Solution:** I built a complete real-time monitoring system with an interactive dashboard, comprehensive REST API, WebSocket streaming, and manual control actions. Now you have full transparency and control over your crawling infrastructure.
|
||||
|
||||
### The Self-Hosting Value Proposition
|
||||
|
||||
Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:
|
||||
|
||||
- **🔒 Data Privacy**: Your data never leaves your infrastructure
|
||||
- **💰 Cost Control**: No per-request pricing or rate limits
|
||||
- **🎯 Full Customization**: Complete control over configurations and strategies
|
||||
- **📊 Complete Transparency**: Real-time visibility into every aspect
|
||||
- **⚡ Performance**: Direct access without network overhead
|
||||
- **🛡️ Enterprise Security**: Keep workflows behind your firewall
|
||||
|
||||
### Interactive Monitoring Dashboard
|
||||
|
||||
Access the dashboard at `http://localhost:11235/dashboard` to see:
|
||||
|
||||
- **System Health Overview**: CPU, memory, network, and uptime in real-time
|
||||
- **Live Request Tracking**: Active and completed requests with full details
|
||||
- **Browser Pool Management**: Interactive table with permanent/hot/cold browsers
|
||||
- **Janitor Events Log**: Automatic cleanup activities
|
||||
- **Error Monitoring**: Full context error logs
|
||||
|
||||
The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.
|
||||
|
||||
## 🔌 Monitor API: Programmatic Access
|
||||
|
||||
**The Problem:** Monitoring dashboards are great for humans, but automation and integration require programmatic access.
|
||||
|
||||
**My Solution:** A comprehensive REST API that exposes all monitoring data for integration with your existing infrastructure.
|
||||
|
||||
### System Health Endpoint
|
||||
|
||||
```python
|
||||
import httpx
|
||||
import asyncio
|
||||
|
||||
async def monitor_system_health():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/health")
|
||||
health = response.json()
|
||||
|
||||
print(f"Container Metrics:")
|
||||
print(f" CPU: {health['container']['cpu_percent']:.1f}%")
|
||||
print(f" Memory: {health['container']['memory_percent']:.1f}%")
|
||||
print(f" Uptime: {health['container']['uptime_seconds']}s")
|
||||
|
||||
print(f"\nBrowser Pool:")
|
||||
print(f" Permanent: {health['pool']['permanent']['active']} active")
|
||||
print(f" Hot Pool: {health['pool']['hot']['count']} browsers")
|
||||
print(f" Cold Pool: {health['pool']['cold']['count']} browsers")
|
||||
|
||||
print(f"\nStatistics:")
|
||||
print(f" Total Requests: {health['stats']['total_requests']}")
|
||||
print(f" Success Rate: {health['stats']['success_rate_percent']:.1f}%")
|
||||
print(f" Avg Latency: {health['stats']['avg_latency_ms']:.0f}ms")
|
||||
|
||||
asyncio.run(monitor_system_health())
|
||||
```
|
||||
|
||||
### Request Tracking
|
||||
|
||||
```python
|
||||
async def track_requests():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/requests")
|
||||
requests_data = response.json()
|
||||
|
||||
print(f"Active Requests: {len(requests_data['active'])}")
|
||||
print(f"Completed Requests: {len(requests_data['completed'])}")
|
||||
|
||||
# See details of recent requests
|
||||
for req in requests_data['completed'][:5]:
|
||||
status_icon = "✅" if req['success'] else "❌"
|
||||
print(f"{status_icon} {req['endpoint']} - {req['latency_ms']:.0f}ms")
|
||||
```
|
||||
|
||||
### Browser Pool Management
|
||||
|
||||
```python
|
||||
async def monitor_browser_pool():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/browsers")
|
||||
browsers = response.json()
|
||||
|
||||
print(f"Pool Summary:")
|
||||
print(f" Total Browsers: {browsers['summary']['total_count']}")
|
||||
print(f" Total Memory: {browsers['summary']['total_memory_mb']} MB")
|
||||
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
|
||||
|
||||
# List all browsers
|
||||
for browser in browsers['permanent']:
|
||||
print(f"🔥 Permanent: {browser['browser_id'][:8]}... | "
|
||||
f"Requests: {browser['request_count']} | "
|
||||
f"Memory: {browser['memory_mb']:.0f} MB")
|
||||
```
|
||||
|
||||
### Endpoint Performance Statistics
|
||||
|
||||
```python
|
||||
async def get_endpoint_stats():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/endpoints/stats")
|
||||
stats = response.json()
|
||||
|
||||
print("Endpoint Analytics:")
|
||||
for endpoint, data in stats.items():
|
||||
print(f" {endpoint}:")
|
||||
print(f" Requests: {data['count']}")
|
||||
print(f" Avg Latency: {data['avg_latency_ms']:.0f}ms")
|
||||
print(f" Success Rate: {data['success_rate_percent']:.1f}%")
|
||||
```
|
||||
|
||||
### Complete API Reference
|
||||
|
||||
The Monitor API includes these endpoints:
|
||||
|
||||
- `GET /monitor/health` - System health with pool statistics
|
||||
- `GET /monitor/requests` - Active and completed request tracking
|
||||
- `GET /monitor/browsers` - Browser pool details and efficiency
|
||||
- `GET /monitor/endpoints/stats` - Per-endpoint performance analytics
|
||||
- `GET /monitor/timeline?minutes=5` - Time-series data for charts
|
||||
- `GET /monitor/logs/janitor?limit=10` - Cleanup activity logs
|
||||
- `GET /monitor/logs/errors?limit=10` - Error logs with context
|
||||
- `POST /monitor/actions/cleanup` - Force immediate cleanup
|
||||
- `POST /monitor/actions/kill_browser` - Kill specific browser
|
||||
- `POST /monitor/actions/restart_browser` - Restart browser
|
||||
- `POST /monitor/stats/reset` - Reset accumulated statistics
|
||||
|
||||
## ⚡ WebSocket Streaming: Real-time Updates
|
||||
|
||||
**The Problem:** Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.
|
||||
|
||||
**My Solution:** WebSocket streaming with 2-second update intervals for building custom real-time dashboards.
|
||||
|
||||
### WebSocket Integration Example
|
||||
|
||||
```python
|
||||
import websockets
|
||||
import json
|
||||
import asyncio
|
||||
|
||||
async def monitor_realtime():
|
||||
uri = "ws://localhost:11235/monitor/ws"
|
||||
|
||||
async with websockets.connect(uri) as websocket:
|
||||
print("Connected to real-time monitoring stream")
|
||||
|
||||
while True:
|
||||
# Receive update every 2 seconds
|
||||
data = await websocket.recv()
|
||||
update = json.loads(data)
|
||||
|
||||
# Access all monitoring data
|
||||
print(f"\n--- Update at {update['timestamp']} ---")
|
||||
print(f"Memory: {update['health']['container']['memory_percent']:.1f}%")
|
||||
print(f"Active Requests: {len(update['requests']['active'])}")
|
||||
print(f"Total Browsers: {update['browsers']['summary']['total_count']}")
|
||||
|
||||
if update['errors']:
|
||||
print(f"⚠️ Recent Errors: {len(update['errors'])}")
|
||||
|
||||
asyncio.run(monitor_realtime())
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Custom Dashboards**: Build tailored monitoring UIs for your team
|
||||
- **Real-time Alerting**: Trigger alerts instantly when metrics exceed thresholds
|
||||
- **Integration**: Feed live data into monitoring tools like Grafana
|
||||
- **Automation**: React to events in real-time without polling
|
||||
|
||||
## 🔥 Smart Browser Pool: 3-Tier Architecture
|
||||
|
||||
**The Problem:** Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.
|
||||
|
||||
**My Solution:** A smart 3-tier browser pool that automatically adapts to usage patterns.
|
||||
|
||||
### How It Works
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async def demonstrate_browser_pool():
|
||||
async with httpx.AsyncClient() as client:
|
||||
# Request 1-3: Default config → Uses permanent browser
|
||||
print("Phase 1: Using permanent browser")
|
||||
for i in range(3):
|
||||
await client.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={"urls": [f"https://httpbin.org/html?req={i}"]}
|
||||
)
|
||||
print(f" Request {i+1}: Reused permanent browser")
|
||||
|
||||
# Request 4-6: Custom viewport → Cold pool (first use)
|
||||
print("\nPhase 2: Custom config creates cold pool browser")
|
||||
viewport_config = {"viewport": {"width": 1280, "height": 720}}
|
||||
for i in range(4):
|
||||
await client.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={
|
||||
"urls": [f"https://httpbin.org/json?v={i}"],
|
||||
"browser_config": viewport_config
|
||||
}
|
||||
)
|
||||
if i < 2:
|
||||
print(f" Request {i+1}: Cold pool browser")
|
||||
else:
|
||||
print(f" Request {i+1}: Promoted to hot pool! (after 3 uses)")
|
||||
|
||||
# Check pool status
|
||||
response = await client.get("http://localhost:11235/monitor/browsers")
|
||||
browsers = response.json()
|
||||
|
||||
print(f"\nPool Status:")
|
||||
print(f" Permanent: {len(browsers['permanent'])} (always active)")
|
||||
print(f" Hot: {len(browsers['hot'])} (frequently used configs)")
|
||||
print(f" Cold: {len(browsers['cold'])} (on-demand)")
|
||||
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
|
||||
|
||||
asyncio.run(demonstrate_browser_pool())
|
||||
```
|
||||
|
||||
**Pool Tiers:**
|
||||
|
||||
- **🔥 Permanent Browser**: Always-on, default configuration, instant response
|
||||
- **♨️ Hot Pool**: Browsers promoted after 3+ uses, kept warm for quick access
|
||||
- **❄️ Cold Pool**: On-demand browsers for variant configs, cleaned up when idle
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Memory Efficiency**: 10x reduction in memory usage vs creating browsers per request
|
||||
- **Performance**: Instant access to frequently-used configurations
|
||||
- **Automatic Optimization**: Pool adapts to your usage patterns
|
||||
- **Resource Management**: Janitor automatically cleans up idle browsers
|
||||
|
||||
## 🧹 Janitor System: Automatic Cleanup
|
||||
|
||||
**The Problem:** Long-running crawlers accumulate idle browsers and consume memory over time.
|
||||
|
||||
**My Solution:** An automatic janitor system that monitors and cleans up idle resources.
|
||||
|
||||
```python
|
||||
async def monitor_janitor_activity():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/logs/janitor?limit=5")
|
||||
logs = response.json()
|
||||
|
||||
print("Recent Cleanup Activities:")
|
||||
for log in logs:
|
||||
print(f" {log['timestamp']}: {log['message']}")
|
||||
|
||||
# Example output:
|
||||
# 2025-11-14 10:30:00: Cleaned up 2 cold pool browsers (idle > 5min)
|
||||
# 2025-11-14 10:25:00: Browser reuse rate: 85.3%
|
||||
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)
|
||||
```
|
||||
|
||||
## 🎮 Control Actions: Manual Management
|
||||
|
||||
**The Problem:** Sometimes you need to manually intervene—kill a stuck browser, force cleanup, or restart resources.
|
||||
|
||||
**My Solution:** Manual control actions via the API for operational troubleshooting.
|
||||
|
||||
### Force Cleanup
|
||||
|
||||
```python
|
||||
async def force_cleanup():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/monitor/actions/cleanup")
|
||||
result = response.json()
|
||||
|
||||
print(f"Cleanup completed:")
|
||||
print(f" Browsers cleaned: {result.get('cleaned_count', 0)}")
|
||||
print(f" Memory freed: {result.get('memory_freed_mb', 0):.1f} MB")
|
||||
```
|
||||
|
||||
### Kill Specific Browser
|
||||
|
||||
```python
|
||||
async def kill_stuck_browser(browser_id: str):
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
"http://localhost:11235/monitor/actions/kill_browser",
|
||||
json={"browser_id": browser_id}
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
print(f"✅ Browser {browser_id} killed successfully")
|
||||
```
|
||||
|
||||
### Reset Statistics
|
||||
|
||||
```python
|
||||
async def reset_stats():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/monitor/stats/reset")
|
||||
print("📊 Statistics reset for fresh monitoring")
|
||||
```
|
||||
|
||||
## 📈 Production Integration Patterns
|
||||
|
||||
### Prometheus Integration
|
||||
|
||||
```python
|
||||
# Export metrics for Prometheus scraping
|
||||
async def export_prometheus_metrics():
|
||||
async with httpx.AsyncClient() as client:
|
||||
health = await client.get("http://localhost:11235/monitor/health")
|
||||
data = health.json()
|
||||
|
||||
# Export in Prometheus format
|
||||
metrics = f"""
|
||||
# HELP crawl4ai_memory_usage_percent Memory usage percentage
|
||||
# TYPE crawl4ai_memory_usage_percent gauge
|
||||
crawl4ai_memory_usage_percent {data['container']['memory_percent']}
|
||||
|
||||
# HELP crawl4ai_request_success_rate Request success rate
|
||||
# TYPE crawl4ai_request_success_rate gauge
|
||||
crawl4ai_request_success_rate {data['stats']['success_rate_percent']}
|
||||
|
||||
# HELP crawl4ai_browser_pool_count Total browsers in pool
|
||||
# TYPE crawl4ai_browser_pool_count gauge
|
||||
crawl4ai_browser_pool_count {data['pool']['permanent']['active'] + data['pool']['hot']['count'] + data['pool']['cold']['count']}
|
||||
"""
|
||||
return metrics
|
||||
```
|
||||
|
||||
### Alerting Example
|
||||
|
||||
```python
|
||||
async def check_alerts():
|
||||
async with httpx.AsyncClient() as client:
|
||||
health = await client.get("http://localhost:11235/monitor/health")
|
||||
data = health.json()
|
||||
|
||||
# Memory alert
|
||||
if data['container']['memory_percent'] > 80:
|
||||
print("🚨 ALERT: Memory usage above 80%")
|
||||
# Trigger cleanup
|
||||
await client.post("http://localhost:11235/monitor/actions/cleanup")
|
||||
|
||||
# Success rate alert
|
||||
if data['stats']['success_rate_percent'] < 90:
|
||||
print("🚨 ALERT: Success rate below 90%")
|
||||
# Check error logs
|
||||
errors = await client.get("http://localhost:11235/monitor/logs/errors")
|
||||
print(f"Recent errors: {len(errors.json())}")
|
||||
|
||||
# Latency alert
|
||||
if data['stats']['avg_latency_ms'] > 5000:
|
||||
print("🚨 ALERT: Average latency above 5s")
|
||||
```
|
||||
|
||||
### Key Metrics to Track
|
||||
|
||||
```python
|
||||
CRITICAL_METRICS = {
|
||||
"memory_usage": {
|
||||
"current": "container.memory_percent",
|
||||
"target": "<80%",
|
||||
"alert_threshold": ">80%",
|
||||
"action": "Force cleanup or scale"
|
||||
},
|
||||
"success_rate": {
|
||||
"current": "stats.success_rate_percent",
|
||||
"target": ">95%",
|
||||
"alert_threshold": "<90%",
|
||||
"action": "Check error logs"
|
||||
},
|
||||
"avg_latency": {
|
||||
"current": "stats.avg_latency_ms",
|
||||
"target": "<2000ms",
|
||||
"alert_threshold": ">5000ms",
|
||||
"action": "Investigate slow requests"
|
||||
},
|
||||
"browser_reuse_rate": {
|
||||
"current": "browsers.summary.reuse_rate_percent",
|
||||
"target": ">80%",
|
||||
"alert_threshold": "<60%",
|
||||
"action": "Check pool configuration"
|
||||
},
|
||||
"total_browsers": {
|
||||
"current": "browsers.summary.total_count",
|
||||
"target": "<15",
|
||||
"alert_threshold": ">20",
|
||||
"action": "Check for browser leaks"
|
||||
},
|
||||
"error_frequency": {
|
||||
"current": "len(errors)",
|
||||
"target": "<5/hour",
|
||||
"alert_threshold": ">10/hour",
|
||||
"action": "Review error patterns"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🐛 Critical Bug Fixes
|
||||
|
||||
This release includes significant bug fixes that improve stability and performance:
|
||||
|
||||
### Async LLM Extraction (#1590)
|
||||
|
||||
**The Problem:** LLM extraction was blocking async execution, causing URLs to be processed sequentially instead of in parallel (issue #1055).
|
||||
|
||||
**The Fix:** Resolved the blocking issue to enable true parallel processing for LLM extraction.
|
||||
|
||||
```python
|
||||
# Before v0.7.7: Sequential processing
|
||||
# After v0.7.7: True parallel processing
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
urls = ["url1", "url2", "url3", "url4"]
|
||||
|
||||
# Now processes truly in parallel with LLM extraction
|
||||
results = await crawler.arun_many(
|
||||
urls,
|
||||
config=CrawlerRunConfig(
|
||||
extraction_strategy=LLMExtractionStrategy(...)
|
||||
)
|
||||
)
|
||||
# 4x faster for parallel LLM extraction!
|
||||
```
|
||||
|
||||
**Expected Impact:** Major performance improvement for batch LLM extraction workflows.
|
||||
|
||||
### DFS Deep Crawling (#1607)
|
||||
|
||||
**The Problem:** DFS (Depth-First Search) deep crawl strategy had implementation issues.
|
||||
|
||||
**The Fix:** Enhanced DFSDeepCrawlStrategy with proper seen URL tracking and improved documentation.
|
||||
|
||||
### Browser & Crawler Config Documentation (#1609)
|
||||
|
||||
**The Problem:** Documentation didn't match the actual `async_configs.py` implementation.
|
||||
|
||||
**The Fix:** Updated all configuration documentation to accurately reflect the current implementation.
|
||||
|
||||
### Sitemap Seeder (#1598)
|
||||
|
||||
**The Problem:** Sitemap parsing and URL normalization issues in AsyncUrlSeeder (issue #1559).
|
||||
|
||||
**The Fix:** Added comprehensive tests and fixes for sitemap namespace parsing and URL normalization.
|
||||
|
||||
### Remove Overlay Elements (#1529)
|
||||
|
||||
**The Problem:** The `remove_overlay_elements` functionality wasn't working (issue #1396).
|
||||
|
||||
**The Fix:** Fixed by properly calling the injected JavaScript function.
|
||||
|
||||
### Viewport Configuration (#1495)
|
||||
|
||||
**The Problem:** Viewport configuration wasn't working in managed browsers (issue #1490).
|
||||
|
||||
**The Fix:** Added proper viewport size configuration support for browser launch.
|
||||
|
||||
### Managed Browser CDP Timing (#1528)
|
||||
|
||||
**The Problem:** CDP (Chrome DevTools Protocol) endpoint verification had timing issues causing connection failures (issue #1445).
|
||||
|
||||
**The Fix:** Added exponential backoff for CDP endpoint verification to handle timing variations.
|
||||
|
||||
### Security Updates
|
||||
|
||||
- **pyOpenSSL**: Updated from >=24.3.0 to >=25.3.0 to address security vulnerability
|
||||
- Added verification tests for the security update
|
||||
|
||||
### Docker Fixes
|
||||
|
||||
- **Port Standardization**: Fixed inconsistent port usage (11234 vs 11235) - now standardized to 11235
|
||||
- **LLM Environment**: Fixed LLM API key handling for multi-provider support (PR #1537)
|
||||
- **Error Handling**: Improved Docker API error messages with comprehensive status codes
|
||||
- **Serialization**: Fixed `fit_html` property serialization in `/crawl` and `/crawl/stream` endpoints
|
||||
|
||||
### Other Important Fixes
|
||||
|
||||
- **arun_many Returns**: Fixed function to always return a list, even on exception (PR #1530)
|
||||
- **Webhook Serialization**: Properly serialize Pydantic HttpUrl in webhook config
|
||||
- **LLMConfig Documentation**: Fixed casing and variable name consistency (issue #1551)
|
||||
- **Python Version**: Dropped Python 3.9 support, now requires Python >=3.10
|
||||
|
||||
## 📊 Expected Real-World Impact
|
||||
|
||||
### For DevOps & Infrastructure Teams
|
||||
- **Full Visibility**: Know exactly what's happening inside your crawling infrastructure
|
||||
- **Proactive Monitoring**: Catch issues before they become problems
|
||||
- **Resource Optimization**: Identify memory leaks and performance bottlenecks
|
||||
- **Operational Control**: Manual intervention when automated systems need help
|
||||
|
||||
### For Production Deployments
|
||||
- **Enterprise Observability**: Prometheus, Grafana, and alerting integration
|
||||
- **Debugging**: Real-time logs and error tracking
|
||||
- **Capacity Planning**: Historical metrics for scaling decisions
|
||||
- **SLA Monitoring**: Track success rates and latency against targets
|
||||
|
||||
### For Development Teams
|
||||
- **Local Monitoring**: Understand crawler behavior during development
|
||||
- **Performance Testing**: Measure impact of configuration changes
|
||||
- **Troubleshooting**: Quickly identify and fix issues
|
||||
- **Learning**: See exactly how the browser pool works
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- All existing Docker configurations continue to work
|
||||
- No API changes to existing endpoints
|
||||
- Monitoring is additive functionality
|
||||
- No migration required
|
||||
|
||||
## 🚀 Upgrade Instructions
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.7
|
||||
|
||||
# Or use the latest tag
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run with monitoring enabled (default)
|
||||
docker run -d \
|
||||
-p 11235:11235 \
|
||||
--shm-size=1g \
|
||||
--name crawl4ai \
|
||||
unclecode/crawl4ai:0.7.7
|
||||
|
||||
# Access the monitoring dashboard
|
||||
open http://localhost:11235/dashboard
|
||||
```
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
# Upgrade to latest version
|
||||
pip install --upgrade crawl4ai
|
||||
|
||||
# Or install specific version
|
||||
pip install crawl4ai==0.7.7
|
||||
```
|
||||
|
||||
## 🎬 Try the Demo
|
||||
|
||||
Run the comprehensive demo that showcases all monitoring features:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.7.py
|
||||
```
|
||||
|
||||
**The demo includes:**
|
||||
1. System health overview with live metrics
|
||||
2. Request tracking with active/completed monitoring
|
||||
3. Browser pool management (permanent/hot/cold)
|
||||
4. Complete Monitor API endpoint examples
|
||||
5. WebSocket streaming demonstration
|
||||
6. Control actions (cleanup, kill, restart)
|
||||
7. Production metrics and alerting patterns
|
||||
8. Self-hosting value proposition
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### New Documentation
|
||||
- **[Self-Hosting Guide](https://docs.crawl4ai.com/core/self-hosting/)** - Complete self-hosting documentation with monitoring
|
||||
- **Demo Script**: `docs/releases_review/demo_v0.7.7.py` - Working examples
|
||||
|
||||
### Updated Documentation
|
||||
- **Docker Deployment** → **Self-Hosting** (renamed for better positioning)
|
||||
- Added comprehensive monitoring sections
|
||||
- Production integration patterns
|
||||
- WebSocket streaming examples
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Start with the dashboard** - Visit `/dashboard` to get familiar with the monitoring system
|
||||
2. **Track the 6 key metrics** - Memory, success rate, latency, reuse rate, browser count, errors
|
||||
3. **Set up alerting early** - Use the Monitor API to build alerts before issues occur
|
||||
4. **Monitor browser pool efficiency** - Aim for >80% reuse rate for optimal performance
|
||||
5. **Use WebSocket for custom dashboards** - Build tailored monitoring UIs for your team
|
||||
6. **Leverage Prometheus integration** - Export metrics for long-term storage and analysis
|
||||
7. **Check janitor logs** - Understand automatic cleanup patterns
|
||||
8. **Use control actions judiciously** - Manual interventions are for exceptional cases
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.
|
||||
|
||||
The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.
|
||||
|
||||
## 📞 Support & Resources
|
||||
|
||||
- **📖 Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **🐙 GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **💬 Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **🐦 Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
- **📊 Dashboard**: `http://localhost:11235/dashboard` (when running)
|
||||
|
||||
---
|
||||
|
||||
**Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platform—it's a game changer for operational excellence!**
|
||||
|
||||
**Happy crawling with full visibility!** 🕷️📊
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,327 @@
|
||||
# Crawl4AI v0.7.8: Stability & Bug Fix Release
|
||||
|
||||
*December 2025*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.7.8—a focused stability release that addresses 11 bugs reported by the community. While there are no new features in this release, these fixes resolve important issues affecting Docker deployments, LLM extraction, URL handling, and dependency compatibility.
|
||||
|
||||
## What's Fixed at a Glance
|
||||
|
||||
- **Docker API**: Fixed ContentRelevanceFilter deserialization, ProxyConfig serialization, and cache folder permissions
|
||||
- **LLM Extraction**: Configurable rate limiter backoff, HTML input format support, and proper URL handling for raw HTML
|
||||
- **URL Handling**: Correct relative URL resolution after JavaScript redirects
|
||||
- **Dependencies**: Replaced deprecated PyPDF2 with pypdf, Pydantic v2 ConfigDict compatibility
|
||||
- **AdaptiveCrawler**: Fixed query expansion to actually use LLM instead of hardcoded mock data
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### Docker & API Fixes
|
||||
|
||||
#### ContentRelevanceFilter Deserialization (#1642)
|
||||
|
||||
**The Problem:** When sending deep crawl requests to the Docker API with `ContentRelevanceFilter`, the server failed to deserialize the filter, causing requests to fail.
|
||||
|
||||
**The Fix:** I added `ContentRelevanceFilter` to the public exports and enhanced the deserialization logic with dynamic imports.
|
||||
|
||||
```python
|
||||
# This now works correctly in Docker API
|
||||
import httpx
|
||||
|
||||
request = {
|
||||
"urls": ["https://docs.example.com"],
|
||||
"crawler_config": {
|
||||
"deep_crawl_strategy": {
|
||||
"type": "BFSDeepCrawlStrategy",
|
||||
"max_depth": 2,
|
||||
"filter_chain": [
|
||||
{
|
||||
"type": "ContentRelevanceFilter",
|
||||
"query": "API documentation",
|
||||
"threshold": 0.3
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/crawl", json=request)
|
||||
# Previously failed, now works!
|
||||
```
|
||||
|
||||
#### ProxyConfig JSON Serialization (#1629)
|
||||
|
||||
**The Problem:** `BrowserConfig.to_dict()` failed when `proxy_config` was set because `ProxyConfig` wasn't being serialized to a dictionary.
|
||||
|
||||
**The Fix:** `ProxyConfig.to_dict()` is now called during serialization.
|
||||
|
||||
```python
|
||||
from crawl4ai import BrowserConfig
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
proxy = ProxyConfig(
|
||||
server="http://proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass"
|
||||
)
|
||||
|
||||
config = BrowserConfig(headless=True, proxy_config=proxy)
|
||||
|
||||
# Previously raised TypeError, now works
|
||||
config_dict = config.to_dict()
|
||||
json.dumps(config_dict) # Valid JSON
|
||||
```
|
||||
|
||||
#### Docker Cache Folder Permissions (#1638)
|
||||
|
||||
**The Problem:** The `.cache` folder in the Docker image had incorrect permissions, causing crawling to fail when caching was enabled.
|
||||
|
||||
**The Fix:** Corrected ownership and permissions during image build.
|
||||
|
||||
```bash
|
||||
# Cache now works correctly in Docker
|
||||
docker run -d -p 11235:11235 \
|
||||
--shm-size=1g \
|
||||
-v ./my-cache:/app/.cache \
|
||||
unclecode/crawl4ai:0.7.8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### LLM & Extraction Fixes
|
||||
|
||||
#### Configurable Rate Limiter Backoff (#1269)
|
||||
|
||||
**The Problem:** The LLM rate limiting backoff parameters were hardcoded, making it impossible to adjust retry behavior for different API rate limits.
|
||||
|
||||
**The Fix:** `LLMConfig` now accepts three new parameters for complete control over retry behavior.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMConfig
|
||||
|
||||
# Default behavior (unchanged)
|
||||
default_config = LLMConfig(provider="openai/gpt-4o-mini")
|
||||
# backoff_base_delay=2, backoff_max_attempts=3, backoff_exponential_factor=2
|
||||
|
||||
# Custom configuration for APIs with strict rate limits
|
||||
custom_config = LLMConfig(
|
||||
provider="openai/gpt-4o-mini",
|
||||
backoff_base_delay=5, # Wait 5 seconds on first retry
|
||||
backoff_max_attempts=5, # Try up to 5 times
|
||||
backoff_exponential_factor=3 # Multiply delay by 3 each attempt
|
||||
)
|
||||
|
||||
# Retry sequence: 5s -> 15s -> 45s -> 135s -> 405s
|
||||
```
|
||||
|
||||
#### LLM Strategy HTML Input Support (#1178)
|
||||
|
||||
**The Problem:** `LLMExtractionStrategy` always sent markdown to the LLM, but some extraction tasks work better with HTML structure preserved.
|
||||
|
||||
**The Fix:** Added `input_format` parameter supporting `"markdown"`, `"html"`, `"fit_markdown"`, `"cleaned_html"`, and `"fit_html"`.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMExtractionStrategy, LLMConfig
|
||||
|
||||
# Default: markdown input (unchanged)
|
||||
markdown_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Extract product information"
|
||||
)
|
||||
|
||||
# NEW: HTML input - preserves table/list structure
|
||||
html_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Extract the data table preserving structure",
|
||||
input_format="html"
|
||||
)
|
||||
|
||||
# NEW: Filtered markdown - only relevant content
|
||||
fit_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Summarize the main content",
|
||||
input_format="fit_markdown"
|
||||
)
|
||||
```
|
||||
|
||||
#### Raw HTML URL Variable (#1116)
|
||||
|
||||
**The Problem:** When using `url="raw:<html>..."`, the entire HTML content was being passed to extraction strategies as the URL parameter, polluting LLM prompts.
|
||||
|
||||
**The Fix:** The URL is now correctly set to `"Raw HTML"` for raw HTML inputs.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
html = "<html><body><h1>Test</h1></body></html>"
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url=f"raw:{html}",
|
||||
config=CrawlerRunConfig(extraction_strategy=my_strategy)
|
||||
)
|
||||
# extraction_strategy receives url="Raw HTML" instead of the HTML blob
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### URL Handling Fix
|
||||
|
||||
#### Relative URLs After Redirects (#1268)
|
||||
|
||||
**The Problem:** When JavaScript caused a page redirect, relative links were resolved against the original URL instead of the final URL.
|
||||
|
||||
**The Fix:** `redirected_url` now captures the actual page URL after all JavaScript execution completes.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Page at /old-page redirects via JS to /new-page
|
||||
result = await crawler.arun(url="https://example.com/old-page")
|
||||
|
||||
# BEFORE: redirected_url = "https://example.com/old-page"
|
||||
# AFTER: redirected_url = "https://example.com/new-page"
|
||||
|
||||
# Links are now correctly resolved against the final URL
|
||||
for link in result.links['internal']:
|
||||
print(link['href']) # Relative links resolved correctly
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Dependency & Compatibility Fixes
|
||||
|
||||
#### PyPDF2 Replaced with pypdf (#1412)
|
||||
|
||||
**The Problem:** PyPDF2 was deprecated in 2022 and is no longer maintained.
|
||||
|
||||
**The Fix:** Replaced with the actively maintained `pypdf` library.
|
||||
|
||||
```python
|
||||
# Installation (unchanged)
|
||||
pip install crawl4ai[pdf]
|
||||
|
||||
# The PDF processor now uses pypdf internally
|
||||
# No code changes required - API remains the same
|
||||
```
|
||||
|
||||
#### Pydantic v2 ConfigDict Compatibility (#678)
|
||||
|
||||
**The Problem:** Using the deprecated `class Config` syntax caused deprecation warnings with Pydantic v2.
|
||||
|
||||
**The Fix:** Migrated to `model_config = ConfigDict(...)` syntax.
|
||||
|
||||
```python
|
||||
# No more deprecation warnings when importing crawl4ai models
|
||||
from crawl4ai.models import CrawlResult
|
||||
from crawl4ai import CrawlerRunConfig, BrowserConfig
|
||||
|
||||
# All models are now Pydantic v2 compatible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### AdaptiveCrawler Fix
|
||||
|
||||
#### Query Expansion Using LLM (#1621)
|
||||
|
||||
**The Problem:** The `EmbeddingStrategy` in AdaptiveCrawler had commented-out LLM code and was using hardcoded mock query variations instead.
|
||||
|
||||
**The Fix:** Uncommented and activated the LLM call for actual query expansion.
|
||||
|
||||
```python
|
||||
# AdaptiveCrawler query expansion now actually uses the LLM
|
||||
# Instead of hardcoded variations like:
|
||||
# variations = {'queries': ['what are the best vegetables...']}
|
||||
|
||||
# The LLM generates relevant query variations based on your actual query
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Code Formatting Fix
|
||||
|
||||
#### Import Statement Formatting (#1181)
|
||||
|
||||
**The Problem:** When extracting code from web pages, import statements were sometimes concatenated without proper line separation.
|
||||
|
||||
**The Fix:** Import statements now maintain proper newline separation.
|
||||
|
||||
```python
|
||||
# BEFORE: "import osimport sysfrom pathlib import Path"
|
||||
# AFTER:
|
||||
# import os
|
||||
# import sys
|
||||
# from pathlib import Path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- All existing code continues to work without modification
|
||||
- New parameters have sensible defaults matching previous behavior
|
||||
- No API changes to existing functionality
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Instructions
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
# or
|
||||
pip install crawl4ai==0.7.8
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.8
|
||||
|
||||
# Run
|
||||
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.7.8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Run the verification tests to confirm all fixes are working:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.8.py
|
||||
```
|
||||
|
||||
This runs actual tests that verify each bug fix is properly implemented.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thank you to everyone who reported these issues and provided detailed reproduction steps. Your bug reports make Crawl4AI better for everyone.
|
||||
|
||||
Issues fixed: #1642, #1638, #1629, #1621, #1412, #1269, #1268, #1181, #1178, #1116, #678
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
---
|
||||
|
||||
**This stability release ensures Crawl4AI works reliably across Docker deployments, LLM extraction workflows, and various edge cases. Thank you for your continued support and feedback!**
|
||||
|
||||
**Happy crawling!**
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,243 @@
|
||||
# Crawl4AI v0.8.0 Release Notes
|
||||
|
||||
**Release Date**: January 2026
|
||||
**Previous Version**: v0.7.6
|
||||
**Status**: Release Candidate
|
||||
|
||||
---
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Critical Security Fixes** for Docker API deployment
|
||||
- **11 New Features** including crash recovery, prefetch mode, and proxy improvements
|
||||
- **Breaking Changes** - see migration guide below
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### 1. Docker API: Hooks Disabled by Default
|
||||
|
||||
**What changed**: Hooks are now disabled by default on the Docker API.
|
||||
|
||||
**Why**: Security fix for Remote Code Execution (RCE) vulnerability.
|
||||
|
||||
**Who is affected**: Users of the Docker API who use the `hooks` parameter in `/crawl` requests.
|
||||
|
||||
**Migration**:
|
||||
```bash
|
||||
# To re-enable hooks (only if you trust all API users):
|
||||
export CRAWL4AI_HOOKS_ENABLED=true
|
||||
```
|
||||
|
||||
### 2. Docker API: file:// URLs Blocked
|
||||
|
||||
**What changed**: The endpoints `/execute_js`, `/screenshot`, `/pdf`, and `/html` now reject `file://` URLs.
|
||||
|
||||
**Why**: Security fix for Local File Inclusion (LFI) vulnerability.
|
||||
|
||||
**Who is affected**: Users who were reading local files via the Docker API.
|
||||
|
||||
**Migration**: Use the Python library directly for local file processing:
|
||||
```python
|
||||
# Instead of API call with file:// URL, use library:
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="file:///path/to/file.html")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: Remote Code Execution via Hooks (CVE Pending)
|
||||
|
||||
**Severity**: CRITICAL (CVSS 10.0)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /crawl` with malicious `hooks` parameter
|
||||
|
||||
**Details**: The `__import__` builtin was available in hook code, allowing attackers to import `os`, `subprocess`, etc. and execute arbitrary commands.
|
||||
|
||||
**Fix**:
|
||||
1. Removed `__import__` from allowed builtins
|
||||
2. Hooks disabled by default (`CRAWL4AI_HOOKS_ENABLED=false`)
|
||||
|
||||
### High: Local File Inclusion via file:// URLs (CVE Pending)
|
||||
|
||||
**Severity**: HIGH (CVSS 8.6)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /execute_js` (and other endpoints) with `file:///etc/passwd`
|
||||
|
||||
**Details**: API endpoints accepted `file://` URLs, allowing attackers to read arbitrary files from the server.
|
||||
|
||||
**Fix**: URL scheme validation now only allows `http://`, `https://`, and `raw:` URLs.
|
||||
|
||||
### Credits
|
||||
|
||||
Discovered by **Neo by ProjectDiscovery** ([projectdiscovery.io](https://projectdiscovery.io)) - December 2025
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. init_scripts Support for BrowserConfig
|
||||
|
||||
Pre-page-load JavaScript injection for stealth evasions.
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
init_scripts=[
|
||||
"Object.defineProperty(navigator, 'webdriver', {get: () => false})"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### 2. CDP Connection Improvements
|
||||
|
||||
- WebSocket URL support (`ws://`, `wss://`)
|
||||
- Proper cleanup with `cdp_cleanup_on_close=True`
|
||||
- Browser reuse across multiple connections
|
||||
|
||||
### 3. Crash Recovery for Deep Crawl Strategies
|
||||
|
||||
All deep crawl strategies (BFS, DFS, Best-First) now support crash recovery:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
|
||||
|
||||
strategy = BFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
resume_state=saved_state, # Resume from checkpoint
|
||||
on_state_change=save_callback # Persist state in real-time
|
||||
)
|
||||
```
|
||||
|
||||
### 4. PDF and MHTML for raw:/file:// URLs
|
||||
|
||||
Generate PDFs and MHTML from cached HTML content.
|
||||
|
||||
### 5. Screenshots for raw:/file:// URLs
|
||||
|
||||
Render cached HTML and capture screenshots.
|
||||
|
||||
### 6. base_url Parameter for CrawlerRunConfig
|
||||
|
||||
Proper URL resolution for raw: HTML processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(base_url='https://example.com')
|
||||
result = await crawler.arun(url='raw:{html}', config=config)
|
||||
```
|
||||
|
||||
### 7. Prefetch Mode for Two-Phase Deep Crawling
|
||||
|
||||
Fast link extraction without full page processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(prefetch=True)
|
||||
```
|
||||
|
||||
### 8. Proxy Rotation and Configuration
|
||||
|
||||
Enhanced proxy rotation with sticky sessions support.
|
||||
|
||||
### 9. Proxy Support for HTTP Strategy
|
||||
|
||||
Non-browser crawler now supports proxies.
|
||||
|
||||
### 10. Browser Pipeline for raw:/file:// URLs
|
||||
|
||||
New `process_in_browser` parameter for browser operations on local content:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
process_in_browser=True, # Force browser processing
|
||||
screenshot=True
|
||||
)
|
||||
result = await crawler.arun(url='raw:<html>...</html>', config=config)
|
||||
```
|
||||
|
||||
### 11. Smart TTL Cache for Sitemap URL Seeder
|
||||
|
||||
Intelligent cache invalidation for sitemaps:
|
||||
|
||||
```python
|
||||
config = SeedingConfig(
|
||||
cache_ttl_hours=24,
|
||||
validate_sitemap_lastmod=True
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### raw: URL Parsing Truncates at # Character
|
||||
|
||||
**Problem**: CSS color codes like `#eee` were being truncated.
|
||||
|
||||
**Before**: `raw:body{background:#eee}` → `body{background:`
|
||||
**After**: `raw:body{background:#eee}` → `body{background:#eee}`
|
||||
|
||||
### Caching System Improvements
|
||||
|
||||
Various fixes to cache validation and persistence.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- Multi-sample schema generation documentation
|
||||
- URL seeder smart TTL cache parameters
|
||||
- Security documentation (SECURITY.md)
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Guide
|
||||
|
||||
### From v0.7.x to v0.8.0
|
||||
|
||||
1. **Update the package**:
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
2. **Docker API users**:
|
||||
- Hooks are now disabled by default
|
||||
- If you need hooks: `export CRAWL4AI_HOOKS_ENABLED=true`
|
||||
- `file://` URLs no longer work on API (use library directly)
|
||||
|
||||
3. **Review security settings**:
|
||||
```yaml
|
||||
# config.yml - recommended for production
|
||||
security:
|
||||
enabled: true
|
||||
jwt_enabled: true
|
||||
```
|
||||
|
||||
4. **Test your integration** before deploying to production
|
||||
|
||||
### Breaking Change Checklist
|
||||
|
||||
- [ ] Check if you use `hooks` parameter in API calls
|
||||
- [ ] Check if you use `file://` URLs via the API
|
||||
- [ ] Update environment variables if needed
|
||||
- [ ] Review security configuration
|
||||
|
||||
---
|
||||
|
||||
## Full Changelog
|
||||
|
||||
See [CHANGELOG.md](../CHANGELOG.md) for complete version history.
|
||||
|
||||
---
|
||||
|
||||
## Contributors
|
||||
|
||||
Thanks to all contributors who made this release possible.
|
||||
|
||||
Special thanks to **Neo by ProjectDiscovery** for responsible security disclosure.
|
||||
|
||||
---
|
||||
|
||||
*For questions or issues, please open a [GitHub Issue](https://github.com/unclecode/crawl4ai/issues).*
|
||||
@@ -0,0 +1,384 @@
|
||||
# Crawl4AI v0.8.5: Anti-Bot, Shadow DOM & 60+ Bug Fixes
|
||||
|
||||
*March 2026 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.5—our biggest release since v0.8.0. This update brings automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes from both our team and the community. If you're running crawls at scale or dealing with protected sites, this one's for you.
|
||||
|
||||
## What's New at a Glance
|
||||
|
||||
- **Anti-Bot Detection & Proxy Escalation**: 3-tier detection with automatic retry, proxy chain, and fallback
|
||||
- **Shadow DOM Flattening**: Extract content hidden inside shadow DOM components
|
||||
- **Deep Crawl Cancellation**: Stop long crawls gracefully with `cancel()` or `should_cancel` callback
|
||||
- **Config Defaults API**: Set once, apply everywhere with `set_defaults()` / `get_defaults()` / `reset_defaults()`
|
||||
- **Source/Sibling Selector**: Extract data spanning sibling elements in JSON extraction schemas
|
||||
- **Consent Popup Removal**: Auto-dismiss cookie banners from 40+ CMP platforms
|
||||
- **Resource Filtering**: Block ads and CSS at the network level with `avoid_ads` / `avoid_css`
|
||||
- **Browser Recycling**: Memory-saving mode and automatic browser restart for long sessions
|
||||
- **GFM Table Compliance**: Proper `| col1 | col2 |` pipe delimiters in markdown output
|
||||
- **60+ Bug Fixes**: Security patches, browser stability, extraction accuracy, and more
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. Anti-Bot Detection, Retry & Fallback
|
||||
|
||||
This is the headline feature. Crawl4AI now automatically detects when a page is blocked by anti-bot protection and takes action—retrying with different proxies or falling back to an alternative fetch method.
|
||||
|
||||
The detection uses three tiers:
|
||||
- **Tier 1**: Known vendor patterns (Cloudflare, Akamai, DataDome, PerimeterX, etc.)
|
||||
- **Tier 2**: Generic block indicators on small pages
|
||||
- **Tier 3**: Structural integrity checks (empty shells, script-heavy pages with no content)
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
# Try direct first, then proxy on bot detection
|
||||
proxy_config=[
|
||||
ProxyConfig.DIRECT,
|
||||
ProxyConfig(server="http://my-proxy:8080"),
|
||||
],
|
||||
max_retries=2,
|
||||
# Optional: fallback when all proxies fail
|
||||
fallback_fetch_function=my_web_unlocker_function,
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://protected-site.com", config=config)
|
||||
|
||||
# Check what happened
|
||||
stats = result.crawl_stats
|
||||
print(f"Resolved by: {stats['resolved_by']}") # "direct", "proxy", or "fallback_fetch"
|
||||
print(f"Proxies tried: {len(stats['proxies_used'])}")
|
||||
```
|
||||
|
||||
The system errs on the side of caution—false positives are cheap (the fallback rescues them), but false negatives mean garbage results. After 5 iterations of real-world testing, it handles everything from Cloudflare challenges to Reddit's 180KB SPA block pages.
|
||||
|
||||
### 2. Shadow DOM Flattening
|
||||
|
||||
Web components with shadow DOM hide their content from regular DOM traversal. The new `flatten_shadow_dom` option serializes shadow DOM content into the light DOM before extraction.
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(flatten_shadow_dom=True)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://some-web-component-site.com", config=config)
|
||||
# Shadow DOM content is now visible in result.html, cleaned_html, and markdown
|
||||
```
|
||||
|
||||
The implementation patches `attachShadow` to force-open closed shadow roots, recursively resolves `<slot>` projections, and strips only shadow-scoped `<style>` tags. It also reorders the JS execution pipeline—`js_code` now runs after `wait_for` + `delay_before_return_html` so your scripts operate on the fully-hydrated page. If you need JS to run before waiting, use the new `js_code_before_wait` parameter.
|
||||
|
||||
### 3. Deep Crawl Cancellation
|
||||
|
||||
All deep crawl strategies (BFS, DFS, BestFirst) now support graceful cancellation:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import DFSDeepCrawlStrategy
|
||||
|
||||
pages_found = 0
|
||||
|
||||
def should_stop():
|
||||
return pages_found >= 50 # Stop after finding enough pages
|
||||
|
||||
async def on_state(state):
|
||||
nonlocal pages_found
|
||||
pages_found = state["pages_crawled"]
|
||||
|
||||
strategy = DFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
max_pages=1000,
|
||||
should_cancel=should_stop, # Sync or async callback
|
||||
on_state_change=on_state,
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(deep_crawl_strategy=strategy)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun("https://example.com", config=config)
|
||||
print(f"Cancelled: {strategy.cancelled}")
|
||||
```
|
||||
|
||||
You can also call `strategy.cancel()` directly from another thread or coroutine.
|
||||
|
||||
### 4. Config Defaults API
|
||||
|
||||
Tired of repeating the same parameters? Set defaults once and they apply to every new instance:
|
||||
|
||||
```python
|
||||
from crawl4ai import BrowserConfig, CrawlerRunConfig
|
||||
|
||||
# Set organization-wide defaults
|
||||
BrowserConfig.set_defaults(headless=True, text_mode=True)
|
||||
CrawlerRunConfig.set_defaults(verbose=False, remove_consent_popups=True)
|
||||
|
||||
# All new instances inherit defaults
|
||||
bc = BrowserConfig() # headless=True, text_mode=True
|
||||
rc = CrawlerRunConfig() # verbose=False, remove_consent_popups=True
|
||||
|
||||
# Explicit params always override
|
||||
bc2 = BrowserConfig(text_mode=False) # text_mode=False, headless still True
|
||||
|
||||
# Inspect and reset
|
||||
print(BrowserConfig.get_defaults()) # {"headless": True, "text_mode": True}
|
||||
BrowserConfig.reset_defaults() # Back to normal
|
||||
```
|
||||
|
||||
### 5. Source/Sibling Selector in JSON Extraction
|
||||
|
||||
Many sites split a single item's data across sibling elements (think Hacker News, where title and score are in separate `<tr>` rows). The new `"source"` field navigates to a sibling before extracting:
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
schema = {
|
||||
"name": "HackerNewsItems",
|
||||
"baseSelector": "tr.athing",
|
||||
"fields": [
|
||||
{"name": "title", "selector": ".titleline > a", "type": "text"},
|
||||
{"name": "link", "selector": ".titleline > a", "type": "attribute", "attribute": "href"},
|
||||
# Navigate to the NEXT sibling <tr> to get the score
|
||||
{"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
|
||||
{"name": "author", "selector": ".hnuser", "type": "text", "source": "+ tr"},
|
||||
]
|
||||
}
|
||||
|
||||
strategy = JsonCssExtractionStrategy(schema=schema)
|
||||
```
|
||||
|
||||
Works in both `JsonCssExtractionStrategy` and `JsonXPathExtractionStrategy`. Falls back gracefully when siblings don't exist.
|
||||
|
||||
### 6. Consent Popup Removal
|
||||
|
||||
A single flag auto-dismisses cookie consent banners from 40+ CMP platforms:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(remove_consent_popups=True)
|
||||
```
|
||||
|
||||
Covers OneTrust, Cookiebot, Didomi, Quantcast, Sourcepoint, Google FundingChoices, TrustArc, ConsentManager, Osano, Iubenda, Complianz, LiveRamp, CookieYes, Klaro, Termly, and many more.
|
||||
|
||||
### 7. Resource Filtering: avoid_ads / avoid_css
|
||||
|
||||
Block ad trackers and CSS resources at the network level for faster, leaner crawls:
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
avoid_ads=True, # Blocks doubleclick, google-analytics, etc.
|
||||
avoid_css=True, # Blocks .css, .less, .scss resources
|
||||
)
|
||||
```
|
||||
|
||||
### 8. Browser Recycling & Memory-Saving Mode
|
||||
|
||||
For long-running crawl sessions:
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
memory_saving_mode=True, # Aggressive cache/V8 heap flags
|
||||
max_pages_before_recycle=100, # Auto-restart browser after N pages
|
||||
)
|
||||
```
|
||||
|
||||
This prevents memory leaks during sustained crawling. The recycling uses a version-based approach that's safe under concurrent load—we fixed three separate deadlock bugs to get this right.
|
||||
|
||||
### 9. GFM Table Compliance
|
||||
|
||||
Tables in markdown output now have proper GitHub-Flavored Markdown pipe delimiters:
|
||||
|
||||
**Before (v0.8.0)**:
|
||||
```
|
||||
Name | Age | City
|
||||
---|---|---
|
||||
Alice | 30 | NYC
|
||||
```
|
||||
|
||||
**After (v0.8.5)**:
|
||||
```
|
||||
| Name | Age | City |
|
||||
| --- | --- | --- |
|
||||
| Alice | 30 | NYC |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Minor Features
|
||||
|
||||
- **`query_llm_config`**: Separate LLM config for adaptive crawler query expansion (#1682)
|
||||
- **`force_viewport_screenshot`**: Screenshot only the viewport, not the full page
|
||||
- **`device_scale_factor`**: Configurable screenshot DPI via BrowserConfig (#1463)
|
||||
- **`redirected_status_code`**: Now available on CrawlResult (#1435)
|
||||
- **`wait_for_images`**: Wait for images to load before taking screenshots (#1792)
|
||||
- **`score_threshold`**: Filter low-quality URLs in BestFirstCrawlingStrategy (#1804)
|
||||
- **`link_preview_timeout`**: Configurable timeout in AdaptiveConfig (#1793)
|
||||
- **`--json-ensure-ascii`**: CLI flag for Unicode preservation in JSON output (#1668)
|
||||
- **`type-list` pipeline**: Chained extraction like `["attribute", "regex"]` in JsonCssExtractionStrategy (#1290)
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: RCE via Deserialization in Docker /crawl Endpoint
|
||||
|
||||
**Severity**: CRITICAL
|
||||
**Affected**: Docker API deployment (v0.8.0 and earlier)
|
||||
|
||||
The `/crawl` endpoint's deserialization logic used `eval()` for certain object types. I removed this entirely and added an allowlist (`ALLOWED_DESERIALIZE_TYPES`) so only known config classes can be instantiated.
|
||||
|
||||
### Critical: Redis CVE-2025-49844 (CVSS 10.0)
|
||||
|
||||
**Affected**: Docker deployments using Redis
|
||||
|
||||
Upgraded Redis to 7.2.7 which patches the Lua use-after-free vulnerability.
|
||||
|
||||
### Additional Security
|
||||
|
||||
- **XSS prevention**: Use DOMParser instead of innerHTML in iframe processing (#1796)
|
||||
- **API token enforcement**: `/token` endpoint now requires `api_token` when configured (#1795)
|
||||
- **Stealth improvements**: `sec-ch-ua` synced with User-Agent, WebGL kept alive in stealth mode
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### Browser & Page Management
|
||||
|
||||
- Fix page reuse race condition when `create_isolated_context=False`
|
||||
- Fix browser context memory leak — signature shrink + LRU eviction (#943)
|
||||
- Fix cascading context crash from duplicate `add_init_script` (#1768)
|
||||
- Fix `simulate_user` destroying page content via ArrowDown keypress
|
||||
- Fix browser recycling deadlock under sustained concurrent load (#1640)
|
||||
- Fix Docker monitor LOCK contention causing pod deadlock (#1754)
|
||||
|
||||
### Proxy & Network
|
||||
|
||||
- Fix proxy auth `ERR_INVALID_AUTH_CREDENTIALS` (#1281)
|
||||
- Fix proxy auth for persistent browser contexts
|
||||
- Fix proxy escalation not re-raising on first exception when chain has alternatives
|
||||
- Fix fallback fetch: run when all proxies crash, skip re-check, never return None
|
||||
|
||||
### Deep Crawling
|
||||
|
||||
- Fix `can_process_url()` to receive normalized URL
|
||||
- Fix `total_score` not calculated for links that fail head extraction
|
||||
- Fix `FilterChain.add_filter` AttributeError on tuple immutability
|
||||
- Fix URL Seeder forcing Common Crawl index for sitemaps (#1746)
|
||||
- Fix `is_external_url` port comparison (#1783)
|
||||
- Prevent AdaptiveCrawler from crawling external domains (#1805)
|
||||
|
||||
### Extraction & Content
|
||||
|
||||
- Fix `<base>` tag ignored in html2text relative link resolution (#1721)
|
||||
- Fix script tag removal losing adjacent text in `cleaned_html` (#1364)
|
||||
- Preserve `class` and `id` attributes in `cleaned_html` (#1782)
|
||||
- Fix nested brackets/parentheses in LINK_PATTERN regex (#1790)
|
||||
- Strip markdown fences in `force_json_response` path for LLM extraction
|
||||
- Guard against None LLM content, propagate `finish_reason` (#1788)
|
||||
- Fix `agenerate_schema()` JSON parsing for Anthropic models
|
||||
- Fix `from_serializable_dict` ignoring plain data dicts with "type" key
|
||||
- Fix MediaItem crash on non-numeric width values like "100%" (#1635)
|
||||
- Fix BM25ContentFilter returning duplicate chunks (#1213)
|
||||
- Fix `css_selector` ignored in LXML scraping for `raw://` URLs (#1484)
|
||||
|
||||
### CLI & Docker
|
||||
|
||||
- Fix deep-crawl CLI outputting only the first page (#1667)
|
||||
- Fix VersionManager ignoring `CRAWL4_AI_BASE_DIRECTORY` env var (#1296)
|
||||
- Fix Docker health endpoint to use dynamic version (#1686)
|
||||
- Add explicit UTF-8 encoding to CLI file output (#1789)
|
||||
- Handle `UnicodeEncodeError` in URL seeder, strip zero-width chars (#1784)
|
||||
- Add TTL expiry for Redis task data to prevent memory growth (#1730)
|
||||
- Add Windows support for crawler monitor keyboard input (#1794)
|
||||
- Fix `scroll_delay` ignored in full-page screenshot scroller
|
||||
- Fix MCP SSE endpoint crash — mount via raw ASGI Route (#1594)
|
||||
- Fix `/llm` per-request provider override, Redis config from host/port/password (#1611, #1817)
|
||||
- Fix screenshot respects `scan_full_page=False` (#1750)
|
||||
- Fix screenshot distortion on Elementor sites (#1370)
|
||||
- Fix deep crawl timeout and `arun_many` dispatcher bypass (#1818, #1509)
|
||||
|
||||
### Other
|
||||
|
||||
- Replace `tf-playwright-stealth` with `playwright-stealth` (#1553)
|
||||
- Allow local embeddings by removing OpenAI fallback (#1658)
|
||||
- Include GoogleSearchCrawler `script.js` in package distribution (#1711)
|
||||
- Fix bs4 deprecation warning (`text` → `string`) (#1077)
|
||||
- Run blocking `chardet.detect` in thread executor (#1751)
|
||||
- Wire `mean_delay`/`max_range` from CrawlerRunConfig into dispatcher rate limiter (#1786)
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
Added a comprehensive **291-test regression suite** covering all major subsystems: core crawl, content processing, extraction strategies, deep crawling, browser management, config serialization, utilities, and edge cases.
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### `cleaned_html` Now Preserves `class` and `id` Attributes
|
||||
|
||||
If you have downstream code that parses `cleaned_html` and assumes no class/id attributes are present, this may need updating. This change enables users to do CSS-based analysis on cleaned HTML.
|
||||
|
||||
### Docker: Redis Upgraded to 7.2.7
|
||||
|
||||
If you pin Redis versions in your deployment, update to 7.2.7 or later.
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Instructions
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
# or
|
||||
pip install crawl4ai==0.8.5
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker pull unclecode/crawl4ai:0.8.5
|
||||
|
||||
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.8.5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Run the verification tests to confirm all features are working:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.8.5.py
|
||||
```
|
||||
|
||||
This runs 13 actual tests that crawl real URLs and verify each feature end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
This release includes contributions from a large number of community members. Thank you to everyone who submitted PRs, reported issues, and provided reproduction steps. Special thanks to all contributors listed in [CONTRIBUTORS.md](../CONTRIBUTORS.md).
|
||||
|
||||
Issues fixed: #462, #880, #943, #1031, #1077, #1183, #1213, #1251, #1281, #1290, #1296, #1308, #1354, #1364, #1370, #1374, #1424, #1435, #1463, #1484, #1487, #1489, #1494, #1503, #1509, #1512, #1520, #1553, #1594, #1601, #1606, #1611, #1622, #1635, #1640, #1658, #1666, #1667, #1668, #1671, #1682, #1686, #1711, #1715, #1716, #1721, #1730, #1731, #1746, #1750, #1751, #1754, #1758, #1762, #1768, #1770, #1776, #1782, #1783, #1784, #1786, #1788, #1789, #1790, #1792, #1793, #1794, #1795, #1796, #1797, #1801, #1803, #1804, #1805, #1815, #1817, #1818, #1824
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
---
|
||||
|
||||
**This is a massive release—10 new features, critical security patches, and 60+ bug fixes. Whether you're dealing with anti-bot protection, shadow DOM sites, or just want more reliable crawls at scale, v0.8.5 has you covered. Thank you for your continued support!**
|
||||
|
||||
**Happy crawling!**
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,90 @@
|
||||
# Crawl4AI v0.9.1: Bug Fixes & PruningContentFilter Whitelist
|
||||
|
||||
*July 2026 - 3 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.9.1, a patch release that ships 12 bug fixes across Docker, browser, core, and extraction, plus one new feature for `PruningContentFilter`.
|
||||
|
||||
No breaking changes. If you're on v0.9.0, upgrade freely.
|
||||
|
||||
## What's new at a glance
|
||||
|
||||
- **PruningContentFilter whitelist**: New `preserve_classes` / `preserve_tags` parameters to protect specific elements from density-based pruning
|
||||
- **Windows browser fix**: Default `channel='chromium'` no longer crashes Playwright on Windows
|
||||
- **Docker hardening**: 6 fixes for auth gate, supervisord, redis, tmpfs, and FastAPI compatibility
|
||||
- **HTTP timeout fix**: `page_timeout` was passed in milliseconds to aiohttp (which expects seconds), effectively disabling timeouts in HTTP mode
|
||||
- **Dependency**: lxml ceiling widened to allow 6.x
|
||||
|
||||
## PruningContentFilter whitelist
|
||||
|
||||
PruningContentFilter's density-based scoring is great at stripping boilerplate, but it sometimes takes short metadata elements — author names, timestamps, attribution lines — along with it. The new `preserve_classes` and `preserve_tags` parameters let you whitelist specific CSS classes or HTML tags that should never be pruned, regardless of their density score.
|
||||
|
||||
```python
|
||||
from crawl4ai.content_filter_strategy import PruningContentFilter
|
||||
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
|
||||
|
||||
filter = PruningContentFilter(
|
||||
threshold=0.48,
|
||||
preserve_classes=["author", "byline", "dateline"],
|
||||
preserve_tags=["time", "address"],
|
||||
)
|
||||
generator = DefaultMarkdownGenerator(content_filter=filter)
|
||||
|
||||
config = CrawlerRunConfig(markdown_generator=generator)
|
||||
```
|
||||
|
||||
Whitelisted nodes skip scoring entirely. Default is empty sets — no behavior change for existing users. (#1900, thanks @hafezparast)
|
||||
|
||||
## Bug fixes
|
||||
|
||||
### Docker (6 fixes)
|
||||
|
||||
- **Auth gate UI**: Dashboard and playground now load when JWT auth is active. Token input bar added to both UIs; all fetch calls use `authFetch()` with Bearer token. API routes remain fail-closed. (#2037)
|
||||
- **Supervisord/Redis dirs**: Pidfile and RDB snapshot dirs moved to writable paths for read-only rootfs deployments. (#2047, thanks @TobiasWallura-xitaso)
|
||||
- **tmpfs writable**: Read-only tmpfs mounts made writable. (#2027, thanks @nightcityblade)
|
||||
- **FastAPI cap**: Pinned FastAPI below 0.137 to avoid compatibility breakage. (#2025, thanks @nightcityblade)
|
||||
- **Redis auth**: Rate-limit Redis storage now authenticates with the configured password. (#2040, thanks @harshmathurx)
|
||||
- **Posture tests updated**: Dashboard/playground moved to public UI assertions to match auth gate fix.
|
||||
|
||||
### Browser (2 fixes)
|
||||
|
||||
- **Windows channel crash**: `channel='chromium'` (the default) caused Playwright to look for a system Chrome install instead of the bundled binary, crashing on Windows with `TargetClosedError`. The default channel is no longer passed to Playwright. (#2051, thanks @fstark96)
|
||||
- **Context snapshot leak**: Browser contexts from snapshot are now properly closed. (#1999, thanks @nightcityblade)
|
||||
|
||||
### Core (2 fixes)
|
||||
|
||||
- **HTTP timeout**: `page_timeout` (60000ms) was passed directly to `aiohttp.ClientTimeout` which expects seconds, making the effective timeout 16.7 hours. Now correctly divided by 1000. (#1894, thanks @hafezparast)
|
||||
- **Best-first ordering**: Batch ordering in `BestFirstCrawlingStrategy` stabilized for deterministic crawl order. (#1998, thanks @nightcityblade)
|
||||
|
||||
### Extraction (1 fix)
|
||||
|
||||
- **Table attributes**: `html2text` now preserves all attributes on table tags when `bypass_tables` is enabled. (#2007)
|
||||
|
||||
### Dependencies (1 fix)
|
||||
|
||||
- **lxml 6.x**: Widened lxml ceiling from `<6` to `<7` so crawl4ai can co-install with packages requiring lxml 6.x (e.g. scrapling). (#2019)
|
||||
|
||||
### Housekeeping
|
||||
|
||||
- Removed dead `normalize_url` duplicates and accidental `adaptive_crawler` copy. (thanks @RajanChavada)
|
||||
- Sponsor logos hosted locally to fix broken GitHub rendering.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
crawl4ai-doctor # verify installation
|
||||
```
|
||||
|
||||
Docker users: pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thanks to the community contributors who made this release possible: @hafezparast (#1894, #1900), @nightcityblade (#1998, #1999, #2025, #2027), @fstark96 (#2051), @TobiasWallura-xitaso (#2047), @harshmathurx (#2040), @RajanChavada (#2042).
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- [Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Issues](https://github.com/unclecode/crawl4ai/issues)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
Reference in New Issue
Block a user