chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,369 @@
|
||||
# Adaptive Crawling: Building Dynamic Knowledge That Grows on Demand
|
||||
|
||||
*Published on January 29, 2025 • 8 min read*
|
||||
|
||||
*By [unclecode](https://x.com/unclecode) • Follow me on [X/Twitter](https://x.com/unclecode) for more web scraping insights*
|
||||
|
||||
---
|
||||
|
||||
## The Knowledge Capacitor
|
||||
|
||||
Imagine a capacitor that stores energy, releasing it precisely when needed. Now imagine that for information. That's Adaptive Crawling—a term I coined to describe a fundamentally different approach to web crawling. Instead of the brute force of traditional deep crawling, we build knowledge dynamically, growing it based on queries and circumstances, like a living organism responding to its environment.
|
||||
|
||||
This isn't just another crawling optimization. It's a paradigm shift from "crawl everything, hope for the best" to "crawl intelligently, know when to stop."
|
||||
|
||||
## Why I Built This
|
||||
|
||||
I've watched too many startups burn through resources with a dangerous misconception: that LLMs make everything efficient. They don't. They make things *possible*, not necessarily *smart*. When you combine brute-force crawling with LLM processing, you're not just wasting time—you're hemorrhaging money on tokens, compute, and opportunity cost.
|
||||
|
||||
Consider this reality:
|
||||
- **Traditional deep crawling**: 500 pages → 50 useful → $15 in LLM tokens → 2 hours wasted
|
||||
- **Adaptive crawling**: 15 pages → 14 useful → $2 in tokens → 10 minutes → **7.5x cost reduction**
|
||||
|
||||
But it's not about crawling less. It's about crawling *right*.
|
||||
|
||||
## The Information Theory Foundation
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
### 🧮 **Pure Statistics, No Magic**
|
||||
|
||||
My first principle was crucial: start with classic statistical approaches. No embeddings. No LLMs. Just pure information theory:
|
||||
|
||||
```python
|
||||
# Information gain calculation - the heart of adaptive crawling
|
||||
def calculate_information_gain(new_page, knowledge_base):
|
||||
new_terms = extract_terms(new_page) - existing_terms(knowledge_base)
|
||||
overlap = calculate_overlap(new_page, knowledge_base)
|
||||
|
||||
# High gain = many new terms + low overlap
|
||||
gain = len(new_terms) / (1 + overlap)
|
||||
return gain
|
||||
```
|
||||
|
||||
This isn't regression to older methods—it's recognition that we've forgotten powerful, efficient solutions in our rush to apply LLMs everywhere.
|
||||
|
||||
</div>
|
||||
|
||||
## The A* of Web Crawling
|
||||
|
||||
Adaptive crawling implements what I call "information scenting"—like A* pathfinding but for knowledge acquisition. Each link is evaluated not randomly, but by its probability of contributing meaningful information toward answering current and future queries.
|
||||
|
||||
<div style="display: flex; align-items: center; background-color: #3f3f44; padding: 20px; margin: 20px 0; border-left: 4px solid #09b5a5;">
|
||||
<div style="font-size: 48px; margin-right: 20px;">🎯</div>
|
||||
<div>
|
||||
<strong>The Scenting Algorithm:</strong><br>
|
||||
From available links, we select those with highest information gain. It's not about following every path—it's about following the <em>right</em> paths. Like a bloodhound following the strongest scent to its target.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
## The Three Pillars of Intelligence
|
||||
|
||||
### 1. Coverage: The Breadth Sensor
|
||||
Measures how well your knowledge spans the query space. Not just "do we have pages?" but "do we have the RIGHT pages?"
|
||||
|
||||
### 2. Consistency: The Coherence Detector
|
||||
Information from multiple sources should align. When pages agree, confidence rises. When they conflict, we need more data.
|
||||
|
||||
### 3. Saturation: The Efficiency Guardian
|
||||
The most crucial metric. When new pages stop adding information, we stop crawling. Simple. Powerful. Ignored by everyone else.
|
||||
|
||||
## Real Impact: Time, Money, and Sanity
|
||||
|
||||
Let me show you what this means for your bottom line:
|
||||
|
||||
### Building a Customer Support Knowledge Base
|
||||
|
||||
**Traditional Approach:**
|
||||
```python
|
||||
# Crawl entire documentation site
|
||||
results = await crawler.crawl_bfs("https://docs.company.com", max_depth=5)
|
||||
# Result: 1,200 pages, 18 hours, $150 in API costs
|
||||
# Useful content: ~100 pages scattered throughout
|
||||
```
|
||||
|
||||
**Adaptive Approach:**
|
||||
```python
|
||||
# Grow knowledge based on actual support queries
|
||||
knowledge = await adaptive.digest(
|
||||
start_url="https://docs.company.com",
|
||||
query="payment processing errors refund policies"
|
||||
)
|
||||
# Result: 45 pages, 12 minutes, $8 in API costs
|
||||
# Useful content: 42 pages, all relevant
|
||||
```
|
||||
|
||||
**Savings: 93% time reduction, 95% cost reduction, 100% more sanity**
|
||||
|
||||
## The Dynamic Growth Pattern
|
||||
|
||||
<div style="text-align: center; padding: 40px; background-color: #1a1a1c; border: 1px dashed #3f3f44; margin: 30px 0;">
|
||||
<div style="font-size: 24px; color: #09b5a5; margin-bottom: 10px;">
|
||||
Knowledge grows like crystals in a supersaturated solution
|
||||
</div>
|
||||
<div style="color: #a3abba;">
|
||||
Add a query (seed), and relevant information crystallizes around it.<br>
|
||||
Change the query, and the knowledge structure adapts.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
This is the beauty of adaptive crawling: your knowledge base becomes a living entity that grows based on actual needs, not hypothetical completeness.
|
||||
|
||||
## Why "Adaptive"?
|
||||
|
||||
I specifically chose "Adaptive" because it captures the essence: the system adapts to what it finds. Dense technical documentation might need 20 pages for confidence. A simple FAQ might need just 5. The crawler doesn't follow a recipe—it reads the room and adjusts.
|
||||
|
||||
This is my term, my concept, and I have extensive plans for its evolution.
|
||||
|
||||
## The Progressive Roadmap
|
||||
|
||||
This is just the beginning. My roadmap for Adaptive Crawling:
|
||||
|
||||
### Phase 1 (Current): Statistical Foundation
|
||||
- Pure information theory approach
|
||||
- No dependencies on expensive models
|
||||
- Proven efficiency gains
|
||||
|
||||
### Phase 2 (Now Available): Embedding Enhancement
|
||||
- Semantic understanding layered onto statistical base
|
||||
- Still efficient, now even smarter
|
||||
- Optional, not required
|
||||
|
||||
### Phase 3 (Future): LLM Integration
|
||||
- LLMs for complex reasoning tasks only
|
||||
- Used surgically, not wastefully
|
||||
- Always with statistical foundation underneath
|
||||
|
||||
## The Efficiency Revolution
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
### 💰 **The Economics of Intelligence**
|
||||
|
||||
For a typical SaaS documentation crawl:
|
||||
|
||||
**Traditional Deep Crawling:**
|
||||
- Pages crawled: 1,000
|
||||
- Useful pages: 80
|
||||
- Time spent: 3 hours
|
||||
- LLM tokens used: 2.5M
|
||||
- Cost: $75
|
||||
- Efficiency: 8%
|
||||
|
||||
**Adaptive Crawling:**
|
||||
- Pages crawled: 95
|
||||
- Useful pages: 88
|
||||
- Time spent: 15 minutes
|
||||
- LLM tokens used: 200K
|
||||
- Cost: $6
|
||||
- Efficiency: 93%
|
||||
|
||||
**That's not optimization. That's transformation.**
|
||||
|
||||
</div>
|
||||
|
||||
## Missing the Forest for the Trees
|
||||
|
||||
The startup world has a dangerous blind spot. We're so enamored with LLMs that we forget: just because you CAN process everything with an LLM doesn't mean you SHOULD.
|
||||
|
||||
Classic NLP and statistical methods can:
|
||||
- Filter irrelevant content before it reaches LLMs
|
||||
- Identify patterns without expensive inference
|
||||
- Make intelligent decisions in microseconds
|
||||
- Scale without breaking the bank
|
||||
|
||||
Adaptive crawling proves this. It uses battle-tested information theory to make smart decisions BEFORE expensive processing.
|
||||
|
||||
## Your Knowledge, On Demand
|
||||
|
||||
```python
|
||||
# Monday: Customer asks about authentication
|
||||
auth_knowledge = await adaptive.digest(
|
||||
"https://docs.api.com",
|
||||
"oauth jwt authentication"
|
||||
)
|
||||
|
||||
# Tuesday: They ask about rate limiting
|
||||
# The crawler adapts, builds on existing knowledge
|
||||
rate_limit_knowledge = await adaptive.digest(
|
||||
"https://docs.api.com",
|
||||
"rate limiting throttling quotas"
|
||||
)
|
||||
|
||||
# Your knowledge base grows intelligently, not indiscriminately
|
||||
```
|
||||
|
||||
## The Competitive Edge
|
||||
|
||||
Companies using adaptive crawling will have:
|
||||
- **90% lower crawling costs**
|
||||
- **Knowledge bases that actually answer questions**
|
||||
- **Update cycles in minutes, not days**
|
||||
- **Happy customers who find answers fast**
|
||||
- **Engineers who sleep at night**
|
||||
|
||||
Those still using brute force? They'll wonder why their infrastructure costs keep rising while their customers keep complaining.
|
||||
|
||||
## The Embedding Evolution (Now Available!)
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
### 🧠 **Semantic Understanding Without the Cost**
|
||||
|
||||
The embedding strategy brings semantic intelligence while maintaining efficiency:
|
||||
|
||||
```python
|
||||
# Statistical strategy - great for exact terms
|
||||
config_statistical = AdaptiveConfig(
|
||||
strategy="statistical" # Default
|
||||
)
|
||||
|
||||
# Embedding strategy - understands concepts
|
||||
config_embedding = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
embedding_model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
n_query_variations=10
|
||||
)
|
||||
```
|
||||
|
||||
**The magic**: It automatically expands your query into semantic variations, maps the coverage space, and identifies gaps to fill intelligently.
|
||||
|
||||
</div>
|
||||
|
||||
### Real-World Comparison
|
||||
|
||||
<div style="display: flex; gap: 20px; margin: 20px 0;">
|
||||
<div style="flex: 1; background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px;">
|
||||
|
||||
**Query**: "authentication oauth"
|
||||
|
||||
**Statistical Strategy**:
|
||||
- Searches for exact terms
|
||||
- 12 pages crawled
|
||||
- 78% confidence
|
||||
- Fast but literal
|
||||
|
||||
</div>
|
||||
<div style="flex: 1; background-color: #1a1a1c; border: 1px solid #09b5a5; padding: 20px;">
|
||||
|
||||
**Embedding Strategy**:
|
||||
- Understands "auth", "login", "SSO"
|
||||
- 8 pages crawled
|
||||
- 92% confidence
|
||||
- Semantic comprehension
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### Detecting Irrelevance
|
||||
|
||||
One killer feature: the embedding strategy knows when to give up:
|
||||
|
||||
```python
|
||||
# Crawling Python docs with a cooking query
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/",
|
||||
query="how to make spaghetti carbonara"
|
||||
)
|
||||
|
||||
# System detects irrelevance and stops
|
||||
# Confidence: 5% (below threshold)
|
||||
# Pages crawled: 2
|
||||
# Stopped reason: "below_minimum_relevance_threshold"
|
||||
```
|
||||
|
||||
No more crawling hundreds of pages hoping to find something that doesn't exist!
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Choose your strategy
|
||||
config = AdaptiveConfig(
|
||||
strategy="embedding", # or "statistical"
|
||||
embedding_min_confidence_threshold=0.1 # Stop if irrelevant
|
||||
)
|
||||
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Watch intelligence at work
|
||||
result = await adaptive.digest(
|
||||
start_url="https://your-docs.com",
|
||||
query="your users' actual questions"
|
||||
)
|
||||
|
||||
# See the efficiency
|
||||
adaptive.print_stats()
|
||||
print(f"Found {adaptive.confidence:.0%} of needed information")
|
||||
print(f"In just {len(result.crawled_urls)} pages")
|
||||
print(f"Saving you {1000 - len(result.crawled_urls)} unnecessary crawls")
|
||||
```
|
||||
|
||||
## A Personal Note
|
||||
|
||||
I created Adaptive Crawling because I was tired of watching smart people make inefficient choices. We have incredibly powerful statistical tools that we've forgotten in our rush toward LLMs. This is my attempt to bring balance back to the Force.
|
||||
|
||||
This is not just a feature. It's a philosophy: **Grow knowledge on demand. Stop when you have enough. Save time, money, and computational resources for what really matters.**
|
||||
|
||||
## The Future is Adaptive
|
||||
|
||||
<div style="text-align: center; padding: 40px; background-color: #1a1a1c; border: 1px dashed #3f3f44; margin: 30px 0;">
|
||||
<div style="font-size: 24px; color: #09b5a5; margin-bottom: 10px;">
|
||||
Traditional Crawling: Drinking from a firehose<br>
|
||||
Adaptive Crawling: Sipping exactly what you need
|
||||
</div>
|
||||
<div style="color: #a3abba;">
|
||||
The future of web crawling isn't about processing more data.<br>
|
||||
It's about processing the <em>right</em> data.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Join me in making web crawling intelligent, efficient, and actually useful. Because in the age of information overload, the winners won't be those who collect the most data—they'll be those who collect the *right* data.
|
||||
|
||||
---
|
||||
|
||||
*Adaptive Crawling is now part of Crawl4AI. [Get started with the documentation](/core/adaptive-crawling/) or [dive into the mathematical framework](https://github.com/unclecode/crawl4ai/blob/main/PROGRESSIVE_CRAWLING.md). For updates on my work in information theory and efficient AI, follow me on [X/Twitter](https://x.com/unclecode).*
|
||||
|
||||
<style>
|
||||
/* Custom styles for this article */
|
||||
.markdown-body pre {
|
||||
background-color: #1e1e1e !important;
|
||||
border: 1px solid #3f3f44;
|
||||
}
|
||||
|
||||
.markdown-body code {
|
||||
background-color: #3f3f44;
|
||||
color: #50ffff;
|
||||
padding: 2px 6px;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
.markdown-body pre code {
|
||||
background-color: transparent;
|
||||
color: #e8e9ed;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.markdown-body blockquote {
|
||||
border-left: 4px solid #09b5a5;
|
||||
background-color: #1a1a1c;
|
||||
padding: 15px 20px;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.markdown-body h2 {
|
||||
color: #50ffff;
|
||||
border-bottom: 1px dashed #3f3f44;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
.markdown-body h3 {
|
||||
color: #09b5a5;
|
||||
}
|
||||
|
||||
.markdown-body strong {
|
||||
color: #50ffff;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,46 @@
|
||||
## Introducing Event Streams and Interactive Hooks in Crawl4AI
|
||||
|
||||
|
||||
|
||||
In the near future, I’m planning to enhance Crawl4AI’s capabilities by introducing an event stream mechanism that will give clients deeper, real-time insights into the crawling process. Today, hooks are a powerful feature at the code level—they let developers define custom logic at key points in the crawl. However, when using Crawl4AI as a service (e.g., through a Dockerized API), there isn’t an easy way to interact with these hooks at runtime.
|
||||
|
||||
**What’s Changing?**
|
||||
|
||||
I’m working on a solution that will allow the crawler to emit a continuous stream of events, updating clients on the current crawling stage, encountered pages, and any decision points. This event stream could be exposed over a standardized protocol like Server-Sent Events (SSE) or WebSockets, enabling clients to “subscribe” and listen as the crawler works.
|
||||
|
||||
**Interactivity Through Process IDs**
|
||||
|
||||
A key part of this new design is the concept of a unique process ID for each crawl session. Imagine you’re listening to an event stream that informs you:
|
||||
- The crawler just hit a certain page
|
||||
- It triggered a hook and is now pausing for instructions
|
||||
|
||||
With the event stream in place, you can send a follow-up request back to the server—referencing the unique process ID—to provide extra data, instructions, or parameters. This might include selecting which links to follow next, adjusting extraction strategies, or providing authentication tokens for a protected API. Once the crawler receives these instructions, it resumes execution with the updated context.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Server
|
||||
participant Crawler
|
||||
|
||||
Client->>Server: Start crawl request
|
||||
Server->>Crawler: Initiate crawl with Process ID
|
||||
Crawler-->>Server: Event: Page hit
|
||||
Server-->>Client: Stream: Page hit event
|
||||
Client->>Server: Instruction for Process ID
|
||||
Server->>Crawler: Update crawl with new instructions
|
||||
Crawler-->>Server: Event: Crawl completed
|
||||
Server-->>Client: Stream: Crawl completed
|
||||
```
|
||||
|
||||
**Benefits for Developers and Users**
|
||||
|
||||
1. **Fine-Grained Control**: Instead of predefining all logic upfront, you can dynamically guide the crawler in response to actual data and conditions encountered mid-crawl.
|
||||
2. **Real-Time Insights**: Monitor progress, errors, or network bottlenecks as they happen, without waiting for the entire crawl to finish.
|
||||
3. **Enhanced Collaboration**: Different team members or automated systems can watch the same crawl events and provide input, making the crawling process more adaptive and intelligent.
|
||||
|
||||
**Next Steps**
|
||||
|
||||
I’m currently exploring the best APIs, technologies, and patterns to make this vision a reality. My goal is to deliver a seamless developer experience—one that integrates with existing Crawl4AI workflows while offering new flexibility and power.
|
||||
|
||||
Stay tuned for more updates as I continue building this feature out. In the meantime, I’d love to hear any feedback or suggestions you might have to help shape this interactive, event-driven future of web crawling with Crawl4AI.
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
# The LLM Context Protocol: Why Your AI Assistant Needs Memory, Reasoning, and Examples
|
||||
|
||||
*Published on January 24, 2025 • 8 min read*
|
||||
|
||||
---
|
||||
|
||||
## The Problem with Teaching Robots to Code
|
||||
|
||||
Picture this: You hand someone a dictionary and ask them to write poetry. They know every word, its spelling, its definition—but do they know how words dance together? How certain combinations evoke emotion while others fall flat? This is exactly what we're doing when we throw API documentation at our AI assistants and expect magic.
|
||||
|
||||
I've spent countless hours watching my AI coding assistant struggle with my own library, Crawl4AI. Despite feeding it comprehensive documentation, it would generate code that was *technically* correct but practically useless. Like a tourist speaking from a phrasebook—grammatically sound, culturally tone-deaf.
|
||||
|
||||
## Enter the Three-Dimensional Context Protocol
|
||||
|
||||
What if, instead of dumping information, we provided *wisdom*? Not just the "what," but the "how" and "why"? This led me to develop what I call the **LLM Context Protocol**—a structured approach that mirrors how humans actually master libraries.
|
||||
|
||||
Think of it as HTTP for AI context. Just as HTTP doesn't dictate your website's content but provides a reliable structure for communication, this protocol doesn't prescribe *how* you write your documentation—it provides a framework for *what* your AI needs to truly understand your code.
|
||||
|
||||
### The Three Pillars of Library Wisdom
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
#### 🧠 **Memory: The Foundation**
|
||||
```markdown
|
||||
# AsyncWebCrawler.arun() - Memory Context
|
||||
|
||||
## Signature
|
||||
async def arun(
|
||||
url: str,
|
||||
config: CrawlerConfig = None,
|
||||
session_id: str = None,
|
||||
**kwargs
|
||||
) -> CrawlResult
|
||||
|
||||
## Parameters
|
||||
- url: Target URL to crawl
|
||||
- config: Optional configuration object
|
||||
- session_id: Optional session identifier for caching
|
||||
...
|
||||
```
|
||||
|
||||
This is your API reference—the facts, the parameters, the return types. It's the easiest part to generate and, ironically, the least useful in isolation. It's like memorizing a dictionary without understanding grammar.
|
||||
|
||||
</div>
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
#### 🎯 **Reasoning: The Soul**
|
||||
```markdown
|
||||
# AsyncWebCrawler Design Philosophy - Reasoning Context
|
||||
|
||||
## Why Async-First Architecture?
|
||||
|
||||
Crawl4AI uses AsyncWebCrawler as its primary interface because modern web
|
||||
scraping demands concurrency. Here's the thinking:
|
||||
|
||||
1. **Network I/O is slow**: Waiting synchronously wastes 90% of execution time
|
||||
2. **Modern sites are complex**: Multiple resources load in parallel
|
||||
3. **Scale matters**: You're rarely crawling just one page
|
||||
|
||||
## When to Use Session Management
|
||||
|
||||
Session management isn't just about performance—it's about appearing human:
|
||||
- Use sessions when crawling multiple pages from the same domain
|
||||
- Reuse browser contexts to maintain cookies and local storage
|
||||
- But don't overdo it: too long sessions look suspicious
|
||||
|
||||
## The Cache Strategy Decision Tree
|
||||
if static_content and infrequent_updates:
|
||||
use_cache_mode('read_write')
|
||||
elif dynamic_content and real_time_needed:
|
||||
use_cache_mode('bypass')
|
||||
else:
|
||||
use_cache_mode('read_only') # Safe default
|
||||
```
|
||||
|
||||
This is where the library creator's philosophy lives. It's not just *what* the library does, but *why* it does it that way. This is the hardest part to write because it requires genuine understanding—and it's a red flag when a library lacks it.
|
||||
|
||||
</div>
|
||||
|
||||
<div style="background-color: #1a1a1c; border: 1px solid #3f3f44; padding: 20px; margin: 20px 0;">
|
||||
|
||||
#### 💻 **Examples: The Practice**
|
||||
```python
|
||||
# Crawling with JavaScript execution
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="css:.lazy-loaded-content"
|
||||
)
|
||||
|
||||
# Extracting structured data with CSS selectors
|
||||
result = await crawler.arun(
|
||||
url="https://shop.example.com",
|
||||
extraction_strategy=CSSExtractionStrategy({
|
||||
"prices": "span.price::text",
|
||||
"titles": "h2.product-title::text"
|
||||
})
|
||||
)
|
||||
|
||||
# Session-based crawling with custom headers
|
||||
async with crawler:
|
||||
result1 = await crawler.arun(url1, session_id="product_scan")
|
||||
result2 = await crawler.arun(url2, session_id="product_scan")
|
||||
```
|
||||
|
||||
Pure code. No fluff. Just patterns in action. Because sometimes, you just need to see how it's done.
|
||||
|
||||
</div>
|
||||
|
||||
## Why This Matters (Especially for Smaller LLMs)
|
||||
|
||||
Here's the thing about AI assistants: the smaller ones can't think their way out of a paper bag. They're like eager interns—full of potential but needing clear guidance. When you rely on a large language model to "figure it out" from raw API docs, you're asking it to reinvent your library's philosophy from scratch. Every. Single. Time.
|
||||
|
||||
By providing structured context across these three dimensions, we're not just documenting—we're teaching. We're transferring not just knowledge, but wisdom.
|
||||
|
||||
## The Cultural DNA of Your Library
|
||||
|
||||
<div style="display: flex; align-items: center; background-color: #3f3f44; padding: 20px; margin: 20px 0; border-left: 4px solid #09b5a5;">
|
||||
<div style="font-size: 48px; margin-right: 20px;">🧬</div>
|
||||
<div>
|
||||
<strong>Your library's reasoning is its cultural DNA.</strong><br>
|
||||
It reflects your taste, your architectural decisions, your opinions about how things should be done. A library without reasoning is like a recipe without techniques—sure, you have the ingredients, but good luck making something edible.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Think about it: When you learn a new library, what are you really after? You want mastery. And mastery comes from understanding:
|
||||
- **Memory** tells you what's possible
|
||||
- **Reasoning** tells you what's sensible
|
||||
- **Examples** show you what's practical
|
||||
|
||||
Together, they create wisdom.
|
||||
|
||||
## Beyond Manual Documentation
|
||||
|
||||
Now, here's where it gets interesting. I didn't hand-craft thousands of lines of structured documentation for Crawl4AI. Who has that kind of time? Instead, I built a tool that:
|
||||
|
||||
1. Analyzes your codebase
|
||||
2. Extracts API signatures and structures (Memory)
|
||||
3. Identifies patterns and architectural decisions (Reasoning)
|
||||
4. Collects real-world usage from tests and examples (Examples)
|
||||
5. Generates structured LLM context files
|
||||
|
||||
The beauty? This tool is becoming part of Crawl4AI itself. Because if we're going to revolutionize how AI understands our code, we might as well automate it.
|
||||
|
||||
## The Protocol, Not the Prescription
|
||||
|
||||
Remember: this is a protocol, not a prescription. Just as HTTP doesn't tell you what website to build, the LLM Context Protocol doesn't dictate your documentation style. It simply says:
|
||||
|
||||
> "If you want an AI to truly understand your library, it needs three things: facts, philosophy, and patterns."
|
||||
|
||||
How you deliver those is up to you. The protocol just ensures nothing important gets lost in translation.
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
Curious about implementing this for your own library? The context generation tool will be open-sourced as part of Crawl4AI. If you're interested in early access or want to discuss the approach, drop me a DM on X [@unclecode](https://twitter.com/unclecode).
|
||||
|
||||
Because let's face it: if we're going to live in a world where AI writes half our code, we might as well teach it properly.
|
||||
|
||||
---
|
||||
|
||||
## A Final Thought
|
||||
|
||||
<div style="text-align: center; padding: 40px; background-color: #1a1a1c; border: 1px dashed #3f3f44; margin: 30px 0;">
|
||||
<div style="font-size: 24px; color: #09b5a5; margin-bottom: 10px;">
|
||||
Memory + Reasoning + Examples = Wisdom
|
||||
</div>
|
||||
<div style="color: #a3abba;">
|
||||
And wisdom, not information, is what makes great developers—human or artificial.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
*Want to see this in action? Check out the [Crawl4AI LLM Context Builder](/core/llmtxt/) and experience the difference structured context makes.*
|
||||
|
||||
<style>
|
||||
/* Custom styles for this article */
|
||||
.markdown-body pre {
|
||||
background-color: #1e1e1e !important;
|
||||
border: 1px solid #3f3f44;
|
||||
}
|
||||
|
||||
.markdown-body code {
|
||||
background-color: #3f3f44;
|
||||
color: #50ffff;
|
||||
padding: 2px 6px;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
.markdown-body pre code {
|
||||
background-color: transparent;
|
||||
color: #e8e9ed;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.markdown-body blockquote {
|
||||
border-left: 4px solid #09b5a5;
|
||||
background-color: #1a1a1c;
|
||||
padding: 15px 20px;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.markdown-body h2 {
|
||||
color: #50ffff;
|
||||
border-bottom: 1px dashed #3f3f44;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
.markdown-body h3 {
|
||||
color: #09b5a5;
|
||||
}
|
||||
|
||||
.markdown-body strong {
|
||||
color: #50ffff;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,355 @@
|
||||
# Solving the Virtual Scroll Puzzle: How Crawl4AI Captures What Others Miss
|
||||
|
||||
*Published on June 29, 2025 • 10 min read*
|
||||
|
||||
*By [unclecode](https://x.com/unclecode) • Follow me on [X/Twitter](https://x.com/unclecode) for more web scraping insights*
|
||||
|
||||
---
|
||||
|
||||
## The Invisible Content Crisis
|
||||
|
||||
You know that feeling when you're scrolling through Twitter, and suddenly realize you can't scroll back to that brilliant tweet from an hour ago? It's not your browser being quirky—it's virtual scrolling at work. And if this frustrates you as a user, imagine being a web scraper trying to capture all those tweets.
|
||||
|
||||
Here's the dirty secret of modern web development: **most of the content you see doesn't actually exist**.
|
||||
|
||||
Let me explain. Open Twitter right now and scroll for a bit. Now inspect the DOM. You'll find maybe 20-30 tweet elements, yet you just scrolled past hundreds. Where did they go? They were never really there—just temporary ghosts passing through a revolving door of DOM elements.
|
||||
|
||||
This is virtual scrolling, and it's everywhere: Twitter, Instagram, LinkedIn, Reddit, data tables, analytics dashboards. It's brilliant for performance but catastrophic for traditional web scraping.
|
||||
|
||||
## The Great DOM Disappearing Act
|
||||
|
||||
Let's visualize what's happening:
|
||||
|
||||
```
|
||||
Traditional Infinite Scroll: Virtual Scroll:
|
||||
┌─────────────┐ ┌─────────────┐
|
||||
│ Item 1 │ │ Item 11 │ ← Items 1-10? Gone.
|
||||
│ Item 2 │ │ Item 12 │ ← Only what's visible
|
||||
│ ... │ │ Item 13 │ exists in the DOM
|
||||
│ Item 10 │ │ Item 14 │
|
||||
│ Item 11 NEW │ │ Item 15 │
|
||||
│ Item 12 NEW │ └─────────────┘
|
||||
└─────────────┘
|
||||
DOM: 12 items & growing DOM: Always ~5 items
|
||||
```
|
||||
|
||||
Traditional scrapers see this and capture... 5 items. Out of thousands. It's like trying to photograph a train by taking a picture of one window.
|
||||
|
||||
## Why Virtual Scroll Broke Everything
|
||||
|
||||
When I first encountered this with Crawl4AI, I thought it was a bug. My scraper would perfectly capture the initial tweets, but scrolling did... nothing. The DOM element count stayed constant. The HTML size barely changed. Yet visually, new content kept appearing.
|
||||
|
||||
It took me embarrassingly long to realize: **the website was gaslighting my scraper**.
|
||||
|
||||
Virtual scroll is deceptively simple:
|
||||
1. Keep only visible items in DOM (usually 10-30 elements)
|
||||
2. As user scrolls down, remove top items, add bottom items
|
||||
3. As user scrolls up, remove bottom items, add top items
|
||||
4. Maintain the illusion of a continuous list
|
||||
|
||||
For users, it's seamless. For scrapers, it's a nightmare. Traditional approaches fail because:
|
||||
- `document.scrollingElement.scrollHeight` lies to you
|
||||
- Waiting for new elements is futile—they replace, not append
|
||||
- Screenshots only capture the current viewport
|
||||
- Even browser automation tools get fooled
|
||||
|
||||
## The Three-State Solution
|
||||
|
||||
After much experimentation (and several cups of coffee), I realized we needed to think differently. Instead of fighting virtual scroll, we needed to understand it. This led to identifying three distinct scrolling behaviors:
|
||||
|
||||
### State 1: No Change (The Stubborn Page)
|
||||
```javascript
|
||||
scroll() → same content → continue trying
|
||||
```
|
||||
The page doesn't react to scrolling. Either we've hit the end, or it's not a scrollable container.
|
||||
|
||||
### State 2: Appending (The Traditional Friend)
|
||||
```javascript
|
||||
scroll() → old content + new content → all good!
|
||||
```
|
||||
Classic infinite scroll. New content appends to existing content. Our traditional tools work fine here.
|
||||
|
||||
### State 3: Replacing (The Trickster)
|
||||
```javascript
|
||||
scroll() → completely different content → capture everything!
|
||||
```
|
||||
Virtual scroll detected! Content is being replaced. This is where our new magic happens.
|
||||
|
||||
## Introducing VirtualScrollConfig
|
||||
|
||||
Here's how Crawl4AI solves this puzzle:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, VirtualScrollConfig, CrawlerRunConfig
|
||||
|
||||
# Configure virtual scroll handling
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="#timeline", # What to scroll
|
||||
scroll_count=30, # How many times
|
||||
scroll_by="container_height", # How much each time
|
||||
wait_after_scroll=0.5 # Pause for content to load
|
||||
)
|
||||
|
||||
# Use it in your crawl
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://twitter.com/search?q=AI",
|
||||
config=config
|
||||
)
|
||||
# result.html now contains ALL tweets, not just visible ones!
|
||||
```
|
||||
|
||||
But here's where it gets clever...
|
||||
|
||||
## The Magic Behind the Scenes
|
||||
|
||||
When Crawl4AI encounters a virtual scroll container, it:
|
||||
|
||||
1. **Takes a snapshot** of the initial HTML
|
||||
2. **Scrolls** by the configured amount
|
||||
3. **Waits** for the DOM to update
|
||||
4. **Compares** the new HTML with the previous
|
||||
5. **Detects** which of our three states we're in
|
||||
6. **For State 3** (virtual scroll), stores the HTML chunk
|
||||
7. **Repeats** until done
|
||||
8. **Merges** all chunks intelligently
|
||||
|
||||
The merging is crucial. We can't just concatenate HTML—we'd get duplicates. Instead, we:
|
||||
- Parse each chunk into elements
|
||||
- Create fingerprints using normalized text
|
||||
- Keep only unique elements
|
||||
- Maintain the original order
|
||||
- Return clean, complete HTML
|
||||
|
||||
## Real-World Example: Capturing Twitter Threads
|
||||
|
||||
Let's see this in action with a real Twitter thread:
|
||||
|
||||
```python
|
||||
async def capture_twitter_thread():
|
||||
# Configure for Twitter's specific behavior
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="[data-testid='primaryColumn']",
|
||||
scroll_count=50, # Enough for long threads
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=1.0 # Twitter needs time to load
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config,
|
||||
# Also extract structured data
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o-mini",
|
||||
schema={
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"author": {"type": "string"},
|
||||
"content": {"type": "string"},
|
||||
"timestamp": {"type": "string"},
|
||||
"replies": {"type": "integer"},
|
||||
"retweets": {"type": "integer"},
|
||||
"likes": {"type": "integer"}
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://twitter.com/elonmusk/status/...",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Parse the extracted tweets
|
||||
import json
|
||||
tweets = json.loads(result.extracted_content)
|
||||
|
||||
print(f"Captured {len(tweets)} tweets from the thread")
|
||||
for tweet in tweets[:5]:
|
||||
print(f"@{tweet['author']}: {tweet['content'][:100]}...")
|
||||
```
|
||||
|
||||
## Performance Insights
|
||||
|
||||
During testing, we achieved remarkable results:
|
||||
|
||||
| Site | Without Virtual Scroll | With Virtual Scroll | Improvement |
|
||||
|------|------------------------|---------------------|-------------|
|
||||
| Twitter Timeline | 10 tweets | 490 tweets | **49x** |
|
||||
| Instagram Grid | 12 posts | 999 posts | **83x** |
|
||||
| LinkedIn Feed | 5 posts | 200 posts | **40x** |
|
||||
| Reddit Comments | 25 comments | 500 comments | **20x** |
|
||||
|
||||
The best part? It's automatic. If the page doesn't use virtual scroll, Crawl4AI handles it normally. No configuration changes needed.
|
||||
|
||||
## When to Use Virtual Scroll
|
||||
|
||||
Use `VirtualScrollConfig` when:
|
||||
- ✅ Scrolling seems to "eat" previous content
|
||||
- ✅ DOM element count stays suspiciously constant
|
||||
- ✅ You're scraping Twitter, Instagram, LinkedIn, Reddit
|
||||
- ✅ Working with modern data tables or dashboards
|
||||
- ✅ Traditional scrolling captures only a fraction of content
|
||||
|
||||
Don't use it when:
|
||||
- ❌ Content accumulates normally (use `scan_full_page` instead)
|
||||
- ❌ Page has no scrollable containers
|
||||
- ❌ You only need the initially visible content
|
||||
- ❌ Working with static or traditionally paginated sites
|
||||
|
||||
## Advanced Techniques
|
||||
|
||||
### Handling Mixed Content
|
||||
|
||||
Some sites mix approaches—featured content stays while regular content virtualizes:
|
||||
|
||||
```python
|
||||
# News site with pinned articles + virtual scroll feed
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector=".main-feed", # Only the feed scrolls virtually
|
||||
scroll_count=30,
|
||||
scroll_by="container_height"
|
||||
)
|
||||
|
||||
# Featured articles remain throughout the crawl
|
||||
# Regular articles are captured via virtual scroll
|
||||
```
|
||||
|
||||
### Optimizing Performance
|
||||
|
||||
```python
|
||||
# Fast scrolling for simple content
|
||||
fast_config = VirtualScrollConfig(
|
||||
container_selector="#feed",
|
||||
scroll_count=100,
|
||||
scroll_by=500, # Fixed pixels for speed
|
||||
wait_after_scroll=0.1 # Minimal wait
|
||||
)
|
||||
|
||||
# Careful scrolling for complex content
|
||||
careful_config = VirtualScrollConfig(
|
||||
container_selector=".timeline",
|
||||
scroll_count=50,
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=1.5 # More time for lazy loading
|
||||
)
|
||||
```
|
||||
|
||||
### Debugging Virtual Scroll
|
||||
|
||||
Want to see it in action? Set `headless=False`:
|
||||
|
||||
```python
|
||||
browser_config = BrowserConfig(headless=False)
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Watch the magic happen!
|
||||
result = await crawler.arun(url="...", config=config)
|
||||
```
|
||||
|
||||
## The Technical Deep Dive
|
||||
|
||||
For the curious, here's how our deduplication works:
|
||||
|
||||
```javascript
|
||||
// Simplified version of our deduplication logic
|
||||
function createFingerprint(element) {
|
||||
const text = element.innerText
|
||||
.toLowerCase()
|
||||
.replace(/[\s\W]/g, ''); // Remove spaces and symbols
|
||||
return text;
|
||||
}
|
||||
|
||||
function mergeChunks(chunks) {
|
||||
const seen = new Set();
|
||||
const unique = [];
|
||||
|
||||
for (const chunk of chunks) {
|
||||
const elements = parseHTML(chunk);
|
||||
for (const element of elements) {
|
||||
const fingerprint = createFingerprint(element);
|
||||
if (!seen.has(fingerprint)) {
|
||||
seen.add(fingerprint);
|
||||
unique.push(element);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return unique;
|
||||
}
|
||||
```
|
||||
|
||||
Simple, but effective. We normalize text to catch duplicates even with slight HTML differences.
|
||||
|
||||
## What This Means for Web Scraping
|
||||
|
||||
Virtual scroll support in Crawl4AI represents a paradigm shift. We're no longer limited to what's immediately visible or what traditional scrolling reveals. We can now capture the full content of virtually any modern website.
|
||||
|
||||
This opens new possibilities:
|
||||
- **Complete social media analysis**: Every tweet, every comment, every reaction
|
||||
- **Comprehensive data extraction**: Full tables, complete lists, entire feeds
|
||||
- **Historical research**: Capture entire timelines, not just recent posts
|
||||
- **Competitive intelligence**: See everything your competitors are showing their users
|
||||
|
||||
## Try It Yourself
|
||||
|
||||
Ready to capture what others miss? Here's a complete example to get you started:
|
||||
|
||||
```python
|
||||
# Save this as virtual_scroll_demo.py
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig
|
||||
|
||||
async def main():
|
||||
# Configure virtual scroll
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="#main-content", # Adjust for your target
|
||||
scroll_count=20,
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=0.5
|
||||
)
|
||||
|
||||
# Set up the crawler
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config,
|
||||
verbose=True # See what's happening
|
||||
)
|
||||
|
||||
# Crawl and capture everything
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com/feed", # Your target URL
|
||||
config=config
|
||||
)
|
||||
|
||||
print(f"Captured {len(result.html)} characters of content")
|
||||
print(f"Found {result.html.count('article')} articles") # Adjust selector
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Conclusion: The Future is Already Here
|
||||
|
||||
Virtual scrolling was supposed to be the end of comprehensive web scraping. Instead, it became the catalyst for smarter, more sophisticated tools. With Crawl4AI's virtual scroll support, we're not just keeping up with modern web development—we're staying ahead of it.
|
||||
|
||||
The web is evolving, becoming more dynamic, more efficient, and yes, more challenging to scrape. But with the right tools and understanding, every challenge becomes an opportunity.
|
||||
|
||||
Welcome to the future of web scraping. Welcome to a world where virtual scroll is no longer a barrier, but just another feature we handle seamlessly.
|
||||
|
||||
---
|
||||
|
||||
## Learn More
|
||||
|
||||
- 📖 [Virtual Scroll Documentation](https://docs.crawl4ai.com/advanced/virtual-scroll) - Complete API reference and configuration options
|
||||
- 💻 [Interactive Examples](https://docs.crawl4ai.com/examples/virtual_scroll_example.py) - Try it yourself with our test server
|
||||
- 🚀 [Get Started with Crawl4AI](https://docs.crawl4ai.com/core/quickstart) - Full installation and setup guide
|
||||
- 🤝 [Join our Community](https://github.com/unclecode/crawl4ai) - Share your experiences and get help
|
||||
|
||||
*Have you encountered virtual scroll challenges? How did you solve them? Share your story in our [GitHub discussions](https://github.com/unclecode/crawl4ai/discussions)!*
|
||||
Reference in New Issue
Block a user