chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,243 @@
|
||||
# Crawl4AI v0.8.0 Release Notes
|
||||
|
||||
**Release Date**: January 2026
|
||||
**Previous Version**: v0.7.6
|
||||
**Status**: Release Candidate
|
||||
|
||||
---
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Critical Security Fixes** for Docker API deployment
|
||||
- **11 New Features** including crash recovery, prefetch mode, and proxy improvements
|
||||
- **Breaking Changes** - see migration guide below
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### 1. Docker API: Hooks Disabled by Default
|
||||
|
||||
**What changed**: Hooks are now disabled by default on the Docker API.
|
||||
|
||||
**Why**: Security fix for Remote Code Execution (RCE) vulnerability.
|
||||
|
||||
**Who is affected**: Users of the Docker API who use the `hooks` parameter in `/crawl` requests.
|
||||
|
||||
**Migration**:
|
||||
```bash
|
||||
# To re-enable hooks (only if you trust all API users):
|
||||
export CRAWL4AI_HOOKS_ENABLED=true
|
||||
```
|
||||
|
||||
### 2. Docker API: file:// URLs Blocked
|
||||
|
||||
**What changed**: The endpoints `/execute_js`, `/screenshot`, `/pdf`, and `/html` now reject `file://` URLs.
|
||||
|
||||
**Why**: Security fix for Local File Inclusion (LFI) vulnerability.
|
||||
|
||||
**Who is affected**: Users who were reading local files via the Docker API.
|
||||
|
||||
**Migration**: Use the Python library directly for local file processing:
|
||||
```python
|
||||
# Instead of API call with file:// URL, use library:
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="file:///path/to/file.html")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: Remote Code Execution via Hooks (CVE Pending)
|
||||
|
||||
**Severity**: CRITICAL (CVSS 10.0)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /crawl` with malicious `hooks` parameter
|
||||
|
||||
**Details**: The `__import__` builtin was available in hook code, allowing attackers to import `os`, `subprocess`, etc. and execute arbitrary commands.
|
||||
|
||||
**Fix**:
|
||||
1. Removed `__import__` from allowed builtins
|
||||
2. Hooks disabled by default (`CRAWL4AI_HOOKS_ENABLED=false`)
|
||||
|
||||
### High: Local File Inclusion via file:// URLs (CVE Pending)
|
||||
|
||||
**Severity**: HIGH (CVSS 8.6)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /execute_js` (and other endpoints) with `file:///etc/passwd`
|
||||
|
||||
**Details**: API endpoints accepted `file://` URLs, allowing attackers to read arbitrary files from the server.
|
||||
|
||||
**Fix**: URL scheme validation now only allows `http://`, `https://`, and `raw:` URLs.
|
||||
|
||||
### Credits
|
||||
|
||||
Discovered by **Neo by ProjectDiscovery** ([projectdiscovery.io](https://projectdiscovery.io)) - December 2025
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. init_scripts Support for BrowserConfig
|
||||
|
||||
Pre-page-load JavaScript injection for stealth evasions.
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
init_scripts=[
|
||||
"Object.defineProperty(navigator, 'webdriver', {get: () => false})"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### 2. CDP Connection Improvements
|
||||
|
||||
- WebSocket URL support (`ws://`, `wss://`)
|
||||
- Proper cleanup with `cdp_cleanup_on_close=True`
|
||||
- Browser reuse across multiple connections
|
||||
|
||||
### 3. Crash Recovery for Deep Crawl Strategies
|
||||
|
||||
All deep crawl strategies (BFS, DFS, Best-First) now support crash recovery:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
|
||||
|
||||
strategy = BFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
resume_state=saved_state, # Resume from checkpoint
|
||||
on_state_change=save_callback # Persist state in real-time
|
||||
)
|
||||
```
|
||||
|
||||
### 4. PDF and MHTML for raw:/file:// URLs
|
||||
|
||||
Generate PDFs and MHTML from cached HTML content.
|
||||
|
||||
### 5. Screenshots for raw:/file:// URLs
|
||||
|
||||
Render cached HTML and capture screenshots.
|
||||
|
||||
### 6. base_url Parameter for CrawlerRunConfig
|
||||
|
||||
Proper URL resolution for raw: HTML processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(base_url='https://example.com')
|
||||
result = await crawler.arun(url='raw:{html}', config=config)
|
||||
```
|
||||
|
||||
### 7. Prefetch Mode for Two-Phase Deep Crawling
|
||||
|
||||
Fast link extraction without full page processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(prefetch=True)
|
||||
```
|
||||
|
||||
### 8. Proxy Rotation and Configuration
|
||||
|
||||
Enhanced proxy rotation with sticky sessions support.
|
||||
|
||||
### 9. Proxy Support for HTTP Strategy
|
||||
|
||||
Non-browser crawler now supports proxies.
|
||||
|
||||
### 10. Browser Pipeline for raw:/file:// URLs
|
||||
|
||||
New `process_in_browser` parameter for browser operations on local content:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
process_in_browser=True, # Force browser processing
|
||||
screenshot=True
|
||||
)
|
||||
result = await crawler.arun(url='raw:<html>...</html>', config=config)
|
||||
```
|
||||
|
||||
### 11. Smart TTL Cache for Sitemap URL Seeder
|
||||
|
||||
Intelligent cache invalidation for sitemaps:
|
||||
|
||||
```python
|
||||
config = SeedingConfig(
|
||||
cache_ttl_hours=24,
|
||||
validate_sitemap_lastmod=True
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### raw: URL Parsing Truncates at # Character
|
||||
|
||||
**Problem**: CSS color codes like `#eee` were being truncated.
|
||||
|
||||
**Before**: `raw:body{background:#eee}` → `body{background:`
|
||||
**After**: `raw:body{background:#eee}` → `body{background:#eee}`
|
||||
|
||||
### Caching System Improvements
|
||||
|
||||
Various fixes to cache validation and persistence.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- Multi-sample schema generation documentation
|
||||
- URL seeder smart TTL cache parameters
|
||||
- Security documentation (SECURITY.md)
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Guide
|
||||
|
||||
### From v0.7.x to v0.8.0
|
||||
|
||||
1. **Update the package**:
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
2. **Docker API users**:
|
||||
- Hooks are now disabled by default
|
||||
- If you need hooks: `export CRAWL4AI_HOOKS_ENABLED=true`
|
||||
- `file://` URLs no longer work on API (use library directly)
|
||||
|
||||
3. **Review security settings**:
|
||||
```yaml
|
||||
# config.yml - recommended for production
|
||||
security:
|
||||
enabled: true
|
||||
jwt_enabled: true
|
||||
```
|
||||
|
||||
4. **Test your integration** before deploying to production
|
||||
|
||||
### Breaking Change Checklist
|
||||
|
||||
- [ ] Check if you use `hooks` parameter in API calls
|
||||
- [ ] Check if you use `file://` URLs via the API
|
||||
- [ ] Update environment variables if needed
|
||||
- [ ] Review security configuration
|
||||
|
||||
---
|
||||
|
||||
## Full Changelog
|
||||
|
||||
See [CHANGELOG.md](../CHANGELOG.md) for complete version history.
|
||||
|
||||
---
|
||||
|
||||
## Contributors
|
||||
|
||||
Thanks to all contributors who made this release possible.
|
||||
|
||||
Special thanks to **Neo by ProjectDiscovery** for responsible security disclosure.
|
||||
|
||||
---
|
||||
|
||||
*For questions or issues, please open a [GitHub Issue](https://github.com/unclecode/crawl4ai/issues).*
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,131 @@
|
||||
# Crawl4AI Prospect‑Wizard – step‑by‑step guide
|
||||
|
||||
[](https://colab.research.google.com/drive/10nRCwmfxPjVrRUHyJsYlX7BH5bvPoGpx?usp=sharing)
|
||||
|
||||
A three‑stage demo that goes from **LinkedIn scraping** ➜ **LLM reasoning** ➜ **graph visualisation**.
|
||||
|
||||
**Try it in Google Colab!** Click the badge above to run this demo in a cloud environment with zero setup required.
|
||||
|
||||
```
|
||||
prospect‑wizard/
|
||||
├─ c4ai_discover.py # Stage 1 – scrape companies + people
|
||||
├─ c4ai_insights.py # Stage 2 – embeddings, org‑charts, scores
|
||||
├─ graph_view_template.html # Stage 3 – graph viewer (static HTML)
|
||||
└─ data/ # output lands here (*.jsonl / *.json)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1 Install & boot a LinkedIn profile (one‑time)
|
||||
|
||||
### 1.1 Install dependencies
|
||||
```bash
|
||||
pip install crawl4ai litellm sentence-transformers pandas rich
|
||||
```
|
||||
|
||||
### 1.2 Create / warm a LinkedIn browser profile
|
||||
```bash
|
||||
crwl profiles
|
||||
```
|
||||
1. The interactive shell shows **New profile** – hit **enter**.
|
||||
2. Choose a name, e.g. `profile_linkedin_uc`.
|
||||
3. A Chromium window opens – log in to LinkedIn, solve whatever CAPTCHA, then close.
|
||||
|
||||
> Remember the **profile name**. All future runs take `--profile-name <your_name>`.
|
||||
|
||||
---
|
||||
|
||||
## 2 Discovery – scrape companies & people
|
||||
|
||||
```bash
|
||||
python c4ai_discover.py full \
|
||||
--query "health insurance management" \
|
||||
--geo 102713980 \ # Malaysia geoUrn
|
||||
--title-filters "" \ # or "Product,Engineering"
|
||||
--max-companies 10 \ # default set small for workshops
|
||||
--max-people 20 \ # \^ same
|
||||
--profile-name profile_linkedin_uc \
|
||||
--outdir ./data \
|
||||
--concurrency 2 \
|
||||
--log-level debug
|
||||
```
|
||||
**Outputs** in `./data/`:
|
||||
* `companies.jsonl` – one JSON per company
|
||||
* `people.jsonl` – one JSON per employee
|
||||
|
||||
🛠️ **Dry‑run:** `C4AI_DEMO_DEBUG=1 python c4ai_discover.py full --query coffee` uses bundled HTML snippets, no network.
|
||||
|
||||
### Handy geoUrn cheatsheet
|
||||
| Location | geoUrn |
|
||||
|----------|--------|
|
||||
| Singapore | **103644278** |
|
||||
| Malaysia | **102713980** |
|
||||
| United States | **103644922** |
|
||||
| United Kingdom | **102221843** |
|
||||
| Australia | **101452733** |
|
||||
_See more: <https://www.linkedin.com/search/results/companies/?geoUrn=XXX> – the number after `geoUrn=` is what you need._
|
||||
|
||||
---
|
||||
|
||||
## 3 Insights – embeddings, org‑charts, decision makers
|
||||
|
||||
```bash
|
||||
python c4ai_insights.py \
|
||||
--in ./data \
|
||||
--out ./data \
|
||||
--embed-model all-MiniLM-L6-v2 \
|
||||
--llm-provider gemini/gemini-2.0-flash \
|
||||
--llm-api-key "" \
|
||||
--top-k 10 \
|
||||
--max-llm-tokens 8024 \
|
||||
--llm-temperature 1.0 \
|
||||
--workers 4
|
||||
```
|
||||
Emits next to the Stage‑1 files:
|
||||
* `company_graph.json` – inter‑company similarity graph
|
||||
* `org_chart_<handle>.json` – one per company
|
||||
* `decision_makers.csv` – hand‑picked ‘who to pitch’ list
|
||||
|
||||
Flags reference (straight from `build_arg_parser()`):
|
||||
| Flag | Default | Purpose |
|
||||
|------|---------|---------|
|
||||
| `--in` | `.` | Stage‑1 output dir |
|
||||
| `--out` | `.` | Destination dir |
|
||||
| `--embed_model` | `all-MiniLM-L6-v2` | Sentence‑Transformer model |
|
||||
| `--top_k` | `10` | Neighbours per company in graph |
|
||||
| `--openai_model` | `gpt-4.1` | LLM for scoring decision makers |
|
||||
| `--max_llm_tokens` | `8024` | Token budget per LLM call |
|
||||
| `--llm_temperature` | `1.0` | Creativity knob |
|
||||
| `--stub` | off | Skip OpenAI and fabricate tiny charts |
|
||||
| `--workers` | `4` | Parallel LLM workers |
|
||||
|
||||
---
|
||||
|
||||
## 4 Visualise – interactive graph
|
||||
|
||||
After Stage 2 completes, simply open the HTML viewer from the project root:
|
||||
```bash
|
||||
open graph_view_template.html # or Live Server / Python -http
|
||||
```
|
||||
The page fetches `data/company_graph.json` and the `org_chart_*.json` files automatically; keep the `data/` folder beside the HTML file.
|
||||
|
||||
* Left pane → list of companies (clans).
|
||||
* Click a node to load its org‑chart on the right.
|
||||
* Chat drawer lets you ask follow‑up questions; context is pulled from `people.jsonl`.
|
||||
|
||||
---
|
||||
|
||||
## 5 Common snags
|
||||
|
||||
| Symptom | Fix |
|
||||
|---------|-----|
|
||||
| Infinite CAPTCHA | Use a residential proxy: `--proxy http://user:pass@ip:port` |
|
||||
| 429 Too Many Requests | Lower `--concurrency`, rotate profile, add delay |
|
||||
| Blank graph | Check JSON paths, clear `localStorage` in browser |
|
||||
|
||||
---
|
||||
|
||||
### TL;DR
|
||||
`crwl profiles` → `c4ai_discover.py` → `c4ai_insights.py` → open `graph_view_template.html`.
|
||||
Live long and `import crawl4ai`.
|
||||
|
||||
@@ -0,0 +1,446 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
c4ai-discover — Stage‑1 Discovery CLI
|
||||
|
||||
Scrapes LinkedIn company search + their people pages and dumps two newline‑delimited
|
||||
JSON files: companies.jsonl and people.jsonl.
|
||||
|
||||
Key design rules
|
||||
----------------
|
||||
* No BeautifulSoup — Crawl4AI only for network + HTML fetch.
|
||||
* JsonCssExtractionStrategy for structured scraping; schema auto‑generated once
|
||||
from sample HTML provided by user and then cached under ./schemas/.
|
||||
* Defaults are embedded so the file runs inside VS Code debugger without CLI args.
|
||||
* If executed as a console script (argv > 1), CLI flags win.
|
||||
* Lightweight deps: argparse + Crawl4AI stack.
|
||||
|
||||
Author: Tom @ Kidocode 2025‑04‑26
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import warnings, re
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message=r"The pseudo class ':contains' is deprecated, ':-soup-contains' should be used.*",
|
||||
category=FutureWarning,
|
||||
module=r"soupsieve"
|
||||
)
|
||||
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Imports
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
import argparse
|
||||
import random
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
# 3rd-party rich for pretty logging
|
||||
from rich.console import Console
|
||||
from rich.logging import RichHandler
|
||||
|
||||
from datetime import datetime, UTC
|
||||
from textwrap import dedent
|
||||
from types import SimpleNamespace
|
||||
from typing import Dict, List, Optional
|
||||
from urllib.parse import quote
|
||||
from pathlib import Path
|
||||
from glob import glob
|
||||
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
BrowserConfig,
|
||||
CacheMode,
|
||||
CrawlerRunConfig,
|
||||
JsonCssExtractionStrategy,
|
||||
BrowserProfiler,
|
||||
LLMConfig,
|
||||
)
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Constants / paths
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
BASE_DIR = pathlib.Path(__file__).resolve().parent
|
||||
SCHEMA_DIR = BASE_DIR / "schemas"
|
||||
SCHEMA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
COMPANY_SCHEMA_PATH = SCHEMA_DIR / "company_card.json"
|
||||
PEOPLE_SCHEMA_PATH = SCHEMA_DIR / "people_card.json"
|
||||
|
||||
# ---------- deterministic target JSON examples ----------
|
||||
_COMPANY_SCHEMA_EXAMPLE = {
|
||||
"handle": "/company/posify/",
|
||||
"profile_image": "https://media.licdn.com/dms/image/v2/.../logo.jpg",
|
||||
"name": "Management Research Services, Inc. (MRS, Inc)",
|
||||
"descriptor": "Insurance • Milwaukee, Wisconsin",
|
||||
"about": "Insurance • Milwaukee, Wisconsin",
|
||||
"followers": 1000
|
||||
}
|
||||
|
||||
_PEOPLE_SCHEMA_EXAMPLE = {
|
||||
"profile_url": "https://www.linkedin.com/in/lily-ng/",
|
||||
"name": "Lily Ng",
|
||||
"headline": "VP Product @ Posify",
|
||||
"followers": 890,
|
||||
"connection_degree": "2nd",
|
||||
"avatar_url": "https://media.licdn.com/dms/image/v2/.../lily.jpg"
|
||||
}
|
||||
|
||||
# Provided sample HTML snippets (trimmed) — used exactly once to cold‑generate schema.
|
||||
_SAMPLE_COMPANY_HTML = (Path(__file__).resolve().parent / "snippets/company.html").read_text()
|
||||
_SAMPLE_PEOPLE_HTML = (Path(__file__).resolve().parent / "snippets/people.html").read_text()
|
||||
|
||||
# --------- tighter schema prompts ----------
|
||||
_COMPANY_SCHEMA_QUERY = dedent(
|
||||
"""
|
||||
Using the supplied <li> company-card HTML, build a JsonCssExtractionStrategy schema that,
|
||||
for every card, outputs *exactly* the keys shown in the example JSON below.
|
||||
JSON spec:
|
||||
• handle – href of the outermost <a> that wraps the logo/title, e.g. "/company/posify/"
|
||||
• profile_image – absolute URL of the <img> inside that link
|
||||
• name – text of the <a> inside the <span class*='t-16'>
|
||||
• descriptor – text line with industry • location
|
||||
• about – text of the <div class*='t-normal'> below the name (industry + geo)
|
||||
• followers – integer parsed from the <div> containing 'followers'
|
||||
|
||||
IMPORTANT: Do not use the base64 kind of classes to target element. It's not reliable.
|
||||
The main div parent contains these li element is "div.search-results-container" you can use this.
|
||||
The <ul> parent has "role" equal to "list". Using these two should be enough to target the <li> elements.
|
||||
|
||||
IMPORTANT: Remember there might be multiple <a> tags that start with https://www.linkedin.com/company/[NAME],
|
||||
so in case you refer to them for different fields, make sure to be more specific. One has the image, and one
|
||||
has the person's name.
|
||||
|
||||
IMPORTANT: Be very smart in selecting the correct and unique way to address the element. You should ensure
|
||||
your selector points to a single element and is unique to the place that contains the information.
|
||||
"""
|
||||
)
|
||||
|
||||
_PEOPLE_SCHEMA_QUERY = dedent(
|
||||
"""
|
||||
Using the supplied <li> people-card HTML, build a JsonCssExtractionStrategy schema that
|
||||
outputs exactly the keys in the example JSON below.
|
||||
Fields:
|
||||
• profile_url – href of the outermost profile link
|
||||
• name – text inside artdeco-entity-lockup__title
|
||||
• headline – inner text of artdeco-entity-lockup__subtitle
|
||||
• followers – integer parsed from the span inside lt-line-clamp--multi-line
|
||||
• connection_degree – '1st', '2nd', etc. from artdeco-entity-lockup__badge
|
||||
• avatar_url – src of the <img> within artdeco-entity-lockup__image
|
||||
|
||||
IMPORTANT: Do not use the base64 kind of classes to target element. It's not reliable.
|
||||
The main div parent contains these li element is a "div" has these classes "artdeco-card org-people-profile-card__card-spacing org-people__card-margin-bottom".
|
||||
"""
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Utility helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _load_or_build_schema(
|
||||
path: pathlib.Path,
|
||||
sample_html: str,
|
||||
query: str,
|
||||
example_json: Dict,
|
||||
force = False
|
||||
) -> Dict:
|
||||
"""Load schema from path, else call generate_schema once and persist."""
|
||||
if path.exists() and not force:
|
||||
return json.loads(path.read_text())
|
||||
|
||||
logging.info("[SCHEMA] Generating schema %s", path.name)
|
||||
schema = JsonCssExtractionStrategy.generate_schema(
|
||||
html=sample_html,
|
||||
llm_config=LLMConfig(
|
||||
provider=os.getenv("C4AI_SCHEMA_PROVIDER", "openai/gpt-4o"),
|
||||
api_token=os.getenv("OPENAI_API_KEY", "env:OPENAI_API_KEY"),
|
||||
),
|
||||
query=query,
|
||||
target_json_example=json.dumps(example_json, indent=2),
|
||||
)
|
||||
path.write_text(json.dumps(schema, indent=2))
|
||||
return schema
|
||||
|
||||
|
||||
def _openai_friendly_number(text: str) -> Optional[int]:
|
||||
"""Extract first int from text like '1K followers' (returns 1000)."""
|
||||
import re
|
||||
|
||||
m = re.search(r"(\d[\d,]*)", text.replace(",", ""))
|
||||
if not m:
|
||||
return None
|
||||
val = int(m.group(1))
|
||||
if "k" in text.lower():
|
||||
val *= 1000
|
||||
if "m" in text.lower():
|
||||
val *= 1_000_000
|
||||
return val
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core async workers
|
||||
# ---------------------------------------------------------------------------
|
||||
async def crawl_company_search(crawler: AsyncWebCrawler, url: str, schema: Dict, limit: int) -> List[Dict]:
|
||||
"""Paginate 10-item company search pages until `limit` reached."""
|
||||
extraction = JsonCssExtractionStrategy(schema)
|
||||
cfg = CrawlerRunConfig(
|
||||
extraction_strategy=extraction,
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
wait_for = ".search-marvel-srp",
|
||||
session_id="company_search",
|
||||
delay_before_return_html=1,
|
||||
magic = True,
|
||||
verbose= False,
|
||||
)
|
||||
companies, page = [], 1
|
||||
while len(companies) < max(limit, 10):
|
||||
paged_url = f"{url}&page={page}"
|
||||
res = await crawler.arun(paged_url, config=cfg)
|
||||
batch = json.loads(res[0].extracted_content)
|
||||
if not batch:
|
||||
break
|
||||
for item in batch:
|
||||
name = item.get("name", "").strip()
|
||||
handle = item.get("handle", "").strip()
|
||||
if not handle or not name:
|
||||
continue
|
||||
descriptor = item.get("descriptor")
|
||||
about = item.get("about")
|
||||
followers = _openai_friendly_number(str(item.get("followers", "")))
|
||||
companies.append(
|
||||
{
|
||||
"handle": handle,
|
||||
"name": name,
|
||||
"descriptor": descriptor,
|
||||
"about": about,
|
||||
"followers": followers,
|
||||
"people_url": f"{handle}people/",
|
||||
"captured_at": datetime.now(UTC).isoformat(timespec="seconds") + "Z",
|
||||
}
|
||||
)
|
||||
page += 1
|
||||
logging.info(
|
||||
f"[dim]Page {page}[/] — running total: {len(companies)}/{limit} companies"
|
||||
)
|
||||
|
||||
return companies[:max(limit, 10)]
|
||||
|
||||
|
||||
async def crawl_people_page(
|
||||
crawler: AsyncWebCrawler,
|
||||
people_url: str,
|
||||
schema: Dict,
|
||||
limit: int,
|
||||
title_kw: str,
|
||||
) -> List[Dict]:
|
||||
people_u = f"{people_url}?keywords={quote(title_kw)}"
|
||||
extraction = JsonCssExtractionStrategy(schema)
|
||||
cfg = CrawlerRunConfig(
|
||||
extraction_strategy=extraction,
|
||||
# scan_full_page=True,
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
magic=True,
|
||||
wait_for=".org-people-profile-card__card-spacing",
|
||||
wait_for_images=5000,
|
||||
delay_before_return_html=1,
|
||||
session_id="people_search",
|
||||
)
|
||||
res = await crawler.arun(people_u, config=cfg)
|
||||
if not res[0].success:
|
||||
return []
|
||||
raw = json.loads(res[0].extracted_content)
|
||||
people = []
|
||||
for p in raw[:limit]:
|
||||
followers = _openai_friendly_number(str(p.get("followers", "")))
|
||||
people.append(
|
||||
{
|
||||
"profile_url": p.get("profile_url"),
|
||||
"name": p.get("name"),
|
||||
"headline": p.get("headline"),
|
||||
"followers": followers,
|
||||
"connection_degree": p.get("connection_degree"),
|
||||
"avatar_url": p.get("avatar_url"),
|
||||
}
|
||||
)
|
||||
return people
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI + main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_arg_parser() -> argparse.ArgumentParser:
|
||||
ap = argparse.ArgumentParser("c4ai-discover — Crawl4AI LinkedIn discovery")
|
||||
sub = ap.add_subparsers(dest="cmd", required=False, help="run scope")
|
||||
|
||||
def add_flags(parser: argparse.ArgumentParser):
|
||||
parser.add_argument("--query", required=False, help="query keyword(s)")
|
||||
parser.add_argument("--geo", required=False, type=int, help="LinkedIn geoUrn")
|
||||
parser.add_argument("--title-filters", default="Product,Engineering", help="comma list of job keywords")
|
||||
parser.add_argument("--max-companies", type=int, default=1000)
|
||||
parser.add_argument("--max-people", type=int, default=500)
|
||||
parser.add_argument("--profile-name", default=str(pathlib.Path.home() / ".crawl4ai/profiles/profile_linkedin_uc"))
|
||||
parser.add_argument("--outdir", default="./output")
|
||||
parser.add_argument("--concurrency", type=int, default=4)
|
||||
parser.add_argument("--log-level", default="info", choices=["debug", "info", "warn", "error"])
|
||||
|
||||
add_flags(sub.add_parser("full"))
|
||||
add_flags(sub.add_parser("companies"))
|
||||
add_flags(sub.add_parser("people"))
|
||||
|
||||
# global flags
|
||||
ap.add_argument(
|
||||
"--debug",
|
||||
action="store_true",
|
||||
help="Use built-in demo defaults (same as C4AI_DEMO_DEBUG=1)",
|
||||
)
|
||||
return ap
|
||||
|
||||
|
||||
def detect_debug_defaults(force = False) -> SimpleNamespace:
|
||||
if not force and sys.gettrace() is None and not os.getenv("C4AI_DEMO_DEBUG"):
|
||||
return SimpleNamespace()
|
||||
# ----- debug‑friendly defaults -----
|
||||
return SimpleNamespace(
|
||||
cmd="full",
|
||||
query="health insurance management",
|
||||
geo=102713980,
|
||||
# title_filters="Product,Engineering",
|
||||
title_filters="",
|
||||
max_companies=10,
|
||||
max_people=5,
|
||||
profile_name="profile_linkedin_uc",
|
||||
outdir="./debug_out",
|
||||
concurrency=2,
|
||||
log_level="debug",
|
||||
)
|
||||
|
||||
|
||||
async def async_main(opts):
|
||||
# ─────────── logging setup ───────────
|
||||
console = Console()
|
||||
logging.basicConfig(
|
||||
level=opts.log_level.upper(),
|
||||
format="%(message)s",
|
||||
handlers=[RichHandler(console=console, markup=True, rich_tracebacks=True)],
|
||||
)
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Load or build schemas (one‑time LLM call each)
|
||||
# -------------------------------------------------------------------
|
||||
company_schema = _load_or_build_schema(
|
||||
COMPANY_SCHEMA_PATH,
|
||||
_SAMPLE_COMPANY_HTML,
|
||||
_COMPANY_SCHEMA_QUERY,
|
||||
_COMPANY_SCHEMA_EXAMPLE,
|
||||
# True
|
||||
)
|
||||
people_schema = _load_or_build_schema(
|
||||
PEOPLE_SCHEMA_PATH,
|
||||
_SAMPLE_PEOPLE_HTML,
|
||||
_PEOPLE_SCHEMA_QUERY,
|
||||
_PEOPLE_SCHEMA_EXAMPLE,
|
||||
# True
|
||||
)
|
||||
|
||||
outdir = BASE_DIR / pathlib.Path(opts.outdir)
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
f_companies = (BASE_DIR / outdir / "companies.jsonl").open("a", encoding="utf-8")
|
||||
f_people = (BASE_DIR / outdir / "people.jsonl").open("a", encoding="utf-8")
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Prepare crawler with cookie pool rotation
|
||||
# -------------------------------------------------------------------
|
||||
profiler = BrowserProfiler()
|
||||
path = profiler.get_profile_path(opts.profile_name)
|
||||
bc = BrowserConfig(
|
||||
headless=False,
|
||||
verbose=False,
|
||||
user_data_dir=path,
|
||||
use_managed_browser=True,
|
||||
user_agent_mode = "random",
|
||||
user_agent_generator_config= {
|
||||
"platforms": "mobile",
|
||||
"os": "Android"
|
||||
}
|
||||
)
|
||||
crawler = AsyncWebCrawler(config=bc)
|
||||
|
||||
await crawler.start()
|
||||
|
||||
# Single worker for simplicity; concurrency can be scaled by arun_many if needed.
|
||||
# crawler = await next_crawler().start()
|
||||
try:
|
||||
# Build LinkedIn search URL
|
||||
search_url = f'https://www.linkedin.com/search/results/companies/?keywords={quote(opts.query)}&companyHqGeo="{opts.geo}"'
|
||||
logging.info("Seed URL => %s", search_url)
|
||||
|
||||
companies: List[Dict] = []
|
||||
if opts.cmd in ("companies", "full"):
|
||||
companies = await crawl_company_search(
|
||||
crawler, search_url, company_schema, opts.max_companies
|
||||
)
|
||||
for c in companies:
|
||||
f_companies.write(json.dumps(c, ensure_ascii=False) + "\n")
|
||||
logging.info(f"[bold green]✓[/] Companies scraped so far: {len(companies)}")
|
||||
|
||||
if opts.cmd in ("people", "full"):
|
||||
if not companies:
|
||||
# load from previous run
|
||||
src = outdir / "companies.jsonl"
|
||||
if not src.exists():
|
||||
logging.error("companies.jsonl missing — run companies/full first")
|
||||
return 10
|
||||
companies = [json.loads(l) for l in src.read_text().splitlines()]
|
||||
total_people = 0
|
||||
title_kw = " ".join([t.strip() for t in opts.title_filters.split(",") if t.strip()]) if opts.title_filters else ""
|
||||
for comp in companies:
|
||||
people = await crawl_people_page(
|
||||
crawler,
|
||||
comp["people_url"],
|
||||
people_schema,
|
||||
opts.max_people,
|
||||
title_kw,
|
||||
)
|
||||
for p in people:
|
||||
rec = p | {
|
||||
"company_handle": comp["handle"],
|
||||
# "captured_at": datetime.now(UTC).isoformat(timespec="seconds") + "Z",
|
||||
"captured_at": datetime.now(UTC).isoformat(timespec="seconds") + "Z",
|
||||
}
|
||||
f_people.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
total_people += len(people)
|
||||
logging.info(
|
||||
f"{comp['name']} — [cyan]{len(people)}[/] people extracted"
|
||||
)
|
||||
await asyncio.sleep(random.uniform(0.5, 1))
|
||||
logging.info("Total people scraped: %d", total_people)
|
||||
finally:
|
||||
await crawler.close()
|
||||
f_companies.close()
|
||||
f_people.close()
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
parser = build_arg_parser()
|
||||
cli_opts = parser.parse_args()
|
||||
|
||||
# decide on debug defaults
|
||||
if cli_opts.debug:
|
||||
opts = detect_debug_defaults(force=True)
|
||||
cli_opts = opts
|
||||
else:
|
||||
env_defaults = detect_debug_defaults()
|
||||
opts = env_defaults if env_defaults else cli_opts
|
||||
|
||||
if not getattr(opts, "cmd", None):
|
||||
opts.cmd = "full"
|
||||
|
||||
exit_code = asyncio.run(async_main(cli_opts))
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,381 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Stage-2 Insights builder
|
||||
------------------------
|
||||
Reads companies.jsonl & people.jsonl (Stage-1 output) and produces:
|
||||
• company_graph.json
|
||||
• org_chart_<handle>.json (one per company)
|
||||
• decision_makers.csv
|
||||
• graph_view.html (interactive visualisation)
|
||||
|
||||
Run:
|
||||
python c4ai_insights.py --in ./stage1_out --out ./stage2_out
|
||||
|
||||
Author : Tom @ Kidocode, 2025-04-28
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Imports & Third-party
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
import argparse, asyncio, json, pathlib, random
|
||||
from datetime import datetime, UTC
|
||||
from types import SimpleNamespace
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any
|
||||
# Pretty CLI UX
|
||||
from rich.console import Console
|
||||
from rich.logging import RichHandler
|
||||
from rich.progress import Progress, SpinnerColumn, BarColumn, TextColumn, TimeElapsedColumn
|
||||
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
BASE_DIR = pathlib.Path(__file__).resolve().parent
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# 3rd-party deps
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
import numpy as np
|
||||
# from sentence_transformers import SentenceTransformer
|
||||
# from sklearn.metrics.pairwise import cosine_similarity
|
||||
import pandas as pd
|
||||
import hashlib
|
||||
|
||||
from litellm import completion #Support any LLM Provider
|
||||
|
||||
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Utils
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
def load_jsonl(path: Path) -> List[Dict[str, Any]]:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return [json.loads(l) for l in f]
|
||||
|
||||
def dump_json(obj, path: Path):
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(obj, f, ensure_ascii=False, indent=2)
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Constants
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
BASE_DIR = pathlib.Path(__file__).resolve().parent
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Debug defaults (mirrors Stage-1 trick)
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
def dev_defaults() -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
in_dir="./samples",
|
||||
out_dir="./samples/insights",
|
||||
embed_model="all-MiniLM-L6-v2",
|
||||
top_k=10,
|
||||
llm_provider="openai/gpt-4.1",
|
||||
llm_api_key=None,
|
||||
max_llm_tokens=8000,
|
||||
llm_temperature=1.0,
|
||||
stub=False, # Set to True to use a stub for org-chart inference
|
||||
llm_base_url=None, # e.g., "https://api.openai.com/v1" for OpenAI
|
||||
workers=4
|
||||
)
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Graph builders
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
def embed_descriptions(companies, model_name:str, opts) -> np.ndarray:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
console = Console()
|
||||
console.print(f"Using embedding model: [bold cyan]{model_name}[/]")
|
||||
cache_path = BASE_DIR / Path(opts.out_dir) / "embeds_cache.json"
|
||||
cache = {}
|
||||
if cache_path.exists():
|
||||
with open(cache_path) as f:
|
||||
cache = json.load(f)
|
||||
# flush cache if model differs
|
||||
if cache.get("_model") != model_name:
|
||||
cache = {}
|
||||
|
||||
model = SentenceTransformer(model_name)
|
||||
new_texts, new_indices = [], []
|
||||
vectors = np.zeros((len(companies), 384), dtype=np.float32)
|
||||
|
||||
for idx, comp in enumerate(companies):
|
||||
text = comp.get("about") or comp.get("descriptor","")
|
||||
h = hashlib.sha1(text.encode("utf-8")).hexdigest()
|
||||
cached = cache.get(comp["handle"])
|
||||
if cached and cached["hash"] == h:
|
||||
vectors[idx] = np.array(cached["vector"], dtype=np.float32)
|
||||
else:
|
||||
new_texts.append(text)
|
||||
new_indices.append((idx, comp["handle"], h))
|
||||
|
||||
if new_texts:
|
||||
embeds = model.encode(new_texts, show_progress_bar=False, convert_to_numpy=True)
|
||||
for vec, (idx, handle, h) in zip(embeds, new_indices):
|
||||
vectors[idx] = vec
|
||||
cache[handle] = {"hash": h, "vector": vec.tolist()}
|
||||
cache["_model"] = model_name
|
||||
with open(cache_path, "w") as f:
|
||||
json.dump(cache, f)
|
||||
|
||||
return vectors
|
||||
|
||||
def build_company_graph(companies, embeds:np.ndarray, top_k:int) -> Dict[str,Any]:
|
||||
from sklearn.metrics.pairwise import cosine_similarity
|
||||
sims = cosine_similarity(embeds)
|
||||
nodes, edges = [], []
|
||||
for i,c in enumerate(companies):
|
||||
node = dict(
|
||||
id=c["handle"].strip("/"),
|
||||
name=c["name"],
|
||||
handle=c["handle"],
|
||||
about=c.get("about",""),
|
||||
people_url=c.get("people_url",""),
|
||||
industry=c.get("descriptor","").split("•")[0].strip(),
|
||||
geoUrn=c.get("geoUrn"),
|
||||
followers=c.get("followers",0),
|
||||
# desc_embed=embeds[i].tolist(),
|
||||
desc_embed=[],
|
||||
)
|
||||
nodes.append(node)
|
||||
# pick top-k most similar except itself
|
||||
top_idx = np.argsort(sims[i])[::-1][1:top_k+1]
|
||||
for j in top_idx:
|
||||
tgt = companies[j]
|
||||
weight = float(sims[i,j])
|
||||
if node["industry"] == tgt.get("descriptor","").split("•")[0].strip():
|
||||
weight += 0.10
|
||||
if node["geoUrn"] == tgt.get("geoUrn"):
|
||||
weight += 0.05
|
||||
tgt['followers'] = tgt.get("followers", None) or 1
|
||||
node["followers"] = node.get("followers", None) or 1
|
||||
follower_ratio = min(node["followers"], tgt.get("followers",1)) / max(node["followers"] or 1, tgt.get("followers",1))
|
||||
weight += 0.05 * follower_ratio
|
||||
edges.append(dict(
|
||||
source=node["id"],
|
||||
target=tgt["handle"].strip("/"),
|
||||
weight=round(weight,4),
|
||||
drivers=dict(
|
||||
embed_sim=round(float(sims[i,j]),4),
|
||||
industry_match=0.10 if node["industry"] == tgt.get("descriptor","").split("•")[0].strip() else 0,
|
||||
geo_overlap=0.05 if node["geoUrn"] == tgt.get("geoUrn") else 0,
|
||||
)
|
||||
))
|
||||
# return {"nodes":nodes,"edges":edges,"meta":{"generated_at":datetime.now(UTC).isoformat()}}
|
||||
return {"nodes":nodes,"edges":edges,"meta":{"generated_at":datetime.now(UTC).isoformat()}}
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Org-chart via LLM
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
async def infer_org_chart_llm(company, people, llm_provider:str, api_key:str, max_tokens:int, temperature:float, stub:bool=False, base_url:str=None):
|
||||
if stub:
|
||||
# Tiny fake org-chart when debugging offline
|
||||
chief = random.choice(people)
|
||||
nodes = [{
|
||||
"id": chief["profile_url"],
|
||||
"name": chief["name"],
|
||||
"title": chief["headline"],
|
||||
"dept": chief["headline"].split()[:1][0],
|
||||
"yoe_total": 8,
|
||||
"yoe_current": 2,
|
||||
"seniority_score": 0.8,
|
||||
"decision_score": 0.9,
|
||||
"avatar_url": chief.get("avatar_url")
|
||||
}]
|
||||
return {"nodes":nodes,"edges":[],"meta":{"debug_stub":True,"generated_at":datetime.now(UTC).isoformat()}}
|
||||
|
||||
prompt = [
|
||||
{"role":"system","content":"You are an expert B2B org-chart reasoner."},
|
||||
{"role":"user","content":f"""Here is the company description:
|
||||
|
||||
<company>
|
||||
{json.dumps(company, ensure_ascii=False)}
|
||||
</company>
|
||||
|
||||
Here is a JSON list of employees:
|
||||
<employees>
|
||||
{json.dumps(people, ensure_ascii=False)}
|
||||
</employees>
|
||||
|
||||
1) Build a reporting tree (manager -> direct reports)
|
||||
2) For each person output a decision_score 0-1 for buying new software
|
||||
|
||||
Return JSON: {{ "nodes":[{{id,name,title,dept,yoe_total,yoe_current,seniority_score,decision_score,avatar_url,profile_url}}], "edges":[{{source,target,type,confidence}}] }}
|
||||
"""}
|
||||
]
|
||||
resp = completion(
|
||||
model=llm_provider,
|
||||
messages=prompt,
|
||||
max_tokens=max_tokens,
|
||||
temperature=temperature,
|
||||
response_format={"type":"json_object"},
|
||||
api_key=api_key,
|
||||
base_url=base_url
|
||||
)
|
||||
chart = json.loads(resp.choices[0].message.content)
|
||||
chart["meta"] = dict(
|
||||
model=llm_provider,
|
||||
generated_at=datetime.now(UTC).isoformat()
|
||||
)
|
||||
return chart
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# CSV flatten
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
def export_decision_makers(charts_dir:Path, csv_path:Path, threshold:float=0.5):
|
||||
rows=[]
|
||||
for p in charts_dir.glob("org_chart_*.json"):
|
||||
data=json.loads(p.read_text())
|
||||
comp = p.stem.split("org_chart_")[1]
|
||||
for n in data.get("nodes",[]):
|
||||
if n.get("decision_score",0)>=threshold:
|
||||
rows.append(dict(
|
||||
company=comp,
|
||||
person=n["name"],
|
||||
title=n["title"],
|
||||
decision_score=n["decision_score"],
|
||||
profile_url=n["id"]
|
||||
))
|
||||
pd.DataFrame(rows).to_csv(csv_path,index=False)
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# HTML rendering
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
def render_html(out:Path, template_dir:Path):
|
||||
# From template folder cp graph_view.html and ai.js in out folder
|
||||
import shutil
|
||||
shutil.copy(template_dir/"graph_view_template.html", out / "graph_view.html")
|
||||
shutil.copy(template_dir/"ai.js", out)
|
||||
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# Main async pipeline
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
async def run(opts):
|
||||
# ── silence SDK noise ──────────────────────────────────────────────────────
|
||||
# for noisy in ("openai", "httpx", "httpcore"):
|
||||
# lg = logging.getLogger(noisy)
|
||||
# lg.setLevel(logging.WARNING) # or ERROR if you want total silence
|
||||
# lg.propagate = False # optional: stop them reaching root
|
||||
|
||||
# ────────────── logging bootstrap ──────────────
|
||||
console = Console()
|
||||
# logging.basicConfig(
|
||||
# level="INFO",
|
||||
# format="%(message)s",
|
||||
# handlers=[RichHandler(console=console, markup=True, rich_tracebacks=True)],
|
||||
# )
|
||||
|
||||
in_dir = BASE_DIR / Path(opts.in_dir)
|
||||
out_dir = BASE_DIR / Path(opts.out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
companies = load_jsonl(in_dir/"companies.jsonl")
|
||||
people = load_jsonl(in_dir/"people.jsonl")
|
||||
|
||||
console.print(f"[bold cyan]Loaded[/] {len(companies)} companies, {len(people)} people")
|
||||
|
||||
console.print("[bold]⇢[/] Embedding company descriptions…")
|
||||
embeds = embed_descriptions(companies, opts.embed_model, opts)
|
||||
|
||||
console.print("[bold]⇢[/] Building similarity graph")
|
||||
company_graph = build_company_graph(companies, embeds, opts.top_k)
|
||||
dump_json(company_graph, out_dir/"company_graph.json")
|
||||
|
||||
# Filter companies that need processing
|
||||
to_process = []
|
||||
for comp in companies:
|
||||
handle = comp["handle"].strip("/").replace("/","_")
|
||||
out_file = out_dir/f"org_chart_{handle}.json"
|
||||
if out_file.exists():
|
||||
console.print(f"[green]✓[/] Skipping existing {comp['name']}")
|
||||
continue
|
||||
to_process.append(comp)
|
||||
|
||||
|
||||
if not to_process:
|
||||
console.print("[yellow]All companies already processed[/]")
|
||||
else:
|
||||
workers = getattr(opts, 'workers', 1)
|
||||
parallel = workers > 1
|
||||
|
||||
console.print(f"[bold]⇢[/] Inferring org-charts via LLM {f'(parallel={workers} workers)' if parallel else ''}")
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
TimeElapsedColumn(),
|
||||
console=console,
|
||||
) as progress:
|
||||
task = progress.add_task("Org charts", total=len(to_process))
|
||||
|
||||
async def process_one(comp):
|
||||
handle = comp["handle"].strip("/").replace("/","_")
|
||||
persons = [p for p in people if p["company_handle"].strip("/") == comp["handle"].strip("/")]
|
||||
chart = await infer_org_chart_llm(
|
||||
comp, persons,
|
||||
llm_provider=opts.llm_provider,
|
||||
api_key=opts.llm_api_key or None,
|
||||
max_tokens=opts.max_llm_tokens,
|
||||
temperature=opts.llm_temperature,
|
||||
stub=opts.stub or False,
|
||||
base_url=opts.llm_base_url or None
|
||||
)
|
||||
chart["meta"]["company"] = comp["name"]
|
||||
|
||||
# Save the result immediately
|
||||
dump_json(chart, out_dir/f"org_chart_{handle}.json")
|
||||
|
||||
progress.update(task, advance=1, description=f"{comp['name']} ({len(persons)} ppl)")
|
||||
|
||||
# Create tasks for all companies
|
||||
tasks = [process_one(comp) for comp in to_process]
|
||||
|
||||
# Process in batches based on worker count
|
||||
semaphore = asyncio.Semaphore(workers)
|
||||
|
||||
async def bounded_process(coro):
|
||||
async with semaphore:
|
||||
return await coro
|
||||
|
||||
# Run with concurrency control
|
||||
await asyncio.gather(*(bounded_process(task) for task in tasks))
|
||||
|
||||
console.print("[bold]⇢[/] Flattening decision-makers CSV")
|
||||
export_decision_makers(out_dir, out_dir/"decision_makers.csv")
|
||||
|
||||
render_html(out_dir, template_dir=BASE_DIR/"templates")
|
||||
console.print(f"[bold green]✓[/] Stage-2 artefacts written to {out_dir}")
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
# CLI
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
def build_arg_parser():
|
||||
p = argparse.ArgumentParser(description="Build graphs & visualisation from Stage-1 output")
|
||||
p.add_argument("--in", dest="in_dir", required=False, help="Stage-1 output dir", default=".")
|
||||
p.add_argument("--out", dest="out_dir", required=False, help="Destination dir", default=".")
|
||||
p.add_argument("--embed-model", default="all-MiniLM-L6-v2")
|
||||
p.add_argument("--top-k", type=int, default=10, help="Top-k neighbours per company")
|
||||
p.add_argument("--llm-provider", default="openai/gpt-4.1",
|
||||
help="LLM model to use in format 'provider/model_name' (e.g., 'anthropic/claude-3')")
|
||||
p.add_argument("--llm-api-key", help="API key for LLM provider (defaults to env vars)")
|
||||
p.add_argument("--llm-base-url", help="Base URL for LLM API endpoint")
|
||||
p.add_argument("--max-llm-tokens", type=int, default=8024)
|
||||
p.add_argument("--llm-temperature", type=float, default=1.0)
|
||||
p.add_argument("--stub", action="store_true", help="Skip OpenAI call and generate tiny fake org charts")
|
||||
p.add_argument("--workers", type=int, default=4, help="Number of parallel workers for LLM inference")
|
||||
return p
|
||||
|
||||
def main():
|
||||
dbg = dev_defaults()
|
||||
opts = dbg if True else build_arg_parser().parse_args()
|
||||
# opts = build_arg_parser().parse_args()
|
||||
asyncio.run(run(opts))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,9 @@
|
||||
{"handle": "https://www.linkedin.com/company/healthpartnersng/", "name": "Health Partners HMO", "descriptor": "Hospitals and Health Care • Ikoyi, LAGOS", "about": "Healthpartners Ltd is a leading HMO in Nigeria providing affordablehealthinsuranceandhealthmanagementservices for companies and individuals in Nigeria. We have several individual and group plans that meets yourhealthmanagementneeds. Call us now at 0807-460-9165, 0807-714-0759 or email...", "followers": null, "people_url": "https://www.linkedin.com/company/healthpartnersng/people/", "captured_at": "2025-04-29T10:46:08Z"}
|
||||
{"handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "name": "Health & Insurance Management Services Organization", "descriptor": "Non-profit Organizations • Mbeya", "about": "Health&InsuranceManagementServices Organization (HIMSO) was established and registered in 2012 as a Non- Government Organization (NGO) with the aim...", "followers": 35, "people_url": "https://www.linkedin.com/company/health-insurance-management-services-organization/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "name": "National Health Insurance Management Authority", "descriptor": "Insurance • Lusaka, Lusaka", "about": "The NationalHealthInsuranceManagementAuthority (NHIMA) is established pursuant to section 4 of the NationalHealthInsurance(NHI) Act No. 2 of 2018. The compulsory NationalHealthInsurancescheme seeks to provide for a sound and reliable healthcare financing for Zambian households and the entirehealthsector...", "followers": null, "people_url": "https://www.linkedin.com/company/national-health-insurance-management-authority/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/health-alliance-plan/", "name": "Health Alliance Plan", "descriptor": "Hospitals and Health Care • Detroit, MI", "about": "...organizations to enhance the lives of those we touch. We offer six distincthealthinsurancelines: • Group Insured Commercial • Individual • Medicare • Medicaid • Self-Funded • Network Leasing HAP also provides: • Award-winning wellness programs • Community outreach • Digitalhealthtools • Diseasemanagement...", "followers": null, "people_url": "https://www.linkedin.com/company/health-alliance-plan/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "name": "Insurance Recruiting Solutions", "descriptor": "Insurance • Waukee, Iowa", "about": "InsuranceRecruiting Solutions provides staffing and recruiting services exclusively to theinsuranceindustry. We are committed to providing highly personalized recruiting services, tailored to each candidate and employer. With years ofinsuranceindustry experience, we speak your language. As a leading national...", "followers": null, "people_url": "https://www.linkedin.com/company/insurance-recruiting-solutions/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "name": "Health Plan of San Mateo (HPSM)", "descriptor": "Hospitals and Health Care • South San Francisco, California", "about": "TheHealthPlan of San Mateo (HPSM) is a local non-profithealthcare plan that offershealthcoverage and a provider network to San Mateo County's under-insured population. We currently serve more than 145,000 County residents.", "followers": null, "people_url": "https://www.linkedin.com/company/healthplanofsanmateo/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/insurance-management-group_2/", "name": "Insurance Management Group", "descriptor": "Insurance • Marion, Indiana", "about": "InsuranceManagementGroup is an all-riskinsuranceagency with over 140 years of experience, specializing in Home, Auto, BusinessInsurance, Individual Life &Health, and Employee Benefits. We represent highly rated and financially soundinsurancecarriers, to ensure that our clients are getting the best coverage...", "followers": null, "people_url": "https://www.linkedin.com/company/insurance-management-group_2/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "name": "CareCard Health Insurance Management Co", "descriptor": "Insurance • Damascus", "about": "CareCard offers Business Process Outsourcing (BPO) services toInsurance, Self Funded and Retireehealthplan market. CareCard provides operational outsourcing...", "followers": 187, "people_url": "https://www.linkedin.com/company/carecard-health-insurance-management-co/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
{"handle": "https://www.linkedin.com/company/healthcluster/", "name": "Health Cluster", "descriptor": "Technology, Information and Internet • Dubai", "about": "..., knowledge and interaction. The company has solutions and products inHealthTech, eHealth, DigitalHealth, Revenue CycleManagement– RCM Solutions, AI & ML, Internet...", "followers": null, "people_url": "https://www.linkedin.com/company/healthcluster/people/", "captured_at": "2025-04-29T13:15:04Z"}
|
||||
@@ -0,0 +1,108 @@
|
||||
{"profile_url": null, "name": "Yahya Ipuge", "headline": "Senior Health Specialist, Independent Consultant, Certified Board Director, Board Chair in NGO and Private Entities", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFuqPObSyLPMQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1517757008397?e=1751500800&v=beta&t=zaHc2CY7AJ-eX1MCSvazp8ny37iBAu3YsyaZjwq6gB0", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-29T13:15:33Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Field officer at Health and Insurance Management Services Organization", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5103AQEVmdDwTIhsjQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1540989154156?e=1751500800&v=beta&t=7N0baJNfZ26dbrNNbv2055sbGlacQUwQu07wUTN0whs", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-29T13:15:33Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Medical Practitioner @ Health & Insurance | Master's Degree in Infection Control", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQHjMXy7dSmmLg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1725975429410?e=1751500800&v=beta&t=lDIL2KhDw471XYvtCrRfkHAnG3Q-npDJnwDdK0sYvpA", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-29T13:15:34Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "--", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-29T13:15:38Z"}
|
||||
{"profile_url": null, "name": "Fadhy Mtanga", "headline": "Executive Director at Health & Insurance Management Services Organization (HIMSO) Author | Creative Writer | Social Scientist", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQEloEreyg3qVQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1704391866585?e=1751500800&v=beta&t=86am-v3cjBPBldLTwgt8-AY-YbxFY6QZQzObwLTtMEA", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-29T13:15:38Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Business Administrator at Consultancy Business investments", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQEuKXJmknr2YA/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1714545221728?e=1751500800&v=beta&t=zJG-rDZgYJJ0eROibf-Wag-v_JecCghwU3ul4TaH2Eg", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-29T13:15:48Z"}
|
||||
{"profile_url": null, "name": "Tamani Phiri", "headline": "Corporate Business Strategy | Thought Leadership | Corporate Governance", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQF4mFx8jY2n-w/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1730302954035?e=1751500800&v=beta&t=i4QIrHA6A9eLtKolwTRNhuoiaTad28sf5KHxAFuXG-w", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-29T13:15:48Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Member Service Assistant @ National Health Insurance Management Authority (NHIMA) | Clinical Officer | Health Insurance & Public Health | Claims Processing & Customer Support | Data & Policy Analyst", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQGob13KyxrB0g/profile-displayphoto-shrink_100_100/B4DZYCgreeHIAU-/0/1743798848889?e=1751500800&v=beta&t=uXxTsMLi5s7hr8FBEzVTDw7V3eJ85kpTaIC7i_5fM-Y", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-29T13:15:48Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Economist/ Development Analyst/ Planner/ Customer Care", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQFEc3EgfdpZeg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1727782989867?e=1751500800&v=beta&t=dWjKzSu5FDRgmxAVret9jQPhWF2VjcrnmEpR2LDMC1Q", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-29T13:15:48Z"}
|
||||
{"profile_url": null, "name": "Samantha Ngandwe", "headline": "Quality Assurance and Accreditation Officer at National Health Insurance Management Authority", "followers": 382, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQHyOjyoz7d95g/profile-displayphoto-shrink_100_100/B4DZYvvhP5GwAY-/0/1744557712084?e=1751500800&v=beta&t=DLYRpz20zmwUWx1UY1Dn-ykvgWBnwn8XHWLaDMf199M", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-29T13:15:48Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Dental Surgery Assistant at Health Promotion Board", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-29T13:16:11Z"}
|
||||
{"profile_url": null, "name": "Liz England Tucker", "headline": "Medical Performance Optimization", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQFY6yx360QunQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1713831102587?e=1751500800&v=beta&t=u-C8Ozpl_ITkTpdgt5QD-C5_Qt7MA0DagLRmiuGKngQ", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-29T13:16:11Z"}
|
||||
{"profile_url": null, "name": "Merrill Hausenfluck", "headline": "Chief Financial Officer", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQGKxDKRJM_BCg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1696292650180?e=1751500800&v=beta&t=NbUVC-QP-XL3frBpQcn3GtGrZ04Fl0xdko4V-mHxPag", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-29T13:16:11Z"}
|
||||
{"profile_url": null, "name": "Mike Treash", "headline": "Senior Vice President and Chief Operating Officer at Health Alliance Plan", "followers": 2000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQH_c6tIq929gw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1720478900599?e=1751500800&v=beta&t=l9RLnLDKBBJjJQTsFMJMa_1MpWCKcV4AUa3dcjGnSXQ", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-29T13:16:11Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Manager at Health Alliance Plan", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-29T13:16:11Z"}
|
||||
{"profile_url": null, "name": "Scot Dickerson", "headline": "Insurance Industry Specialist, Insurance Recruiter, Talent Acquisition, Talent Sourcing, Hiring Consultant, Career Consultant, Staffing, Executive Recruiter at Insurance Recruiting Solutions #insurancejobs #insurance", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQGLFvtPPU3HEw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1724950672124?e=1751500800&v=beta&t=uT4SFSMF32O1d50Z0dbnd6zRRKdABHxSGlOZdxWdXBM", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-29T13:16:24Z"}
|
||||
{"profile_url": null, "name": "Steele Dickerson", "headline": "Insurance Recruiting Solutions", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQEyICWaE_PvXA/profile-displayphoto-shrink_100_100/B56ZQuDHyZH0Ac-/0/1735939358232?e=1751500800&v=beta&t=9FdnWHrjnPQ7LQ5FdwC7sY8sS6hm-R4zfWO5Vmwm46w", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-29T13:16:24Z"}
|
||||
{"profile_url": null, "name": "Madeline Judas", "headline": "Recruiting Operations & Business Development Specialist", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQG6xiTaJ71UiA/profile-displayphoto-shrink_100_100/B56ZU_N_jPHoAY-/0/1740522388021?e=1751500800&v=beta&t=CxvAsYgU0zelghZsRhUJOC26ILVovP3ZPn4nMnWkEJE", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-29T13:16:24Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "All Lines Claims Adjuster / General Lines Agent (Property & Casualty : Life, Accident, Health & HMO)", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQFTjkb7SxTWWg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1725920318474?e=1751500800&v=beta&t=BGEzQg1c2l8qxuy2iKJ896nElsiYcaWnhkf-mqc-KhY", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-29T13:16:24Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Clinical Pharmacy Manager at Health Plan of San Mateo (HPSM)", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQEPO0pZOxznoA/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1551565536585?e=1751500800&v=beta&t=qwMGzWX_Zefkciq8h2m9daLMflT0WoDr5F1R5pXvyM4", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-29T13:16:40Z"}
|
||||
{"profile_url": null, "name": "Tamana M.", "headline": "MPH Candidate at Brown University | Data Coordinator", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQEY3iDtFmpzlg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1714197678074?e=1751500800&v=beta&t=IsVT0uC7A-T-Tp22gZFDG9wiT7LMB5GmhccuI8f9c-I", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-29T13:16:40Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Program Manager", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-29T13:16:40Z"}
|
||||
{"profile_url": null, "name": "Mackenzie Baysinger Moniz, MSW", "headline": "Program Manager at Health Plan of San Mateo", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQHAd3A4zLyuWA/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1675716742150?e=1751500800&v=beta&t=ot3fMyJFnHwwNfKJiA_YxZp6MOK_iVGtSCUgVNq867g", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-29T13:16:40Z"}
|
||||
{"profile_url": null, "name": "John O.", "headline": "Healthcare Delivery Strategy Execution", "followers": null, "connection_degree": "· 3rd", "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-29T13:16:40Z"}
|
||||
{"profile_url": null, "name": "Daniel McQuilkin", "headline": "Senior Vice President", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFkScOqwhxvfQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1521406683682?e=1751500800&v=beta&t=iohhak3lrV1gpmA6dnoCxTRJidskfgmZUXKbNQbkxjs", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-29T13:17:05Z"}
|
||||
{"profile_url": null, "name": "Tony Bonacuse", "headline": "Senior Vice President at Insurance Management Group", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQF_JJOFLjkZoQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1516269003018?e=1751500800&v=beta&t=0APZt5RNhvUj4IxsSdi7JO9KxezZzOH_WQCibn5Szgs", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-29T13:17:05Z"}
|
||||
{"profile_url": null, "name": "Mark Bilger", "headline": "Director - Sr. Vice President at Insurance Management Group", "followers": 1000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQEzX5qUfqhd2g/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1663842785708?e=1751500800&v=beta&t=YyKXRQol0cDntoq8vbdxyaRvEFf0vWKNHPxk0cyWiG8", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-29T13:17:05Z"}
|
||||
{"profile_url": null, "name": "Adam Young, MBA", "headline": "Husband | Father | Traveler | Sports Fanatic | Food Enthusiast | Independent Insurance Professional", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQErWIq1AVyxKg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1601480475688?e=1751500800&v=beta&t=jK_mhX0PkDdG8WBZaipIIYRDm1PnWIuFR7sCKDhDi6s", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-29T13:17:05Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Senior Vice President at Insurance Management Group / Partner", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQH3dm30dXH82w/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1572228299104?e=1751500800&v=beta&t=iuBQYs4iLHJgRgjFbSA2YiNiAI8zDILqg-nVsLR9Qjk", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-29T13:17:05Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Doctor at CareCard Health Insurance Management Co", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-29T13:17:09Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Pharmacist", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQHyPi4Amu_Dkw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1640460490377?e=1751500800&v=beta&t=q7R_b7bD9CR-1-Dvu81WoEHN_ljHK16l6ioTIA0LN7Q", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-29T13:17:09Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "IT Manager at CareCard Health Insurance Management Co", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-29T13:17:09Z"}
|
||||
{"profile_url": null, "name": "Amal Shabani", "headline": "at carecard", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQFLzeP3yPkjgg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1519625412373?e=1751500800&v=beta&t=GULSoesSn83F_fYkkH_nPxWIjjs1d9Pucc3dUDNei6I", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-29T13:17:09Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "--", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-29T13:17:09Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Biologiste | Pharmaco-épidemiologie & Pharmaco-économie | Software Helath Care Management", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQHOPXrX5-oeug/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1663013895834?e=1751500800&v=beta&t=yE2RGp0rfhcJkjh_vdM0VwpaPUtoPewM80lTlr20OHU", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-29T13:17:14Z"}
|
||||
{"profile_url": null, "name": "Ruqaia Ali Alkhalifa", "headline": " RN,BSN, MSN,NE Database Officer for Scholarship Programs and Central Committee rapporteur at Al-Ahsa Health Cluster.", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQGfNujqDnuZDA/profile-displayphoto-shrink_100_100/B4EZOvsQThH0AU-/0/1733819436577?e=1751500800&v=beta&t=jleAVvhbg0H85tSi9TG96x0fqdkS1oytfaU02LHsFEI", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-29T13:17:14Z"}
|
||||
{"profile_url": null, "name": "Fahad Mohyuddin", "headline": "Healthcare AI Strategist | Digital Health | SaaS | Telehealth | HIS | EHR | IoT", "followers": 7000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFLnPh8fu-HHg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1647320077586?e=1751500800&v=beta&t=S__knVzEVrGZuyqwszCe_5V_kawbG5tejmmEe3fkMJE", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-29T13:17:14Z"}
|
||||
{"profile_url": null, "name": "Muhammad Moid Shams", "headline": "Azure DevOps | AWS Cloud Infrastructure| Freight Tech | Health Tech | HL7- NABIDH | HL7+ FHIR | KSA -NPHIES | FHIR - MOPH | HL7- Riayati | Freight Tech | Insure Tech | with Azure, Azure AI , PowerApps, D365 , M365", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQEzousRurY2Zg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1711283874675?e=1751500800&v=beta&t=ZheuoRIAkS_9M8WXafdwB1nJEuy-a5HEsrXlfOANx80", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-29T13:17:14Z"}
|
||||
{"profile_url": null, "name": "Muhammad Shahzaib (PMP® - SCRUM®)", "headline": "PMP-Certified Project Manager | Health Care & Web Solutions Expert | Customer Success & Operations Management Expert | Business Transformation Expert", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D35AQFyp7WcBZinYA/profile-framedphoto-shrink_100_100/profile-framedphoto-shrink_100_100/0/1730638721808?e=1746540000&v=beta&t=QoGze1AlotUfm3K9kMWG6ZGVHS3ADu38THVPlxlYUys", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-29T13:17:14Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Music Professional at Health Options Worldwide", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5103AQGF-Dp6v6nkGw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1585401654822?e=1751500800&v=beta&t=7yeO-dGz1p_B66cJVSlTSdAYJLMFFwxPIhwwcR8uWWo", "company_handle": "https://www.linkedin.com/company/health-options-worldwide/", "captured_at": "2025-04-29T13:17:17Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Trainer/instructor at Health Options Worldwide", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-options-worldwide/", "captured_at": "2025-04-29T13:17:17Z"}
|
||||
{"profile_url": null, "name": "Michael Akpoarebe-Isaac", "headline": "Chief Operating officer, Health Partners HMO", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQE7KNFaLMyqYg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1602714385413?e=1751500800&v=beta&t=In5GaREqoXtO3sPCx9ZJJBwIPY4008ii13RPRl0w0Fw", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-29T13:17:22Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "REGISTERED NURSE/CLAIMS SUPERVISOR/HEALTH EDUCATOR/ CASE MANAGER/ Lekki.", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4E35AQEEqf5i5pD76g/profile-framedphoto-shrink_100_100/profile-framedphoto-shrink_100_100/0/1724219552412?e=1746540000&v=beta&t=1PAfKEpQFL196LZHfY0wHAZ35TH0fRjku9ihSfDdOk4", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-29T13:17:22Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Data Analyst|Dedicated Retention Officer Boosting Customer Loyalty| Business Developer/ Event planner", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D35AQHIgeS1H7w65w/profile-framedphoto-shrink_100_100/B4DZV1QfcCGcAk-/0/1741429012517?e=1746540000&v=beta&t=NCIbW7MWY7Cy4YEC2xzLoX54-Lm5CNhorbuSQe0lZSk", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-29T13:17:22Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Former Group managing director at Health Partners Ltd", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQHPQPvIQbPQPg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1583328612508?e=1751500800&v=beta&t=LpynArccJCWrdWMSBvYLH4SI5G-xae7ECoWUUAl_CeU", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-29T13:17:22Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "HEAD, FINCON, @ HEALTH PARTNERS (HMO) LTD", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQG8XOvnazEibQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1518882054975?e=1751500800&v=beta&t=5gT6GAWGTqYfpvkjOk0ArvV73I_KspkWXgoG-VhoStg", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-29T13:17:22Z"}
|
||||
{"profile_url": null, "name": "Yahya Ipuge", "headline": "Senior Health Specialist, Independent Consultant, Certified Board Director, Board Chair in NGO and Private Entities", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFuqPObSyLPMQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1517757008397?e=1751500800&v=beta&t=zaHc2CY7AJ-eX1MCSvazp8ny37iBAu3YsyaZjwq6gB0", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:36:39Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Field officer at Health and Insurance Management Services Organization", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5103AQEVmdDwTIhsjQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1540989154156?e=1751500800&v=beta&t=7N0baJNfZ26dbrNNbv2055sbGlacQUwQu07wUTN0whs", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:36:39Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Medical Practitioner @ Health & Insurance | Master's Degree in Infection Control", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQHjMXy7dSmmLg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1725975429410?e=1751500800&v=beta&t=lDIL2KhDw471XYvtCrRfkHAnG3Q-npDJnwDdK0sYvpA", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:36:39Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "--", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:36:39Z"}
|
||||
{"profile_url": null, "name": "Fadhy Mtanga", "headline": "Executive Director at Health & Insurance Management Services Organization (HIMSO) Author | Creative Writer | Social Scientist", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQEloEreyg3qVQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1704391866585?e=1751500800&v=beta&t=86am-v3cjBPBldLTwgt8-AY-YbxFY6QZQzObwLTtMEA", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:36:39Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Member Service Assistant @ National Health Insurance Management Authority (NHIMA) | Clinical Officer | Health Insurance & Public Health | Claims Processing & Customer Support | Data & Policy Analyst", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQGob13KyxrB0g/profile-displayphoto-shrink_100_100/B4DZYCgreeHIAU-/0/1743798848889?e=1751500800&v=beta&t=uXxTsMLi5s7hr8FBEzVTDw7V3eJ85kpTaIC7i_5fM-Y", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:36:45Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Business Administrator at Consultancy Business investments", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQEuKXJmknr2YA/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1714545221728?e=1751500800&v=beta&t=zJG-rDZgYJJ0eROibf-Wag-v_JecCghwU3ul4TaH2Eg", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:36:45Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Economist/ Development Analyst/ Planner/ Customer Care", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQFEc3EgfdpZeg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1727782989867?e=1751500800&v=beta&t=dWjKzSu5FDRgmxAVret9jQPhWF2VjcrnmEpR2LDMC1Q", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:36:45Z"}
|
||||
{"profile_url": null, "name": "Tamani Phiri", "headline": "Corporate Business Strategy | Thought Leadership | Corporate Governance", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQF4mFx8jY2n-w/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1730302954035?e=1751500800&v=beta&t=i4QIrHA6A9eLtKolwTRNhuoiaTad28sf5KHxAFuXG-w", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:36:45Z"}
|
||||
{"profile_url": null, "name": "Samantha Ngandwe", "headline": "Quality Assurance and Accreditation Officer at National Health Insurance Management Authority", "followers": 382, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQHyOjyoz7d95g/profile-displayphoto-shrink_100_100/B4DZYvvhP5GwAY-/0/1744557712084?e=1751500800&v=beta&t=DLYRpz20zmwUWx1UY1Dn-ykvgWBnwn8XHWLaDMf199M", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:36:45Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Dental Surgery Assistant at Health Promotion Board", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:36:51Z"}
|
||||
{"profile_url": null, "name": "Merrill Hausenfluck", "headline": "Chief Financial Officer", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQGKxDKRJM_BCg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1696292650180?e=1751500800&v=beta&t=NbUVC-QP-XL3frBpQcn3GtGrZ04Fl0xdko4V-mHxPag", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:36:51Z"}
|
||||
{"profile_url": null, "name": "Mike Treash", "headline": "Senior Vice President and Chief Operating Officer at Health Alliance Plan", "followers": 2000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQH_c6tIq929gw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1720478900599?e=1751500800&v=beta&t=l9RLnLDKBBJjJQTsFMJMa_1MpWCKcV4AUa3dcjGnSXQ", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:36:51Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Manager at Health Alliance Plan", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:36:51Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Manager, Government Programs at Health Alliance Plan", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQF473eFGZeIpQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1654455840818?e=1751500800&v=beta&t=FllKCznSi0Ndm75QYy2i5UDtflCojNGkVzRpoChPC8c", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:36:51Z"}
|
||||
{"profile_url": null, "name": "Steele Dickerson", "headline": "Insurance Recruiting Solutions", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQEyICWaE_PvXA/profile-displayphoto-shrink_100_100/B56ZQuDHyZH0Ac-/0/1735939358232?e=1751500800&v=beta&t=9FdnWHrjnPQ7LQ5FdwC7sY8sS6hm-R4zfWO5Vmwm46w", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-30T07:36:56Z"}
|
||||
{"profile_url": null, "name": "Yahya Ipuge", "headline": "Senior Health Specialist, Independent Consultant, Certified Board Director, Board Chair in NGO and Private Entities", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFuqPObSyLPMQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1517757008397?e=1751500800&v=beta&t=zaHc2CY7AJ-eX1MCSvazp8ny37iBAu3YsyaZjwq6gB0", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:44:32+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Field officer at Health and Insurance Management Services Organization", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5103AQEVmdDwTIhsjQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1540989154156?e=1751500800&v=beta&t=7N0baJNfZ26dbrNNbv2055sbGlacQUwQu07wUTN0whs", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:44:32+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Medical Practitioner @ Health & Insurance | Master's Degree in Infection Control", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQHjMXy7dSmmLg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1725975429410?e=1751500800&v=beta&t=lDIL2KhDw471XYvtCrRfkHAnG3Q-npDJnwDdK0sYvpA", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:44:32+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "--", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:44:32+00:00Z"}
|
||||
{"profile_url": null, "name": "Fadhy Mtanga", "headline": "Executive Director at Health & Insurance Management Services Organization (HIMSO) Author | Creative Writer | Social Scientist", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQEloEreyg3qVQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1704391866585?e=1751500800&v=beta&t=86am-v3cjBPBldLTwgt8-AY-YbxFY6QZQzObwLTtMEA", "company_handle": "https://www.linkedin.com/company/health-insurance-management-services-organization/", "captured_at": "2025-04-30T07:44:32+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Member Service Assistant @ National Health Insurance Management Authority (NHIMA) | Clinical Officer | Health Insurance & Public Health | Claims Processing & Customer Support | Data & Policy Analyst", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQGob13KyxrB0g/profile-displayphoto-shrink_100_100/B4DZYCgreeHIAU-/0/1743798848889?e=1751500800&v=beta&t=uXxTsMLi5s7hr8FBEzVTDw7V3eJ85kpTaIC7i_5fM-Y", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:44:38+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Business Administrator at Consultancy Business investments", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQEuKXJmknr2YA/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1714545221728?e=1751500800&v=beta&t=zJG-rDZgYJJ0eROibf-Wag-v_JecCghwU3ul4TaH2Eg", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:44:38+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Economist/ Development Analyst/ Planner/ Customer Care", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQFEc3EgfdpZeg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1727782989867?e=1751500800&v=beta&t=dWjKzSu5FDRgmxAVret9jQPhWF2VjcrnmEpR2LDMC1Q", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:44:38+00:00Z"}
|
||||
{"profile_url": null, "name": "Tamani Phiri", "headline": "Corporate Business Strategy | Thought Leadership | Corporate Governance", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQF4mFx8jY2n-w/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1730302954035?e=1751500800&v=beta&t=i4QIrHA6A9eLtKolwTRNhuoiaTad28sf5KHxAFuXG-w", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:44:38+00:00Z"}
|
||||
{"profile_url": null, "name": "Samantha Ngandwe", "headline": "Quality Assurance and Accreditation Officer at National Health Insurance Management Authority", "followers": 382, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQHyOjyoz7d95g/profile-displayphoto-shrink_100_100/B4DZYvvhP5GwAY-/0/1744557712084?e=1751500800&v=beta&t=DLYRpz20zmwUWx1UY1Dn-ykvgWBnwn8XHWLaDMf199M", "company_handle": "https://www.linkedin.com/company/national-health-insurance-management-authority/", "captured_at": "2025-04-30T07:44:38+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Dental Surgery Assistant at Health Promotion Board", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:44:43+00:00Z"}
|
||||
{"profile_url": null, "name": "Merrill Hausenfluck", "headline": "Chief Financial Officer", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQGKxDKRJM_BCg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1696292650180?e=1751500800&v=beta&t=NbUVC-QP-XL3frBpQcn3GtGrZ04Fl0xdko4V-mHxPag", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:44:43+00:00Z"}
|
||||
{"profile_url": null, "name": "Mike Treash", "headline": "Senior Vice President and Chief Operating Officer at Health Alliance Plan", "followers": 2000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQH_c6tIq929gw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1720478900599?e=1751500800&v=beta&t=l9RLnLDKBBJjJQTsFMJMa_1MpWCKcV4AUa3dcjGnSXQ", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:44:43+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Manager at Health Alliance Plan", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:44:43+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Manager, Government Programs at Health Alliance Plan", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQF473eFGZeIpQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1654455840818?e=1751500800&v=beta&t=FllKCznSi0Ndm75QYy2i5UDtflCojNGkVzRpoChPC8c", "company_handle": "https://www.linkedin.com/company/health-alliance-plan/", "captured_at": "2025-04-30T07:44:43+00:00Z"}
|
||||
{"profile_url": null, "name": "Steele Dickerson", "headline": "Insurance Recruiting Solutions", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQEyICWaE_PvXA/profile-displayphoto-shrink_100_100/B56ZQuDHyZH0Ac-/0/1735939358232?e=1751500800&v=beta&t=9FdnWHrjnPQ7LQ5FdwC7sY8sS6hm-R4zfWO5Vmwm46w", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-30T07:44:48+00:00Z"}
|
||||
{"profile_url": null, "name": "Scot Dickerson", "headline": "Insurance Industry Specialist, Insurance Recruiter, Talent Acquisition, Talent Sourcing, Hiring Consultant, Career Consultant, Staffing, Executive Recruiter at Insurance Recruiting Solutions #insurancejobs #insurance", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQGLFvtPPU3HEw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1724950672124?e=1751500800&v=beta&t=uT4SFSMF32O1d50Z0dbnd6zRRKdABHxSGlOZdxWdXBM", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-30T07:44:48+00:00Z"}
|
||||
{"profile_url": null, "name": "Madeline Judas", "headline": "Recruiting Operations & Business Development Specialist", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQG6xiTaJ71UiA/profile-displayphoto-shrink_100_100/B56ZU_N_jPHoAY-/0/1740522388021?e=1751500800&v=beta&t=CxvAsYgU0zelghZsRhUJOC26ILVovP3ZPn4nMnWkEJE", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-30T07:44:48+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "All Lines Claims Adjuster / General Lines Agent (Property & Casualty : Life, Accident, Health & HMO)", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQFTjkb7SxTWWg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1725920318474?e=1751500800&v=beta&t=BGEzQg1c2l8qxuy2iKJ896nElsiYcaWnhkf-mqc-KhY", "company_handle": "https://www.linkedin.com/company/insurance-recruiting-solutions/", "captured_at": "2025-04-30T07:44:48+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Clinical Pharmacy Manager at Health Plan of San Mateo (HPSM)", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQEPO0pZOxznoA/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1551565536585?e=1751500800&v=beta&t=qwMGzWX_Zefkciq8h2m9daLMflT0WoDr5F1R5pXvyM4", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-30T07:44:54+00:00Z"}
|
||||
{"profile_url": null, "name": "Tamana M.", "headline": "MPH Candidate at Brown University | Data Coordinator", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQEY3iDtFmpzlg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1714197678074?e=1751500800&v=beta&t=IsVT0uC7A-T-Tp22gZFDG9wiT7LMB5GmhccuI8f9c-I", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-30T07:44:54+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Program Manager", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-30T07:44:54+00:00Z"}
|
||||
{"profile_url": null, "name": "Mackenzie Baysinger Moniz, MSW", "headline": "Program Manager at Health Plan of San Mateo", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D5603AQHAd3A4zLyuWA/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1675716742150?e=1751500800&v=beta&t=ot3fMyJFnHwwNfKJiA_YxZp6MOK_iVGtSCUgVNq867g", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-30T07:44:54+00:00Z"}
|
||||
{"profile_url": null, "name": "John O.", "headline": "Healthcare Delivery Strategy Execution", "followers": null, "connection_degree": "· 3rd", "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/healthplanofsanmateo/", "captured_at": "2025-04-30T07:44:54+00:00Z"}
|
||||
{"profile_url": null, "name": "Daniel McQuilkin", "headline": "Senior Vice President", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFkScOqwhxvfQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1521406683682?e=1751500800&v=beta&t=iohhak3lrV1gpmA6dnoCxTRJidskfgmZUXKbNQbkxjs", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-30T07:44:59+00:00Z"}
|
||||
{"profile_url": null, "name": "Tony Bonacuse", "headline": "Senior Vice President at Insurance Management Group", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQF_JJOFLjkZoQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1516269003018?e=1751500800&v=beta&t=0APZt5RNhvUj4IxsSdi7JO9KxezZzOH_WQCibn5Szgs", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-30T07:44:59+00:00Z"}
|
||||
{"profile_url": null, "name": "Mark Bilger", "headline": "Director - Sr. Vice President at Insurance Management Group", "followers": 1000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQEzX5qUfqhd2g/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1663842785708?e=1751500800&v=beta&t=YyKXRQol0cDntoq8vbdxyaRvEFf0vWKNHPxk0cyWiG8", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-30T07:44:59+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Senior Vice President at Insurance Management Group / Partner", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQH3dm30dXH82w/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1572228299104?e=1751500800&v=beta&t=iuBQYs4iLHJgRgjFbSA2YiNiAI8zDILqg-nVsLR9Qjk", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-30T07:44:59+00:00Z"}
|
||||
{"profile_url": null, "name": "Adam Young, MBA", "headline": "Husband | Father | Traveler | Sports Fanatic | Food Enthusiast | Independent Insurance Professional", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQErWIq1AVyxKg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1601480475688?e=1751500800&v=beta&t=jK_mhX0PkDdG8WBZaipIIYRDm1PnWIuFR7sCKDhDi6s", "company_handle": "https://www.linkedin.com/company/insurance-management-group_2/", "captured_at": "2025-04-30T07:44:59+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Doctor at CareCard Health Insurance Management Co", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-30T07:45:04+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Pharmacist", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQHyPi4Amu_Dkw/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1640460490377?e=1751500800&v=beta&t=q7R_b7bD9CR-1-Dvu81WoEHN_ljHK16l6ioTIA0LN7Q", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-30T07:45:04+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "IT Manager at CareCard Health Insurance Management Co", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-30T07:45:04+00:00Z"}
|
||||
{"profile_url": null, "name": "Amal Shabani", "headline": "at carecard", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C5603AQFLzeP3yPkjgg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1519625412373?e=1751500800&v=beta&t=GULSoesSn83F_fYkkH_nPxWIjjs1d9Pucc3dUDNei6I", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-30T07:45:04+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "--", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/carecard-health-insurance-management-co/", "captured_at": "2025-04-30T07:45:04+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Biologiste | Pharmaco-épidemiologie & Pharmaco-économie | Software Helath Care Management", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQHOPXrX5-oeug/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1663013895834?e=1751500800&v=beta&t=yE2RGp0rfhcJkjh_vdM0VwpaPUtoPewM80lTlr20OHU", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-30T07:45:09+00:00Z"}
|
||||
{"profile_url": null, "name": "Ruqaia Ali Alkhalifa", "headline": " RN,BSN, MSN,NE Database Officer for Scholarship Programs and Central Committee rapporteur at Al-Ahsa Health Cluster.", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4E03AQGfNujqDnuZDA/profile-displayphoto-shrink_100_100/B4EZOvsQThH0AU-/0/1733819436577?e=1751500800&v=beta&t=jleAVvhbg0H85tSi9TG96x0fqdkS1oytfaU02LHsFEI", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-30T07:45:09+00:00Z"}
|
||||
{"profile_url": null, "name": "Fahad Mohyuddin", "headline": "Healthcare AI Strategist | Digital Health | SaaS | Telehealth | HIS | EHR | IoT", "followers": 7000, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQFLnPh8fu-HHg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1647320077586?e=1751500800&v=beta&t=S__knVzEVrGZuyqwszCe_5V_kawbG5tejmmEe3fkMJE", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-30T07:45:09+00:00Z"}
|
||||
{"profile_url": null, "name": "Muhammad Moid Shams", "headline": "Azure DevOps | AWS Cloud Infrastructure| Freight Tech | Health Tech | HL7- NABIDH | HL7+ FHIR | KSA -NPHIES | FHIR - MOPH | HL7- Riayati | Freight Tech | Insure Tech | with Azure, Azure AI , PowerApps, D365 , M365", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D03AQEzousRurY2Zg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1711283874675?e=1751500800&v=beta&t=ZheuoRIAkS_9M8WXafdwB1nJEuy-a5HEsrXlfOANx80", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-30T07:45:09+00:00Z"}
|
||||
{"profile_url": null, "name": "Muhammad Shahzaib (PMP® - SCRUM®)", "headline": "PMP-Certified Project Manager | Health Care & Web Solutions Expert | Customer Success & Operations Management Expert | Business Transformation Expert", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/D4D35AQFyp7WcBZinYA/profile-framedphoto-shrink_100_100/profile-framedphoto-shrink_100_100/0/1730638721808?e=1746604800&v=beta&t=oewST3uZcxrt48z76eiJgTxl1EPoo63Cq-JcTwrFTbs", "company_handle": "https://www.linkedin.com/company/healthcluster/", "captured_at": "2025-04-30T07:45:09+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Music Professional at Health Options Worldwide", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C5103AQGF-Dp6v6nkGw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1585401654822?e=1751500800&v=beta&t=7yeO-dGz1p_B66cJVSlTSdAYJLMFFwxPIhwwcR8uWWo", "company_handle": "https://www.linkedin.com/company/health-options-worldwide/", "captured_at": "2025-04-30T07:45:13+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Trainer/instructor at Health Options Worldwide", "followers": null, "connection_degree": null, "avatar_url": "data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7", "company_handle": "https://www.linkedin.com/company/health-options-worldwide/", "captured_at": "2025-04-30T07:45:13+00:00Z"}
|
||||
{"profile_url": null, "name": "Michael Akpoarebe-Isaac", "headline": "Chief Operating officer, Health Partners HMO", "followers": null, "connection_degree": "· 3rd", "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQE7KNFaLMyqYg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1602714385413?e=1751500800&v=beta&t=In5GaREqoXtO3sPCx9ZJJBwIPY4008ii13RPRl0w0Fw", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-30T07:45:19+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "REGISTERED NURSE/CLAIMS SUPERVISOR/HEALTH EDUCATOR/ CASE MANAGER/ Lekki.", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4E35AQEEqf5i5pD76g/profile-framedphoto-shrink_100_100/profile-framedphoto-shrink_100_100/0/1724219552412?e=1746604800&v=beta&t=h0kqmp2KnpqQxsCCwyy7NpA8CAkSQ6qgbsZ0p0H7mXM", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-30T07:45:19+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Data Analyst|Dedicated Retention Officer Boosting Customer Loyalty| Business Developer/ Event planner", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/D4D35AQHIgeS1H7w65w/profile-framedphoto-shrink_100_100/B4DZV1QfcCGcAk-/0/1741429012517?e=1746604800&v=beta&t=zZi8WjnLpDrQD271jAId2mnfld_hO538QrN1-q2G4Zw", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-30T07:45:19+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "Former Group managing director at Health Partners Ltd", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4E03AQHPQPvIQbPQPg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1583328612508?e=1751500800&v=beta&t=LpynArccJCWrdWMSBvYLH4SI5G-xae7ECoWUUAl_CeU", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-30T07:45:19+00:00Z"}
|
||||
{"profile_url": null, "name": "LinkedIn Member", "headline": "HEAD, FINCON, @ HEALTH PARTNERS (HMO) LTD", "followers": null, "connection_degree": null, "avatar_url": "https://media.licdn.com/dms/image/v2/C4D03AQG8XOvnazEibQ/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1518882054975?e=1751500800&v=beta&t=5gT6GAWGTqYfpvkjOk0ArvV73I_KspkWXgoG-VhoStg", "company_handle": "https://www.linkedin.com/company/healthpartnersng/", "captured_at": "2025-04-30T07:45:19+00:00Z"}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"name": "LinkedIn Company Search Result Card",
|
||||
"baseSelector": "div[data-chameleon-result-urn][data-view-name=\"search-entity-result-universal-template\"]",
|
||||
"baseFields": [
|
||||
{
|
||||
"name": "chameleon_result_urn",
|
||||
"type": "attribute",
|
||||
"attribute": "data-chameleon-result-urn"
|
||||
},
|
||||
{
|
||||
"name": "view_name",
|
||||
"type": "attribute",
|
||||
"attribute": "data-view-name"
|
||||
}
|
||||
],
|
||||
"fields": [
|
||||
{
|
||||
"name": "handle",
|
||||
"selector": "div.mb1 div.display-flex span a[data-test-app-aware-link]",
|
||||
"type": "attribute",
|
||||
"attribute": "href"
|
||||
},
|
||||
{
|
||||
"name": "profile_image",
|
||||
"selector": "div.ivm-image-view-model img",
|
||||
"type": "attribute",
|
||||
"attribute": "src"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"selector": "div.mb1 div.display-flex span a[data-test-app-aware-link]",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "descriptor",
|
||||
"selector": "div.mb1 > div[class*=\"t-14 t-black\"]",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "about",
|
||||
"selector": "p.entity-result__summary--2-lines",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "followers",
|
||||
"selector": "div.mb1 > div:nth-of-type(3)",
|
||||
"type": "regex",
|
||||
"pattern": "(\\d+[KM]?) followers"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"name": "LinkedIn People Profile Card",
|
||||
"baseSelector": "li.org-people-profile-card__profile-card-spacing",
|
||||
"baseFields": [],
|
||||
"fields": [
|
||||
{
|
||||
"name": "profile_url",
|
||||
"selector": "div.artdeco-entity-lockup__title a[data-test-app-aware-link]",
|
||||
"type": "attribute",
|
||||
"attribute": "href"
|
||||
},
|
||||
{
|
||||
"name": "avatar_url",
|
||||
"selector": "div.artdeco-entity-lockup__image img",
|
||||
"type": "attribute",
|
||||
"attribute": "src"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"selector": "div.artdeco-entity-lockup__title a div.lt-line-clamp--single-line",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "headline",
|
||||
"selector": "div.artdeco-entity-lockup__subtitle div.lt-line-clamp--multi-line",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "followers",
|
||||
"selector": "span.text-align-center span.lt-line-clamp--multi-line",
|
||||
"type": "regex",
|
||||
"pattern": "(\\d+)"
|
||||
},
|
||||
{
|
||||
"name": "connection_degree",
|
||||
"selector": "span.artdeco-entity-lockup__degree",
|
||||
"type": "regex",
|
||||
"pattern": "(\\d+\\w+)"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
<li class="kZRArQqqhjjrHYceWaFbyEGWHRZbtqjTMawKA">
|
||||
<!----><!---->
|
||||
|
||||
|
||||
|
||||
<div class="xAuWirHJDUTuhkfOpmJApZWziplUyPIc" data-chameleon-result-urn="urn:li:company:2095237"
|
||||
data-view-name="search-entity-result-universal-template">
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="linked-area flex-1
|
||||
cursor-pointer">
|
||||
|
||||
<div class="qMGLeKnJyQnibGOueKodvnfLgWpsuA">
|
||||
<div class="cBPGFfFovHsbNhBFmECDIsPgMWmtMozOUfIAbs">
|
||||
<div class="display-flex align-items-center">
|
||||
<!---->
|
||||
|
||||
<a class="sDWEFrcVubKuUVGggeBOYqLlgYgPbojOc scale-down " aria-hidden="true" tabindex="-1"
|
||||
href="https://www.linkedin.com/company/health-insurance/" data-test-app-aware-link="">
|
||||
|
||||
<div class="ivm-image-view-model ">
|
||||
|
||||
<div class="ivm-view-attr__img-wrapper
|
||||
|
||||
">
|
||||
<!---->
|
||||
<!----> <img width="48"
|
||||
src="https://media.licdn.com/dms/image/v2/C560BAQEXIoLSJbShlw/company-logo_100_100/company-logo_100_100/0/1662748332921/health_insurance_logo?e=1753920000&v=beta&t=p2ZNMYNsC9KSlp-sIqMYuc88avBTjKF4CqDobq1Xr2M"
|
||||
loading="lazy" height="48" alt="Health Insurance" id="ember28"
|
||||
class="ivm-view-attr__img--centered EntityPhoto-square-3 evi-image lazy-image ember-view">
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</a>
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div
|
||||
class="BNxZPngZfeRnDrIUbICgBZvQjRvMAUnwCHuDrmRg yNRlrJOHDflDBnYPLbVmiAkUsCUZKUznmAc pt3 pb3 t-12 t-black--light">
|
||||
<div class="mb1">
|
||||
|
||||
<div class="t-roman t-sans">
|
||||
|
||||
|
||||
|
||||
<div class="display-flex">
|
||||
<span
|
||||
class="kmApjJVnFerynwITxTBSCqzqgoHwVfkiA HHGiVqODTCkszDUDWwPGPJGUPfAeRpygAKwwLePrQ ">
|
||||
<span class="OjTMoZLoiuspGuWWptwqxZRcMcHZBoSDxfig
|
||||
t-16">
|
||||
<a class="sDWEFrcVubKuUVGggeBOYqLlgYgPbojOc "
|
||||
href="https://www.linkedin.com/company/health-insurance/"
|
||||
data-test-app-aware-link="">
|
||||
<!---->Health Insurance<!---->
|
||||
<!----> </a>
|
||||
<!----> </span>
|
||||
</span>
|
||||
<!---->
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<div class="kFTZPhxHBbvnnRxiRPmTxafKGLUNSiaeInag
|
||||
t-14 t-black t-normal">
|
||||
<!---->Insurance ⢠Cardiff, CA<!---->
|
||||
</div>
|
||||
|
||||
<div class="FlWUwyrEUZpkVCgzGTDwUHTLntfZNseavlY
|
||||
t-14 t-normal">
|
||||
<!---->3K followers<!---->
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<!---->
|
||||
<p class="JBUEKeXhPyClEtYwdsASPYsZsCkTvUBqsDUs
|
||||
entity-result__summary--2-lines
|
||||
t-12 t-black--light
|
||||
">
|
||||
<!---->Your<span class="white-space-pre"> </span><strong><!---->health<!----></strong><span
|
||||
class="white-space-pre"> </span><!----><!----><strong><!---->insurance<!----></strong><span
|
||||
class="white-space-pre"> </span>expert for all stages of your life; Medicare, Individuals,
|
||||
Families, Small Groups, CoveredCA.<!---->
|
||||
</p>
|
||||
|
||||
<!---->
|
||||
</div>
|
||||
<div class="JZcKRppsWfaxfMaqtvfVwEeAtzNwryBOMdo yNRlrJOHDflDBnYPLbVmiAkUsCUZKUznmAc">
|
||||
<!---->
|
||||
|
||||
|
||||
<div>
|
||||
|
||||
|
||||
|
||||
|
||||
<button aria-label="Follow Health Insurance" id="ember49"
|
||||
class="artdeco-button artdeco-button--2 artdeco-button--secondary ember-view"
|
||||
type="button"><!---->
|
||||
<span class="artdeco-button__text">
|
||||
Follow
|
||||
</span></button>
|
||||
|
||||
|
||||
|
||||
<!---->
|
||||
<!---->
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
@@ -0,0 +1,93 @@
|
||||
<li class="grid grid__col--lg-8 block org-people-profile-card__profile-card-spacing">
|
||||
<div>
|
||||
|
||||
|
||||
<section class="artdeco-card full-width IxXiAcHfbZpayHVZUYdQwfYOkMbOirmr">
|
||||
<!---->
|
||||
|
||||
<img width="210" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
|
||||
ariarole="presentation" loading="lazy" height="210" alt="" id="ember102"
|
||||
class="evi-image lazy-image ghost-default ember-view org-people-profile-card__cover-photo org-people-profile-card__cover-photo--people">
|
||||
|
||||
<div class="org-people-profile-card__profile-info">
|
||||
<div id="ember103"
|
||||
class="artdeco-entity-lockup artdeco-entity-lockup--stacked-center artdeco-entity-lockup--size-7 ember-view">
|
||||
<div id="ember104"
|
||||
class="artdeco-entity-lockup__image artdeco-entity-lockup__image--type-circle ember-view"
|
||||
type="circle">
|
||||
|
||||
<a class="sDWEFrcVubKuUVGggeBOYqLlgYgPbojOc " id="org-people-profile-card__profile-image-0"
|
||||
href="https://www.linkedin.com/in/ericweberhcbd?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAABVh2MBFoyTaAxDqYQQcW8oGxVsqsKioHw"
|
||||
data-test-app-aware-link="">
|
||||
<img width="104"
|
||||
src="https://media.licdn.com/dms/image/v2/C4D03AQHNP9KoXtSrkg/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1573501774845?e=1753920000&v=beta&t=JYsY56biGUmDzbYj2ORZMcd1dSm2IRWCA-IM3KNFLw8"
|
||||
loading="lazy" height="104" alt="Eric Weber" id="ember105"
|
||||
class="evi-image lazy-image ember-view">
|
||||
</a>
|
||||
|
||||
|
||||
</div>
|
||||
<div id="ember106" class="artdeco-entity-lockup__content ember-view">
|
||||
<div id="ember107" class="artdeco-entity-lockup__title ember-view">
|
||||
<a class="sDWEFrcVubKuUVGggeBOYqLlgYgPbojOc link-without-visited-state"
|
||||
aria-label="View Eric Weberâs profile"
|
||||
href="https://www.linkedin.com/in/ericweberhcbd?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAABVh2MBFoyTaAxDqYQQcW8oGxVsqsKioHw"
|
||||
data-test-app-aware-link="">
|
||||
<div id="ember109" class="ember-view lt-line-clamp lt-line-clamp--single-line rMKrzkehlCEvJWoQjDQJFaHmBFAYQLMGrNY
|
||||
t-black" style="">
|
||||
Eric Weber
|
||||
|
||||
<!---->
|
||||
</div>
|
||||
|
||||
</a>
|
||||
|
||||
</div>
|
||||
<div id="ember110" class="artdeco-entity-lockup__badge ember-view"> <span class="a11y-text">3rd+
|
||||
degree connection</span>
|
||||
<span class="artdeco-entity-lockup__degree" aria-hidden="true">
|
||||
· 3rd
|
||||
</span>
|
||||
<!----><!---->
|
||||
</div>
|
||||
<div id="ember111" class="artdeco-entity-lockup__subtitle ember-view">
|
||||
<div class="t-14 t-black--light t-normal">
|
||||
<div id="ember113" class="ember-view lt-line-clamp lt-line-clamp--multi-line"
|
||||
style="-webkit-line-clamp: 2">
|
||||
HIPN Executive Editor | Healthcare BizDev CEO â Health Insurance Plan News.
|
||||
|
||||
<!---->
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div id="ember114" class="artdeco-entity-lockup__caption ember-view"></div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<span class="text-align-center">
|
||||
<span id="ember116"
|
||||
class="ember-view lt-line-clamp lt-line-clamp--multi-line t-12 t-black--light mt2"
|
||||
style="-webkit-line-clamp: 3">
|
||||
10K followers
|
||||
|
||||
<!----> </span>
|
||||
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<footer class="ph3 pb3">
|
||||
<button aria-label="Follow Eric Weber" id="ember117"
|
||||
class="artdeco-button artdeco-button--2 artdeco-button--secondary ember-view full-width"
|
||||
type="button"><!---->
|
||||
<span class="artdeco-button__text">
|
||||
Follow
|
||||
</span></button>
|
||||
</footer>
|
||||
|
||||
</section>
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
</li>
|
||||
@@ -0,0 +1,50 @@
|
||||
// ==== File: ai.js ====
|
||||
|
||||
class ApiHandler {
|
||||
constructor(apiKey = null) {
|
||||
this.apiKey = apiKey || localStorage.getItem("openai_api_key") || "";
|
||||
console.log("ApiHandler ready");
|
||||
}
|
||||
|
||||
setApiKey(k) {
|
||||
this.apiKey = k.trim();
|
||||
if (this.apiKey) localStorage.setItem("openai_api_key", this.apiKey);
|
||||
}
|
||||
|
||||
async *chatStream(messages, {model = "gpt-4o", temperature = 0.7} = {}) {
|
||||
if (!this.apiKey) throw new Error("OpenAI API key missing");
|
||||
const payload = {model, messages, stream: true, max_tokens: 1024};
|
||||
const controller = new AbortController();
|
||||
|
||||
const res = await fetch("https://api.openai.com/v1/chat/completions", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
},
|
||||
body: JSON.stringify(payload),
|
||||
signal: controller.signal,
|
||||
});
|
||||
if (!res.ok) throw new Error(`OpenAI: ${res.statusText}`);
|
||||
const reader = res.body.getReader();
|
||||
const dec = new TextDecoder();
|
||||
|
||||
let buf = "";
|
||||
while (true) {
|
||||
const {done, value} = await reader.read();
|
||||
if (done) break;
|
||||
buf += dec.decode(value, {stream: true});
|
||||
for (const line of buf.split("\n")) {
|
||||
if (!line.startsWith("data: ")) continue;
|
||||
if (line.includes("[DONE]")) return;
|
||||
const json = JSON.parse(line.slice(6));
|
||||
const delta = json.choices?.[0]?.delta?.content;
|
||||
if (delta) yield delta;
|
||||
}
|
||||
buf = buf.endsWith("\n") ? "" : buf; // keep partial line
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
window.API = new ApiHandler();
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
@@ -0,0 +1,64 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 500">
|
||||
<!-- Background -->
|
||||
<rect width="800" height="500" fill="#1a1a1a"/>
|
||||
|
||||
<!-- Opportunities Section -->
|
||||
<g transform="translate(50,50)">
|
||||
<!-- Opportunity 1 Box -->
|
||||
<rect x="0" y="0" width="300" height="150" rx="10" fill="#1a2d3d" stroke="#64b5f6" stroke-width="2"/>
|
||||
<text x="150" y="30" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#64b5f6">Data Capitalization Opportunity</text>
|
||||
<text x="150" y="60" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
|
||||
<tspan x="150" dy="0">Transform digital footprints into assets</tspan>
|
||||
<tspan x="150" dy="20">Personal data as capital</tspan>
|
||||
<tspan x="150" dy="20">Enterprise knowledge valuation</tspan>
|
||||
<tspan x="150" dy="20">New form of wealth creation</tspan>
|
||||
</text>
|
||||
|
||||
<!-- Opportunity 2 Box -->
|
||||
<rect x="0" y="200" width="300" height="150" rx="10" fill="#1a2d1a" stroke="#81c784" stroke-width="2"/>
|
||||
<text x="150" y="230" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#81c784">Authentic Data Potential</text>
|
||||
<text x="150" y="260" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">
|
||||
<tspan x="150" dy="0">Vast reservoir of real insights</tspan>
|
||||
<tspan x="150" dy="20">Enhanced AI development</tspan>
|
||||
<tspan x="150" dy="20">Diverse human knowledge</tspan>
|
||||
<tspan x="150" dy="20">Willing participation model</tspan>
|
||||
</text>
|
||||
</g>
|
||||
|
||||
<!-- Development Pathway -->
|
||||
<g transform="translate(450,50)">
|
||||
<!-- Step 1 Box -->
|
||||
<rect x="0" y="0" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
|
||||
<text x="150" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">1. Open-Source Foundation</text>
|
||||
<text x="150" y="65" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Data extraction engine & community development</text>
|
||||
|
||||
<!-- Step 2 Box -->
|
||||
<rect x="0" y="125" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
|
||||
<text x="150" y="160" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">2. Data Capitalization Platform</text>
|
||||
<text x="150" y="190" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Tools to structure & value digital assets</text>
|
||||
|
||||
<!-- Step 3 Box -->
|
||||
<rect x="0" y="250" width="300" height="100" rx="10" fill="#2d1a2d" stroke="#ce93d8" stroke-width="2"/>
|
||||
<text x="150" y="285" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ce93d8">3. Shared Data Marketplace</text>
|
||||
<text x="150" y="315" text-anchor="middle" font-family="Arial" font-size="12" fill="#e0e0e0">Economic platform for data exchange</text>
|
||||
</g>
|
||||
|
||||
<!-- Connecting Arrows -->
|
||||
<g transform="translate(400,125)">
|
||||
<path d="M-20,0 L40,0" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
|
||||
<path d="M-20,200 L40,200" stroke="#666" stroke-width="2" marker-end="url(#arrowhead)"/>
|
||||
</g>
|
||||
|
||||
<!-- Arrow Marker -->
|
||||
<defs>
|
||||
<marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
|
||||
<polygon points="0 0, 10 3.5, 0 7" fill="#666"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- Vision Box at Bottom -->
|
||||
<g transform="translate(200,420)">
|
||||
<rect x="0" y="0" width="400" height="60" rx="10" fill="#2d2613" stroke="#ffd54f" stroke-width="2"/>
|
||||
<text x="200" y="35" text-anchor="middle" font-family="Arial" font-weight="bold" font-size="16" fill="#ffd54f">Economic Vision: Shared Data Economy</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.8 KiB |
@@ -0,0 +1,25 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="120" height="35" viewBox="0 0 120 35">
|
||||
<!-- Dark Theme -->
|
||||
<g>
|
||||
<defs>
|
||||
<pattern id="halftoneDark" width="4" height="4" patternUnits="userSpaceOnUse">
|
||||
<circle cx="2" cy="2" r="1" fill="#eee" opacity="0.1"/>
|
||||
</pattern>
|
||||
<pattern id="halftoneTextDark" width="3" height="3" patternUnits="userSpaceOnUse">
|
||||
<circle cx="1.5" cy="1.5" r="2" fill="#aaa" opacity="0.2"/>
|
||||
</pattern>
|
||||
</defs>
|
||||
<!-- White border - added as outer rectangle -->
|
||||
<rect width="120" height="35" rx="5" fill="#111"/>
|
||||
<!-- Dark background slightly smaller to show thicker border -->
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="#1a1a1a"/>
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="url(#halftoneDark)"/>
|
||||
|
||||
<!-- Logo with halftone -->
|
||||
<path d="M30 17.5 a7.5 7.5 0 1 1 -15 0 a7.5 7.5 0 1 1 15 0" fill="none" stroke="#eee" stroke-width="2"/>
|
||||
<path d="M18 17.5 L27 17.5" stroke="#eee" stroke-width="2"/>
|
||||
<circle cx="22.5" cy="17.5" r="2" fill="#eee"/>
|
||||
|
||||
<text x="40" y="23" fill="#eee" font-family="Arial, sans-serif" font-weight="500" font-size="14">Crawl4AI</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.2 KiB |
@@ -0,0 +1,64 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="120" height="35" viewBox="0 0 120 35">
|
||||
<g>
|
||||
<defs>
|
||||
<pattern id="cyberdots" width="4" height="4" patternUnits="userSpaceOnUse">
|
||||
<circle cx="2" cy="2" r="1">
|
||||
<animate attributeName="fill"
|
||||
values="#FF2EC4;#8B5CF6;#0BC5EA;#FF2EC4"
|
||||
dur="6s"
|
||||
repeatCount="indefinite"/>
|
||||
<animate attributeName="opacity"
|
||||
values="0.2;0.4;0.2"
|
||||
dur="4s"
|
||||
repeatCount="indefinite"/>
|
||||
</circle>
|
||||
</pattern>
|
||||
<filter id="neonGlow" x="-20%" y="-20%" width="140%" height="140%">
|
||||
<feGaussianBlur stdDeviation="1" result="blur"/>
|
||||
<feFlood flood-color="#FF2EC4" flood-opacity="0.2">
|
||||
<animate attributeName="flood-color"
|
||||
values="#FF2EC4;#8B5CF6;#0BC5EA;#FF2EC4"
|
||||
dur="8s"
|
||||
repeatCount="indefinite"/>
|
||||
</feFlood>
|
||||
<feComposite in2="blur" operator="in"/>
|
||||
<feMerge>
|
||||
<feMergeNode/>
|
||||
<feMergeNode in="SourceGraphic"/>
|
||||
</feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
|
||||
<rect width="120" height="35" rx="5" fill="#0A0A0F"/>
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="#16161E"/>
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="url(#cyberdots)"/>
|
||||
|
||||
<!-- Logo with animated neon -->
|
||||
<path d="M30 17.5 a7.5 7.5 0 1 1 -15 0 a7.5 7.5 0 1 1 15 0" fill="none" stroke="#8B5CF6" stroke-width="2" filter="url(#neonGlow)">
|
||||
<animate attributeName="stroke"
|
||||
values="#FF2EC4;#8B5CF6;#0BC5EA;#FF2EC4"
|
||||
dur="8s"
|
||||
repeatCount="indefinite"/>
|
||||
</path>
|
||||
<path d="M18 17.5 L27 17.5" stroke="#8B5CF6" stroke-width="2" filter="url(#neonGlow)">
|
||||
<animate attributeName="stroke"
|
||||
values="#FF2EC4;#8B5CF6;#0BC5EA;#FF2EC4"
|
||||
dur="8s"
|
||||
repeatCount="indefinite"/>
|
||||
</path>
|
||||
<circle cx="22.5" cy="17.5" r="2" fill="#0BC5EA">
|
||||
<animate attributeName="fill"
|
||||
values="#0BC5EA;#FF2EC4;#8B5CF6;#0BC5EA"
|
||||
dur="8s"
|
||||
repeatCount="indefinite"/>
|
||||
</circle>
|
||||
|
||||
<text x="40" y="23" font-family="Arial, sans-serif" font-weight="500" font-size="14" filter="url(#neonGlow)">
|
||||
<animate attributeName="fill"
|
||||
values="#FF2EC4;#8B5CF6;#0BC5EA;#FF2EC4"
|
||||
dur="8s"
|
||||
repeatCount="indefinite"/>
|
||||
Crawl4AI
|
||||
</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.5 KiB |
@@ -0,0 +1,21 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="120" height="35" viewBox="0 0 120 35">
|
||||
<g>
|
||||
<defs>
|
||||
<pattern id="halftoneLight" width="4" height="4" patternUnits="userSpaceOnUse">
|
||||
<circle cx="2" cy="2" r="1" fill="#111" opacity="0.1"/>
|
||||
</pattern>
|
||||
</defs>
|
||||
<!-- Dark border -->
|
||||
<rect width="120" height="35" rx="5" fill="#DDD"/>
|
||||
<!-- Light background -->
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="#fff"/>
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="url(#halftoneLight)"/>
|
||||
|
||||
<!-- Logo -->
|
||||
<path d="M30 17.5 a7.5 7.5 0 1 1 -15 0 a7.5 7.5 0 1 1 15 0" fill="none" stroke="#111" stroke-width="2"/>
|
||||
<path d="M18 17.5 L27 17.5" stroke="#111" stroke-width="2"/>
|
||||
<circle cx="22.5" cy="17.5" r="2" fill="#111"/>
|
||||
|
||||
<text x="40" y="23" fill="#111" font-family="Arial, sans-serif" font-weight="500" font-size="14">Crawl4AI</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 925 B |
@@ -0,0 +1,28 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="120" height="35" viewBox="0 0 120 35">
|
||||
<g>
|
||||
<defs>
|
||||
<pattern id="halftoneDark" width="4" height="4" patternUnits="userSpaceOnUse">
|
||||
<circle cx="2" cy="2" r="1" fill="#8B5CF6" opacity="0.1"/>
|
||||
</pattern>
|
||||
<filter id="neonGlow" x="-20%" y="-20%" width="140%" height="140%">
|
||||
<feGaussianBlur stdDeviation="1" result="blur"/>
|
||||
<feFlood flood-color="#8B5CF6" flood-opacity="0.2"/>
|
||||
<feComposite in2="blur" operator="in"/>
|
||||
<feMerge>
|
||||
<feMergeNode/>
|
||||
<feMergeNode in="SourceGraphic"/>
|
||||
</feMerge>
|
||||
</filter>
|
||||
</defs>
|
||||
<rect width="120" height="35" rx="5" fill="#0A0A0F"/>
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="#16161E"/>
|
||||
<rect x="2" y="2" width="116" height="31" rx="4" fill="url(#halftoneDark)"/>
|
||||
|
||||
<!-- Logo with neon glow -->
|
||||
<path d="M30 17.5 a7.5 7.5 0 1 1 -15 0 a7.5 7.5 0 1 1 15 0" fill="none" stroke="#8B5CF6" stroke-width="2" filter="url(#neonGlow)"/>
|
||||
<path d="M18 17.5 L27 17.5" stroke="#8B5CF6" stroke-width="2" filter="url(#neonGlow)"/>
|
||||
<circle cx="22.5" cy="17.5" r="2" fill="#8B5CF6"/>
|
||||
|
||||
<text x="40" y="23" fill="#fff" font-family="Arial, sans-serif" font-weight="500" font-size="14" filter="url(#neonGlow)">Crawl4AI</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.3 KiB |
@@ -0,0 +1,25 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" width="580" height="126">
|
||||
<path d="M0 0 C12.1179958 -0.85875561 22.12023468 2.20651341 31.375 10 C33.9588012 14.77009452 33.87070291 19.70535807 33 25 C30.77052562 30.98269025 27.15351506 35.59987457 22.9375 40.3125 C22.40833984 40.91900391 21.87917969 41.52550781 21.33398438 42.15039062 C18.96447825 44.85287628 16.55997075 47.47583963 14 50 C15.45042909 53.34584964 17.27738825 55.50054036 19.87890625 58.0390625 C20.63494141 58.78414062 21.39097656 59.52921875 22.16992188 60.296875 C23.35424805 61.44929688 23.35424805 61.44929688 24.5625 62.625 C26.12847891 64.15663169 27.69238939 65.69038195 29.25390625 67.2265625 C29.94814697 67.90283691 30.6423877 68.57911133 31.35766602 69.27587891 C34.45581673 72.52831541 34.34790986 75.39098231 34.265625 79.7109375 C33.85513193 83.24842192 32.23104931 85.37661393 30 88 C29.25105469 88.928125 28.50210938 89.85625 27.73046875 90.8125 C25.50640826 93.55842665 23.2611955 96.28463276 21 99 C17.79614904 97.58912451 15.66752174 95.9074692 13.21289062 93.42407227 C12.48533569 92.69313858 11.75778076 91.9622049 11.00817871 91.2091217 C10.23204224 90.4195256 9.45590576 89.6299295 8.65625 88.81640625 C7.85014282 88.0050975 7.04403564 87.19378876 6.21350098 86.3578949 C3.63754598 83.76412665 1.068366 81.16378144 -1.5 78.5625 C-4.06920944 75.96565173 -6.63986464 73.37027798 -9.21350098 70.77781677 C-10.81451003 69.16457674 -12.41283817 67.5486707 -14.00817871 65.92982483 C-14.73573364 65.19573898 -15.46328857 64.46165314 -16.21289062 63.70532227 C-16.84924438 63.06071548 -17.48559814 62.4161087 -18.14123535 61.75196838 C-22.03088131 58.085804 -24.61197596 56.61227546 -30 56 C-30.07227711 63.58830253 -30.07227711 63.58830253 -26.4375 69.9375 C-24.82588941 71.04727981 -24.82588941 71.04727981 -23 72 C-22.28199219 72.38929687 -21.56398437 72.77859375 -20.82421875 73.1796875 C-18.90661636 74.16744007 -16.9555406 75.08966213 -15 76 C-16.63963796 81.94615694 -19.62375462 87.03522078 -22.625 92.375 C-23.140625 93.30183594 -23.65625 94.22867188 -24.1875 95.18359375 C-25.45382811 97.45829425 -26.72475925 99.73028792 -28 102 C-28.66 102 -29.32 102 -30 102 C-30.82177734 100.60974609 -30.82177734 100.60974609 -31.66015625 99.19140625 C-34.44251387 95.39643767 -37.12592087 94.33254662 -41.4375 92.9375 C-46.66232548 91.20616648 -50.57943739 89.50277451 -54 85 C-55.64091654 77.93009362 -54.79786553 71.36338683 -51 65.1875 C-48.45285245 61.62461876 -45.62399459 58.55902389 -42.5625 55.4375 C-39.69642145 52.30220076 -38.35441699 50.14733571 -38 45.875 C-38.3468606 41.79465803 -39.50438354 39.98760383 -42.5625 37.3125 C-44.04011809 36.20626186 -45.51932214 35.10213934 -47 34 C-48.35675038 32.71172113 -49.69183763 31.40008934 -51 30.0625 C-51.5775 29.47597656 -52.155 28.88945313 -52.75 28.28515625 C-53.1625 27.86105469 -53.575 27.43695313 -54 27 C-51.97191687 21.52163591 -48.89585901 17.05542502 -45.625 12.25 C-45.0475 11.37472656 -44.47 10.49945313 -43.875 9.59765625 C-43.02808594 8.35048828 -43.02808594 8.35048828 -42.1640625 7.078125 C-41.65311035 6.31918945 -41.1421582 5.56025391 -40.61572266 4.77832031 C-38.7788438 2.75658782 -37.66673296 2.36269302 -35 2 C-34.10539062 2.96679688 -33.21078125 3.93359375 -32.2890625 4.9296875 C-25.32821084 12.43185297 -18.31333171 19.83832724 -11 27 C-7.22933534 18.04050033 -3.59082425 9.03258884 0 0 Z M-10 28 C-4.72898586 34.41069287 1.09128152 40.18590512 7 46 C9.2477975 43.94766315 10.0022242 42.99332739 11 40 C11.16677282 36.41656024 11.16677282 36.41656024 11 33 C10.01732141 31.79438816 10.01732141 31.79438816 8.15234375 31.90234375 C6.1015625 31.93489583 4.05078125 31.96744792 2 32 C2 30.68 2 29.36 2 28 C-1.96 28 -5.92 28 -10 28 Z " fill="#8154F1" transform="translate(54,24)"/>
|
||||
<path d="M0 0 C3.38476564 2.35606236 6.21063206 4.96784375 9 8 C9.55300781 8.60070313 10.10601562 9.20140625 10.67578125 9.8203125 C16.15254582 17.44150038 14.41778301 28.08515337 14 37 C13.34 37 12.68 37 12 37 C11.814375 37.66 11.62875 38.32 11.4375 39 C8.08008874 46.00677133 2.89061193 50.96935134 -4.1875 54.15234375 C-13.42096561 56.93520769 -19.45572708 55.27213646 -28 51 C-28.33 58.26 -28.66 65.52 -29 73 C-34.28 73 -39.56 73 -45 73 C-45.02255209 63.36886982 -45.0409157 53.73775705 -45.05181217 44.10660648 C-45.05704074 39.63494451 -45.06413789 35.16330147 -45.07543945 30.69165039 C-45.08626929 26.37988027 -45.09227704 22.06812841 -45.09487724 17.75634575 C-45.09673132 16.10763702 -45.1003516 14.45892934 -45.10573006 12.81022835 C-45.11294178 10.50896128 -45.11399089 8.2077706 -45.11352539 5.90649414 C-45.115746 4.5941127 -45.11796661 3.28173126 -45.12025452 1.92958069 C-45 -1 -45 -1 -44 -2 C-41.64686445 -2.07271741 -39.29166332 -2.08370868 -36.9375 -2.0625 C-35.64714844 -2.05347656 -34.35679687 -2.04445312 -33.02734375 -2.03515625 C-32.02832031 -2.02355469 -31.02929688 -2.01195312 -30 -2 C-29.67 -0.68 -29.34 0.64 -29 2 C-28.484375 1.54625 -27.96875 1.0925 -27.4375 0.625 C-19.32020499 -4.78653001 -8.47872855 -4.35631225 0 0 Z M-24 14 C-28.17753855 18.397409 -29.31526252 21.86049191 -29.21484375 27.94140625 C-28.76016001 32.29810316 -26.74299239 35.144433 -23.9375 38.375 C-19.62339005 40.76152891 -15.82145828 40.74948951 -11 40 C-6.94423079 38.15646854 -5.40268437 36.64949093 -3.0625 32.875 C-1.20382923 27.69965925 -1.58282678 22.08108641 -3.875 17.09765625 C-6.14022321 13.88072468 -8.71339602 12.48099465 -12.375 11.4375 C-17.01182731 10.68141169 -19.94013876 11.53224121 -24 14 Z " fill="#000000" transform="translate(324,51)"/>
|
||||
<path d="M0 0 C5.28 0 10.56 0 16 0 C16.33 7.92 16.66 15.84 17 24 C19.64 22.68 22.28 21.36 25 20 C30.79515734 18.8354668 36.95136189 19.94208504 42 23 C47.9048799 27.95173 49.76651132 32.563617 51 40 C51.2310069 42.92494721 51.23616441 45.82135668 51.1953125 48.75390625 C51.1924826 49.54022934 51.18965271 50.32655243 51.18673706 51.13670349 C51.17567289 53.61213359 51.15059591 56.08718199 51.125 58.5625 C51.11495234 60.2558541 51.10582965 61.94921395 51.09765625 63.64257812 C51.07749276 67.76197886 51.03832028 71.88056963 51 76 C45.72 76 40.44 76 35 76 C34.97421875 73.53144531 34.9484375 71.06289062 34.921875 68.51953125 C34.88342532 66.11992147 34.83827477 63.72052445 34.79248047 61.32104492 C34.76431351 59.65773737 34.74307767 57.99429704 34.72900391 56.33081055 C34.70751408 53.93469475 34.66083712 51.54016816 34.609375 49.14453125 C34.60814636 48.4047554 34.60691772 47.66497955 34.60565186 46.90278625 C34.46797124 42.01276895 33.54228573 39.39650767 30 36 C26.14448235 34.65895038 24.50052951 34.80167699 20.6875 36.3125 C17.92091219 37.72511938 17.92091219 37.72511938 17 40 C16.84323225 42.69655446 16.74163685 45.37129977 16.68359375 48.0703125 C16.66281265 48.86654022 16.64203156 49.66276794 16.62062073 50.48312378 C16.55515851 53.03032685 16.49625585 55.57763425 16.4375 58.125 C16.3943116 59.85027127 16.35069721 61.57553193 16.30664062 63.30078125 C16.19867755 67.53374872 16.09957993 71.76679371 16 76 C10.72 76 5.44 76 0 76 C0 50.92 0 25.84 0 0 Z " fill="#000000" transform="translate(344,28)"/>
|
||||
<path d="M0 0 C4.596825 3.51710564 7.4389019 8.37920571 9.25 13.8125 C9.25 18.7625 9.25 23.7125 9.25 28.8125 C-3.29 28.8125 -15.83 28.8125 -28.75 28.8125 C-25.6660386 35.49712981 -25.6660386 35.49712981 -22.4375 37.25 C-17.89993673 38.19972255 -14.27077596 38.59057142 -10.0625 36.4375 C-9.299375 35.90125 -8.53625 35.365 -7.75 34.8125 C-5.96484375 33.5625 -5.96484375 33.5625 -3.75 32.8125 C-0.953125 33.7109375 -0.953125 33.7109375 2 35.1875 C2.98484375 35.66960937 3.9696875 36.15171875 4.984375 36.6484375 C6.10585938 37.22464844 6.10585938 37.22464844 7.25 37.8125 C4.96980879 44.03877553 1.20244001 47.83627999 -4.75 50.8125 C-5.74 50.8125 -6.73 50.8125 -7.75 50.8125 C-7.75 51.4725 -7.75 52.1325 -7.75 52.8125 C-18.86485662 53.23938639 -27.92116401 52.9319381 -36.890625 45.828125 C-43.28550316 38.89586212 -46.03948779 30.01177861 -45.75 20.8125 C-45.76804688 20.09578125 -45.78609375 19.3790625 -45.8046875 18.640625 C-45.62359436 11.34515857 -41.87362618 6.75194354 -36.75 1.8125 C-25.3156908 -6.74590719 -12.23266757 -7.83738009 0 0 Z M-26.75 11.8125 C-27.41 13.4625 -28.07 15.1125 -28.75 16.8125 C-21.82 16.8125 -14.89 16.8125 -7.75 16.8125 C-7.75 15.4925 -7.75 14.1725 -7.75 12.8125 C-8.41 12.8125 -9.07 12.8125 -9.75 12.8125 C-10.183125 11.8534375 -10.183125 11.8534375 -10.625 10.875 C-11.57823451 8.66043921 -11.57823451 8.66043921 -13.75 7.8125 C-19.54730856 7.33731077 -22.6050575 7.30182728 -26.75 11.8125 Z " fill="#000000" transform="translate(263.75,53.1875)"/>
|
||||
<path d="M0 0 C4.32910275 3.64646978 7.66772623 8.20398511 8.1743927 13.99232483 C8.23104843 16.41010116 8.22470999 18.82001506 8.1953125 21.23828125 C8.1924826 22.11904831 8.18965271 22.99981537 8.18673706 23.90727234 C8.17557833 26.70915545 8.15048171 29.51071416 8.125 32.3125 C8.11496463 34.21679225 8.10583975 36.12108952 8.09765625 38.02539062 C8.07565309 42.68371797 8.04115294 47.34180406 8 52 C3.05 52 -1.9 52 -7 52 C-7.33 51.01 -7.66 50.02 -8 49 C-8.495 49.45375 -8.99 49.9075 -9.5 50.375 C-15.29801973 54.14371282 -23.22880965 55.2016773 -30 54 C-34.3928156 51.97306811 -37.80456866 49.30410692 -41.0625 45.75 C-42.91931383 40.3033461 -42.71169624 35.02271562 -40.8125 29.625 C-36.94789201 24.02798153 -31.57537025 20.73422782 -24.9375 19.37109375 C-19.57363295 18.49946535 -14.44183784 17.89345799 -9 18 C-9.495 15.03 -9.495 15.03 -10 12 C-14.41501433 10.28805567 -16.10525832 9.7315775 -20.75 11.125 C-23.78019531 12.8731896 -25.25741527 14.02735546 -27 17 C-31.85963179 16.39254603 -35.7405518 14.27241033 -40 12 C-39.58188612 7.12200473 -37.63597431 5.15117773 -34 2 C-23.66058923 -5.86694298 -11.30746775 -6.07088535 0 0 Z M-19.5625 31.5 C-20.57441406 31.7784375 -21.58632812 32.056875 -22.62890625 32.34375 C-23.41136719 32.5603125 -24.19382813 32.776875 -25 33 C-25.33 34.98 -25.66 36.96 -26 39 C-23.68515576 41.1367793 -22.27316474 41.95843145 -19.125 42.4375 C-15.20927421 41.88929839 -13.04990083 40.46896734 -10 38 C-10 37.34 -10 36.68 -10 36 C-10.99 36 -11.98 36 -13 36 C-13 34.02 -13 32.04 -13 30 C-15.19848934 30 -17.46622194 30.9217164 -19.5625 31.5 Z " fill="#000000" transform="translate(181,52)"/>
|
||||
<path d="M0 0 C4.59650402 3.88149228 7.78723415 8.06356011 9 14 C9.08231525 15.34273815 9.11789309 16.68892787 9.11352539 18.03417969 C9.11344986 18.81468689 9.11337433 19.59519409 9.11329651 20.39935303 C9.10813522 21.23178528 9.10297394 22.06421753 9.09765625 22.921875 C9.0962413 23.78186096 9.09482635 24.64184692 9.09336853 25.52789307 C9.08778097 28.26863109 9.07523075 31.00928676 9.0625 33.75 C9.05748339 35.6106759 9.05292077 37.47135309 9.04882812 39.33203125 C9.03781561 43.88805246 9.02055951 48.44401228 9 53 C3.72 53 -1.56 53 -7 53 C-7.04898438 50.51984375 -7.09796875 48.0396875 -7.1484375 45.484375 C-7.2024716 43.07822721 -7.25976949 40.67217771 -7.31738281 38.26611328 C-7.35562857 36.59594059 -7.39048915 34.92568677 -7.421875 33.25537109 C-7.46762161 30.85224443 -7.52559925 28.44967416 -7.5859375 26.046875 C-7.59748871 25.30193787 -7.60903992 24.55700073 -7.62094116 23.78948975 C-7.73913631 18.56517651 -7.73913631 18.56517651 -10 14 C-13.83496597 12.3775144 -16.20258323 11.84954401 -20.3125 12.625 C-23.46078829 14.23575215 -23.84153717 14.73626008 -25 18 C-25.23582402 20.71215682 -25.37196504 23.33201064 -25.4140625 26.046875 C-25.43342865 26.81807678 -25.4527948 27.58927856 -25.4727478 28.3838501 C-25.53237077 30.83912205 -25.57885914 33.29444133 -25.625 35.75 C-25.66319575 37.41798882 -25.70224841 39.08595826 -25.7421875 40.75390625 C-25.83966639 44.83583474 -25.91936979 48.91756492 -26 53 C-31.28 53 -36.56 53 -42 53 C-42 34.85 -42 16.7 -42 -2 C-37.05 -2 -32.1 -2 -27 -2 C-26.67 -0.68 -26.34 0.64 -26 2 C-25.0925 1.360625 -24.185 0.72125 -23.25 0.0625 C-15.90966364 -4.59579038 -7.55358672 -4.0685628 0 0 Z " fill="#8154F3" transform="translate(459,51)"/>
|
||||
<path d="M0 0 C4.95 0 9.9 0 15 0 C15.03738281 2.46855469 15.07476562 4.93710937 15.11328125 7.48046875 C15.15953688 9.88005261 15.21075571 12.27948297 15.26245117 14.67895508 C15.29563655 16.34227108 15.32368834 18.00569796 15.34643555 19.66918945 C15.38009999 22.06509499 15.4324194 24.45999035 15.48828125 26.85546875 C15.49467117 27.5952446 15.5010611 28.33502045 15.50764465 29.09721375 C15.63329971 33.54308862 16.2548976 36.46144732 19 40 C21.24512262 41.12256131 22.51456649 41.13413762 25 41.125 C25.70125 41.12757812 26.4025 41.13015625 27.125 41.1328125 C29.09153775 41.11377236 29.09153775 41.11377236 31 40 C31.65213292 37.97536745 31.65213292 37.97536745 32 36 C32.66 36 33.32 36 34 36 C33.99476318 34.94498291 33.98952637 33.88996582 33.98413086 32.80297852 C33.96608099 28.9024337 33.95456805 25.0018926 33.94506836 21.10131836 C33.9400281 19.41055684 33.93319595 17.71979971 33.92456055 16.02905273 C33.91248143 13.60399293 33.90676877 11.17899056 33.90234375 8.75390625 C33.89718246 7.99348526 33.89202118 7.23306427 33.88670349 6.44960022 C33.88650762 4.6326743 33.93802723 2.81586873 34 1 C35 0 35 0 38.125 -0.1328125 C39.40375 -0.13023438 40.6825 -0.12765625 42 -0.125 C43.27875 -0.12757812 44.5575 -0.13015625 45.875 -0.1328125 C49 0 49 0 50 1 C50.09353163 2.68801985 50.11744791 4.37995207 50.11352539 6.07055664 C50.11344986 7.15160172 50.11337433 8.23264679 50.11329651 9.34645081 C50.10813522 10.5224736 50.10297394 11.6984964 50.09765625 12.91015625 C50.0962413 14.10732773 50.09482635 15.30449921 50.09336853 16.53794861 C50.08774979 20.37949066 50.07519411 24.22097543 50.0625 28.0625 C50.05748741 30.65950437 50.05292409 33.25650965 50.04882812 35.85351562 C50.03777427 42.23569932 50.02049708 48.61783993 50 55 C45.05 55 40.1 55 35 55 C34.67 54.01 34.34 53.02 34 52 C33.236875 52.639375 32.47375 53.27875 31.6875 53.9375 C26.66981088 57.30651984 20.17775664 57.46594672 14.3359375 56.609375 C11.52354881 55.87570838 9.41796099 54.58894579 7 53 C6.67 53.66 6.34 54.32 6 55 C5.34 55 4.68 55 4 55 C2.74335022 52.09399739 2 50.20395416 2 47 C1.67402832 46.31268799 1.34805664 45.62537598 1.01220703 44.91723633 C-0.44721623 40.71109824 -0.24640517 36.74368469 -0.1953125 32.32421875 C-0.1924826 31.3990596 -0.18965271 30.47390045 -0.18673706 29.52070618 C-0.17557392 26.57617058 -0.15047637 23.63194312 -0.125 20.6875 C-0.11496521 18.68685336 -0.10584023 16.68620194 -0.09765625 14.68554688 C-0.07564722 9.7902465 -0.04114391 4.89517543 0 0 Z " fill="#8053EF" transform="translate(474,49)"/>
|
||||
<path d="M0 0 C0.33 0.66 0.66 1.32 1 2 C5.95 2 10.9 2 16 2 C16 27.08 16 52.16 16 78 C10.72 78 5.44 78 0 78 C0 52.26 0 26.52 0 0 Z " fill="#8255F3" transform="translate(532,26)"/>
|
||||
<path d="M0 0 C5.28 0 10.56 0 16 0 C16 25.08 16 50.16 16 76 C10.72 76 5.44 76 0 76 C0 50.92 0 25.84 0 0 Z " fill="#000000" transform="translate(197,28)"/>
|
||||
<path d="M0 0 C4.95 0 9.9 0 15 0 C16.66165207 1.66165207 16.11959725 3.6651603 16.12442017 5.88401794 C16.12082489 6.58172806 16.11722961 7.27943817 16.11352539 7.99829102 C16.11367142 8.72995483 16.11381744 9.46161865 16.1139679 10.2154541 C16.11326867 12.64496784 16.10547585 15.07440583 16.09765625 17.50390625 C16.0957921 19.18399514 16.09436819 20.86408457 16.09336853 22.54417419 C16.08954307 26.97473643 16.07971256 31.4052661 16.06866455 35.83581543 C16.05845103 40.35342976 16.05387104 44.87104934 16.04882812 49.38867188 C16.03809031 58.25913123 16.02049329 67.12955781 16 76 C10.72 76 5.44 76 0 76 C0 50.92 0 25.84 0 0 Z " fill="#8354F4" transform="translate(556,28)"/>
|
||||
<path d="M0 0 C1.32 1.32 2.64 2.64 4 4 C2.35 4 0.7 4 -1 4 C-1 7.3 -1 10.6 -1 14 C-3 12 -3 12 -3.3125 8.6875 C-2.98867205 4.86633018 -2.22414284 3.05819641 0 0 Z " fill="#000400" transform="translate(25,76)"/>
|
||||
<path d="M0 0 C2.02045442 0.60183749 4.02111039 1.27306096 6 2 C6.33 2.66 6.66 3.32 7 4 C4.69 4 2.38 4 0 4 C0 2.68 0 1.36 0 0 Z " fill="#000100" transform="translate(56,52)"/>
|
||||
<path d="M0 0 C1.65 0 3.3 0 5 0 C5 0.33 5 0.66 5 1 C3.35 1 1.7 1 0 1 C0 0.67 0 0.34 0 0 Z " fill="#000000" transform="translate(272,123)"/>
|
||||
<path d="M0 0 C2 1 2 1 2 1 Z " fill="#000000" transform="translate(481,104)"/>
|
||||
<path d="M0 0 C2 1 2 1 2 1 Z " fill="#000000" transform="translate(144,104)"/>
|
||||
<path d="" fill="#000000" transform="translate(0,0)"/>
|
||||
<path d="" fill="#000100" transform="translate(0,0)"/>
|
||||
<path d="M0 0 C2 1 2 1 2 1 Z " fill="#000100" transform="translate(70,24)"/>
|
||||
<path d="" fill="#000000" transform="translate(0,0)"/>
|
||||
<path d="" fill="#000000" transform="translate(0,0)"/>
|
||||
<path d="" fill="#000100" transform="translate(0,0)"/>
|
||||
<path d="" fill="#000000" transform="translate(0,0)"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 16 KiB |
@@ -0,0 +1,25 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" width="580" height="126">
|
||||
<path d="M0 0 C12.1179958 -0.85875561 22.12023468 2.20651341 31.375 10 C33.9588012 14.77009452 33.87070291 19.70535807 33 25 C30.77052562 30.98269025 27.15351506 35.59987457 22.9375 40.3125 C22.40833984 40.91900391 21.87917969 41.52550781 21.33398438 42.15039062 C18.96447825 44.85287628 16.55997075 47.47583963 14 50 C15.45042909 53.34584964 17.27738825 55.50054036 19.87890625 58.0390625 C20.63494141 58.78414062 21.39097656 59.52921875 22.16992188 60.296875 C23.35424805 61.44929688 23.35424805 61.44929688 24.5625 62.625 C26.12847891 64.15663169 27.69238939 65.69038195 29.25390625 67.2265625 C29.94814697 67.90283691 30.6423877 68.57911133 31.35766602 69.27587891 C34.45581673 72.52831541 34.34790986 75.39098231 34.265625 79.7109375 C33.85513193 83.24842192 32.23104931 85.37661393 30 88 C29.25105469 88.928125 28.50210938 89.85625 27.73046875 90.8125 C25.50640826 93.55842665 23.2611955 96.28463276 21 99 C17.79614904 97.58912451 15.66752174 95.9074692 13.21289062 93.42407227 C12.48533569 92.69313858 11.75778076 91.9622049 11.00817871 91.2091217 C10.23204224 90.4195256 9.45590576 89.6299295 8.65625 88.81640625 C7.85014282 88.0050975 7.04403564 87.19378876 6.21350098 86.3578949 C3.63754598 83.76412665 1.068366 81.16378144 -1.5 78.5625 C-4.06920944 75.96565173 -6.63986464 73.37027798 -9.21350098 70.77781677 C-10.81451003 69.16457674 -12.41283817 67.5486707 -14.00817871 65.92982483 C-14.73573364 65.19573898 -15.46328857 64.46165314 -16.21289062 63.70532227 C-16.84924438 63.06071548 -17.48559814 62.4161087 -18.14123535 61.75196838 C-22.03088131 58.085804 -24.61197596 56.61227546 -30 56 C-30.07227711 63.58830253 -30.07227711 63.58830253 -26.4375 69.9375 C-24.82588941 71.04727981 -24.82588941 71.04727981 -23 72 C-22.28199219 72.38929687 -21.56398437 72.77859375 -20.82421875 73.1796875 C-18.90661636 74.16744007 -16.9555406 75.08966213 -15 76 C-16.63963796 81.94615694 -19.62375462 87.03522078 -22.625 92.375 C-23.140625 93.30183594 -23.65625 94.22867188 -24.1875 95.18359375 C-25.45382811 97.45829425 -26.72475925 99.73028792 -28 102 C-28.66 102 -29.32 102 -30 102 C-30.82177734 100.60974609 -30.82177734 100.60974609 -31.66015625 99.19140625 C-34.44251387 95.39643767 -37.12592087 94.33254662 -41.4375 92.9375 C-46.66232548 91.20616648 -50.57943739 89.50277451 -54 85 C-55.64091654 77.93009362 -54.79786553 71.36338683 -51 65.1875 C-48.45285245 61.62461876 -45.62399459 58.55902389 -42.5625 55.4375 C-39.69642145 52.30220076 -38.35441699 50.14733571 -38 45.875 C-38.3468606 41.79465803 -39.50438354 39.98760383 -42.5625 37.3125 C-44.04011809 36.20626186 -45.51932214 35.10213934 -47 34 C-48.35675038 32.71172113 -49.69183763 31.40008934 -51 30.0625 C-51.5775 29.47597656 -52.155 28.88945313 -52.75 28.28515625 C-53.1625 27.86105469 -53.575 27.43695313 -54 27 C-51.97191687 21.52163591 -48.89585901 17.05542502 -45.625 12.25 C-45.0475 11.37472656 -44.47 10.49945313 -43.875 9.59765625 C-43.02808594 8.35048828 -43.02808594 8.35048828 -42.1640625 7.078125 C-41.65311035 6.31918945 -41.1421582 5.56025391 -40.61572266 4.77832031 C-38.7788438 2.75658782 -37.66673296 2.36269302 -35 2 C-34.10539062 2.96679688 -33.21078125 3.93359375 -32.2890625 4.9296875 C-25.32821084 12.43185297 -18.31333171 19.83832724 -11 27 C-7.22933534 18.04050033 -3.59082425 9.03258884 0 0 Z M-10 28 C-4.72898586 34.41069287 1.09128152 40.18590512 7 46 C9.2477975 43.94766315 10.0022242 42.99332739 11 40 C11.16677282 36.41656024 11.16677282 36.41656024 11 33 C10.01732141 31.79438816 10.01732141 31.79438816 8.15234375 31.90234375 C6.1015625 31.93489583 4.05078125 31.96744792 2 32 C2 30.68 2 29.36 2 28 C-1.96 28 -5.92 28 -10 28 Z " fill="#8154F1" transform="translate(54,24)"/>
|
||||
<path d="M0 0 C3.38476564 2.35606236 6.21063206 4.96784375 9 8 C9.55300781 8.60070313 10.10601562 9.20140625 10.67578125 9.8203125 C16.15254582 17.44150038 14.41778301 28.08515337 14 37 C13.34 37 12.68 37 12 37 C11.814375 37.66 11.62875 38.32 11.4375 39 C8.08008874 46.00677133 2.89061193 50.96935134 -4.1875 54.15234375 C-13.42096561 56.93520769 -19.45572708 55.27213646 -28 51 C-28.33 58.26 -28.66 65.52 -29 73 C-34.28 73 -39.56 73 -45 73 C-45.02255209 63.36886982 -45.0409157 53.73775705 -45.05181217 44.10660648 C-45.05704074 39.63494451 -45.06413789 35.16330147 -45.07543945 30.69165039 C-45.08626929 26.37988027 -45.09227704 22.06812841 -45.09487724 17.75634575 C-45.09673132 16.10763702 -45.1003516 14.45892934 -45.10573006 12.81022835 C-45.11294178 10.50896128 -45.11399089 8.2077706 -45.11352539 5.90649414 C-45.115746 4.5941127 -45.11796661 3.28173126 -45.12025452 1.92958069 C-45 -1 -45 -1 -44 -2 C-41.64686445 -2.07271741 -39.29166332 -2.08370868 -36.9375 -2.0625 C-35.64714844 -2.05347656 -34.35679687 -2.04445312 -33.02734375 -2.03515625 C-32.02832031 -2.02355469 -31.02929688 -2.01195312 -30 -2 C-29.67 -0.68 -29.34 0.64 -29 2 C-28.484375 1.54625 -27.96875 1.0925 -27.4375 0.625 C-19.32020499 -4.78653001 -8.47872855 -4.35631225 0 0 Z M-24 14 C-28.17753855 18.397409 -29.31526252 21.86049191 -29.21484375 27.94140625 C-28.76016001 32.29810316 -26.74299239 35.144433 -23.9375 38.375 C-19.62339005 40.76152891 -15.82145828 40.74948951 -11 40 C-6.94423079 38.15646854 -5.40268437 36.64949093 -3.0625 32.875 C-1.20382923 27.69965925 -1.58282678 22.08108641 -3.875 17.09765625 C-6.14022321 13.88072468 -8.71339602 12.48099465 -12.375 11.4375 C-17.01182731 10.68141169 -19.94013876 11.53224121 -24 14 Z " fill="#ffffff" transform="translate(324,51)"/>
|
||||
<path d="M0 0 C5.28 0 10.56 0 16 0 C16.33 7.92 16.66 15.84 17 24 C19.64 22.68 22.28 21.36 25 20 C30.79515734 18.8354668 36.95136189 19.94208504 42 23 C47.9048799 27.95173 49.76651132 32.563617 51 40 C51.2310069 42.92494721 51.23616441 45.82135668 51.1953125 48.75390625 C51.1924826 49.54022934 51.18965271 50.32655243 51.18673706 51.13670349 C51.17567289 53.61213359 51.15059591 56.08718199 51.125 58.5625 C51.11495234 60.2558541 51.10582965 61.94921395 51.09765625 63.64257812 C51.07749276 67.76197886 51.03832028 71.88056963 51 76 C45.72 76 40.44 76 35 76 C34.97421875 73.53144531 34.9484375 71.06289062 34.921875 68.51953125 C34.88342532 66.11992147 34.83827477 63.72052445 34.79248047 61.32104492 C34.76431351 59.65773737 34.74307767 57.99429704 34.72900391 56.33081055 C34.70751408 53.93469475 34.66083712 51.54016816 34.609375 49.14453125 C34.60814636 48.4047554 34.60691772 47.66497955 34.60565186 46.90278625 C34.46797124 42.01276895 33.54228573 39.39650767 30 36 C26.14448235 34.65895038 24.50052951 34.80167699 20.6875 36.3125 C17.92091219 37.72511938 17.92091219 37.72511938 17 40 C16.84323225 42.69655446 16.74163685 45.37129977 16.68359375 48.0703125 C16.66281265 48.86654022 16.64203156 49.66276794 16.62062073 50.48312378 C16.55515851 53.03032685 16.49625585 55.57763425 16.4375 58.125 C16.3943116 59.85027127 16.35069721 61.57553193 16.30664062 63.30078125 C16.19867755 67.53374872 16.09957993 71.76679371 16 76 C10.72 76 5.44 76 0 76 C0 50.92 0 25.84 0 0 Z " fill="#ffffff" transform="translate(344,28)"/>
|
||||
<path d="M0 0 C4.596825 3.51710564 7.4389019 8.37920571 9.25 13.8125 C9.25 18.7625 9.25 23.7125 9.25 28.8125 C-3.29 28.8125 -15.83 28.8125 -28.75 28.8125 C-25.6660386 35.49712981 -25.6660386 35.49712981 -22.4375 37.25 C-17.89993673 38.19972255 -14.27077596 38.59057142 -10.0625 36.4375 C-9.299375 35.90125 -8.53625 35.365 -7.75 34.8125 C-5.96484375 33.5625 -5.96484375 33.5625 -3.75 32.8125 C-0.953125 33.7109375 -0.953125 33.7109375 2 35.1875 C2.98484375 35.66960937 3.9696875 36.15171875 4.984375 36.6484375 C6.10585938 37.22464844 6.10585938 37.22464844 7.25 37.8125 C4.96980879 44.03877553 1.20244001 47.83627999 -4.75 50.8125 C-5.74 50.8125 -6.73 50.8125 -7.75 50.8125 C-7.75 51.4725 -7.75 52.1325 -7.75 52.8125 C-18.86485662 53.23938639 -27.92116401 52.9319381 -36.890625 45.828125 C-43.28550316 38.89586212 -46.03948779 30.01177861 -45.75 20.8125 C-45.76804688 20.09578125 -45.78609375 19.3790625 -45.8046875 18.640625 C-45.62359436 11.34515857 -41.87362618 6.75194354 -36.75 1.8125 C-25.3156908 -6.74590719 -12.23266757 -7.83738009 0 0 Z M-26.75 11.8125 C-27.41 13.4625 -28.07 15.1125 -28.75 16.8125 C-21.82 16.8125 -14.89 16.8125 -7.75 16.8125 C-7.75 15.4925 -7.75 14.1725 -7.75 12.8125 C-8.41 12.8125 -9.07 12.8125 -9.75 12.8125 C-10.183125 11.8534375 -10.183125 11.8534375 -10.625 10.875 C-11.57823451 8.66043921 -11.57823451 8.66043921 -13.75 7.8125 C-19.54730856 7.33731077 -22.6050575 7.30182728 -26.75 11.8125 Z " fill="#ffffff" transform="translate(263.75,53.1875)"/>
|
||||
<path d="M0 0 C4.32910275 3.64646978 7.66772623 8.20398511 8.1743927 13.99232483 C8.23104843 16.41010116 8.22470999 18.82001506 8.1953125 21.23828125 C8.1924826 22.11904831 8.18965271 22.99981537 8.18673706 23.90727234 C8.17557833 26.70915545 8.15048171 29.51071416 8.125 32.3125 C8.11496463 34.21679225 8.10583975 36.12108952 8.09765625 38.02539062 C8.07565309 42.68371797 8.04115294 47.34180406 8 52 C3.05 52 -1.9 52 -7 52 C-7.33 51.01 -7.66 50.02 -8 49 C-8.495 49.45375 -8.99 49.9075 -9.5 50.375 C-15.29801973 54.14371282 -23.22880965 55.2016773 -30 54 C-34.3928156 51.97306811 -37.80456866 49.30410692 -41.0625 45.75 C-42.91931383 40.3033461 -42.71169624 35.02271562 -40.8125 29.625 C-36.94789201 24.02798153 -31.57537025 20.73422782 -24.9375 19.37109375 C-19.57363295 18.49946535 -14.44183784 17.89345799 -9 18 C-9.495 15.03 -9.495 15.03 -10 12 C-14.41501433 10.28805567 -16.10525832 9.7315775 -20.75 11.125 C-23.78019531 12.8731896 -25.25741527 14.02735546 -27 17 C-31.85963179 16.39254603 -35.7405518 14.27241033 -40 12 C-39.58188612 7.12200473 -37.63597431 5.15117773 -34 2 C-23.66058923 -5.86694298 -11.30746775 -6.07088535 0 0 Z M-19.5625 31.5 C-20.57441406 31.7784375 -21.58632812 32.056875 -22.62890625 32.34375 C-23.41136719 32.5603125 -24.19382813 32.776875 -25 33 C-25.33 34.98 -25.66 36.96 -26 39 C-23.68515576 41.1367793 -22.27316474 41.95843145 -19.125 42.4375 C-15.20927421 41.88929839 -13.04990083 40.46896734 -10 38 C-10 37.34 -10 36.68 -10 36 C-10.99 36 -11.98 36 -13 36 C-13 34.02 -13 32.04 -13 30 C-15.19848934 30 -17.46622194 30.9217164 -19.5625 31.5 Z " fill="#ffffff" transform="translate(181,52)"/>
|
||||
<path d="M0 0 C4.59650402 3.88149228 7.78723415 8.06356011 9 14 C9.08231525 15.34273815 9.11789309 16.68892787 9.11352539 18.03417969 C9.11344986 18.81468689 9.11337433 19.59519409 9.11329651 20.39935303 C9.10813522 21.23178528 9.10297394 22.06421753 9.09765625 22.921875 C9.0962413 23.78186096 9.09482635 24.64184692 9.09336853 25.52789307 C9.08778097 28.26863109 9.07523075 31.00928676 9.0625 33.75 C9.05748339 35.6106759 9.05292077 37.47135309 9.04882812 39.33203125 C9.03781561 43.88805246 9.02055951 48.44401228 9 53 C3.72 53 -1.56 53 -7 53 C-7.04898438 50.51984375 -7.09796875 48.0396875 -7.1484375 45.484375 C-7.2024716 43.07822721 -7.25976949 40.67217771 -7.31738281 38.26611328 C-7.35562857 36.59594059 -7.39048915 34.92568677 -7.421875 33.25537109 C-7.46762161 30.85224443 -7.52559925 28.44967416 -7.5859375 26.046875 C-7.59748871 25.30193787 -7.60903992 24.55700073 -7.62094116 23.78948975 C-7.73913631 18.56517651 -7.73913631 18.56517651 -10 14 C-13.83496597 12.3775144 -16.20258323 11.84954401 -20.3125 12.625 C-23.46078829 14.23575215 -23.84153717 14.73626008 -25 18 C-25.23582402 20.71215682 -25.37196504 23.33201064 -25.4140625 26.046875 C-25.43342865 26.81807678 -25.4527948 27.58927856 -25.4727478 28.3838501 C-25.53237077 30.83912205 -25.57885914 33.29444133 -25.625 35.75 C-25.66319575 37.41798882 -25.70224841 39.08595826 -25.7421875 40.75390625 C-25.83966639 44.83583474 -25.91936979 48.91756492 -26 53 C-31.28 53 -36.56 53 -42 53 C-42 34.85 -42 16.7 -42 -2 C-37.05 -2 -32.1 -2 -27 -2 C-26.67 -0.68 -26.34 0.64 -26 2 C-25.0925 1.360625 -24.185 0.72125 -23.25 0.0625 C-15.90966364 -4.59579038 -7.55358672 -4.0685628 0 0 Z " fill="#8154F3" transform="translate(459,51)"/>
|
||||
<path d="M0 0 C4.95 0 9.9 0 15 0 C15.03738281 2.46855469 15.07476562 4.93710937 15.11328125 7.48046875 C15.15953688 9.88005261 15.21075571 12.27948297 15.26245117 14.67895508 C15.29563655 16.34227108 15.32368834 18.00569796 15.34643555 19.66918945 C15.38009999 22.06509499 15.4324194 24.45999035 15.48828125 26.85546875 C15.49467117 27.5952446 15.5010611 28.33502045 15.50764465 29.09721375 C15.63329971 33.54308862 16.2548976 36.46144732 19 40 C21.24512262 41.12256131 22.51456649 41.13413762 25 41.125 C25.70125 41.12757812 26.4025 41.13015625 27.125 41.1328125 C29.09153775 41.11377236 29.09153775 41.11377236 31 40 C31.65213292 37.97536745 31.65213292 37.97536745 32 36 C32.66 36 33.32 36 34 36 C33.99476318 34.94498291 33.98952637 33.88996582 33.98413086 32.80297852 C33.96608099 28.9024337 33.95456805 25.0018926 33.94506836 21.10131836 C33.9400281 19.41055684 33.93319595 17.71979971 33.92456055 16.02905273 C33.91248143 13.60399293 33.90676877 11.17899056 33.90234375 8.75390625 C33.89718246 7.99348526 33.89202118 7.23306427 33.88670349 6.44960022 C33.88650762 4.6326743 33.93802723 2.81586873 34 1 C35 0 35 0 38.125 -0.1328125 C39.40375 -0.13023438 40.6825 -0.12765625 42 -0.125 C43.27875 -0.12757812 44.5575 -0.13015625 45.875 -0.1328125 C49 0 49 0 50 1 C50.09353163 2.68801985 50.11744791 4.37995207 50.11352539 6.07055664 C50.11344986 7.15160172 50.11337433 8.23264679 50.11329651 9.34645081 C50.10813522 10.5224736 50.10297394 11.6984964 50.09765625 12.91015625 C50.0962413 14.10732773 50.09482635 15.30449921 50.09336853 16.53794861 C50.08774979 20.37949066 50.07519411 24.22097543 50.0625 28.0625 C50.05748741 30.65950437 50.05292409 33.25650965 50.04882812 35.85351562 C50.03777427 42.23569932 50.02049708 48.61783993 50 55 C45.05 55 40.1 55 35 55 C34.67 54.01 34.34 53.02 34 52 C33.236875 52.639375 32.47375 53.27875 31.6875 53.9375 C26.66981088 57.30651984 20.17775664 57.46594672 14.3359375 56.609375 C11.52354881 55.87570838 9.41796099 54.58894579 7 53 C6.67 53.66 6.34 54.32 6 55 C5.34 55 4.68 55 4 55 C2.74335022 52.09399739 2 50.20395416 2 47 C1.67402832 46.31268799 1.34805664 45.62537598 1.01220703 44.91723633 C-0.44721623 40.71109824 -0.24640517 36.74368469 -0.1953125 32.32421875 C-0.1924826 31.3990596 -0.18965271 30.47390045 -0.18673706 29.52070618 C-0.17557392 26.57617058 -0.15047637 23.63194312 -0.125 20.6875 C-0.11496521 18.68685336 -0.10584023 16.68620194 -0.09765625 14.68554688 C-0.07564722 9.7902465 -0.04114391 4.89517543 0 0 Z " fill="#8053EF" transform="translate(474,49)"/>
|
||||
<path d="M0 0 C0.33 0.66 0.66 1.32 1 2 C5.95 2 10.9 2 16 2 C16 27.08 16 52.16 16 78 C10.72 78 5.44 78 0 78 C0 52.26 0 26.52 0 0 Z " fill="#8255F3" transform="translate(532,26)"/>
|
||||
<path d="M0 0 C5.28 0 10.56 0 16 0 C16 25.08 16 50.16 16 76 C10.72 76 5.44 76 0 76 C0 50.92 0 25.84 0 0 Z " fill="#ffffff" transform="translate(197,28)"/>
|
||||
<path d="M0 0 C4.95 0 9.9 0 15 0 C16.66165207 1.66165207 16.11959725 3.6651603 16.12442017 5.88401794 C16.12082489 6.58172806 16.11722961 7.27943817 16.11352539 7.99829102 C16.11367142 8.72995483 16.11381744 9.46161865 16.1139679 10.2154541 C16.11326867 12.64496784 16.10547585 15.07440583 16.09765625 17.50390625 C16.0957921 19.18399514 16.09436819 20.86408457 16.09336853 22.54417419 C16.08954307 26.97473643 16.07971256 31.4052661 16.06866455 35.83581543 C16.05845103 40.35342976 16.05387104 44.87104934 16.04882812 49.38867188 C16.03809031 58.25913123 16.02049329 67.12955781 16 76 C10.72 76 5.44 76 0 76 C0 50.92 0 25.84 0 0 Z " fill="#8354F4" transform="translate(556,28)"/>
|
||||
<path d="M0 0 C1.32 1.32 2.64 2.64 4 4 C2.35 4 0.7 4 -1 4 C-1 7.3 -1 10.6 -1 14 C-3 12 -3 12 -3.3125 8.6875 C-2.98867205 4.86633018 -2.22414284 3.05819641 0 0 Z " fill="#ffffff" transform="translate(25,76)"/>
|
||||
<path d="M0 0 C2.02045442 0.60183749 4.02111039 1.27306096 6 2 C6.33 2.66 6.66 3.32 7 4 C4.69 4 2.38 4 0 4 C0 2.68 0 1.36 0 0 Z " fill="#ffffff" transform="translate(56,52)"/>
|
||||
<path d="M0 0 C1.65 0 3.3 0 5 0 C5 0.33 5 0.66 5 1 C3.35 1 1.7 1 0 1 C0 0.67 0 0.34 0 0 Z " fill="#ffffff" transform="translate(272,123)"/>
|
||||
<path d="M0 0 C2 1 2 1 2 1 Z " fill="#ffffff" transform="translate(481,104)"/>
|
||||
<path d="M0 0 C2 1 2 1 2 1 Z " fill="#ffffff" transform="translate(144,104)"/>
|
||||
<path d="" fill="#ffffff" transform="translate(0,0)"/>
|
||||
<path d="" fill="#ffffff" transform="translate(0,0)"/>
|
||||
<path d="M0 0 C2 1 2 1 2 1 Z " fill="#ffffff" transform="translate(70,24)"/>
|
||||
<path d="" fill="#ffffff" transform="translate(0,0)"/>
|
||||
<path d="" fill="#ffffff" transform="translate(0,0)"/>
|
||||
<path d="" fill="#ffffff" transform="translate(0,0)"/>
|
||||
<path d="" fill="#ffffff" transform="translate(0,0)"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 16 KiB |
@@ -0,0 +1,10 @@
|
||||
<svg width="290" height="79" viewBox="0 0 290 79" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M9.29035 49.4468L28.7567 29.5561C32.3895 25.8441 38.3259 25.7973 42.0161 29.4517C42.0212 29.4567 42.0262 29.4617 42.0313 29.4667L56.5094 43.8693C56.6443 44.0035 56.8618 44.0023 56.9952 43.8666C57.1226 43.737 57.1281 43.5302 57.0077 43.394L51.0902 36.6964C50.4099 35.9264 50.3618 34.7813 50.9751 33.9563L53.6623 30.3414C55.3367 28.0889 58.5094 27.6283 60.7486 29.3126C61.0308 29.5249 61.2902 29.7662 61.5226 30.0326L78.5752 49.5857C79.7704 50.9562 79.6349 53.0418 78.2725 54.2441C77.6738 54.7724 76.9046 55.0637 76.1083 55.0637H11.6289C9.81652 55.0637 8.34727 53.5858 8.34727 51.7626C8.34727 50.8961 8.686 50.0643 9.29035 49.4468Z" fill="url(#paint0_linear_1393_1443)"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M96.6631 54.3223L101.635 24.6779H109.17L115.035 40.483L120.861 24.6779H128.396L133.367 54.3223H125.794L123.269 37.2591L116.355 54.3223H113.326L106.762 37.2591L104.237 54.3223H96.6631ZM153.922 43.2058L150.618 33.5714L147.315 43.2058H153.922ZM156.463 49.1719H145.508L143.747 54.3223H135.57L146.799 24.6779H155.172L166.401 54.3223H158.224L156.463 49.1719ZM186.317 32.305C185.473 31.6272 184.63 31.1254 183.786 30.7995C182.943 30.4736 182.126 30.3107 181.335 30.3107C180.333 30.3107 179.516 30.5453 178.883 31.0146C178.251 31.4838 177.934 32.0964 177.934 32.8525C177.934 33.3738 178.093 33.804 178.409 34.1429C178.725 34.4818 179.14 34.7751 179.654 35.0227C180.168 35.2704 180.748 35.4855 181.394 35.6679C182.04 35.8504 182.679 36.0459 183.312 36.2545C185.842 37.0887 187.694 38.2032 188.867 39.5979C190.04 40.9926 190.626 42.8109 190.626 45.0529C190.626 46.5649 190.369 47.9335 189.855 49.1588C189.341 50.3841 188.59 51.4333 187.601 52.3067C186.613 53.18 185.401 53.8578 183.964 54.3401C182.527 54.8223 180.9 55.0635 179.081 55.0635C175.312 55.0635 171.819 53.9555 168.604 51.7396L171.925 45.5612C173.085 46.5779 174.231 47.3339 175.365 47.8293C176.498 48.3246 177.618 48.5722 178.725 48.5722C179.99 48.5722 180.933 48.2855 181.552 47.7119C182.172 47.1384 182.481 46.4867 182.481 45.7568C182.481 45.3136 182.402 44.9291 182.244 44.6032C182.086 44.2773 181.822 43.9775 181.453 43.7038C181.084 43.4301 180.603 43.1759 180.01 42.9413C179.417 42.7066 178.699 42.446 177.855 42.1592C176.854 41.8464 175.872 41.5009 174.91 41.1229C173.948 40.7449 173.091 40.2431 172.34 39.6174C171.589 38.9918 170.982 38.2032 170.521 37.2517C170.06 36.3001 169.829 35.0944 169.829 33.6345C169.829 32.1747 170.073 30.8516 170.561 29.6655C171.048 28.4793 171.734 27.4626 172.617 26.6154C173.5 25.7681 174.58 25.1099 175.859 24.6406C177.137 24.1714 178.567 23.9368 180.149 23.9368C181.625 23.9368 183.167 24.1388 184.775 24.5429C186.382 24.947 187.924 25.54 189.4 26.3221L186.317 32.305ZM211.275 32.305C210.432 31.6272 209.588 31.1254 208.745 30.7995C207.902 30.4736 207.084 30.3107 206.294 30.3107C205.292 30.3107 204.475 30.5453 203.842 31.0146C203.21 31.4838 202.893 32.0964 202.893 32.8525C202.893 33.3738 203.052 33.804 203.368 34.1429C203.684 34.4818 204.099 34.7751 204.613 35.0227C205.127 35.2704 205.707 35.4855 206.353 35.6679C206.999 35.8504 207.638 36.0459 208.271 36.2545C210.801 37.0887 212.653 38.2032 213.826 39.5979C214.999 40.9926 215.585 42.8109 215.585 45.0529C215.585 46.5649 215.328 47.9335 214.814 49.1588C214.3 50.3841 213.549 51.4333 212.56 52.3067C211.572 53.18 210.359 53.8578 208.923 54.3401C207.486 54.8223 205.859 55.0635 204.04 55.0635C200.271 55.0635 196.778 53.9555 193.562 51.7396L196.884 45.5612C198.043 46.5779 199.19 47.3339 200.323 47.8293C201.457 48.3246 202.577 48.5722 203.684 48.5722C204.949 48.5722 205.892 48.2855 206.511 47.7119C207.131 47.1384 207.44 46.4867 207.44 45.7568C207.44 45.3136 207.361 44.9291 207.203 44.6032C207.045 44.2773 206.781 43.9775 206.412 43.7038C206.043 43.4301 205.562 43.1759 204.969 42.9413C204.376 42.7066 203.658 42.446 202.814 42.1592C201.813 41.8464 200.831 41.5009 199.869 41.1229C198.907 40.7449 198.05 40.2431 197.299 39.6174C196.548 38.9918 195.941 38.2032 195.48 37.2517C195.019 36.3001 194.788 35.0944 194.788 33.6345C194.788 32.1747 195.032 30.8516 195.52 29.6655C196.007 28.4793 196.693 27.4626 197.576 26.6154C198.459 25.7681 199.539 25.1099 200.818 24.6406C202.096 24.1714 203.526 23.9368 205.108 23.9368C206.584 23.9368 208.126 24.1388 209.733 24.5429C211.341 24.947 212.883 25.54 214.359 26.3221L211.275 32.305ZM227.33 24.6779V54.3223H219.99V24.6779H227.33ZM239.303 24.6779L246.397 42.724L253.53 24.6779H261.832L249.437 54.3223H243.357L231.001 24.6779H239.303ZM281.653 31.2044H272.483V36.1582H281.141V42.6847H272.483V47.7958H281.653V54.3223H264.769V24.6779H281.653V31.2044Z" fill="#1B222D"/>
|
||||
<defs>
|
||||
<linearGradient id="paint0_linear_1393_1443" x1="27.9096" y1="53.8722" x2="57.3312" y2="32.653" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0.288159" stop-color="#FF9964"/>
|
||||
<stop offset="1" stop-color="#FF6E63"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.9 KiB |
@@ -0,0 +1,10 @@
|
||||
<svg width="290" height="79" viewBox="0 0 290 79" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M8.82332 49.1503L28.2896 29.2597C31.9225 25.5476 37.8589 25.5009 41.5491 29.1552C41.5541 29.1602 41.5592 29.1652 41.5642 29.1702L56.0423 43.5728C56.1772 43.707 56.3948 43.7058 56.5282 43.5701C56.6556 43.4405 56.661 43.2337 56.5407 43.0975L50.6232 36.4C49.9429 35.63 49.8948 34.4849 50.508 33.6598L53.1952 30.0449C54.8697 27.7924 58.0423 27.3318 60.2816 29.0162C60.5638 29.2284 60.8232 29.4698 61.0555 29.7362L78.1082 49.2892C79.3034 50.6597 79.1679 52.7454 77.8054 53.9477C77.2068 54.476 76.4376 54.7673 75.6413 54.7673H11.1619C9.34949 54.7673 7.88025 53.2893 7.88025 51.4662C7.88025 50.5996 8.21898 49.7679 8.82332 49.1503Z" fill="url(#paint0_linear_1401_462)"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M96.196 54.0261L101.168 24.3817H108.703L114.568 40.1868L120.394 24.3817H127.929L132.9 54.0261H125.326L122.802 36.9629L115.888 54.0261H112.859L106.295 36.9629L103.77 54.0261H96.196ZM153.455 42.9095L150.151 33.275L146.848 42.9095H153.455ZM155.996 48.8757H145.041L143.28 54.0261H135.103L146.332 24.3817H154.705L165.934 54.0261H157.757L155.996 48.8757ZM185.849 32.0088C185.006 31.331 184.163 30.8292 183.319 30.5033C182.476 30.1774 181.658 30.0145 180.868 30.0145C179.866 30.0145 179.049 30.2491 178.416 30.7184C177.784 31.1876 177.467 31.8002 177.467 32.5563C177.467 33.0776 177.626 33.5078 177.942 33.8467C178.258 34.1856 178.673 34.4789 179.187 34.7265C179.701 34.9742 180.281 35.1893 180.927 35.3717C181.573 35.5542 182.212 35.7497 182.845 35.9583C185.375 36.7925 187.227 37.907 188.4 39.3017C189.573 40.6964 190.159 42.5147 190.159 44.7567C190.159 46.2687 189.902 47.6373 189.388 48.8626C188.874 50.0878 188.123 51.1371 187.134 52.0105C186.146 52.8838 184.934 53.5616 183.497 54.0439C182.06 54.5261 180.433 54.7673 178.614 54.7673C174.845 54.7673 171.352 53.6593 168.137 51.4434L171.458 45.265C172.617 46.2817 173.764 47.0377 174.897 47.5331C176.031 48.0284 177.151 48.276 178.258 48.276C179.523 48.276 180.466 47.9893 181.085 47.4157C181.705 46.8422 182.014 46.1905 182.014 45.4605C182.014 45.0174 181.935 44.6328 181.777 44.307C181.619 43.9811 181.355 43.6813 180.986 43.4076C180.617 43.1339 180.136 42.8797 179.543 42.6451C178.95 42.4104 178.232 42.1498 177.388 41.863C176.387 41.5502 175.405 41.2047 174.443 40.8267C173.481 40.4487 172.624 39.9469 171.873 39.3212C171.122 38.6956 170.515 37.907 170.054 36.9554C169.593 36.0039 169.362 34.7982 169.362 33.3383C169.362 31.8784 169.606 30.5554 170.094 29.3693C170.581 28.1831 171.267 27.1664 172.15 26.3192C173.033 25.4719 174.113 24.8137 175.392 24.3444C176.67 23.8752 178.1 23.6406 179.682 23.6406C181.158 23.6406 182.7 23.8426 184.308 24.2467C185.915 24.6508 187.457 25.2438 188.933 26.0259L185.849 32.0088ZM210.808 32.0088C209.965 31.331 209.121 30.8292 208.278 30.5033C207.434 30.1774 206.617 30.0145 205.826 30.0145C204.825 30.0145 204.008 30.2491 203.375 30.7184C202.743 31.1876 202.426 31.8002 202.426 32.5563C202.426 33.0776 202.584 33.5078 202.901 33.8467C203.217 34.1856 203.632 34.4789 204.146 34.7265C204.66 34.9742 205.24 35.1893 205.886 35.3717C206.532 35.5542 207.171 35.7497 207.803 35.9583C210.334 36.7925 212.185 37.907 213.358 39.3017C214.531 40.6964 215.118 42.5147 215.118 44.7567C215.118 46.2687 214.861 47.6373 214.347 48.8626C213.833 50.0878 213.082 51.1371 212.093 52.0105C211.105 52.8838 209.892 53.5616 208.456 54.0439C207.019 54.5261 205.392 54.7673 203.573 54.7673C199.804 54.7673 196.311 53.6593 193.095 51.4434L196.416 45.265C197.576 46.2817 198.723 47.0377 199.856 47.5331C200.99 48.0284 202.11 48.276 203.217 48.276C204.482 48.276 205.425 47.9893 206.044 47.4157C206.663 46.8422 206.973 46.1905 206.973 45.4605C206.973 45.0174 206.894 44.6328 206.736 44.307C206.578 43.9811 206.314 43.6813 205.945 43.4076C205.576 43.1339 205.095 42.8797 204.502 42.6451C203.909 42.4104 203.191 42.1498 202.347 41.863C201.346 41.5502 200.364 41.2047 199.402 40.8267C198.439 40.4487 197.583 39.9469 196.832 39.3212C196.08 38.6956 195.474 37.907 195.013 36.9554C194.552 36.0039 194.321 34.7982 194.321 33.3383C194.321 31.8784 194.565 30.5554 195.052 29.3693C195.54 28.1831 196.225 27.1664 197.108 26.3192C197.991 25.4719 199.072 24.8137 200.35 24.3444C201.629 23.8752 203.059 23.6406 204.64 23.6406C206.116 23.6406 207.658 23.8426 209.266 24.2467C210.874 24.6508 212.416 25.2438 213.892 26.0259L210.808 32.0088ZM226.863 24.3817V54.0261H219.522V24.3817H226.863ZM238.836 24.3817L245.93 42.4278L253.063 24.3817H261.365L248.97 54.0261H242.89L230.534 24.3817H238.836ZM281.186 30.9082H272.016V35.862H280.674V42.3885H272.016V47.4996H281.186V54.0261H264.302V24.3817H281.186V30.9082Z" fill="white"/>
|
||||
<defs>
|
||||
<linearGradient id="paint0_linear_1401_462" x1="27.4425" y1="53.5758" x2="56.8642" y2="32.3566" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0.288159" stop-color="#FF9964"/>
|
||||
<stop offset="1" stop-color="#FF6E63"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.9 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 8.3 KiB |
@@ -0,0 +1,18 @@
|
||||
<svg width="150" height="32" viewBox="0 0 150 32" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M1 25.305L2.17278 25.3059L20.2143 7.26444L20.213 6.09197L24.908 10.787L23.7353 10.7857L5.69392 28.8271L5.69499 30L1 25.305ZM17.2209 5.02285L14.2901 7.95366L12.1732 5.8368L7.61418 10.3958L9.73104 12.5127L6.80023 15.4435L2.77739 11.4204V9.37021L11.1477 1H13.1983L17.2209 5.02285ZM25.9771 13.7791L23.0463 16.7099L25.1632 18.8268L20.6042 23.3858L18.4873 21.269L15.5565 24.1998L19.5796 28.2226H21.6298L30 19.8523V17.8016L25.9771 13.7791Z" fill="#25B894"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M33.6071 8.00558H51.0965V11.5035H33.6071V8.00558Z" fill="#171717"/>
|
||||
<path d="M133.949 12.1593H124.767V15.6572H133.949V12.1593Z" fill="#171717"/>
|
||||
<path d="M44.5397 11.2849H40.1674V25.4951H44.5397V11.2849Z" fill="#171717"/>
|
||||
<path d="M56.3425 8.00558H52.4073V25.495H56.3425V8.00558Z" fill="#171717"/>
|
||||
<path d="M87.1673 12.1593H83.4508V25.495H87.1673V12.1593Z" fill="#171717"/>
|
||||
<path d="M107.937 8.00558H104.002V25.495H107.937V8.00558Z" fill="#171717"/>
|
||||
<path d="M130.237 8.00558V21.5599C130.237 21.6806 130.334 21.7785 130.455 21.7785H133.953V25.7136H130.455C128.161 25.7136 126.301 23.854 126.301 21.5599V8.00558H130.237Z" fill="#171717"/>
|
||||
<path d="M60.0603 12.0499C63.0184 12.0499 65.4164 14.448 65.4164 17.4061V25.4949H61.6999V17.4061C61.6999 16.5005 60.9658 15.7664 60.0603 15.7664H54.5948V12.0499H60.0603Z" fill="#171717"/>
|
||||
<path d="M77.0041 12.0499C79.5396 12.0499 81.5951 14.1054 81.5951 16.6409V21.0133C81.5951 23.5488 79.5396 25.6043 77.0041 25.6043H71.9759C69.4403 25.6043 67.3849 23.5488 67.3849 21.0133V16.6409C67.3849 14.1054 69.4403 12.0499 71.9759 12.0499H77.0041ZM77.0041 15.5478H71.9759C71.3722 15.5478 70.8828 16.0372 70.8828 16.6409V21.0133C70.8828 21.617 71.3722 22.1064 71.9759 22.1064H77.0041C77.6078 22.1064 78.0972 21.617 78.0972 21.0133V16.6409C78.0972 16.0372 77.6078 15.5478 77.0041 15.5478Z" fill="#171717"/>
|
||||
<path d="M93.8343 12.0499V15.5478H90.9922C88.8189 15.5478 87.0571 17.3096 87.0571 19.483V25.6043H83.5592V19.483C83.5592 15.3778 86.8871 12.0499 90.9922 12.0499H93.8343Z" fill="#171717"/>
|
||||
<path d="M104.44 12.1593V15.6572H99.4886C98.9518 15.6572 98.537 16.0602 98.537 16.5317V21.1227C98.537 21.5941 98.9518 21.9971 99.4886 21.9971H104.44V25.495H99.4886C97.0423 25.495 95.0391 23.549 95.0391 21.1227V16.5317C95.0391 14.1054 97.0423 12.1593 99.4886 12.1593H104.44Z" fill="#171717"/>
|
||||
<path d="M119.742 12.1593C121.674 12.1593 123.24 13.7254 123.24 15.6572V25.495H119.742V15.6572H113.839C113.718 15.6572 113.621 15.7551 113.621 15.8758H110.123C110.123 13.8233 111.787 12.1593 113.839 12.1593H119.742Z" fill="#171717"/>
|
||||
<path d="M145.322 12.1593C147.254 12.1593 148.82 13.7254 148.82 15.6572V25.495H145.322V15.6572H139.42C139.299 15.6572 139.201 15.7551 139.201 15.8758H135.703C135.703 13.8233 137.367 12.1593 139.42 12.1593H145.322Z" fill="#171717"/>
|
||||
<path d="M120.179 16.9688C121.869 16.9688 123.24 18.3391 123.24 20.0294V21.2318C123.24 23.5862 121.331 25.4949 118.977 25.4949H113.183C111.493 25.4949 110.123 24.1246 110.123 22.4342V20.0294C110.123 18.3391 111.493 16.9688 113.183 16.9688H120.179ZM119.742 20.4666H113.621V21.997H118.977C119.385 21.997 119.719 21.6768 119.741 21.2738L119.742 21.2318V20.4666Z" fill="#171717"/>
|
||||
<path d="M145.76 16.9688C147.45 16.9688 148.82 18.3391 148.82 20.0294V21.2318C148.82 23.5862 146.912 25.4949 144.557 25.4949H138.764C137.073 25.4949 135.703 24.1246 135.703 22.4342V20.0294C135.703 18.3391 137.073 16.9688 138.764 16.9688H145.76ZM145.322 20.4666H139.201V21.997H144.557C144.966 21.997 145.299 21.6768 145.321 21.2738L145.322 21.2318V20.4666Z" fill="#171717"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.6 KiB |
@@ -0,0 +1,18 @@
|
||||
<svg width="150" height="32" viewBox="0 0 150 32" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M1 25.305L2.17278 25.3059L20.2143 7.26444L20.213 6.09197L24.908 10.787L23.7353 10.7857L5.69392 28.8271L5.69499 30L1 25.305ZM17.2209 5.02285L14.2901 7.95366L12.1732 5.8368L7.61418 10.3958L9.73104 12.5127L6.80023 15.4435L2.77739 11.4204V9.37021L11.1477 1H13.1983L17.2209 5.02285ZM25.9771 13.7791L23.0463 16.7099L25.1632 18.8268L20.6042 23.3858L18.4873 21.269L15.5565 24.1998L19.5796 28.2226H21.6298L30 19.8523V17.8016L25.9771 13.7791Z" fill="#25B894"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M33.6071 8.00558H51.0965V11.5035H33.6071V8.00558Z" fill="#ffffff"/>
|
||||
<path d="M133.949 12.1593H124.767V15.6572H133.949V12.1593Z" fill="#ffffff"/>
|
||||
<path d="M44.5397 11.2849H40.1674V25.4951H44.5397V11.2849Z" fill="#ffffff"/>
|
||||
<path d="M56.3425 8.00558H52.4073V25.495H56.3425V8.00558Z" fill="#ffffff"/>
|
||||
<path d="M87.1673 12.1593H83.4508V25.495H87.1673V12.1593Z" fill="#ffffff"/>
|
||||
<path d="M107.937 8.00558H104.002V25.495H107.937V8.00558Z" fill="#ffffff"/>
|
||||
<path d="M130.237 8.00558V21.5599C130.237 21.6806 130.334 21.7785 130.455 21.7785H133.953V25.7136H130.455C128.161 25.7136 126.301 23.854 126.301 21.5599V8.00558H130.237Z" fill="#ffffff"/>
|
||||
<path d="M60.0603 12.0499C63.0184 12.0499 65.4164 14.448 65.4164 17.4061V25.4949H61.6999V17.4061C61.6999 16.5005 60.9658 15.7664 60.0603 15.7664H54.5948V12.0499H60.0603Z" fill="#ffffff"/>
|
||||
<path d="M77.0041 12.0499C79.5396 12.0499 81.5951 14.1054 81.5951 16.6409V21.0133C81.5951 23.5488 79.5396 25.6043 77.0041 25.6043H71.9759C69.4403 25.6043 67.3849 23.5488 67.3849 21.0133V16.6409C67.3849 14.1054 69.4403 12.0499 71.9759 12.0499H77.0041ZM77.0041 15.5478H71.9759C71.3722 15.5478 70.8828 16.0372 70.8828 16.6409V21.0133C70.8828 21.617 71.3722 22.1064 71.9759 22.1064H77.0041C77.6078 22.1064 78.0972 21.617 78.0972 21.0133V16.6409C78.0972 16.0372 77.6078 15.5478 77.0041 15.5478Z" fill="#ffffff"/>
|
||||
<path d="M93.8343 12.0499V15.5478H90.9922C88.8189 15.5478 87.0571 17.3096 87.0571 19.483V25.6043H83.5592V19.483C83.5592 15.3778 86.8871 12.0499 90.9922 12.0499H93.8343Z" fill="#ffffff"/>
|
||||
<path d="M104.44 12.1593V15.6572H99.4886C98.9518 15.6572 98.537 16.0602 98.537 16.5317V21.1227C98.537 21.5941 98.9518 21.9971 99.4886 21.9971H104.44V25.495H99.4886C97.0423 25.495 95.0391 23.549 95.0391 21.1227V16.5317C95.0391 14.1054 97.0423 12.1593 99.4886 12.1593H104.44Z" fill="#ffffff"/>
|
||||
<path d="M119.742 12.1593C121.674 12.1593 123.24 13.7254 123.24 15.6572V25.495H119.742V15.6572H113.839C113.718 15.6572 113.621 15.7551 113.621 15.8758H110.123C110.123 13.8233 111.787 12.1593 113.839 12.1593H119.742Z" fill="#ffffff"/>
|
||||
<path d="M145.322 12.1593C147.254 12.1593 148.82 13.7254 148.82 15.6572V25.495H145.322V15.6572H139.42C139.299 15.6572 139.201 15.7551 139.201 15.8758H135.703C135.703 13.8233 137.367 12.1593 139.42 12.1593H145.322Z" fill="#ffffff"/>
|
||||
<path d="M120.179 16.9688C121.869 16.9688 123.24 18.3391 123.24 20.0294V21.2318C123.24 23.5862 121.331 25.4949 118.977 25.4949H113.183C111.493 25.4949 110.123 24.1246 110.123 22.4342V20.0294C110.123 18.3391 111.493 16.9688 113.183 16.9688H120.179ZM119.742 20.4666H113.621V21.997H118.977C119.385 21.997 119.719 21.6768 119.741 21.2738L119.742 21.2318V20.4666Z" fill="#ffffff"/>
|
||||
<path d="M145.76 16.9688C147.45 16.9688 148.82 18.3391 148.82 20.0294V21.2318C148.82 23.5862 146.912 25.4949 144.557 25.4949H138.764C137.073 25.4949 135.703 24.1246 135.703 22.4342V20.0294C135.703 18.3391 137.073 16.9688 138.764 16.9688H145.76ZM145.322 20.4666H139.201V21.997H144.557C144.966 21.997 145.299 21.6768 145.321 21.2738L145.322 21.2318V20.4666Z" fill="#ffffff"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.6 KiB |
@@ -0,0 +1,343 @@
|
||||
# 🚀 Crawl4AI v0.7.0: The Adaptive Intelligence Update
|
||||
|
||||
*January 28, 2025 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.0—the Adaptive Intelligence Update. This release introduces fundamental improvements in how Crawl4AI handles modern web complexity through adaptive learning, intelligent content discovery, and advanced extraction capabilities.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Adaptive Crawling**: Your crawler now learns and adapts to website patterns
|
||||
- **Virtual Scroll Support**: Complete content extraction from infinite scroll pages
|
||||
- **Link Preview with Intelligent Scoring**: Intelligent link analysis and prioritization
|
||||
- **Async URL Seeder**: Discover thousands of URLs in seconds with intelligent filtering
|
||||
- **Performance Optimizations**: Significant speed and memory improvements
|
||||
|
||||
## 🧠 Adaptive Crawling: Intelligence Through Pattern Learning
|
||||
|
||||
**The Problem:** Websites change. Class names shift. IDs disappear. Your carefully crafted selectors break at 3 AM, and you wake up to empty datasets and angry stakeholders.
|
||||
|
||||
**My Solution:** I implemented an adaptive learning system that observes patterns, builds confidence scores, and adjusts extraction strategies on the fly. It's like having a junior developer who gets better at their job with every page they scrape.
|
||||
|
||||
### Technical Deep-Dive
|
||||
|
||||
The Adaptive Crawler maintains a persistent state for each domain, tracking:
|
||||
- Pattern success rates
|
||||
- Selector stability over time
|
||||
- Content structure variations
|
||||
- Extraction confidence scores
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
|
||||
# Configure adaptive crawler
|
||||
config = AdaptiveConfig(
|
||||
strategy="statistical", # or "embedding" for semantic understanding
|
||||
max_pages=10,
|
||||
confidence_threshold=0.7, # Stop at 70% confidence
|
||||
top_k_links=3, # Follow top 3 links per page
|
||||
min_gain_threshold=0.05 # Need 5% information gain to continue
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
print("Starting adaptive crawl about Python decorators...")
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/glossary.html",
|
||||
query="python decorators functions wrapping"
|
||||
)
|
||||
|
||||
print(f"\n✅ Crawling Complete!")
|
||||
print(f"• Confidence Level: {adaptive.confidence:.0%}")
|
||||
print(f"• Pages Crawled: {len(result.crawled_urls)}")
|
||||
print(f"• Knowledge Base: {len(adaptive.state.knowledge_base)} documents")
|
||||
|
||||
# Get most relevant content
|
||||
relevant = adaptive.get_relevant_content(top_k=3)
|
||||
print(f"\nMost Relevant Pages:")
|
||||
for i, page in enumerate(relevant, 1):
|
||||
print(f"{i}. {page['url']} (relevance: {page['score']:.2%})")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **News Aggregation**: Maintain 95%+ extraction accuracy even as news sites update their templates
|
||||
- **E-commerce Monitoring**: Track product changes across hundreds of stores without constant maintenance
|
||||
- **Research Data Collection**: Build robust academic datasets that survive website redesigns
|
||||
- **Reduced Maintenance**: Cut selector update time by 80% for frequently-changing sites
|
||||
|
||||
## 🌊 Virtual Scroll: Complete Content Capture
|
||||
|
||||
**The Problem:** Modern web apps only render what's visible. Scroll down, new content appears, old content vanishes into the void. Traditional crawlers capture that first viewport and miss 90% of the content. It's like reading only the first page of every book.
|
||||
|
||||
**My Solution:** I built Virtual Scroll support that mimics human browsing behavior, capturing content as it loads and preserving it before the browser's garbage collector strikes.
|
||||
|
||||
### Implementation Details
|
||||
|
||||
```python
|
||||
from crawl4ai import VirtualScrollConfig
|
||||
|
||||
# For social media feeds (Twitter/X style)
|
||||
twitter_config = VirtualScrollConfig(
|
||||
container_selector="[data-testid='primaryColumn']",
|
||||
scroll_count=20, # Number of scrolls
|
||||
scroll_by="container_height", # Smart scrolling by container size
|
||||
wait_after_scroll=1.0 # Let content load
|
||||
)
|
||||
|
||||
# For e-commerce product grids (Instagram style)
|
||||
grid_config = VirtualScrollConfig(
|
||||
container_selector="main .product-grid",
|
||||
scroll_count=30,
|
||||
scroll_by=800, # Fixed pixel scrolling
|
||||
wait_after_scroll=1.5 # Images need time
|
||||
)
|
||||
|
||||
# For news feeds with lazy loading
|
||||
news_config = VirtualScrollConfig(
|
||||
container_selector=".article-feed",
|
||||
scroll_count=50,
|
||||
scroll_by="page_height", # Viewport-based scrolling
|
||||
wait_after_scroll=0.5 # Wait for content to load
|
||||
)
|
||||
|
||||
# Use it in your crawl
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://twitter.com/trending",
|
||||
config=CrawlerRunConfig(
|
||||
virtual_scroll_config=twitter_config,
|
||||
# Combine with other features
|
||||
extraction_strategy=JsonCssExtractionStrategy({
|
||||
"tweets": {
|
||||
"selector": "[data-testid='tweet']",
|
||||
"fields": {
|
||||
"text": {"selector": "[data-testid='tweetText']", "type": "text"},
|
||||
"likes": {"selector": "[data-testid='like']", "type": "text"}
|
||||
}
|
||||
}
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
print(f"Captured {len(result.extracted_content['tweets'])} tweets")
|
||||
```
|
||||
|
||||
**Key Capabilities:**
|
||||
- **DOM Recycling Awareness**: Detects and handles virtual DOM element recycling
|
||||
- **Smart Scroll Physics**: Three modes - container height, page height, or fixed pixels
|
||||
- **Content Preservation**: Captures content before it's destroyed
|
||||
- **Intelligent Stopping**: Stops when no new content appears
|
||||
- **Memory Efficient**: Streams content instead of holding everything in memory
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Social Media Analysis**: Capture entire Twitter threads with hundreds of replies, not just top 10
|
||||
- **E-commerce Scraping**: Extract 500+ products from infinite scroll catalogs vs. 20-50 with traditional methods
|
||||
- **News Aggregation**: Get all articles from modern news sites, not just above-the-fold content
|
||||
- **Research Applications**: Complete data extraction from academic databases using virtual pagination
|
||||
|
||||
## 🔗 Link Preview: Intelligent Link Analysis and Scoring
|
||||
|
||||
**The Problem:** You crawl a page and get 200 links. Which ones matter? Which lead to the content you actually want? Traditional crawlers force you to follow everything or build complex filters.
|
||||
|
||||
**My Solution:** I implemented a three-layer scoring system that analyzes links like a human would—considering their position, context, and relevance to your goals.
|
||||
|
||||
### Intelligent Link Analysis and Scoring
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import CrawlerRunConfig, CacheMode, AsyncWebCrawler
|
||||
from crawl4ai.adaptive_crawler import LinkPreviewConfig
|
||||
|
||||
async def main():
|
||||
# Configure intelligent link analysis
|
||||
link_config = LinkPreviewConfig(
|
||||
include_internal=True,
|
||||
include_external=False,
|
||||
max_links=10,
|
||||
concurrency=5,
|
||||
query="python tutorial", # For contextual scoring
|
||||
score_threshold=0.3,
|
||||
verbose=True
|
||||
)
|
||||
# Use in your crawl
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://www.geeksforgeeks.org/",
|
||||
config=CrawlerRunConfig(
|
||||
link_preview_config=link_config,
|
||||
score_links=True, # Enable intrinsic scoring
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
)
|
||||
|
||||
# Access scored and sorted links
|
||||
if result.success and result.links:
|
||||
for link in result.links.get("internal", []):
|
||||
text = link.get('text', 'No text')[:40]
|
||||
print(
|
||||
text,
|
||||
f"{link.get('intrinsic_score', 0):.1f}/10" if link.get('intrinsic_score') is not None else "0.0/10",
|
||||
f"{link.get('contextual_score', 0):.2f}/1" if link.get('contextual_score') is not None else "0.00/1",
|
||||
f"{link.get('total_score', 0):.3f}" if link.get('total_score') is not None else "0.000"
|
||||
)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Scoring Components:**
|
||||
|
||||
1. **Intrinsic Score**: Based on link quality indicators
|
||||
- Position on page (navigation, content, footer)
|
||||
- Link attributes (rel, title, class names)
|
||||
- Anchor text quality and length
|
||||
- URL structure and depth
|
||||
|
||||
2. **Contextual Score**: Relevance to your query using BM25 algorithm
|
||||
- Keyword matching in link text and title
|
||||
- Meta description analysis
|
||||
- Content preview scoring
|
||||
|
||||
3. **Total Score**: Combined score for final ranking
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Research Efficiency**: Find relevant papers 10x faster by following only high-score links
|
||||
- **Competitive Analysis**: Automatically identify important pages on competitor sites
|
||||
- **Content Discovery**: Build topic-focused crawlers that stay on track
|
||||
- **SEO Audits**: Identify and prioritize high-value internal linking opportunities
|
||||
|
||||
## 🎣 Async URL Seeder: Automated URL Discovery at Scale
|
||||
|
||||
**The Problem:** You want to crawl an entire domain but only have the homepage. Or worse, you want specific content types across thousands of pages. Manual URL discovery? That's a job for machines, not humans.
|
||||
|
||||
**My Solution:** I built Async URL Seeder—a turbocharged URL discovery engine that combines multiple sources with intelligent filtering and relevance scoring.
|
||||
|
||||
### Technical Architecture
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncUrlSeeder, SeedingConfig
|
||||
|
||||
async def main():
|
||||
async with AsyncUrlSeeder() as seeder:
|
||||
# Discover Python tutorial URLs
|
||||
config = SeedingConfig(
|
||||
source="sitemap", # Use sitemap
|
||||
pattern="*python*", # URL pattern filter
|
||||
extract_head=True, # Get metadata
|
||||
query="python tutorial", # For relevance scoring
|
||||
scoring_method="bm25",
|
||||
score_threshold=0.2,
|
||||
max_urls=10
|
||||
)
|
||||
|
||||
print("Discovering Python async tutorial URLs...")
|
||||
urls = await seeder.urls("https://www.geeksforgeeks.org/", config)
|
||||
|
||||
print(f"\n✅ Found {len(urls)} relevant URLs:")
|
||||
for i, url_info in enumerate(urls[:5], 1):
|
||||
print(f"\n{i}. {url_info['url']}")
|
||||
if url_info.get('relevance_score'):
|
||||
print(f" Relevance: {url_info['relevance_score']:.3f}")
|
||||
if url_info.get('head_data', {}).get('title'):
|
||||
print(f" Title: {url_info['head_data']['title'][:60]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Discovery Methods:**
|
||||
- **Sitemap Mining**: Parses robots.txt and all linked sitemaps
|
||||
- **Common Crawl**: Queries the Common Crawl index for historical URLs
|
||||
- **Intelligent Crawling**: Follows links with smart depth control
|
||||
- **Pattern Analysis**: Learns URL structures and generates variations
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Migration Projects**: Discover 10,000+ URLs from legacy sites in under 60 seconds
|
||||
- **Market Research**: Map entire competitor ecosystems automatically
|
||||
- **Academic Research**: Build comprehensive datasets without manual URL collection
|
||||
- **SEO Audits**: Find every indexable page with content scoring
|
||||
- **Content Archival**: Ensure no content is left behind during site migrations
|
||||
|
||||
## ⚡ Performance Optimizations
|
||||
|
||||
This release includes significant performance improvements through optimized resource handling, better concurrency management, and reduced memory footprint.
|
||||
|
||||
### What We Optimized
|
||||
|
||||
```python
|
||||
# Optimized crawling with v0.7.0 improvements
|
||||
results = []
|
||||
for url in urls:
|
||||
result = await crawler.arun(
|
||||
url,
|
||||
config=CrawlerRunConfig(
|
||||
# Performance optimizations
|
||||
wait_until="domcontentloaded", # Faster than networkidle
|
||||
cache_mode=CacheMode.ENABLED # Enable caching
|
||||
)
|
||||
)
|
||||
results.append(result)
|
||||
```
|
||||
|
||||
**Performance Gains:**
|
||||
- **Startup Time**: 70% faster browser initialization
|
||||
- **Page Loading**: 40% reduction with smart resource blocking
|
||||
- **Extraction**: 3x faster with compiled CSS selectors
|
||||
- **Memory Usage**: 60% reduction with streaming processing
|
||||
- **Concurrent Crawls**: Handle 5x more parallel requests
|
||||
|
||||
|
||||
## 🔧 Important Changes
|
||||
|
||||
### Breaking Changes
|
||||
- `link_extractor` renamed to `link_preview` (better reflects functionality)
|
||||
- Minimum Python version now 3.9
|
||||
- `CrawlerConfig` split into `CrawlerRunConfig` and `BrowserConfig`
|
||||
|
||||
### Migration Guide
|
||||
```python
|
||||
# Old (v0.6.x)
|
||||
from crawl4ai import CrawlerConfig
|
||||
config = CrawlerConfig(timeout=30000)
|
||||
|
||||
# New (v0.7.0)
|
||||
from crawl4ai import CrawlerRunConfig, BrowserConfig
|
||||
browser_config = BrowserConfig(timeout=30000)
|
||||
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
|
||||
```
|
||||
|
||||
## 🤖 Coming Soon: Intelligent Web Automation
|
||||
|
||||
I'm currently working on bringing advanced automation capabilities to Crawl4AI. This includes:
|
||||
|
||||
- **Crawl Agents**: Autonomous crawlers that understand your goals and adapt their strategies
|
||||
- **Auto JS Generation**: Automatic JavaScript code generation for complex interactions
|
||||
- **Smart Form Handling**: Intelligent form detection and filling
|
||||
- **Context-Aware Actions**: Crawlers that understand page context and make decisions
|
||||
|
||||
These features are under active development and will revolutionize how we approach web automation. Stay tuned!
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.0
|
||||
```
|
||||
|
||||
Check out the [updated documentation](https://docs.crawl4ai.com).
|
||||
|
||||
Questions? Issues? I'm always listening:
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
|
||||
---
|
||||
|
||||
*P.S. If you're using Crawl4AI in production, I'd love to hear about it. Your use cases inspire the next features.*
|
||||
@@ -0,0 +1,43 @@
|
||||
# 🛠️ Crawl4AI v0.7.1: Minor Cleanup Update
|
||||
|
||||
*July 17, 2025 • 2 min read*
|
||||
|
||||
---
|
||||
|
||||
A small maintenance release that removes unused code and improves documentation.
|
||||
|
||||
## 🎯 What's Changed
|
||||
|
||||
- **Removed unused StealthConfig** from `crawl4ai/browser_manager.py`
|
||||
- **Updated documentation** with better examples and parameter explanations
|
||||
- **Fixed virtual scroll configuration** examples in docs
|
||||
|
||||
## 🧹 Code Cleanup
|
||||
|
||||
Removed unused `StealthConfig` import and configuration that wasn't being used anywhere in the codebase. The project uses its own custom stealth implementation through JavaScript injection instead.
|
||||
|
||||
```python
|
||||
# Removed unused code:
|
||||
from playwright_stealth import StealthConfig
|
||||
stealth_config = StealthConfig(...) # This was never used
|
||||
```
|
||||
|
||||
## 📖 Documentation Updates
|
||||
|
||||
- Fixed adaptive crawling parameter examples
|
||||
- Updated session management documentation
|
||||
- Corrected virtual scroll configuration examples
|
||||
|
||||
## 🚀 Installation
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.1
|
||||
```
|
||||
|
||||
No breaking changes - upgrade directly from v0.7.0.
|
||||
|
||||
---
|
||||
|
||||
Questions? Issues?
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
@@ -0,0 +1,350 @@
|
||||
# 🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update
|
||||
|
||||
*August 6, 2025 • 5 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.3—the Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **🕵️ Undetected Browser Support**: Stealth mode for bypassing bot detection systems
|
||||
- **🎨 Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
|
||||
- **🐳 Flexible Docker LLM Providers**: Configure LLM providers via environment variables
|
||||
- **🧠 Memory Monitoring**: Enhanced memory usage tracking and optimization tools
|
||||
- **📊 Enhanced Table Extraction**: Improved table access and DataFrame conversion
|
||||
- **💰 GitHub Sponsors**: 4-tier sponsorship system with custom arrangements
|
||||
- **🔧 Bug Fixes**: Resolved several critical issues for better stability
|
||||
- **📚 Documentation Updates**: Clearer examples and improved API documentation
|
||||
|
||||
## 🎨 Multi-URL Configurations: One Size Doesn't Fit All
|
||||
|
||||
**The Problem:** You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handling—caching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.
|
||||
|
||||
**My Solution:** I implemented URL-specific configurations that let you define different strategies for different URL patterns in a single crawl batch. First match wins, with optional fallback support.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
|
||||
|
||||
# Define specialized configs for different content types
|
||||
configs = [
|
||||
# Documentation sites - aggressive caching, include links
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*docs*", "*documentation*"],
|
||||
cache_mode="write",
|
||||
markdown_generator_options={"include_links": True}
|
||||
),
|
||||
|
||||
# News/blog sites - fresh content, scroll for lazy loading
|
||||
CrawlerRunConfig(
|
||||
url_matcher=lambda url: 'blog' in url or 'news' in url,
|
||||
cache_mode="bypass",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight/2);"
|
||||
),
|
||||
|
||||
# API endpoints - structured extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*.json", "*api*"],
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o-mini",
|
||||
extraction_type="structured"
|
||||
)
|
||||
),
|
||||
|
||||
# Default fallback for everything else
|
||||
CrawlerRunConfig() # No url_matcher = matches everything
|
||||
]
|
||||
|
||||
# Crawl multiple URLs with appropriate configs
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=[
|
||||
"https://docs.python.org/3/", # → Uses documentation config
|
||||
"https://blog.python.org/", # → Uses blog config
|
||||
"https://api.github.com/users", # → Uses API config
|
||||
"https://example.com/" # → Uses default config
|
||||
],
|
||||
config=configs
|
||||
)
|
||||
```
|
||||
|
||||
**Matching Capabilities:**
|
||||
- **String Patterns**: Wildcards like `"*.pdf"`, `"*/blog/*"`
|
||||
- **Function Matchers**: Lambda functions for complex logic
|
||||
- **Mixed Matchers**: Combine strings and functions with AND/OR logic
|
||||
- **Fallback Support**: Default config when nothing matches
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Mixed Content Sites**: Handle blogs, docs, and downloads in one crawl
|
||||
- **Multi-Domain Crawling**: Different strategies per domain without separate runs
|
||||
- **Reduced Complexity**: No more if/else forests in your extraction code
|
||||
- **Better Performance**: Each URL gets exactly the processing it needs
|
||||
|
||||
## 🕵️ Undetected Browser Support: Stealth Mode Activated
|
||||
|
||||
**The Problem:** Modern websites employ sophisticated bot detection systems. Cloudflare, Akamai, and custom solutions block automated crawlers, limiting access to valuable content.
|
||||
|
||||
**My Solution:** I implemented undetected browser support with a flexible adapter pattern. Now Crawl4AI can bypass most bot detection systems using stealth techniques.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig
|
||||
|
||||
# Enable undetected mode for stealth crawling
|
||||
browser_config = BrowserConfig(
|
||||
browser_type="undetected", # Use undetected Chrome
|
||||
headless=True, # Can run headless with stealth
|
||||
extra_args=[
|
||||
"--disable-blink-features=AutomationControlled",
|
||||
"--disable-web-security",
|
||||
"--disable-features=VizDisplayCompositor"
|
||||
]
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# This will bypass most bot detection systems
|
||||
result = await crawler.arun("https://protected-site.com")
|
||||
|
||||
if result.success:
|
||||
print("✅ Successfully bypassed bot detection!")
|
||||
print(f"Content length: {len(result.markdown)}")
|
||||
```
|
||||
|
||||
**Advanced Anti-Bot Strategies:**
|
||||
|
||||
```python
|
||||
# Combine multiple stealth techniques
|
||||
from crawl4ai import CrawlerRunConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
# Random user agents and headers
|
||||
headers={
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"DNT": "1"
|
||||
},
|
||||
|
||||
# Human-like behavior simulation
|
||||
js_code="""
|
||||
// Random mouse movements
|
||||
const simulateHuman = () => {
|
||||
const event = new MouseEvent('mousemove', {
|
||||
clientX: Math.random() * window.innerWidth,
|
||||
clientY: Math.random() * window.innerHeight
|
||||
});
|
||||
document.dispatchEvent(event);
|
||||
};
|
||||
setInterval(simulateHuman, 100 + Math.random() * 200);
|
||||
|
||||
// Random scrolling
|
||||
const randomScroll = () => {
|
||||
const scrollY = Math.random() * (document.body.scrollHeight - window.innerHeight);
|
||||
window.scrollTo(0, scrollY);
|
||||
};
|
||||
setTimeout(randomScroll, 500 + Math.random() * 1000);
|
||||
""",
|
||||
|
||||
# Delay to appear more human
|
||||
delay_before_return_html=2.0
|
||||
)
|
||||
|
||||
result = await crawler.arun("https://bot-protected-site.com", config=config)
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Enterprise Scraping**: Access previously blocked corporate sites and databases
|
||||
- **Market Research**: Gather data from competitor sites with protection
|
||||
- **Price Monitoring**: Track e-commerce sites that block automated access
|
||||
- **Content Aggregation**: Collect news and social media despite anti-bot measures
|
||||
- **Compliance Testing**: Verify your own site's bot protection effectiveness
|
||||
|
||||
## 🧠 Memory Monitoring & Optimization
|
||||
|
||||
**The Problem:** Long-running crawl sessions consuming excessive memory, especially when processing large batches or heavy JavaScript sites.
|
||||
|
||||
**My Solution:** Built comprehensive memory monitoring and optimization utilities that track usage patterns and provide actionable insights.
|
||||
|
||||
### Memory Tracking Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai.memory_utils import MemoryMonitor, get_memory_info
|
||||
|
||||
# Monitor memory during crawling
|
||||
monitor = MemoryMonitor()
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Start monitoring
|
||||
monitor.start_monitoring()
|
||||
|
||||
# Perform memory-intensive operations
|
||||
results = await crawler.arun_many([
|
||||
"https://heavy-js-site.com",
|
||||
"https://large-images-site.com",
|
||||
"https://dynamic-content-site.com"
|
||||
])
|
||||
|
||||
# Get detailed memory report
|
||||
memory_report = monitor.get_report()
|
||||
print(f"Peak memory usage: {memory_report['peak_mb']:.1f} MB")
|
||||
print(f"Memory efficiency: {memory_report['efficiency']:.1f}%")
|
||||
|
||||
# Automatic cleanup suggestions
|
||||
if memory_report['peak_mb'] > 1000: # > 1GB
|
||||
print("💡 Consider batch size optimization")
|
||||
print("💡 Enable aggressive garbage collection")
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Production Stability**: Prevent memory-related crashes in long-running services
|
||||
- **Cost Optimization**: Right-size server resources based on actual usage
|
||||
- **Performance Tuning**: Identify memory bottlenecks and optimization opportunities
|
||||
- **Scalability Planning**: Understand memory patterns for horizontal scaling
|
||||
|
||||
## 📊 Enhanced Table Extraction
|
||||
|
||||
**The Problem:** Table data was accessed through the generic `result.media` interface, making DataFrame conversion cumbersome and unclear.
|
||||
|
||||
**My Solution:** Dedicated `result.tables` interface with direct DataFrame conversion and improved detection algorithms.
|
||||
|
||||
### New Table Access Pattern
|
||||
|
||||
```python
|
||||
# Old way (deprecated)
|
||||
# tables_data = result.media.get('tables', [])
|
||||
|
||||
# New way (v0.7.3+)
|
||||
result = await crawler.arun("https://site-with-tables.com")
|
||||
|
||||
# Direct table access
|
||||
if result.tables:
|
||||
print(f"Found {len(result.tables)} tables")
|
||||
|
||||
# Convert to pandas DataFrame instantly
|
||||
import pandas as pd
|
||||
|
||||
for i, table in enumerate(result.tables):
|
||||
df = pd.DataFrame(table['data'])
|
||||
print(f"Table {i}: {df.shape[0]} rows × {df.shape[1]} columns")
|
||||
print(df.head())
|
||||
|
||||
# Table metadata
|
||||
print(f"Source: {table.get('source_xpath', 'Unknown')}")
|
||||
print(f"Headers: {table.get('headers', [])}")
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Data Analysis**: Faster transition from web data to analysis-ready DataFrames
|
||||
- **ETL Pipelines**: Cleaner integration with data processing workflows
|
||||
- **Reporting**: Simplified table extraction for automated reporting systems
|
||||
|
||||
## 💰 Community Support: GitHub Sponsors
|
||||
|
||||
I've launched GitHub Sponsors to ensure Crawl4AI's continued development and support our growing community.
|
||||
|
||||
**Sponsorship Tiers:**
|
||||
- **🌱 Supporter ($5/month)**: Community support + early feature previews
|
||||
- **🚀 Professional ($25/month)**: Priority support + beta access
|
||||
- **🏢 Business ($100/month)**: Direct consultation + custom integrations
|
||||
- **🏛️ Enterprise ($500/month)**: Dedicated support + feature development
|
||||
|
||||
**Why Sponsor?**
|
||||
- Ensure continuous development and maintenance
|
||||
- Get priority support and feature requests
|
||||
- Access to premium documentation and examples
|
||||
- Direct line to the development team
|
||||
|
||||
[**Become a Sponsor →**](https://github.com/sponsors/unclecode)
|
||||
|
||||
## 🐳 Docker: Flexible LLM Provider Configuration
|
||||
|
||||
**The Problem:** Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.
|
||||
|
||||
**My Solution:** Configure LLM providers via environment variables. Switch providers without touching code or rebuilding images.
|
||||
|
||||
### Deployment Flexibility
|
||||
|
||||
```bash
|
||||
# Option 1: Direct environment variables
|
||||
docker run -d \
|
||||
-e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
|
||||
-e GROQ_API_KEY="your-key" \
|
||||
-p 11235:11235 \
|
||||
unclecode/crawl4ai:latest
|
||||
|
||||
# Option 2: Using .llm.env file (recommended for production)
|
||||
# Create .llm.env file:
|
||||
# LLM_PROVIDER=openai/gpt-4o-mini
|
||||
# OPENAI_API_KEY=your-openai-key
|
||||
# GROQ_API_KEY=your-groq-key
|
||||
|
||||
docker run -d \
|
||||
--env-file .llm.env \
|
||||
-p 11235:11235 \
|
||||
unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
Override per request when needed:
|
||||
```python
|
||||
# Use default provider from .llm.env
|
||||
response = requests.post("http://localhost:11235/crawl", json={
|
||||
"url": "https://example.com",
|
||||
"extraction_strategy": {"type": "llm"}
|
||||
})
|
||||
|
||||
# Override to use different provider for this specific request
|
||||
response = requests.post("http://localhost:11235/crawl", json={
|
||||
"url": "https://complex-page.com",
|
||||
"extraction_strategy": {
|
||||
"type": "llm",
|
||||
"provider": "openai/gpt-4" # Override default
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Cost Optimization**: Use cheaper models for simple tasks, premium for complex
|
||||
- **A/B Testing**: Compare provider performance without deployment changes
|
||||
- **Fallback Strategies**: Switch providers on-the-fly during outages
|
||||
- **Development Flexibility**: Test locally with one provider, deploy with another
|
||||
- **Secure Configuration**: Keep API keys in `.llm.env` file, not in commands
|
||||
|
||||
## 🔧 Bug Fixes & Improvements
|
||||
|
||||
This release includes several important bug fixes that improve stability and reliability:
|
||||
|
||||
- **URL Matcher Fallback**: Fixed edge cases in URL pattern matching logic
|
||||
- **Memory Management**: Resolved memory leaks in long-running crawl sessions
|
||||
- **Sitemap Processing**: Fixed redirect handling in sitemap fetching
|
||||
- **Table Extraction**: Improved table detection and extraction accuracy
|
||||
- **Error Handling**: Better error messages and recovery from network failures
|
||||
|
||||
## 📚 Documentation Enhancements
|
||||
|
||||
Based on community feedback, we've updated:
|
||||
- Clearer examples for multi-URL configuration
|
||||
- Improved CrawlResult documentation with all available fields
|
||||
- Fixed typos and inconsistencies across documentation
|
||||
- Added real-world URLs in examples for better understanding
|
||||
- New comprehensive demo showcasing all v0.7.3 features
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thanks to our contributors and the entire community for feedback and bug reports.
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [Full Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
- [Feature Demo](https://github.com/unclecode/crawl4ai/blob/main/docs/releases_review/demo_v0.7.3.py)
|
||||
|
||||
---
|
||||
|
||||
*Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deployment—they're game changers!*
|
||||
|
||||
**Happy Crawling! 🕷️**
|
||||
|
||||
*- The Crawl4AI Team*
|
||||
@@ -0,0 +1,270 @@
|
||||
# 🚀 Crawl4AI v0.7.4: The Intelligent Table Extraction & Performance Update
|
||||
|
||||
*August 17, 2025 • 6 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.4—the Intelligent Table Extraction & Performance Update. This release introduces revolutionary LLM-powered table extraction with intelligent chunking, significant performance improvements for concurrent crawling, enhanced browser management, and critical stability fixes that make Crawl4AI more robust for production workloads.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **🚀 LLMTableExtraction**: Revolutionary table extraction with intelligent chunking for massive tables
|
||||
- **⚡ Enhanced Concurrency**: True concurrency improvements for fast-completing tasks in batch operations
|
||||
- **🔧 Browser Manager Fixes**: Resolved race conditions in concurrent page creation
|
||||
- **⌨️ Cross-Platform Browser Profiler**: Improved keyboard handling and quit mechanisms
|
||||
- **🔗 Advanced URL Processing**: Better handling of raw URLs and base tag link resolution
|
||||
- **🛡️ Enhanced Proxy Support**: Flexible proxy configuration with dict and string formats
|
||||
- **🐳 Docker Improvements**: Better API handling and raw HTML support
|
||||
|
||||
## 🚀 LLMTableExtraction: Revolutionary Table Processing
|
||||
|
||||
**The Problem:** Complex tables with rowspan, colspan, nested structures, or massive datasets that traditional HTML parsing can't handle effectively. Large tables that exceed token limits crash extraction processes.
|
||||
|
||||
**My Solution:** I developed LLMTableExtraction—an intelligent table extraction strategy that uses Large Language Models with automatic chunking to handle tables of any size and complexity.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
CrawlerRunConfig,
|
||||
LLMConfig,
|
||||
LLMTableExtraction,
|
||||
CacheMode
|
||||
)
|
||||
|
||||
# Configure LLM for table extraction
|
||||
llm_config = LLMConfig(
|
||||
provider="openai/gpt-4.1-mini",
|
||||
api_token="env:OPENAI_API_KEY",
|
||||
temperature=0.1, # Low temperature for consistency
|
||||
max_tokens=32000
|
||||
)
|
||||
|
||||
# Create intelligent table extraction strategy
|
||||
table_strategy = LLMTableExtraction(
|
||||
llm_config=llm_config,
|
||||
verbose=True,
|
||||
max_tries=2,
|
||||
enable_chunking=True, # Handle massive tables
|
||||
chunk_token_threshold=5000, # Smart chunking threshold
|
||||
overlap_threshold=100, # Maintain context between chunks
|
||||
extraction_type="structured" # Get structured data output
|
||||
)
|
||||
|
||||
# Apply to crawler configuration
|
||||
config = CrawlerRunConfig(
|
||||
table_extraction_strategy=table_strategy,
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Extract complex tables with intelligence
|
||||
result = await crawler.arun(
|
||||
"https://en.wikipedia.org/wiki/List_of_countries_by_GDP",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Access extracted tables directly
|
||||
for i, table in enumerate(result.tables):
|
||||
print(f"Table {i}: {len(table['data'])} rows × {len(table['headers'])} columns")
|
||||
|
||||
# Convert to pandas DataFrame instantly
|
||||
import pandas as pd
|
||||
df = pd.DataFrame(table['data'], columns=table['headers'])
|
||||
print(df.head())
|
||||
```
|
||||
|
||||
**Intelligent Chunking for Massive Tables:**
|
||||
|
||||
```python
|
||||
# Handle tables that exceed token limits
|
||||
large_table_strategy = LLMTableExtraction(
|
||||
llm_config=llm_config,
|
||||
enable_chunking=True,
|
||||
chunk_token_threshold=3000, # Conservative threshold
|
||||
overlap_threshold=150, # Preserve context
|
||||
max_concurrent_chunks=3, # Parallel processing
|
||||
merge_strategy="intelligent" # Smart chunk merging
|
||||
)
|
||||
|
||||
# Process Wikipedia comparison tables, financial reports, etc.
|
||||
config = CrawlerRunConfig(
|
||||
table_extraction_strategy=large_table_strategy,
|
||||
# Target specific table containers
|
||||
css_selector="div.wikitable, table.sortable",
|
||||
delay_before_return_html=2.0
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
"https://en.wikipedia.org/wiki/Comparison_of_operating_systems",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Tables are automatically chunked, processed, and merged
|
||||
print(f"Extracted {len(result.tables)} complex tables")
|
||||
for table in result.tables:
|
||||
print(f"Merged table: {len(table['data'])} total rows")
|
||||
```
|
||||
|
||||
**Advanced Features:**
|
||||
|
||||
- **Intelligent Chunking**: Automatically splits massive tables while preserving structure
|
||||
- **Context Preservation**: Overlapping chunks maintain column relationships
|
||||
- **Parallel Processing**: Concurrent chunk processing for speed
|
||||
- **Smart Merging**: Reconstructs complete tables from processed chunks
|
||||
- **Complex Structure Support**: Handles rowspan, colspan, nested tables
|
||||
- **Metadata Extraction**: Captures table context, captions, and relationships
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Financial Analysis**: Extract complex earnings tables and financial statements
|
||||
- **Research & Academia**: Process large datasets from Wikipedia, research papers
|
||||
- **E-commerce**: Handle product comparison tables with complex layouts
|
||||
- **Government Data**: Extract census data, statistical tables from official sources
|
||||
- **Competitive Intelligence**: Process competitor pricing and feature tables
|
||||
|
||||
## ⚡ Enhanced Concurrency: True Performance Gains
|
||||
|
||||
**The Problem:** The `arun_many()` method wasn't achieving true concurrency for fast-completing tasks, leading to sequential processing bottlenecks in batch operations.
|
||||
|
||||
**My Solution:** I implemented true concurrency improvements in the dispatcher that enable genuine parallel processing for fast-completing tasks.
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
```python
|
||||
# Before v0.7.4: Sequential-like behavior for fast tasks
|
||||
# After v0.7.4: True concurrency
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# These will now run with true concurrency
|
||||
urls = [
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1"
|
||||
]
|
||||
|
||||
# Processes in truly parallel fashion
|
||||
results = await crawler.arun_many(urls)
|
||||
|
||||
# Performance improvement: ~4x faster for fast-completing tasks
|
||||
print(f"Processed {len(results)} URLs with true concurrency")
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **API Crawling**: 3-4x faster processing of REST endpoints and API documentation
|
||||
- **Batch URL Processing**: Significant speedup for large URL lists
|
||||
- **Monitoring Systems**: Faster health checks and status page monitoring
|
||||
- **Data Aggregation**: Improved performance for real-time data collection
|
||||
|
||||
## 🔧 Critical Stability Fixes
|
||||
|
||||
### Browser Manager Race Condition Resolution
|
||||
|
||||
**The Problem:** Concurrent page creation in persistent browser contexts caused "Target page/context closed" errors during high-concurrency operations.
|
||||
|
||||
**My Solution:** Implemented thread-safe page creation with proper locking mechanisms.
|
||||
|
||||
```python
|
||||
# Fixed: Safe concurrent page creation
|
||||
browser_config = BrowserConfig(
|
||||
browser_type="chromium",
|
||||
use_persistent_context=True, # Now thread-safe
|
||||
max_concurrent_sessions=10 # Safely handle concurrent requests
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# These concurrent operations are now stable
|
||||
tasks = [crawler.arun(url) for url in url_list]
|
||||
results = await asyncio.gather(*tasks) # No more race conditions
|
||||
```
|
||||
|
||||
### Enhanced Browser Profiler
|
||||
|
||||
**The Problem:** Inconsistent keyboard handling across platforms and unreliable quit mechanisms.
|
||||
|
||||
**My Solution:** Cross-platform keyboard listeners with improved quit handling.
|
||||
|
||||
### Advanced URL Processing
|
||||
|
||||
**The Problem:** Raw URL formats (`raw://` and `raw:`) weren't properly handled, and base tag link resolution was incomplete.
|
||||
|
||||
**My Solution:** Enhanced URL preprocessing and base tag support.
|
||||
|
||||
```python
|
||||
# Now properly handles all URL formats
|
||||
urls = [
|
||||
"https://example.com",
|
||||
"raw://static-html-content",
|
||||
"raw:file://local-file.html"
|
||||
]
|
||||
|
||||
# Base tag links are now correctly resolved
|
||||
config = CrawlerRunConfig(
|
||||
include_links=True, # Links properly resolved with base tags
|
||||
resolve_absolute_urls=True
|
||||
)
|
||||
```
|
||||
|
||||
## 🛡️ Enhanced Proxy Configuration
|
||||
|
||||
**The Problem:** Proxy configuration only accepted specific formats, limiting flexibility.
|
||||
|
||||
**My Solution:** Enhanced ProxyConfig to support both dictionary and string formats.
|
||||
|
||||
```python
|
||||
# Multiple proxy configuration formats now supported
|
||||
from crawl4ai import BrowserConfig, ProxyConfig
|
||||
|
||||
# String format
|
||||
proxy_config = ProxyConfig("http://proxy.example.com:8080")
|
||||
|
||||
# Dictionary format
|
||||
proxy_config = ProxyConfig({
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
})
|
||||
|
||||
# Use with crawler
|
||||
browser_config = BrowserConfig(proxy_config=proxy_config)
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun("https://httpbin.org/ip")
|
||||
```
|
||||
|
||||
## 🐳 Docker & Infrastructure Improvements
|
||||
|
||||
This release includes several Docker and infrastructure improvements:
|
||||
|
||||
- **Better API Token Handling**: Improved Docker example scripts with correct endpoints
|
||||
- **Raw HTML Support**: Enhanced Docker API to handle raw HTML content properly
|
||||
- **Documentation Updates**: Comprehensive Docker deployment examples
|
||||
- **Test Coverage**: Expanded test suite with better coverage
|
||||
|
||||
## 📚 Documentation & Examples
|
||||
|
||||
Enhanced documentation includes:
|
||||
|
||||
- **LLM Table Extraction Guide**: Comprehensive examples and best practices
|
||||
- **Migration Documentation**: Updated patterns for new table extraction methods
|
||||
- **Docker Deployment**: Complete deployment guide with examples
|
||||
- **Performance Optimization**: Guidelines for concurrent crawling
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thanks to our contributors and community for feedback, bug reports, and feature requests that made this release possible.
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [Full Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
- [LLM Table Extraction Examples](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/llm_table_extraction_example.py)
|
||||
|
||||
---
|
||||
|
||||
*Crawl4AI v0.7.4 delivers intelligent table extraction and significant performance improvements. The new LLMTableExtraction strategy handles complex tables that were previously impossible to process, while concurrency improvements make batch operations 3-4x faster. Try the intelligent table extraction—it's a game changer for data extraction workflows!*
|
||||
|
||||
**Happy Crawling! 🕷️**
|
||||
|
||||
*- The Crawl4AI Team*
|
||||
@@ -0,0 +1,318 @@
|
||||
# 🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update
|
||||
|
||||
*September 29, 2025 • 8 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.5—focused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Docker Hooks System**: Custom Python functions at key pipeline points with function-based API
|
||||
- **Function-Based Hooks**: New `hooks_to_string()` utility with Docker client auto-conversion
|
||||
- **Enhanced LLM Integration**: Custom providers with temperature control
|
||||
- **HTTPS Preservation**: Secure internal link handling
|
||||
- **Bug Fixes**: Resolved multiple community-reported issues
|
||||
- **Improved Docker Error Handling**: Better debugging and reliability
|
||||
|
||||
## 🔧 Docker Hooks System: Pipeline Customization
|
||||
|
||||
Every scraping project needs custom logic—authentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.
|
||||
|
||||
### Real Example: Authentication & Performance
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
# Real working hooks for httpbin.org
|
||||
hooks_config = {
|
||||
"on_page_context_created": """
|
||||
async def hook(page, context, **kwargs):
|
||||
print("Hook: Setting up page context")
|
||||
# Block images to speed up crawling
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
print("Hook: Images blocked")
|
||||
return page
|
||||
""",
|
||||
|
||||
"before_retrieve_html": """
|
||||
async def hook(page, context, **kwargs):
|
||||
print("Hook: Before retrieving HTML")
|
||||
# Scroll to bottom to load lazy content
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(1000)
|
||||
print("Hook: Scrolled to bottom")
|
||||
return page
|
||||
""",
|
||||
|
||||
"before_goto": """
|
||||
async def hook(page, context, url, **kwargs):
|
||||
print(f"Hook: About to navigate to {url}")
|
||||
# Add custom headers
|
||||
await page.set_extra_http_headers({
|
||||
'X-Test-Header': 'crawl4ai-hooks-test'
|
||||
})
|
||||
return page
|
||||
"""
|
||||
}
|
||||
|
||||
# Test with Docker API
|
||||
payload = {
|
||||
"urls": ["https://httpbin.org/html"],
|
||||
"hooks": {
|
||||
"code": hooks_config,
|
||||
"timeout": 30
|
||||
}
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/crawl", json=payload)
|
||||
result = response.json()
|
||||
|
||||
if result.get('success'):
|
||||
print("✅ Hooks executed successfully!")
|
||||
print(f"Content length: {len(result.get('markdown', ''))} characters")
|
||||
```
|
||||
|
||||
**Available Hook Points:**
|
||||
- `on_browser_created`: Browser setup
|
||||
- `on_page_context_created`: Page context configuration
|
||||
- `before_goto`: Pre-navigation setup
|
||||
- `after_goto`: Post-navigation processing
|
||||
- `on_user_agent_updated`: User agent changes
|
||||
- `on_execution_started`: Crawl initialization
|
||||
- `before_retrieve_html`: Pre-extraction processing
|
||||
- `before_return_html`: Final HTML processing
|
||||
|
||||
### Function-Based Hooks API
|
||||
|
||||
Writing hooks as strings works, but lacks IDE support and type checking. v0.7.5 introduces a function-based approach with automatic conversion!
|
||||
|
||||
**Option 1: Using the `hooks_to_string()` Utility**
|
||||
|
||||
```python
|
||||
from crawl4ai import hooks_to_string
|
||||
import requests
|
||||
|
||||
# Define hooks as regular Python functions (with full IDE support!)
|
||||
async def on_page_context_created(page, context, **kwargs):
|
||||
"""Block images to speed up crawling"""
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
await page.set_viewport_size({"width": 1920, "height": 1080})
|
||||
return page
|
||||
|
||||
async def before_goto(page, context, url, **kwargs):
|
||||
"""Add custom headers"""
|
||||
await page.set_extra_http_headers({
|
||||
'X-Crawl4AI': 'v0.7.5',
|
||||
'X-Custom-Header': 'my-value'
|
||||
})
|
||||
return page
|
||||
|
||||
# Convert functions to strings
|
||||
hooks_code = hooks_to_string({
|
||||
"on_page_context_created": on_page_context_created,
|
||||
"before_goto": before_goto
|
||||
})
|
||||
|
||||
# Use with REST API
|
||||
payload = {
|
||||
"urls": ["https://httpbin.org/html"],
|
||||
"hooks": {"code": hooks_code, "timeout": 30}
|
||||
}
|
||||
response = requests.post("http://localhost:11235/crawl", json=payload)
|
||||
```
|
||||
|
||||
**Option 2: Docker Client with Automatic Conversion (Recommended!)**
|
||||
|
||||
```python
|
||||
from crawl4ai.docker_client import Crawl4aiDockerClient
|
||||
|
||||
# Define hooks as functions (same as above)
|
||||
async def on_page_context_created(page, context, **kwargs):
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
return page
|
||||
|
||||
async def before_retrieve_html(page, context, **kwargs):
|
||||
# Scroll to load lazy content
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(1000)
|
||||
return page
|
||||
|
||||
# Use Docker client - conversion happens automatically!
|
||||
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
|
||||
|
||||
results = await client.crawl(
|
||||
urls=["https://httpbin.org/html"],
|
||||
hooks={
|
||||
"on_page_context_created": on_page_context_created,
|
||||
"before_retrieve_html": before_retrieve_html
|
||||
},
|
||||
hooks_timeout=30
|
||||
)
|
||||
|
||||
if results and results.success:
|
||||
print(f"✅ Hooks executed! HTML length: {len(results.html)}")
|
||||
```
|
||||
|
||||
**Benefits of Function-Based Hooks:**
|
||||
- ✅ Full IDE support (autocomplete, syntax highlighting)
|
||||
- ✅ Type checking and linting
|
||||
- ✅ Easier to test and debug
|
||||
- ✅ Reusable across projects
|
||||
- ✅ Automatic conversion in Docker client
|
||||
- ✅ No breaking changes - string hooks still work!
|
||||
|
||||
## 🤖 Enhanced LLM Integration
|
||||
|
||||
Enhanced LLM integration with custom providers, temperature control, and base URL configuration.
|
||||
|
||||
### Multi-Provider Support
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
# Test with different providers
|
||||
async def test_llm_providers():
|
||||
# OpenAI with custom temperature
|
||||
openai_strategy = LLMExtractionStrategy(
|
||||
provider="gemini/gemini-2.5-flash-lite",
|
||||
api_token="your-api-token",
|
||||
temperature=0.7, # New in v0.7.5
|
||||
instruction="Summarize this page in one sentence"
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://example.com",
|
||||
config=CrawlerRunConfig(extraction_strategy=openai_strategy)
|
||||
)
|
||||
|
||||
if result.success:
|
||||
print("✅ LLM extraction completed")
|
||||
print(result.extracted_content)
|
||||
|
||||
# Docker API with enhanced LLM config
|
||||
llm_payload = {
|
||||
"url": "https://example.com",
|
||||
"f": "llm",
|
||||
"q": "Summarize this page in one sentence.",
|
||||
"provider": "gemini/gemini-2.5-flash-lite",
|
||||
"temperature": 0.7
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/md", json=llm_payload)
|
||||
```
|
||||
|
||||
**New Features:**
|
||||
- Custom `temperature` parameter for creativity control
|
||||
- `base_url` for custom API endpoints
|
||||
- Multi-provider environment variable support
|
||||
- Docker API integration
|
||||
|
||||
## 🔒 HTTPS Preservation
|
||||
|
||||
**The Problem:** Modern web apps require HTTPS everywhere. When crawlers downgrade internal links from HTTPS to HTTP, authentication breaks and security warnings appear.
|
||||
|
||||
**Solution:** HTTPS preservation maintains secure protocols throughout crawling.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy
|
||||
|
||||
async def test_https_preservation():
|
||||
# Enable HTTPS preservation
|
||||
url_filter = URLPatternFilter(
|
||||
patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
exclude_external_links=True,
|
||||
preserve_https_for_internal_links=True, # New in v0.7.5
|
||||
deep_crawl_strategy=BFSDeepCrawlStrategy(
|
||||
max_depth=2,
|
||||
max_pages=5,
|
||||
filter_chain=FilterChain([url_filter])
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
async for result in await crawler.arun(
|
||||
url="https://quotes.toscrape.com",
|
||||
config=config
|
||||
):
|
||||
# All internal links maintain HTTPS
|
||||
internal_links = [link['href'] for link in result.links['internal']]
|
||||
https_links = [link for link in internal_links if link.startswith('https://')]
|
||||
|
||||
print(f"HTTPS links preserved: {len(https_links)}/{len(internal_links)}")
|
||||
for link in https_links[:3]:
|
||||
print(f" → {link}")
|
||||
```
|
||||
|
||||
## 🛠️ Bug Fixes and Improvements
|
||||
|
||||
### Major Fixes
|
||||
- **URL Processing**: Fixed '+' sign preservation in query parameters (#1332)
|
||||
- **Proxy Configuration**: Enhanced proxy string parsing (old `proxy` parameter deprecated)
|
||||
- **Docker Error Handling**: Comprehensive error messages with status codes
|
||||
- **Memory Management**: Fixed leaks in long-running sessions
|
||||
- **JWT Authentication**: Fixed Docker JWT validation issues (#1442)
|
||||
- **Playwright Stealth**: Fixed stealth features for Playwright integration (#1481)
|
||||
- **API Configuration**: Fixed config handling to prevent overriding user-provided settings (#1505)
|
||||
- **Docker Filter Serialization**: Resolved JSON encoding errors in deep crawl strategy (#1419)
|
||||
- **LLM Provider Support**: Fixed custom LLM provider integration for adaptive crawler (#1291)
|
||||
- **Performance Issues**: Resolved backoff strategy failures and timeout handling (#989)
|
||||
|
||||
### Community-Reported Issues Fixed
|
||||
This release addresses multiple issues reported by the community through GitHub issues and Discord discussions:
|
||||
- Fixed browser configuration reference errors
|
||||
- Resolved dependency conflicts with cssselect
|
||||
- Improved error messaging for failed authentications
|
||||
- Enhanced compatibility with various proxy configurations
|
||||
- Fixed edge cases in URL normalization
|
||||
|
||||
### Configuration Updates
|
||||
```python
|
||||
# Old proxy config (deprecated)
|
||||
# browser_config = BrowserConfig(proxy="http://proxy:8080")
|
||||
|
||||
# New enhanced proxy config
|
||||
browser_config = BrowserConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy:8080",
|
||||
"username": "optional-user",
|
||||
"password": "optional-pass"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
1. **Python 3.10+ Required**: Upgrade from Python 3.9
|
||||
2. **Proxy Parameter Deprecated**: Use new `proxy_config` structure
|
||||
3. **New Dependency**: Added `cssselect` for better CSS handling
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
```bash
|
||||
# Install latest version
|
||||
pip install crawl4ai==0.7.5
|
||||
|
||||
# Docker deployment
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
**Try the Demo:**
|
||||
```bash
|
||||
# Run working examples
|
||||
python docs/releases_review/demo_v0.7.5.py
|
||||
```
|
||||
|
||||
**Resources:**
|
||||
- 📖 Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- 🐙 GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- 💬 Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- 🐦 Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
@@ -0,0 +1,314 @@
|
||||
# Crawl4AI v0.7.6 Release Notes
|
||||
|
||||
*Release Date: October 22, 2025*
|
||||
|
||||
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
|
||||
|
||||
## 🎯 What's New
|
||||
|
||||
### Webhook Support for Docker Job Queue API
|
||||
|
||||
The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!
|
||||
|
||||
**Key Capabilities:**
|
||||
|
||||
- ✅ **Universal Webhook Support**: Both `/crawl/job` and `/llm/job` endpoints now support webhooks
|
||||
- ✅ **Flexible Delivery Modes**: Choose notification-only or include full data in the webhook payload
|
||||
- ✅ **Reliable Delivery**: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
|
||||
- ✅ **Custom Authentication**: Add custom headers for webhook authentication
|
||||
- ✅ **Global Configuration**: Set default webhook URL in `config.yml` for all jobs
|
||||
- ✅ **Task Type Identification**: Distinguish between `crawl` and `llm_extraction` tasks
|
||||
|
||||
### How It Works
|
||||
|
||||
Instead of constantly checking job status:
|
||||
|
||||
**OLD WAY (Polling):**
|
||||
```python
|
||||
# Submit job
|
||||
response = requests.post("http://localhost:11235/crawl/job", json=payload)
|
||||
task_id = response.json()['task_id']
|
||||
|
||||
# Poll until complete
|
||||
while True:
|
||||
status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
|
||||
if status.json()['status'] == 'completed':
|
||||
break
|
||||
time.sleep(5) # Wait and try again
|
||||
```
|
||||
|
||||
**NEW WAY (Webhooks):**
|
||||
```python
|
||||
# Submit job with webhook
|
||||
payload = {
|
||||
"urls": ["https://example.com"],
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhook",
|
||||
"webhook_data_in_payload": True
|
||||
}
|
||||
}
|
||||
response = requests.post("http://localhost:11235/crawl/job", json=payload)
|
||||
|
||||
# Done! Webhook will notify you when complete
|
||||
# Your webhook handler receives the results automatically
|
||||
```
|
||||
|
||||
### Crawl Job Webhooks
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:11235/crawl/job \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"urls": ["https://example.com"],
|
||||
"browser_config": {"headless": true},
|
||||
"crawler_config": {"cache_mode": "bypass"},
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhooks/crawl-complete",
|
||||
"webhook_data_in_payload": false,
|
||||
"webhook_headers": {
|
||||
"X-Webhook-Secret": "your-secret-token"
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### LLM Extraction Job Webhooks (NEW!)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:11235/llm/job \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"url": "https://example.com/article",
|
||||
"q": "Extract the article title, author, and publication date",
|
||||
"schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
|
||||
"provider": "openai/gpt-4o-mini",
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhooks/llm-complete",
|
||||
"webhook_data_in_payload": true
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Webhook Payload Structure
|
||||
|
||||
**Success (with data):**
|
||||
```json
|
||||
{
|
||||
"task_id": "llm_1698765432",
|
||||
"task_type": "llm_extraction",
|
||||
"status": "completed",
|
||||
"timestamp": "2025-10-22T10:30:00.000000+00:00",
|
||||
"urls": ["https://example.com/article"],
|
||||
"data": {
|
||||
"extracted_content": {
|
||||
"title": "Understanding Web Scraping",
|
||||
"author": "John Doe",
|
||||
"date": "2025-10-22"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Failure:**
|
||||
```json
|
||||
{
|
||||
"task_id": "crawl_abc123",
|
||||
"task_type": "crawl",
|
||||
"status": "failed",
|
||||
"timestamp": "2025-10-22T10:30:00.000000+00:00",
|
||||
"urls": ["https://example.com"],
|
||||
"error": "Connection timeout after 30s"
|
||||
}
|
||||
```
|
||||
|
||||
### Simple Webhook Handler Example
|
||||
|
||||
```python
|
||||
from flask import Flask, request, jsonify
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
@app.route('/webhook', methods=['POST'])
|
||||
def handle_webhook():
|
||||
payload = request.json
|
||||
|
||||
task_id = payload['task_id']
|
||||
task_type = payload['task_type']
|
||||
status = payload['status']
|
||||
|
||||
if status == 'completed':
|
||||
if 'data' in payload:
|
||||
# Process data directly
|
||||
data = payload['data']
|
||||
else:
|
||||
# Fetch from API
|
||||
endpoint = 'crawl' if task_type == 'crawl' else 'llm'
|
||||
response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
|
||||
data = response.json()
|
||||
|
||||
# Your business logic here
|
||||
print(f"Job {task_id} completed!")
|
||||
|
||||
elif status == 'failed':
|
||||
error = payload.get('error', 'Unknown error')
|
||||
print(f"Job {task_id} failed: {error}")
|
||||
|
||||
return jsonify({"status": "received"}), 200
|
||||
|
||||
app.run(port=8080)
|
||||
```
|
||||
|
||||
## 📊 Performance Improvements
|
||||
|
||||
- **Reduced Server Load**: Eliminates constant polling requests
|
||||
- **Lower Latency**: Instant notification vs. polling interval delay
|
||||
- **Better Resource Usage**: Frees up client connections while jobs run in background
|
||||
- **Scalable Architecture**: Handles high-volume crawling workflows efficiently
|
||||
|
||||
## 🐛 Bug Fixes
|
||||
|
||||
- Fixed webhook configuration serialization for Pydantic HttpUrl fields
|
||||
- Improved error handling in webhook delivery service
|
||||
- Enhanced Redis task storage for webhook config persistence
|
||||
|
||||
## 🌍 Expected Real-World Impact
|
||||
|
||||
### For Web Scraping Workflows
|
||||
- **Reduced Costs**: Less API calls = lower bandwidth and server costs
|
||||
- **Better UX**: Instant notifications improve user experience
|
||||
- **Scalability**: Handle 100s of concurrent jobs without polling overhead
|
||||
|
||||
### For LLM Extraction Pipelines
|
||||
- **Async Processing**: Submit LLM extraction jobs and move on
|
||||
- **Batch Processing**: Queue multiple extractions, get notified as they complete
|
||||
- **Integration**: Easy integration with workflow automation tools (Zapier, n8n, etc.)
|
||||
|
||||
### For Microservices
|
||||
- **Event-Driven**: Perfect for event-driven microservice architectures
|
||||
- **Decoupling**: Decouple job submission from result processing
|
||||
- **Reliability**: Automatic retries ensure webhooks are delivered
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- Webhook configuration is optional
|
||||
- Existing code continues to work without modification
|
||||
- Polling is still supported for jobs without webhook config
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### New Documentation
|
||||
- **[WEBHOOK_EXAMPLES.md](../deploy/docker/WEBHOOK_EXAMPLES.md)** - Comprehensive webhook usage guide
|
||||
- **[docker_webhook_example.py](../docs/examples/docker_webhook_example.py)** - Working code examples
|
||||
|
||||
### Updated Documentation
|
||||
- **[Docker README](../deploy/docker/README.md)** - Added webhook sections
|
||||
- API documentation with webhook examples
|
||||
|
||||
## 🛠️ Migration Guide
|
||||
|
||||
No migration needed! Webhooks are opt-in:
|
||||
|
||||
1. **To use webhooks**: Add `webhook_config` to your job payload
|
||||
2. **To keep polling**: Continue using your existing code
|
||||
|
||||
### Quick Start
|
||||
|
||||
```python
|
||||
# Just add webhook_config to your existing payload
|
||||
payload = {
|
||||
# Your existing configuration
|
||||
"urls": ["https://example.com"],
|
||||
"browser_config": {...},
|
||||
"crawler_config": {...},
|
||||
|
||||
# NEW: Add webhook configuration
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhook",
|
||||
"webhook_data_in_payload": True
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Global Webhook Configuration (config.yml)
|
||||
|
||||
```yaml
|
||||
webhooks:
|
||||
enabled: true
|
||||
default_url: "https://myapp.com/webhooks/default" # Optional
|
||||
data_in_payload: false
|
||||
retry:
|
||||
max_attempts: 5
|
||||
initial_delay_ms: 1000
|
||||
max_delay_ms: 32000
|
||||
timeout_ms: 30000
|
||||
headers:
|
||||
User-Agent: "Crawl4AI-Webhook/1.0"
|
||||
```
|
||||
|
||||
## 🚀 Upgrade Instructions
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest image
|
||||
docker pull unclecode/crawl4ai:0.7.6
|
||||
|
||||
# Or use latest tag
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run with webhook support
|
||||
docker run -d \
|
||||
-p 11235:11235 \
|
||||
--env-file .llm.env \
|
||||
--name crawl4ai \
|
||||
unclecode/crawl4ai:0.7.6
|
||||
```
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Use notification-only mode** for large results - fetch data separately to avoid large webhook payloads
|
||||
2. **Set custom headers** for webhook authentication and request tracking
|
||||
3. **Configure global default webhook** for consistent handling across all jobs
|
||||
4. **Implement idempotent webhook handlers** - same webhook may be delivered multiple times on retry
|
||||
5. **Use structured schemas** with LLM extraction for predictable webhook data
|
||||
|
||||
## 🎬 Demo
|
||||
|
||||
Try the release demo:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.6.py
|
||||
```
|
||||
|
||||
This comprehensive demo showcases:
|
||||
- Crawl job webhooks (notification-only and with data)
|
||||
- LLM extraction webhooks (with JSON schema support)
|
||||
- Custom headers for authentication
|
||||
- Webhook retry mechanism
|
||||
- Real-time webhook receiver
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
|
||||
|
||||
## 📞 Support
|
||||
|
||||
- **Documentation**: https://docs.crawl4ai.com
|
||||
- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
|
||||
- **Discord**: https://discord.gg/crawl4ai
|
||||
|
||||
---
|
||||
|
||||
**Happy crawling with webhooks!** 🕷️🪝
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,626 @@
|
||||
# 🚀 Crawl4AI v0.7.7: The Self-Hosting & Monitoring Update
|
||||
|
||||
*November 14, 2025 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.7—the Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **📊 Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool status
|
||||
- **🔌 Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data
|
||||
- **⚡ WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards
|
||||
- **🎮 Control Actions**: Manual browser management (kill, restart, cleanup)
|
||||
- **🔥 Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion
|
||||
- **🧹 Janitor Cleanup System**: Automatic resource management with event logging
|
||||
- **📈 Production Metrics**: 6 critical metrics for operational excellence
|
||||
- **🏭 Integration Ready**: Prometheus, alerting, and log aggregation examples
|
||||
- **🐛 Critical Bug Fixes**: Async LLM extraction, DFS crawling, viewport config, and more
|
||||
|
||||
## 📊 Real-time Monitoring Dashboard: Complete Visibility
|
||||
|
||||
**The Problem:** Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the container—memory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.
|
||||
|
||||
**My Solution:** I built a complete real-time monitoring system with an interactive dashboard, comprehensive REST API, WebSocket streaming, and manual control actions. Now you have full transparency and control over your crawling infrastructure.
|
||||
|
||||
### The Self-Hosting Value Proposition
|
||||
|
||||
Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:
|
||||
|
||||
- **🔒 Data Privacy**: Your data never leaves your infrastructure
|
||||
- **💰 Cost Control**: No per-request pricing or rate limits
|
||||
- **🎯 Full Customization**: Complete control over configurations and strategies
|
||||
- **📊 Complete Transparency**: Real-time visibility into every aspect
|
||||
- **⚡ Performance**: Direct access without network overhead
|
||||
- **🛡️ Enterprise Security**: Keep workflows behind your firewall
|
||||
|
||||
### Interactive Monitoring Dashboard
|
||||
|
||||
Access the dashboard at `http://localhost:11235/dashboard` to see:
|
||||
|
||||
- **System Health Overview**: CPU, memory, network, and uptime in real-time
|
||||
- **Live Request Tracking**: Active and completed requests with full details
|
||||
- **Browser Pool Management**: Interactive table with permanent/hot/cold browsers
|
||||
- **Janitor Events Log**: Automatic cleanup activities
|
||||
- **Error Monitoring**: Full context error logs
|
||||
|
||||
The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.
|
||||
|
||||
## 🔌 Monitor API: Programmatic Access
|
||||
|
||||
**The Problem:** Monitoring dashboards are great for humans, but automation and integration require programmatic access.
|
||||
|
||||
**My Solution:** A comprehensive REST API that exposes all monitoring data for integration with your existing infrastructure.
|
||||
|
||||
### System Health Endpoint
|
||||
|
||||
```python
|
||||
import httpx
|
||||
import asyncio
|
||||
|
||||
async def monitor_system_health():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/health")
|
||||
health = response.json()
|
||||
|
||||
print(f"Container Metrics:")
|
||||
print(f" CPU: {health['container']['cpu_percent']:.1f}%")
|
||||
print(f" Memory: {health['container']['memory_percent']:.1f}%")
|
||||
print(f" Uptime: {health['container']['uptime_seconds']}s")
|
||||
|
||||
print(f"\nBrowser Pool:")
|
||||
print(f" Permanent: {health['pool']['permanent']['active']} active")
|
||||
print(f" Hot Pool: {health['pool']['hot']['count']} browsers")
|
||||
print(f" Cold Pool: {health['pool']['cold']['count']} browsers")
|
||||
|
||||
print(f"\nStatistics:")
|
||||
print(f" Total Requests: {health['stats']['total_requests']}")
|
||||
print(f" Success Rate: {health['stats']['success_rate_percent']:.1f}%")
|
||||
print(f" Avg Latency: {health['stats']['avg_latency_ms']:.0f}ms")
|
||||
|
||||
asyncio.run(monitor_system_health())
|
||||
```
|
||||
|
||||
### Request Tracking
|
||||
|
||||
```python
|
||||
async def track_requests():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/requests")
|
||||
requests_data = response.json()
|
||||
|
||||
print(f"Active Requests: {len(requests_data['active'])}")
|
||||
print(f"Completed Requests: {len(requests_data['completed'])}")
|
||||
|
||||
# See details of recent requests
|
||||
for req in requests_data['completed'][:5]:
|
||||
status_icon = "✅" if req['success'] else "❌"
|
||||
print(f"{status_icon} {req['endpoint']} - {req['latency_ms']:.0f}ms")
|
||||
```
|
||||
|
||||
### Browser Pool Management
|
||||
|
||||
```python
|
||||
async def monitor_browser_pool():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/browsers")
|
||||
browsers = response.json()
|
||||
|
||||
print(f"Pool Summary:")
|
||||
print(f" Total Browsers: {browsers['summary']['total_count']}")
|
||||
print(f" Total Memory: {browsers['summary']['total_memory_mb']} MB")
|
||||
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
|
||||
|
||||
# List all browsers
|
||||
for browser in browsers['permanent']:
|
||||
print(f"🔥 Permanent: {browser['browser_id'][:8]}... | "
|
||||
f"Requests: {browser['request_count']} | "
|
||||
f"Memory: {browser['memory_mb']:.0f} MB")
|
||||
```
|
||||
|
||||
### Endpoint Performance Statistics
|
||||
|
||||
```python
|
||||
async def get_endpoint_stats():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/endpoints/stats")
|
||||
stats = response.json()
|
||||
|
||||
print("Endpoint Analytics:")
|
||||
for endpoint, data in stats.items():
|
||||
print(f" {endpoint}:")
|
||||
print(f" Requests: {data['count']}")
|
||||
print(f" Avg Latency: {data['avg_latency_ms']:.0f}ms")
|
||||
print(f" Success Rate: {data['success_rate_percent']:.1f}%")
|
||||
```
|
||||
|
||||
### Complete API Reference
|
||||
|
||||
The Monitor API includes these endpoints:
|
||||
|
||||
- `GET /monitor/health` - System health with pool statistics
|
||||
- `GET /monitor/requests` - Active and completed request tracking
|
||||
- `GET /monitor/browsers` - Browser pool details and efficiency
|
||||
- `GET /monitor/endpoints/stats` - Per-endpoint performance analytics
|
||||
- `GET /monitor/timeline?minutes=5` - Time-series data for charts
|
||||
- `GET /monitor/logs/janitor?limit=10` - Cleanup activity logs
|
||||
- `GET /monitor/logs/errors?limit=10` - Error logs with context
|
||||
- `POST /monitor/actions/cleanup` - Force immediate cleanup
|
||||
- `POST /monitor/actions/kill_browser` - Kill specific browser
|
||||
- `POST /monitor/actions/restart_browser` - Restart browser
|
||||
- `POST /monitor/stats/reset` - Reset accumulated statistics
|
||||
|
||||
## ⚡ WebSocket Streaming: Real-time Updates
|
||||
|
||||
**The Problem:** Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.
|
||||
|
||||
**My Solution:** WebSocket streaming with 2-second update intervals for building custom real-time dashboards.
|
||||
|
||||
### WebSocket Integration Example
|
||||
|
||||
```python
|
||||
import websockets
|
||||
import json
|
||||
import asyncio
|
||||
|
||||
async def monitor_realtime():
|
||||
uri = "ws://localhost:11235/monitor/ws"
|
||||
|
||||
async with websockets.connect(uri) as websocket:
|
||||
print("Connected to real-time monitoring stream")
|
||||
|
||||
while True:
|
||||
# Receive update every 2 seconds
|
||||
data = await websocket.recv()
|
||||
update = json.loads(data)
|
||||
|
||||
# Access all monitoring data
|
||||
print(f"\n--- Update at {update['timestamp']} ---")
|
||||
print(f"Memory: {update['health']['container']['memory_percent']:.1f}%")
|
||||
print(f"Active Requests: {len(update['requests']['active'])}")
|
||||
print(f"Total Browsers: {update['browsers']['summary']['total_count']}")
|
||||
|
||||
if update['errors']:
|
||||
print(f"⚠️ Recent Errors: {len(update['errors'])}")
|
||||
|
||||
asyncio.run(monitor_realtime())
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Custom Dashboards**: Build tailored monitoring UIs for your team
|
||||
- **Real-time Alerting**: Trigger alerts instantly when metrics exceed thresholds
|
||||
- **Integration**: Feed live data into monitoring tools like Grafana
|
||||
- **Automation**: React to events in real-time without polling
|
||||
|
||||
## 🔥 Smart Browser Pool: 3-Tier Architecture
|
||||
|
||||
**The Problem:** Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.
|
||||
|
||||
**My Solution:** A smart 3-tier browser pool that automatically adapts to usage patterns.
|
||||
|
||||
### How It Works
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async def demonstrate_browser_pool():
|
||||
async with httpx.AsyncClient() as client:
|
||||
# Request 1-3: Default config → Uses permanent browser
|
||||
print("Phase 1: Using permanent browser")
|
||||
for i in range(3):
|
||||
await client.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={"urls": [f"https://httpbin.org/html?req={i}"]}
|
||||
)
|
||||
print(f" Request {i+1}: Reused permanent browser")
|
||||
|
||||
# Request 4-6: Custom viewport → Cold pool (first use)
|
||||
print("\nPhase 2: Custom config creates cold pool browser")
|
||||
viewport_config = {"viewport": {"width": 1280, "height": 720}}
|
||||
for i in range(4):
|
||||
await client.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={
|
||||
"urls": [f"https://httpbin.org/json?v={i}"],
|
||||
"browser_config": viewport_config
|
||||
}
|
||||
)
|
||||
if i < 2:
|
||||
print(f" Request {i+1}: Cold pool browser")
|
||||
else:
|
||||
print(f" Request {i+1}: Promoted to hot pool! (after 3 uses)")
|
||||
|
||||
# Check pool status
|
||||
response = await client.get("http://localhost:11235/monitor/browsers")
|
||||
browsers = response.json()
|
||||
|
||||
print(f"\nPool Status:")
|
||||
print(f" Permanent: {len(browsers['permanent'])} (always active)")
|
||||
print(f" Hot: {len(browsers['hot'])} (frequently used configs)")
|
||||
print(f" Cold: {len(browsers['cold'])} (on-demand)")
|
||||
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
|
||||
|
||||
asyncio.run(demonstrate_browser_pool())
|
||||
```
|
||||
|
||||
**Pool Tiers:**
|
||||
|
||||
- **🔥 Permanent Browser**: Always-on, default configuration, instant response
|
||||
- **♨️ Hot Pool**: Browsers promoted after 3+ uses, kept warm for quick access
|
||||
- **❄️ Cold Pool**: On-demand browsers for variant configs, cleaned up when idle
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Memory Efficiency**: 10x reduction in memory usage vs creating browsers per request
|
||||
- **Performance**: Instant access to frequently-used configurations
|
||||
- **Automatic Optimization**: Pool adapts to your usage patterns
|
||||
- **Resource Management**: Janitor automatically cleans up idle browsers
|
||||
|
||||
## 🧹 Janitor System: Automatic Cleanup
|
||||
|
||||
**The Problem:** Long-running crawlers accumulate idle browsers and consume memory over time.
|
||||
|
||||
**My Solution:** An automatic janitor system that monitors and cleans up idle resources.
|
||||
|
||||
```python
|
||||
async def monitor_janitor_activity():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/logs/janitor?limit=5")
|
||||
logs = response.json()
|
||||
|
||||
print("Recent Cleanup Activities:")
|
||||
for log in logs:
|
||||
print(f" {log['timestamp']}: {log['message']}")
|
||||
|
||||
# Example output:
|
||||
# 2025-11-14 10:30:00: Cleaned up 2 cold pool browsers (idle > 5min)
|
||||
# 2025-11-14 10:25:00: Browser reuse rate: 85.3%
|
||||
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)
|
||||
```
|
||||
|
||||
## 🎮 Control Actions: Manual Management
|
||||
|
||||
**The Problem:** Sometimes you need to manually intervene—kill a stuck browser, force cleanup, or restart resources.
|
||||
|
||||
**My Solution:** Manual control actions via the API for operational troubleshooting.
|
||||
|
||||
### Force Cleanup
|
||||
|
||||
```python
|
||||
async def force_cleanup():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/monitor/actions/cleanup")
|
||||
result = response.json()
|
||||
|
||||
print(f"Cleanup completed:")
|
||||
print(f" Browsers cleaned: {result.get('cleaned_count', 0)}")
|
||||
print(f" Memory freed: {result.get('memory_freed_mb', 0):.1f} MB")
|
||||
```
|
||||
|
||||
### Kill Specific Browser
|
||||
|
||||
```python
|
||||
async def kill_stuck_browser(browser_id: str):
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
"http://localhost:11235/monitor/actions/kill_browser",
|
||||
json={"browser_id": browser_id}
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
print(f"✅ Browser {browser_id} killed successfully")
|
||||
```
|
||||
|
||||
### Reset Statistics
|
||||
|
||||
```python
|
||||
async def reset_stats():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/monitor/stats/reset")
|
||||
print("📊 Statistics reset for fresh monitoring")
|
||||
```
|
||||
|
||||
## 📈 Production Integration Patterns
|
||||
|
||||
### Prometheus Integration
|
||||
|
||||
```python
|
||||
# Export metrics for Prometheus scraping
|
||||
async def export_prometheus_metrics():
|
||||
async with httpx.AsyncClient() as client:
|
||||
health = await client.get("http://localhost:11235/monitor/health")
|
||||
data = health.json()
|
||||
|
||||
# Export in Prometheus format
|
||||
metrics = f"""
|
||||
# HELP crawl4ai_memory_usage_percent Memory usage percentage
|
||||
# TYPE crawl4ai_memory_usage_percent gauge
|
||||
crawl4ai_memory_usage_percent {data['container']['memory_percent']}
|
||||
|
||||
# HELP crawl4ai_request_success_rate Request success rate
|
||||
# TYPE crawl4ai_request_success_rate gauge
|
||||
crawl4ai_request_success_rate {data['stats']['success_rate_percent']}
|
||||
|
||||
# HELP crawl4ai_browser_pool_count Total browsers in pool
|
||||
# TYPE crawl4ai_browser_pool_count gauge
|
||||
crawl4ai_browser_pool_count {data['pool']['permanent']['active'] + data['pool']['hot']['count'] + data['pool']['cold']['count']}
|
||||
"""
|
||||
return metrics
|
||||
```
|
||||
|
||||
### Alerting Example
|
||||
|
||||
```python
|
||||
async def check_alerts():
|
||||
async with httpx.AsyncClient() as client:
|
||||
health = await client.get("http://localhost:11235/monitor/health")
|
||||
data = health.json()
|
||||
|
||||
# Memory alert
|
||||
if data['container']['memory_percent'] > 80:
|
||||
print("🚨 ALERT: Memory usage above 80%")
|
||||
# Trigger cleanup
|
||||
await client.post("http://localhost:11235/monitor/actions/cleanup")
|
||||
|
||||
# Success rate alert
|
||||
if data['stats']['success_rate_percent'] < 90:
|
||||
print("🚨 ALERT: Success rate below 90%")
|
||||
# Check error logs
|
||||
errors = await client.get("http://localhost:11235/monitor/logs/errors")
|
||||
print(f"Recent errors: {len(errors.json())}")
|
||||
|
||||
# Latency alert
|
||||
if data['stats']['avg_latency_ms'] > 5000:
|
||||
print("🚨 ALERT: Average latency above 5s")
|
||||
```
|
||||
|
||||
### Key Metrics to Track
|
||||
|
||||
```python
|
||||
CRITICAL_METRICS = {
|
||||
"memory_usage": {
|
||||
"current": "container.memory_percent",
|
||||
"target": "<80%",
|
||||
"alert_threshold": ">80%",
|
||||
"action": "Force cleanup or scale"
|
||||
},
|
||||
"success_rate": {
|
||||
"current": "stats.success_rate_percent",
|
||||
"target": ">95%",
|
||||
"alert_threshold": "<90%",
|
||||
"action": "Check error logs"
|
||||
},
|
||||
"avg_latency": {
|
||||
"current": "stats.avg_latency_ms",
|
||||
"target": "<2000ms",
|
||||
"alert_threshold": ">5000ms",
|
||||
"action": "Investigate slow requests"
|
||||
},
|
||||
"browser_reuse_rate": {
|
||||
"current": "browsers.summary.reuse_rate_percent",
|
||||
"target": ">80%",
|
||||
"alert_threshold": "<60%",
|
||||
"action": "Check pool configuration"
|
||||
},
|
||||
"total_browsers": {
|
||||
"current": "browsers.summary.total_count",
|
||||
"target": "<15",
|
||||
"alert_threshold": ">20",
|
||||
"action": "Check for browser leaks"
|
||||
},
|
||||
"error_frequency": {
|
||||
"current": "len(errors)",
|
||||
"target": "<5/hour",
|
||||
"alert_threshold": ">10/hour",
|
||||
"action": "Review error patterns"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🐛 Critical Bug Fixes
|
||||
|
||||
This release includes significant bug fixes that improve stability and performance:
|
||||
|
||||
### Async LLM Extraction (#1590)
|
||||
|
||||
**The Problem:** LLM extraction was blocking async execution, causing URLs to be processed sequentially instead of in parallel (issue #1055).
|
||||
|
||||
**The Fix:** Resolved the blocking issue to enable true parallel processing for LLM extraction.
|
||||
|
||||
```python
|
||||
# Before v0.7.7: Sequential processing
|
||||
# After v0.7.7: True parallel processing
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
urls = ["url1", "url2", "url3", "url4"]
|
||||
|
||||
# Now processes truly in parallel with LLM extraction
|
||||
results = await crawler.arun_many(
|
||||
urls,
|
||||
config=CrawlerRunConfig(
|
||||
extraction_strategy=LLMExtractionStrategy(...)
|
||||
)
|
||||
)
|
||||
# 4x faster for parallel LLM extraction!
|
||||
```
|
||||
|
||||
**Expected Impact:** Major performance improvement for batch LLM extraction workflows.
|
||||
|
||||
### DFS Deep Crawling (#1607)
|
||||
|
||||
**The Problem:** DFS (Depth-First Search) deep crawl strategy had implementation issues.
|
||||
|
||||
**The Fix:** Enhanced DFSDeepCrawlStrategy with proper seen URL tracking and improved documentation.
|
||||
|
||||
### Browser & Crawler Config Documentation (#1609)
|
||||
|
||||
**The Problem:** Documentation didn't match the actual `async_configs.py` implementation.
|
||||
|
||||
**The Fix:** Updated all configuration documentation to accurately reflect the current implementation.
|
||||
|
||||
### Sitemap Seeder (#1598)
|
||||
|
||||
**The Problem:** Sitemap parsing and URL normalization issues in AsyncUrlSeeder (issue #1559).
|
||||
|
||||
**The Fix:** Added comprehensive tests and fixes for sitemap namespace parsing and URL normalization.
|
||||
|
||||
### Remove Overlay Elements (#1529)
|
||||
|
||||
**The Problem:** The `remove_overlay_elements` functionality wasn't working (issue #1396).
|
||||
|
||||
**The Fix:** Fixed by properly calling the injected JavaScript function.
|
||||
|
||||
### Viewport Configuration (#1495)
|
||||
|
||||
**The Problem:** Viewport configuration wasn't working in managed browsers (issue #1490).
|
||||
|
||||
**The Fix:** Added proper viewport size configuration support for browser launch.
|
||||
|
||||
### Managed Browser CDP Timing (#1528)
|
||||
|
||||
**The Problem:** CDP (Chrome DevTools Protocol) endpoint verification had timing issues causing connection failures (issue #1445).
|
||||
|
||||
**The Fix:** Added exponential backoff for CDP endpoint verification to handle timing variations.
|
||||
|
||||
### Security Updates
|
||||
|
||||
- **pyOpenSSL**: Updated from >=24.3.0 to >=25.3.0 to address security vulnerability
|
||||
- Added verification tests for the security update
|
||||
|
||||
### Docker Fixes
|
||||
|
||||
- **Port Standardization**: Fixed inconsistent port usage (11234 vs 11235) - now standardized to 11235
|
||||
- **LLM Environment**: Fixed LLM API key handling for multi-provider support (PR #1537)
|
||||
- **Error Handling**: Improved Docker API error messages with comprehensive status codes
|
||||
- **Serialization**: Fixed `fit_html` property serialization in `/crawl` and `/crawl/stream` endpoints
|
||||
|
||||
### Other Important Fixes
|
||||
|
||||
- **arun_many Returns**: Fixed function to always return a list, even on exception (PR #1530)
|
||||
- **Webhook Serialization**: Properly serialize Pydantic HttpUrl in webhook config
|
||||
- **LLMConfig Documentation**: Fixed casing and variable name consistency (issue #1551)
|
||||
- **Python Version**: Dropped Python 3.9 support, now requires Python >=3.10
|
||||
|
||||
## 📊 Expected Real-World Impact
|
||||
|
||||
### For DevOps & Infrastructure Teams
|
||||
- **Full Visibility**: Know exactly what's happening inside your crawling infrastructure
|
||||
- **Proactive Monitoring**: Catch issues before they become problems
|
||||
- **Resource Optimization**: Identify memory leaks and performance bottlenecks
|
||||
- **Operational Control**: Manual intervention when automated systems need help
|
||||
|
||||
### For Production Deployments
|
||||
- **Enterprise Observability**: Prometheus, Grafana, and alerting integration
|
||||
- **Debugging**: Real-time logs and error tracking
|
||||
- **Capacity Planning**: Historical metrics for scaling decisions
|
||||
- **SLA Monitoring**: Track success rates and latency against targets
|
||||
|
||||
### For Development Teams
|
||||
- **Local Monitoring**: Understand crawler behavior during development
|
||||
- **Performance Testing**: Measure impact of configuration changes
|
||||
- **Troubleshooting**: Quickly identify and fix issues
|
||||
- **Learning**: See exactly how the browser pool works
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- All existing Docker configurations continue to work
|
||||
- No API changes to existing endpoints
|
||||
- Monitoring is additive functionality
|
||||
- No migration required
|
||||
|
||||
## 🚀 Upgrade Instructions
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.7
|
||||
|
||||
# Or use the latest tag
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run with monitoring enabled (default)
|
||||
docker run -d \
|
||||
-p 11235:11235 \
|
||||
--shm-size=1g \
|
||||
--name crawl4ai \
|
||||
unclecode/crawl4ai:0.7.7
|
||||
|
||||
# Access the monitoring dashboard
|
||||
open http://localhost:11235/dashboard
|
||||
```
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
# Upgrade to latest version
|
||||
pip install --upgrade crawl4ai
|
||||
|
||||
# Or install specific version
|
||||
pip install crawl4ai==0.7.7
|
||||
```
|
||||
|
||||
## 🎬 Try the Demo
|
||||
|
||||
Run the comprehensive demo that showcases all monitoring features:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.7.py
|
||||
```
|
||||
|
||||
**The demo includes:**
|
||||
1. System health overview with live metrics
|
||||
2. Request tracking with active/completed monitoring
|
||||
3. Browser pool management (permanent/hot/cold)
|
||||
4. Complete Monitor API endpoint examples
|
||||
5. WebSocket streaming demonstration
|
||||
6. Control actions (cleanup, kill, restart)
|
||||
7. Production metrics and alerting patterns
|
||||
8. Self-hosting value proposition
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### New Documentation
|
||||
- **[Self-Hosting Guide](https://docs.crawl4ai.com/core/self-hosting/)** - Complete self-hosting documentation with monitoring
|
||||
- **Demo Script**: `docs/releases_review/demo_v0.7.7.py` - Working examples
|
||||
|
||||
### Updated Documentation
|
||||
- **Docker Deployment** → **Self-Hosting** (renamed for better positioning)
|
||||
- Added comprehensive monitoring sections
|
||||
- Production integration patterns
|
||||
- WebSocket streaming examples
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Start with the dashboard** - Visit `/dashboard` to get familiar with the monitoring system
|
||||
2. **Track the 6 key metrics** - Memory, success rate, latency, reuse rate, browser count, errors
|
||||
3. **Set up alerting early** - Use the Monitor API to build alerts before issues occur
|
||||
4. **Monitor browser pool efficiency** - Aim for >80% reuse rate for optimal performance
|
||||
5. **Use WebSocket for custom dashboards** - Build tailored monitoring UIs for your team
|
||||
6. **Leverage Prometheus integration** - Export metrics for long-term storage and analysis
|
||||
7. **Check janitor logs** - Understand automatic cleanup patterns
|
||||
8. **Use control actions judiciously** - Manual interventions are for exceptional cases
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.
|
||||
|
||||
The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.
|
||||
|
||||
## 📞 Support & Resources
|
||||
|
||||
- **📖 Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **🐙 GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **💬 Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **🐦 Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
- **📊 Dashboard**: `http://localhost:11235/dashboard` (when running)
|
||||
|
||||
---
|
||||
|
||||
**Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platform—it's a game changer for operational excellence!**
|
||||
|
||||
**Happy crawling with full visibility!** 🕷️📊
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,327 @@
|
||||
# Crawl4AI v0.7.8: Stability & Bug Fix Release
|
||||
|
||||
*December 2025*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.7.8—a focused stability release that addresses 11 bugs reported by the community. While there are no new features in this release, these fixes resolve important issues affecting Docker deployments, LLM extraction, URL handling, and dependency compatibility.
|
||||
|
||||
## What's Fixed at a Glance
|
||||
|
||||
- **Docker API**: Fixed ContentRelevanceFilter deserialization, ProxyConfig serialization, and cache folder permissions
|
||||
- **LLM Extraction**: Configurable rate limiter backoff, HTML input format support, and proper URL handling for raw HTML
|
||||
- **URL Handling**: Correct relative URL resolution after JavaScript redirects
|
||||
- **Dependencies**: Replaced deprecated PyPDF2 with pypdf, Pydantic v2 ConfigDict compatibility
|
||||
- **AdaptiveCrawler**: Fixed query expansion to actually use LLM instead of hardcoded mock data
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### Docker & API Fixes
|
||||
|
||||
#### ContentRelevanceFilter Deserialization (#1642)
|
||||
|
||||
**The Problem:** When sending deep crawl requests to the Docker API with `ContentRelevanceFilter`, the server failed to deserialize the filter, causing requests to fail.
|
||||
|
||||
**The Fix:** I added `ContentRelevanceFilter` to the public exports and enhanced the deserialization logic with dynamic imports.
|
||||
|
||||
```python
|
||||
# This now works correctly in Docker API
|
||||
import httpx
|
||||
|
||||
request = {
|
||||
"urls": ["https://docs.example.com"],
|
||||
"crawler_config": {
|
||||
"deep_crawl_strategy": {
|
||||
"type": "BFSDeepCrawlStrategy",
|
||||
"max_depth": 2,
|
||||
"filter_chain": [
|
||||
{
|
||||
"type": "ContentRelevanceFilter",
|
||||
"query": "API documentation",
|
||||
"threshold": 0.3
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/crawl", json=request)
|
||||
# Previously failed, now works!
|
||||
```
|
||||
|
||||
#### ProxyConfig JSON Serialization (#1629)
|
||||
|
||||
**The Problem:** `BrowserConfig.to_dict()` failed when `proxy_config` was set because `ProxyConfig` wasn't being serialized to a dictionary.
|
||||
|
||||
**The Fix:** `ProxyConfig.to_dict()` is now called during serialization.
|
||||
|
||||
```python
|
||||
from crawl4ai import BrowserConfig
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
proxy = ProxyConfig(
|
||||
server="http://proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass"
|
||||
)
|
||||
|
||||
config = BrowserConfig(headless=True, proxy_config=proxy)
|
||||
|
||||
# Previously raised TypeError, now works
|
||||
config_dict = config.to_dict()
|
||||
json.dumps(config_dict) # Valid JSON
|
||||
```
|
||||
|
||||
#### Docker Cache Folder Permissions (#1638)
|
||||
|
||||
**The Problem:** The `.cache` folder in the Docker image had incorrect permissions, causing crawling to fail when caching was enabled.
|
||||
|
||||
**The Fix:** Corrected ownership and permissions during image build.
|
||||
|
||||
```bash
|
||||
# Cache now works correctly in Docker
|
||||
docker run -d -p 11235:11235 \
|
||||
--shm-size=1g \
|
||||
-v ./my-cache:/app/.cache \
|
||||
unclecode/crawl4ai:0.7.8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### LLM & Extraction Fixes
|
||||
|
||||
#### Configurable Rate Limiter Backoff (#1269)
|
||||
|
||||
**The Problem:** The LLM rate limiting backoff parameters were hardcoded, making it impossible to adjust retry behavior for different API rate limits.
|
||||
|
||||
**The Fix:** `LLMConfig` now accepts three new parameters for complete control over retry behavior.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMConfig
|
||||
|
||||
# Default behavior (unchanged)
|
||||
default_config = LLMConfig(provider="openai/gpt-4o-mini")
|
||||
# backoff_base_delay=2, backoff_max_attempts=3, backoff_exponential_factor=2
|
||||
|
||||
# Custom configuration for APIs with strict rate limits
|
||||
custom_config = LLMConfig(
|
||||
provider="openai/gpt-4o-mini",
|
||||
backoff_base_delay=5, # Wait 5 seconds on first retry
|
||||
backoff_max_attempts=5, # Try up to 5 times
|
||||
backoff_exponential_factor=3 # Multiply delay by 3 each attempt
|
||||
)
|
||||
|
||||
# Retry sequence: 5s -> 15s -> 45s -> 135s -> 405s
|
||||
```
|
||||
|
||||
#### LLM Strategy HTML Input Support (#1178)
|
||||
|
||||
**The Problem:** `LLMExtractionStrategy` always sent markdown to the LLM, but some extraction tasks work better with HTML structure preserved.
|
||||
|
||||
**The Fix:** Added `input_format` parameter supporting `"markdown"`, `"html"`, `"fit_markdown"`, `"cleaned_html"`, and `"fit_html"`.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMExtractionStrategy, LLMConfig
|
||||
|
||||
# Default: markdown input (unchanged)
|
||||
markdown_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Extract product information"
|
||||
)
|
||||
|
||||
# NEW: HTML input - preserves table/list structure
|
||||
html_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Extract the data table preserving structure",
|
||||
input_format="html"
|
||||
)
|
||||
|
||||
# NEW: Filtered markdown - only relevant content
|
||||
fit_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Summarize the main content",
|
||||
input_format="fit_markdown"
|
||||
)
|
||||
```
|
||||
|
||||
#### Raw HTML URL Variable (#1116)
|
||||
|
||||
**The Problem:** When using `url="raw:<html>..."`, the entire HTML content was being passed to extraction strategies as the URL parameter, polluting LLM prompts.
|
||||
|
||||
**The Fix:** The URL is now correctly set to `"Raw HTML"` for raw HTML inputs.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
html = "<html><body><h1>Test</h1></body></html>"
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url=f"raw:{html}",
|
||||
config=CrawlerRunConfig(extraction_strategy=my_strategy)
|
||||
)
|
||||
# extraction_strategy receives url="Raw HTML" instead of the HTML blob
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### URL Handling Fix
|
||||
|
||||
#### Relative URLs After Redirects (#1268)
|
||||
|
||||
**The Problem:** When JavaScript caused a page redirect, relative links were resolved against the original URL instead of the final URL.
|
||||
|
||||
**The Fix:** `redirected_url` now captures the actual page URL after all JavaScript execution completes.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Page at /old-page redirects via JS to /new-page
|
||||
result = await crawler.arun(url="https://example.com/old-page")
|
||||
|
||||
# BEFORE: redirected_url = "https://example.com/old-page"
|
||||
# AFTER: redirected_url = "https://example.com/new-page"
|
||||
|
||||
# Links are now correctly resolved against the final URL
|
||||
for link in result.links['internal']:
|
||||
print(link['href']) # Relative links resolved correctly
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Dependency & Compatibility Fixes
|
||||
|
||||
#### PyPDF2 Replaced with pypdf (#1412)
|
||||
|
||||
**The Problem:** PyPDF2 was deprecated in 2022 and is no longer maintained.
|
||||
|
||||
**The Fix:** Replaced with the actively maintained `pypdf` library.
|
||||
|
||||
```python
|
||||
# Installation (unchanged)
|
||||
pip install crawl4ai[pdf]
|
||||
|
||||
# The PDF processor now uses pypdf internally
|
||||
# No code changes required - API remains the same
|
||||
```
|
||||
|
||||
#### Pydantic v2 ConfigDict Compatibility (#678)
|
||||
|
||||
**The Problem:** Using the deprecated `class Config` syntax caused deprecation warnings with Pydantic v2.
|
||||
|
||||
**The Fix:** Migrated to `model_config = ConfigDict(...)` syntax.
|
||||
|
||||
```python
|
||||
# No more deprecation warnings when importing crawl4ai models
|
||||
from crawl4ai.models import CrawlResult
|
||||
from crawl4ai import CrawlerRunConfig, BrowserConfig
|
||||
|
||||
# All models are now Pydantic v2 compatible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### AdaptiveCrawler Fix
|
||||
|
||||
#### Query Expansion Using LLM (#1621)
|
||||
|
||||
**The Problem:** The `EmbeddingStrategy` in AdaptiveCrawler had commented-out LLM code and was using hardcoded mock query variations instead.
|
||||
|
||||
**The Fix:** Uncommented and activated the LLM call for actual query expansion.
|
||||
|
||||
```python
|
||||
# AdaptiveCrawler query expansion now actually uses the LLM
|
||||
# Instead of hardcoded variations like:
|
||||
# variations = {'queries': ['what are the best vegetables...']}
|
||||
|
||||
# The LLM generates relevant query variations based on your actual query
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Code Formatting Fix
|
||||
|
||||
#### Import Statement Formatting (#1181)
|
||||
|
||||
**The Problem:** When extracting code from web pages, import statements were sometimes concatenated without proper line separation.
|
||||
|
||||
**The Fix:** Import statements now maintain proper newline separation.
|
||||
|
||||
```python
|
||||
# BEFORE: "import osimport sysfrom pathlib import Path"
|
||||
# AFTER:
|
||||
# import os
|
||||
# import sys
|
||||
# from pathlib import Path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- All existing code continues to work without modification
|
||||
- New parameters have sensible defaults matching previous behavior
|
||||
- No API changes to existing functionality
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Instructions
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
# or
|
||||
pip install crawl4ai==0.7.8
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.8
|
||||
|
||||
# Run
|
||||
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.7.8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Run the verification tests to confirm all fixes are working:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.8.py
|
||||
```
|
||||
|
||||
This runs actual tests that verify each bug fix is properly implemented.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thank you to everyone who reported these issues and provided detailed reproduction steps. Your bug reports make Crawl4AI better for everyone.
|
||||
|
||||
Issues fixed: #1642, #1638, #1629, #1621, #1412, #1269, #1268, #1181, #1178, #1116, #678
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
---
|
||||
|
||||
**This stability release ensures Crawl4AI works reliably across Docker deployments, LLM extraction workflows, and various edge cases. Thank you for your continued support and feedback!**
|
||||
|
||||
**Happy crawling!**
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,243 @@
|
||||
# Crawl4AI v0.8.0 Release Notes
|
||||
|
||||
**Release Date**: January 2026
|
||||
**Previous Version**: v0.7.6
|
||||
**Status**: Release Candidate
|
||||
|
||||
---
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Critical Security Fixes** for Docker API deployment
|
||||
- **11 New Features** including crash recovery, prefetch mode, and proxy improvements
|
||||
- **Breaking Changes** - see migration guide below
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### 1. Docker API: Hooks Disabled by Default
|
||||
|
||||
**What changed**: Hooks are now disabled by default on the Docker API.
|
||||
|
||||
**Why**: Security fix for Remote Code Execution (RCE) vulnerability.
|
||||
|
||||
**Who is affected**: Users of the Docker API who use the `hooks` parameter in `/crawl` requests.
|
||||
|
||||
**Migration**:
|
||||
```bash
|
||||
# To re-enable hooks (only if you trust all API users):
|
||||
export CRAWL4AI_HOOKS_ENABLED=true
|
||||
```
|
||||
|
||||
### 2. Docker API: file:// URLs Blocked
|
||||
|
||||
**What changed**: The endpoints `/execute_js`, `/screenshot`, `/pdf`, and `/html` now reject `file://` URLs.
|
||||
|
||||
**Why**: Security fix for Local File Inclusion (LFI) vulnerability.
|
||||
|
||||
**Who is affected**: Users who were reading local files via the Docker API.
|
||||
|
||||
**Migration**: Use the Python library directly for local file processing:
|
||||
```python
|
||||
# Instead of API call with file:// URL, use library:
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="file:///path/to/file.html")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: Remote Code Execution via Hooks (CVE Pending)
|
||||
|
||||
**Severity**: CRITICAL (CVSS 10.0)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /crawl` with malicious `hooks` parameter
|
||||
|
||||
**Details**: The `__import__` builtin was available in hook code, allowing attackers to import `os`, `subprocess`, etc. and execute arbitrary commands.
|
||||
|
||||
**Fix**:
|
||||
1. Removed `__import__` from allowed builtins
|
||||
2. Hooks disabled by default (`CRAWL4AI_HOOKS_ENABLED=false`)
|
||||
|
||||
### High: Local File Inclusion via file:// URLs (CVE Pending)
|
||||
|
||||
**Severity**: HIGH (CVSS 8.6)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /execute_js` (and other endpoints) with `file:///etc/passwd`
|
||||
|
||||
**Details**: API endpoints accepted `file://` URLs, allowing attackers to read arbitrary files from the server.
|
||||
|
||||
**Fix**: URL scheme validation now only allows `http://`, `https://`, and `raw:` URLs.
|
||||
|
||||
### Credits
|
||||
|
||||
Discovered by **Neo by ProjectDiscovery** ([projectdiscovery.io](https://projectdiscovery.io)) - December 2025
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. init_scripts Support for BrowserConfig
|
||||
|
||||
Pre-page-load JavaScript injection for stealth evasions.
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
init_scripts=[
|
||||
"Object.defineProperty(navigator, 'webdriver', {get: () => false})"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### 2. CDP Connection Improvements
|
||||
|
||||
- WebSocket URL support (`ws://`, `wss://`)
|
||||
- Proper cleanup with `cdp_cleanup_on_close=True`
|
||||
- Browser reuse across multiple connections
|
||||
|
||||
### 3. Crash Recovery for Deep Crawl Strategies
|
||||
|
||||
All deep crawl strategies (BFS, DFS, Best-First) now support crash recovery:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
|
||||
|
||||
strategy = BFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
resume_state=saved_state, # Resume from checkpoint
|
||||
on_state_change=save_callback # Persist state in real-time
|
||||
)
|
||||
```
|
||||
|
||||
### 4. PDF and MHTML for raw:/file:// URLs
|
||||
|
||||
Generate PDFs and MHTML from cached HTML content.
|
||||
|
||||
### 5. Screenshots for raw:/file:// URLs
|
||||
|
||||
Render cached HTML and capture screenshots.
|
||||
|
||||
### 6. base_url Parameter for CrawlerRunConfig
|
||||
|
||||
Proper URL resolution for raw: HTML processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(base_url='https://example.com')
|
||||
result = await crawler.arun(url='raw:{html}', config=config)
|
||||
```
|
||||
|
||||
### 7. Prefetch Mode for Two-Phase Deep Crawling
|
||||
|
||||
Fast link extraction without full page processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(prefetch=True)
|
||||
```
|
||||
|
||||
### 8. Proxy Rotation and Configuration
|
||||
|
||||
Enhanced proxy rotation with sticky sessions support.
|
||||
|
||||
### 9. Proxy Support for HTTP Strategy
|
||||
|
||||
Non-browser crawler now supports proxies.
|
||||
|
||||
### 10. Browser Pipeline for raw:/file:// URLs
|
||||
|
||||
New `process_in_browser` parameter for browser operations on local content:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
process_in_browser=True, # Force browser processing
|
||||
screenshot=True
|
||||
)
|
||||
result = await crawler.arun(url='raw:<html>...</html>', config=config)
|
||||
```
|
||||
|
||||
### 11. Smart TTL Cache for Sitemap URL Seeder
|
||||
|
||||
Intelligent cache invalidation for sitemaps:
|
||||
|
||||
```python
|
||||
config = SeedingConfig(
|
||||
cache_ttl_hours=24,
|
||||
validate_sitemap_lastmod=True
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### raw: URL Parsing Truncates at # Character
|
||||
|
||||
**Problem**: CSS color codes like `#eee` were being truncated.
|
||||
|
||||
**Before**: `raw:body{background:#eee}` → `body{background:`
|
||||
**After**: `raw:body{background:#eee}` → `body{background:#eee}`
|
||||
|
||||
### Caching System Improvements
|
||||
|
||||
Various fixes to cache validation and persistence.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- Multi-sample schema generation documentation
|
||||
- URL seeder smart TTL cache parameters
|
||||
- Security documentation (SECURITY.md)
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Guide
|
||||
|
||||
### From v0.7.x to v0.8.0
|
||||
|
||||
1. **Update the package**:
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
2. **Docker API users**:
|
||||
- Hooks are now disabled by default
|
||||
- If you need hooks: `export CRAWL4AI_HOOKS_ENABLED=true`
|
||||
- `file://` URLs no longer work on API (use library directly)
|
||||
|
||||
3. **Review security settings**:
|
||||
```yaml
|
||||
# config.yml - recommended for production
|
||||
security:
|
||||
enabled: true
|
||||
jwt_enabled: true
|
||||
```
|
||||
|
||||
4. **Test your integration** before deploying to production
|
||||
|
||||
### Breaking Change Checklist
|
||||
|
||||
- [ ] Check if you use `hooks` parameter in API calls
|
||||
- [ ] Check if you use `file://` URLs via the API
|
||||
- [ ] Update environment variables if needed
|
||||
- [ ] Review security configuration
|
||||
|
||||
---
|
||||
|
||||
## Full Changelog
|
||||
|
||||
See [CHANGELOG.md](../CHANGELOG.md) for complete version history.
|
||||
|
||||
---
|
||||
|
||||
## Contributors
|
||||
|
||||
Thanks to all contributors who made this release possible.
|
||||
|
||||
Special thanks to **Neo by ProjectDiscovery** for responsible security disclosure.
|
||||
|
||||
---
|
||||
|
||||
*For questions or issues, please open a [GitHub Issue](https://github.com/unclecode/crawl4ai/issues).*
|
||||
@@ -0,0 +1,384 @@
|
||||
# Crawl4AI v0.8.5: Anti-Bot, Shadow DOM & 60+ Bug Fixes
|
||||
|
||||
*March 2026 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.5—our biggest release since v0.8.0. This update brings automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes from both our team and the community. If you're running crawls at scale or dealing with protected sites, this one's for you.
|
||||
|
||||
## What's New at a Glance
|
||||
|
||||
- **Anti-Bot Detection & Proxy Escalation**: 3-tier detection with automatic retry, proxy chain, and fallback
|
||||
- **Shadow DOM Flattening**: Extract content hidden inside shadow DOM components
|
||||
- **Deep Crawl Cancellation**: Stop long crawls gracefully with `cancel()` or `should_cancel` callback
|
||||
- **Config Defaults API**: Set once, apply everywhere with `set_defaults()` / `get_defaults()` / `reset_defaults()`
|
||||
- **Source/Sibling Selector**: Extract data spanning sibling elements in JSON extraction schemas
|
||||
- **Consent Popup Removal**: Auto-dismiss cookie banners from 40+ CMP platforms
|
||||
- **Resource Filtering**: Block ads and CSS at the network level with `avoid_ads` / `avoid_css`
|
||||
- **Browser Recycling**: Memory-saving mode and automatic browser restart for long sessions
|
||||
- **GFM Table Compliance**: Proper `| col1 | col2 |` pipe delimiters in markdown output
|
||||
- **60+ Bug Fixes**: Security patches, browser stability, extraction accuracy, and more
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. Anti-Bot Detection, Retry & Fallback
|
||||
|
||||
This is the headline feature. Crawl4AI now automatically detects when a page is blocked by anti-bot protection and takes action—retrying with different proxies or falling back to an alternative fetch method.
|
||||
|
||||
The detection uses three tiers:
|
||||
- **Tier 1**: Known vendor patterns (Cloudflare, Akamai, DataDome, PerimeterX, etc.)
|
||||
- **Tier 2**: Generic block indicators on small pages
|
||||
- **Tier 3**: Structural integrity checks (empty shells, script-heavy pages with no content)
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
# Try direct first, then proxy on bot detection
|
||||
proxy_config=[
|
||||
ProxyConfig.DIRECT,
|
||||
ProxyConfig(server="http://my-proxy:8080"),
|
||||
],
|
||||
max_retries=2,
|
||||
# Optional: fallback when all proxies fail
|
||||
fallback_fetch_function=my_web_unlocker_function,
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://protected-site.com", config=config)
|
||||
|
||||
# Check what happened
|
||||
stats = result.crawl_stats
|
||||
print(f"Resolved by: {stats['resolved_by']}") # "direct", "proxy", or "fallback_fetch"
|
||||
print(f"Proxies tried: {len(stats['proxies_used'])}")
|
||||
```
|
||||
|
||||
The system errs on the side of caution—false positives are cheap (the fallback rescues them), but false negatives mean garbage results. After 5 iterations of real-world testing, it handles everything from Cloudflare challenges to Reddit's 180KB SPA block pages.
|
||||
|
||||
### 2. Shadow DOM Flattening
|
||||
|
||||
Web components with shadow DOM hide their content from regular DOM traversal. The new `flatten_shadow_dom` option serializes shadow DOM content into the light DOM before extraction.
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(flatten_shadow_dom=True)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://some-web-component-site.com", config=config)
|
||||
# Shadow DOM content is now visible in result.html, cleaned_html, and markdown
|
||||
```
|
||||
|
||||
The implementation patches `attachShadow` to force-open closed shadow roots, recursively resolves `<slot>` projections, and strips only shadow-scoped `<style>` tags. It also reorders the JS execution pipeline—`js_code` now runs after `wait_for` + `delay_before_return_html` so your scripts operate on the fully-hydrated page. If you need JS to run before waiting, use the new `js_code_before_wait` parameter.
|
||||
|
||||
### 3. Deep Crawl Cancellation
|
||||
|
||||
All deep crawl strategies (BFS, DFS, BestFirst) now support graceful cancellation:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import DFSDeepCrawlStrategy
|
||||
|
||||
pages_found = 0
|
||||
|
||||
def should_stop():
|
||||
return pages_found >= 50 # Stop after finding enough pages
|
||||
|
||||
async def on_state(state):
|
||||
nonlocal pages_found
|
||||
pages_found = state["pages_crawled"]
|
||||
|
||||
strategy = DFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
max_pages=1000,
|
||||
should_cancel=should_stop, # Sync or async callback
|
||||
on_state_change=on_state,
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(deep_crawl_strategy=strategy)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun("https://example.com", config=config)
|
||||
print(f"Cancelled: {strategy.cancelled}")
|
||||
```
|
||||
|
||||
You can also call `strategy.cancel()` directly from another thread or coroutine.
|
||||
|
||||
### 4. Config Defaults API
|
||||
|
||||
Tired of repeating the same parameters? Set defaults once and they apply to every new instance:
|
||||
|
||||
```python
|
||||
from crawl4ai import BrowserConfig, CrawlerRunConfig
|
||||
|
||||
# Set organization-wide defaults
|
||||
BrowserConfig.set_defaults(headless=True, text_mode=True)
|
||||
CrawlerRunConfig.set_defaults(verbose=False, remove_consent_popups=True)
|
||||
|
||||
# All new instances inherit defaults
|
||||
bc = BrowserConfig() # headless=True, text_mode=True
|
||||
rc = CrawlerRunConfig() # verbose=False, remove_consent_popups=True
|
||||
|
||||
# Explicit params always override
|
||||
bc2 = BrowserConfig(text_mode=False) # text_mode=False, headless still True
|
||||
|
||||
# Inspect and reset
|
||||
print(BrowserConfig.get_defaults()) # {"headless": True, "text_mode": True}
|
||||
BrowserConfig.reset_defaults() # Back to normal
|
||||
```
|
||||
|
||||
### 5. Source/Sibling Selector in JSON Extraction
|
||||
|
||||
Many sites split a single item's data across sibling elements (think Hacker News, where title and score are in separate `<tr>` rows). The new `"source"` field navigates to a sibling before extracting:
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
schema = {
|
||||
"name": "HackerNewsItems",
|
||||
"baseSelector": "tr.athing",
|
||||
"fields": [
|
||||
{"name": "title", "selector": ".titleline > a", "type": "text"},
|
||||
{"name": "link", "selector": ".titleline > a", "type": "attribute", "attribute": "href"},
|
||||
# Navigate to the NEXT sibling <tr> to get the score
|
||||
{"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
|
||||
{"name": "author", "selector": ".hnuser", "type": "text", "source": "+ tr"},
|
||||
]
|
||||
}
|
||||
|
||||
strategy = JsonCssExtractionStrategy(schema=schema)
|
||||
```
|
||||
|
||||
Works in both `JsonCssExtractionStrategy` and `JsonXPathExtractionStrategy`. Falls back gracefully when siblings don't exist.
|
||||
|
||||
### 6. Consent Popup Removal
|
||||
|
||||
A single flag auto-dismisses cookie consent banners from 40+ CMP platforms:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(remove_consent_popups=True)
|
||||
```
|
||||
|
||||
Covers OneTrust, Cookiebot, Didomi, Quantcast, Sourcepoint, Google FundingChoices, TrustArc, ConsentManager, Osano, Iubenda, Complianz, LiveRamp, CookieYes, Klaro, Termly, and many more.
|
||||
|
||||
### 7. Resource Filtering: avoid_ads / avoid_css
|
||||
|
||||
Block ad trackers and CSS resources at the network level for faster, leaner crawls:
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
avoid_ads=True, # Blocks doubleclick, google-analytics, etc.
|
||||
avoid_css=True, # Blocks .css, .less, .scss resources
|
||||
)
|
||||
```
|
||||
|
||||
### 8. Browser Recycling & Memory-Saving Mode
|
||||
|
||||
For long-running crawl sessions:
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
memory_saving_mode=True, # Aggressive cache/V8 heap flags
|
||||
max_pages_before_recycle=100, # Auto-restart browser after N pages
|
||||
)
|
||||
```
|
||||
|
||||
This prevents memory leaks during sustained crawling. The recycling uses a version-based approach that's safe under concurrent load—we fixed three separate deadlock bugs to get this right.
|
||||
|
||||
### 9. GFM Table Compliance
|
||||
|
||||
Tables in markdown output now have proper GitHub-Flavored Markdown pipe delimiters:
|
||||
|
||||
**Before (v0.8.0)**:
|
||||
```
|
||||
Name | Age | City
|
||||
---|---|---
|
||||
Alice | 30 | NYC
|
||||
```
|
||||
|
||||
**After (v0.8.5)**:
|
||||
```
|
||||
| Name | Age | City |
|
||||
| --- | --- | --- |
|
||||
| Alice | 30 | NYC |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Minor Features
|
||||
|
||||
- **`query_llm_config`**: Separate LLM config for adaptive crawler query expansion (#1682)
|
||||
- **`force_viewport_screenshot`**: Screenshot only the viewport, not the full page
|
||||
- **`device_scale_factor`**: Configurable screenshot DPI via BrowserConfig (#1463)
|
||||
- **`redirected_status_code`**: Now available on CrawlResult (#1435)
|
||||
- **`wait_for_images`**: Wait for images to load before taking screenshots (#1792)
|
||||
- **`score_threshold`**: Filter low-quality URLs in BestFirstCrawlingStrategy (#1804)
|
||||
- **`link_preview_timeout`**: Configurable timeout in AdaptiveConfig (#1793)
|
||||
- **`--json-ensure-ascii`**: CLI flag for Unicode preservation in JSON output (#1668)
|
||||
- **`type-list` pipeline**: Chained extraction like `["attribute", "regex"]` in JsonCssExtractionStrategy (#1290)
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: RCE via Deserialization in Docker /crawl Endpoint
|
||||
|
||||
**Severity**: CRITICAL
|
||||
**Affected**: Docker API deployment (v0.8.0 and earlier)
|
||||
|
||||
The `/crawl` endpoint's deserialization logic used `eval()` for certain object types. I removed this entirely and added an allowlist (`ALLOWED_DESERIALIZE_TYPES`) so only known config classes can be instantiated.
|
||||
|
||||
### Critical: Redis CVE-2025-49844 (CVSS 10.0)
|
||||
|
||||
**Affected**: Docker deployments using Redis
|
||||
|
||||
Upgraded Redis to 7.2.7 which patches the Lua use-after-free vulnerability.
|
||||
|
||||
### Additional Security
|
||||
|
||||
- **XSS prevention**: Use DOMParser instead of innerHTML in iframe processing (#1796)
|
||||
- **API token enforcement**: `/token` endpoint now requires `api_token` when configured (#1795)
|
||||
- **Stealth improvements**: `sec-ch-ua` synced with User-Agent, WebGL kept alive in stealth mode
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### Browser & Page Management
|
||||
|
||||
- Fix page reuse race condition when `create_isolated_context=False`
|
||||
- Fix browser context memory leak — signature shrink + LRU eviction (#943)
|
||||
- Fix cascading context crash from duplicate `add_init_script` (#1768)
|
||||
- Fix `simulate_user` destroying page content via ArrowDown keypress
|
||||
- Fix browser recycling deadlock under sustained concurrent load (#1640)
|
||||
- Fix Docker monitor LOCK contention causing pod deadlock (#1754)
|
||||
|
||||
### Proxy & Network
|
||||
|
||||
- Fix proxy auth `ERR_INVALID_AUTH_CREDENTIALS` (#1281)
|
||||
- Fix proxy auth for persistent browser contexts
|
||||
- Fix proxy escalation not re-raising on first exception when chain has alternatives
|
||||
- Fix fallback fetch: run when all proxies crash, skip re-check, never return None
|
||||
|
||||
### Deep Crawling
|
||||
|
||||
- Fix `can_process_url()` to receive normalized URL
|
||||
- Fix `total_score` not calculated for links that fail head extraction
|
||||
- Fix `FilterChain.add_filter` AttributeError on tuple immutability
|
||||
- Fix URL Seeder forcing Common Crawl index for sitemaps (#1746)
|
||||
- Fix `is_external_url` port comparison (#1783)
|
||||
- Prevent AdaptiveCrawler from crawling external domains (#1805)
|
||||
|
||||
### Extraction & Content
|
||||
|
||||
- Fix `<base>` tag ignored in html2text relative link resolution (#1721)
|
||||
- Fix script tag removal losing adjacent text in `cleaned_html` (#1364)
|
||||
- Preserve `class` and `id` attributes in `cleaned_html` (#1782)
|
||||
- Fix nested brackets/parentheses in LINK_PATTERN regex (#1790)
|
||||
- Strip markdown fences in `force_json_response` path for LLM extraction
|
||||
- Guard against None LLM content, propagate `finish_reason` (#1788)
|
||||
- Fix `agenerate_schema()` JSON parsing for Anthropic models
|
||||
- Fix `from_serializable_dict` ignoring plain data dicts with "type" key
|
||||
- Fix MediaItem crash on non-numeric width values like "100%" (#1635)
|
||||
- Fix BM25ContentFilter returning duplicate chunks (#1213)
|
||||
- Fix `css_selector` ignored in LXML scraping for `raw://` URLs (#1484)
|
||||
|
||||
### CLI & Docker
|
||||
|
||||
- Fix deep-crawl CLI outputting only the first page (#1667)
|
||||
- Fix VersionManager ignoring `CRAWL4_AI_BASE_DIRECTORY` env var (#1296)
|
||||
- Fix Docker health endpoint to use dynamic version (#1686)
|
||||
- Add explicit UTF-8 encoding to CLI file output (#1789)
|
||||
- Handle `UnicodeEncodeError` in URL seeder, strip zero-width chars (#1784)
|
||||
- Add TTL expiry for Redis task data to prevent memory growth (#1730)
|
||||
- Add Windows support for crawler monitor keyboard input (#1794)
|
||||
- Fix `scroll_delay` ignored in full-page screenshot scroller
|
||||
- Fix MCP SSE endpoint crash — mount via raw ASGI Route (#1594)
|
||||
- Fix `/llm` per-request provider override, Redis config from host/port/password (#1611, #1817)
|
||||
- Fix screenshot respects `scan_full_page=False` (#1750)
|
||||
- Fix screenshot distortion on Elementor sites (#1370)
|
||||
- Fix deep crawl timeout and `arun_many` dispatcher bypass (#1818, #1509)
|
||||
|
||||
### Other
|
||||
|
||||
- Replace `tf-playwright-stealth` with `playwright-stealth` (#1553)
|
||||
- Allow local embeddings by removing OpenAI fallback (#1658)
|
||||
- Include GoogleSearchCrawler `script.js` in package distribution (#1711)
|
||||
- Fix bs4 deprecation warning (`text` → `string`) (#1077)
|
||||
- Run blocking `chardet.detect` in thread executor (#1751)
|
||||
- Wire `mean_delay`/`max_range` from CrawlerRunConfig into dispatcher rate limiter (#1786)
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
Added a comprehensive **291-test regression suite** covering all major subsystems: core crawl, content processing, extraction strategies, deep crawling, browser management, config serialization, utilities, and edge cases.
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### `cleaned_html` Now Preserves `class` and `id` Attributes
|
||||
|
||||
If you have downstream code that parses `cleaned_html` and assumes no class/id attributes are present, this may need updating. This change enables users to do CSS-based analysis on cleaned HTML.
|
||||
|
||||
### Docker: Redis Upgraded to 7.2.7
|
||||
|
||||
If you pin Redis versions in your deployment, update to 7.2.7 or later.
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Instructions
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
# or
|
||||
pip install crawl4ai==0.8.5
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker pull unclecode/crawl4ai:0.8.5
|
||||
|
||||
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.8.5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Run the verification tests to confirm all features are working:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.8.5.py
|
||||
```
|
||||
|
||||
This runs 13 actual tests that crawl real URLs and verify each feature end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
This release includes contributions from a large number of community members. Thank you to everyone who submitted PRs, reported issues, and provided reproduction steps. Special thanks to all contributors listed in [CONTRIBUTORS.md](../CONTRIBUTORS.md).
|
||||
|
||||
Issues fixed: #462, #880, #943, #1031, #1077, #1183, #1213, #1251, #1281, #1290, #1296, #1308, #1354, #1364, #1370, #1374, #1424, #1435, #1463, #1484, #1487, #1489, #1494, #1503, #1509, #1512, #1520, #1553, #1594, #1601, #1606, #1611, #1622, #1635, #1640, #1658, #1666, #1667, #1668, #1671, #1682, #1686, #1711, #1715, #1716, #1721, #1730, #1731, #1746, #1750, #1751, #1754, #1758, #1762, #1768, #1770, #1776, #1782, #1783, #1784, #1786, #1788, #1789, #1790, #1792, #1793, #1794, #1795, #1796, #1797, #1801, #1803, #1804, #1805, #1815, #1817, #1818, #1824
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
---
|
||||
|
||||
**This is a massive release—10 new features, critical security patches, and 60+ bug fixes. Whether you're dealing with anti-bot protection, shadow DOM sites, or just want more reliable crawls at scale, v0.8.5 has you covered. Thank you for your continued support!**
|
||||
|
||||
**Happy crawling!**
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,113 @@
|
||||
# Crawl4AI v0.8.7: Security Hardening, DomainMapper & Community Fixes
|
||||
|
||||
*June 2026 - 7 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.7, a security-hardening release. It bundles every responsibly-disclosed vulnerability we patched since v0.8.6, adds the new DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes from our team and the community.
|
||||
|
||||
If you self-host the Docker API server, please upgrade right away. This release closes several critical issues, and two GitHub Security Advisories accompany it.
|
||||
|
||||
## What's New at a Glance
|
||||
|
||||
- **Security hardening**: critical RCE, SSRF, auth-bypass, file-write, XSS, and hardcoded-secret fixes across the Docker API
|
||||
- **DomainMapper**: comprehensive domain URL discovery with subdomain and per-source timeout controls
|
||||
- **arun_many config-list in the Docker API**: send per-URL configs in one request
|
||||
- **Markdown fidelity fixes**: mermaid SVG text, table rowspan/colspan, trailing text, sentence order
|
||||
- **Deep crawl and dispatcher fixes**: streaming ContextVar bug, semaphore wiring
|
||||
- **LLM and provider fixes**: Bedrock auth, schema-first extraction, table-extraction allowlist
|
||||
- **Logging and MCP fixes**: stderr by default, non-TTY width, CJK preservation
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
v0.8.7 is, first and foremost, a security release. Every issue below was reported responsibly by the community, and every reporter is credited in `SECURITY-CREDITS.md` and the published advisories.
|
||||
|
||||
### Remote Code Execution
|
||||
|
||||
- **AST sandbox escape (CVSS 9.8)**: a `gi_frame.f_back` frame-chain walk escaped the computed-field expression sandbox to reach the real `__import__`. We removed `eval()` from computed fields entirely. SDK users can still pass Python callables via the `function` key.
|
||||
- **Hook sandbox escape (CVSS 9.8)**: injected module objects (`asyncio`, `json`, `re`) carried a full `__builtins__`, providing an alternate path to `__import__`. We stripped those builtins and tightened the allowlist.
|
||||
|
||||
### Authentication and Secrets
|
||||
|
||||
- **Hardcoded JWT secret (CVSS 9.8)**: the signing key defaulted to `"mysecret"`. We removed the default, reject weak or short secrets at startup, and auto-generate an ephemeral key when JWT is enabled with no key set.
|
||||
- **Monitor endpoint auth bypass (CVSS 6.5)**: the `/monitor/*` routes, including destructive actions, ran without auth. They now require a token, and the WebSocket endpoint checks the token explicitly.
|
||||
|
||||
### SSRF
|
||||
|
||||
- **Webhook SSRF (CVSS 8.6)**: webhook URLs on `/crawl/job` and `/llm/job` could hit internal and cloud-metadata addresses. We added a blocklist and disabled redirect following.
|
||||
- **Direct crawl-endpoint SSRF (CVSS 8.6)**: `/crawl`, `/md`, and `/llm` fetched arbitrary URLs, and IPv6-mapped IPv4 addresses such as `[::ffff:169.254.169.254]` slipped past naive checks. We added destination validation on all entry points and normalize IPv6-mapped IPv4 before the blocklist check.
|
||||
|
||||
### File Write, XSS, and JS Execution
|
||||
|
||||
- **Arbitrary file write (CVSS 9.1)**: `/screenshot` and `/pdf` honored any `output_path`. Writes are now restricted to `CRAWL4AI_OUTPUT_DIR`, and `..` traversal is rejected.
|
||||
- **Stored XSS in the monitor dashboard (CVSS 6.1)**: crawled URLs were rendered via `innerHTML` without escaping. We escape on both the server and the client now.
|
||||
- **Arbitrary JS execution via `/execute_js` (CVSS 8.1)**: the endpoint is disabled by default behind `CRAWL4AI_EXECUTE_JS_ENABLED`, we removed `--disable-web-security` from default browser args, and added an SSRF blocklist on the destination.
|
||||
|
||||
We also replaced the `eval()` in `/config/dump` with Pydantic-validated JSON input, and added type validation for `markdown_generator` in `CrawlerRunConfig`.
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### DomainMapper
|
||||
|
||||
DomainMapper discovers URLs across an entire domain in one pass, combining multiple discovery sources. It supports an `include_subdomains` flag to widen or narrow the crawl boundary, and a per-source timeout so a single slow source cannot stall the whole map. See the [Domain Mapping guide](../core/domain-mapping.md).
|
||||
|
||||
### arun_many config-list in the Docker API
|
||||
|
||||
The Docker API now accepts a list of configs aligned with the list of URLs, so you can apply a different `CrawlerRunConfig` to each URL in a single `arun_many` request (#1837).
|
||||
|
||||
---
|
||||
|
||||
## Fixes
|
||||
|
||||
**Markdown and scraping**
|
||||
|
||||
- Preserve mermaid diagram text rendered as SVG, and prevent nested code fences (#1043)
|
||||
- Preserve table `rowspan` and `colspan` in cleaned HTML (#1920)
|
||||
- Preserve `.tail` text when removing empty elements (#1938)
|
||||
- Keep sentence order in `NlpSentenceChunking` (#1909)
|
||||
|
||||
**Deep crawl and dispatcher**
|
||||
|
||||
- Fix the deep-crawl streaming ContextVar bug by using `set(False)` instead of `reset(token)` (#1917)
|
||||
- Wire `semaphore_count` into the auto-created `MemoryAdaptiveDispatcher` and default it to 10 (#1927)
|
||||
|
||||
**LLM and providers**
|
||||
|
||||
- Add Bedrock to the provider prefixes so AWS credential auth works
|
||||
- Default `LLMExtractionStrategy.extraction_type` to schema
|
||||
- Add `LLMTableExtraction` to the Docker deserialization allowlist
|
||||
|
||||
**Crawler and downloads**
|
||||
|
||||
- Return `success=True` for binary downloads, and skip the block check when `downloaded_files` is set
|
||||
- Honor `<base href>` in prefetch `quick_extract_links` (#752)
|
||||
|
||||
**Logging and MCP**
|
||||
|
||||
- Route `AsyncLogger` output to stderr by default (#1968) and use `Console(width=200)` for non-TTY contexts
|
||||
- Use `ensure_ascii=False` in the MCP bridge to preserve CJK characters (#1967)
|
||||
|
||||
**Browser and misc**
|
||||
|
||||
- `browser_adapter` now uses the `Stealth` import, fixing a stealth import mismatch (#1960)
|
||||
- Correct the `arun()` return type to `CrawlResultContainer` (#1898)
|
||||
- Log the real failure reason before COMPLETE, fixing a misleading success line (#1949)
|
||||
- Assistant toolbar scroll fix and issue-1973 fix
|
||||
|
||||
---
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
```
|
||||
|
||||
Docker users should pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Security Credits
|
||||
|
||||
Thank you to the researchers who disclosed these issues responsibly: Song Binglin (q1uf3ng), by111 (August829), Jeongbean Jeon, wulonchia, secsys_codex, Velayutham Selvaraj, and IcySun. Full details are in `SECURITY-CREDITS.md`.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Crawl4AI v0.8.8: Docker Server Security Patch
|
||||
|
||||
*June 2026 - 3 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.8, a focused security patch for the self-hosted Docker API server. It is backward compatible: upgrade in place, no configuration changes required.
|
||||
|
||||
If you run the Docker server, please upgrade. If it is exposed to a network, also set `CRAWL4AI_API_TOKEN`. Security advisories accompany this release.
|
||||
|
||||
## What it fixes
|
||||
|
||||
- **SSRF filter gaps**: the SSRF protection now rejects any resolved address that is not globally routable, and it covers IPv6 transition forms that previously slipped past the blocklist (NAT64, 6to4, IPv4-mapped, and the unspecified `::`). These could otherwise reach internal services and cloud-metadata endpoints. Error messages no longer echo the resolved address.
|
||||
- **Arbitrary file write via `output_path`**: `/screenshot` and `/pdf` now resolve symlinks and re-check containment before writing, and write with `O_NOFOLLOW`, closing a symlink/TOCTOU bypass of the output-directory restriction. Normal use is unchanged.
|
||||
- **LLM credential exfiltration**: the LLM endpoints no longer honor a request-supplied `base_url`, so the configured provider key cannot be redirected to an attacker endpoint, and `LLMConfig` will not resolve protected environment variables via `env:`.
|
||||
- **Hardening**: CRLF-safe logging and webhook request-header validation.
|
||||
|
||||
All changes are backward compatible. Details and credits are in the security advisories.
|
||||
|
||||
## Coming next: a secure-by-default Docker server (~1-2 weeks)
|
||||
|
||||
The next release is a larger, secure-by-default update for the Docker API server, and it has intentional breaking changes. I want to give everyone time to prepare, so here is the heads-up.
|
||||
|
||||
If you run the Docker server, plan for these and test in staging before upgrading:
|
||||
|
||||
- **Authentication on by default.** The server binds loopback unless you configure a credential (`CRAWL4AI_API_TOKEN`). Put a TLS-terminating reverse proxy in front to expose it.
|
||||
- **Stricter request validation and safer defaults.** TLS verification on, tighter outbound egress controls, and declarative hook actions instead of inline code.
|
||||
- **A few request options move server-side.** `/screenshot` and `/pdf` return an artifact id instead of a file path, and the LLM endpoint is selected by provider name.
|
||||
- **Hardened container defaults.** Least-privilege compose, Redis authentication, loopback bind.
|
||||
|
||||
A full migration guide will go out with the pre-announcement on Discord and X. Watch those channels.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
# Docker
|
||||
docker pull unclecode/crawl4ai:0.8.8
|
||||
```
|
||||
|
||||
Thanks to everyone who reports issues responsibly. Star and use Crawl4AI: https://github.com/unclecode/crawl4ai
|
||||
|
||||
Live long and import crawl4ai
|
||||
@@ -0,0 +1,37 @@
|
||||
# Crawl4AI v0.8.9: Proxy SSRF Patch
|
||||
|
||||
*June 2026 - 2 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.9, a follow-up security patch for the self-hosted Docker API server. It closes a server-side request forgery path that v0.8.8 did not cover. It is backward compatible: upgrade in place, no configuration changes required.
|
||||
|
||||
If you run the Docker server, please upgrade. If it is exposed to a network, also set `CRAWL4AI_API_TOKEN`. A security advisory accompanies this release.
|
||||
|
||||
## What it fixes
|
||||
|
||||
The SSRF destination check validated the crawl target URL, but not the proxy address. An unauthenticated `/crawl`, `/crawl/stream`, or `/crawl/job` request could point a proxy at an internal IP and route the browser through it, reaching internal services and cloud-metadata endpoints, even with a perfectly valid crawl URL.
|
||||
|
||||
v0.8.9 validates every proxy destination with the same global-routability check before the browser is built:
|
||||
|
||||
- `browser_config.proxy_config.server`
|
||||
- `browser_config.proxy` (deprecated field)
|
||||
- `crawler_config.proxy_config.server`
|
||||
- proxy / DNS-redirecting flags in `extra_args` (`--proxy-server`, `--host-resolver-rules`, `--proxy-bypass-list`, `--proxy-pac-url`) are stripped
|
||||
|
||||
A legitimate public proxy still works. The only behavior change: set proxies through `proxy_config` (which is validated) rather than raw `extra_args` flags.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
docker pull unclecode/crawl4ai:0.8.9
|
||||
```
|
||||
|
||||
## Still coming: a secure-by-default Docker server (~1-2 weeks)
|
||||
|
||||
The next release remains a larger, secure-by-default update for the Docker API server, with intentional breaking changes (authentication on by default, stricter request validation, safer deployment defaults). A full migration guide will accompany the pre-announcement on Discord and X. This proxy fix is already part of that release; 0.8.9 simply brings it forward because it is an unauthenticated SSRF.
|
||||
|
||||
Thanks to Geo for the responsible disclosure.
|
||||
|
||||
Live long and import crawl4ai
|
||||
@@ -0,0 +1,78 @@
|
||||
# Crawl4AI v0.9.0: Secure-by-Default Docker Server
|
||||
|
||||
*June 2026 - 6 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.9.0, a major, secure-by-default release of the Crawl4AI Docker API server. This is the biggest change to the self-hosted HTTP server since we shipped it. It moves the out-of-the-box deployment from an open, trust-the-caller posture to a closed, hardened one with defense in depth.
|
||||
|
||||
This is a breaking release for the Docker server only. The core pip library (the SDK and in-process use) is unchanged. If you only `pip install crawl4ai` and drive it from Python, nothing here affects you and you can upgrade freely.
|
||||
|
||||
If you self-host the Docker API server, please read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) before you upgrade, and roll out behind a staging environment first.
|
||||
|
||||
## Why this release
|
||||
|
||||
Over the last few releases we patched a series of issues in the Docker server one at a time. 0.9.0 finishes the job by changing the architecture instead of patching behavior. The principle is simple: the server should be safe the moment you start it, and the network request body should be treated as untrusted input rather than a trusted control channel.
|
||||
|
||||
That means the permissive defaults are gone. Authentication is on by default. The server binds loopback unless you give it a token. The request body carries declarative options only. Everything that used to let a caller reach into browser internals or supply code now lives server-side, where the operator controls it.
|
||||
|
||||
## What changed at a glance
|
||||
|
||||
- **Auth on by default, loopback bind**: no unauthenticated API on `0.0.0.0`.
|
||||
- **Request trust boundary**: crawl request bodies are declarative and scalar; power fields are rejected at the network edge.
|
||||
- **Declarative hooks**: a fixed action set replaces request-supplied hook code.
|
||||
- **Artifact store**: `output_path` is gone; screenshots and PDFs return an artifact id you fetch with auth.
|
||||
- **Provider by name**: LLM endpoints select a provider by name, configured server-side.
|
||||
- **Hardened transport and infra**: TLS verification on, deny-by-default CORS, strict security headers, password-protected loopback-only Redis, a bounded job queue, and generic error responses with correlation ids.
|
||||
|
||||
## Hardening details
|
||||
|
||||
### Authentication and binding
|
||||
|
||||
The server no longer serves an unauthenticated API on `0.0.0.0`. With no token configured it binds `127.0.0.1` only and prints a one-off token at startup for local use. To expose it, set a token and put a TLS-terminating reverse proxy in front:
|
||||
|
||||
```bash
|
||||
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
Every request except `GET /health` then needs `Authorization: Bearer <token>`. WebSocket clients that cannot set headers may pass `?token=...`. The JWT implementation changed, so tokens from older versions are no longer valid; re-mint via `POST /token`.
|
||||
|
||||
### The request trust boundary
|
||||
|
||||
A crawl request body now carries declarative, scalar options only. Fields that previously let a caller drive browser internals or arbitrary code are rejected with HTTP 400 at the network boundary, including `js_code`, `c4a_script`, `proxy_config`, `extra_args`, `user_data_dir`, `cdp_url`, `cookies`, `headers`, `init_scripts`, `base_url`, `deep_crawl_strategy`, `simulate_user`, `magic`, and `process_in_browser`. Configure these server-side, or use the in-process SDK where you keep full control. Unknown fields are dropped, and timeouts, viewport, and scroll counts are clamped to safe maximums.
|
||||
|
||||
Request-supplied browser launch arguments (`browser_config.extra_args`) are part of this boundary and are now rejected, closing a Chromium launch-argument injection class.
|
||||
|
||||
### Declarative hooks
|
||||
|
||||
`hooks.code` (Python strings) is replaced by a fixed set of declarative actions: `block_resources`, `add_cookies`, `set_headers`, `scroll_to_bottom`, and `wait_for_timeout`. Call `GET /hooks/info` for the parameter schemas. Arbitrary hook code remains available in a self-hosted in-process build.
|
||||
|
||||
### Downloads, screenshots, and PDFs
|
||||
|
||||
Download sinks now confine writes with basename plus realpath plus `O_NOFOLLOW`, removing a path-traversal-to-file-write class. `output_path` is removed from `/screenshot` and `/pdf`; the server stores the result and returns an `artifact_id` plus a URL, which you fetch with authenticated `GET /artifacts/{artifact_id}` (artifacts have a TTL and a storage quota).
|
||||
|
||||
### SSRF on the streaming path
|
||||
|
||||
Destination validation now covers the streaming crawl handler. `/crawl/stream` and `/crawl` with `stream=true` validate the target and return HTTP 400 for disallowed destinations, matching the non-streaming handlers.
|
||||
|
||||
### Transport and infrastructure
|
||||
|
||||
TLS verification is on; self-signed or internal targets fail by default, with explicit escape hatches (`CRAWL4AI_ALLOW_INSECURE_TLS`, `CRAWL4AI_ALLOW_INTERNAL_URLS`) for trusted internal testing. CORS is deny-by-default; allowlist your frontend origin under `security.cors_allow_origins`. Redis runs in-container, loopback-only, password-protected, with its port no longer published. Background jobs run on a bounded queue, and request size, wall-clock, and per-principal concurrency are capped (all configurable, `0` = unbounded). 5xx responses return a generic body with a correlation id you can match in the logs.
|
||||
|
||||
## Migrating
|
||||
|
||||
How much you have to do scales with how much you drove through the API. A plain "crawl these URLs with a normal config" user only needs to set a token and re-issue tokens. Everything else applies only if you used that specific feature.
|
||||
|
||||
Read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) first, then follow `deploy/docker/SECURITY-VERIFY.md` for the deployment checklist.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
```
|
||||
|
||||
Docker users should pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Security Credits
|
||||
|
||||
Thank you to the researchers who disclosed these issues responsibly: Y4tacker, KOH Jun Sheng, and UDU_RisePho ([hoanggxyuuki](https://github.com/hoanggxyuuki)). Full details are in `SECURITY-CREDITS.md`.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Crawl4AI v0.9.1: Bug Fixes & PruningContentFilter Whitelist
|
||||
|
||||
*July 2026 - 3 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.9.1, a patch release that ships 12 bug fixes across Docker, browser, core, and extraction, plus one new feature for `PruningContentFilter`.
|
||||
|
||||
No breaking changes. If you're on v0.9.0, upgrade freely.
|
||||
|
||||
## What's new at a glance
|
||||
|
||||
- **PruningContentFilter whitelist**: New `preserve_classes` / `preserve_tags` parameters to protect specific elements from density-based pruning
|
||||
- **Windows browser fix**: Default `channel='chromium'` no longer crashes Playwright on Windows
|
||||
- **Docker hardening**: 6 fixes for auth gate, supervisord, redis, tmpfs, and FastAPI compatibility
|
||||
- **HTTP timeout fix**: `page_timeout` was passed in milliseconds to aiohttp (which expects seconds), effectively disabling timeouts in HTTP mode
|
||||
- **Dependency**: lxml ceiling widened to allow 6.x
|
||||
|
||||
## PruningContentFilter whitelist
|
||||
|
||||
PruningContentFilter's density-based scoring is great at stripping boilerplate, but it sometimes takes short metadata elements — author names, timestamps, attribution lines — along with it. The new `preserve_classes` and `preserve_tags` parameters let you whitelist specific CSS classes or HTML tags that should never be pruned, regardless of their density score.
|
||||
|
||||
```python
|
||||
from crawl4ai.content_filter_strategy import PruningContentFilter
|
||||
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
|
||||
|
||||
filter = PruningContentFilter(
|
||||
threshold=0.48,
|
||||
preserve_classes=["author", "byline", "dateline"],
|
||||
preserve_tags=["time", "address"],
|
||||
)
|
||||
generator = DefaultMarkdownGenerator(content_filter=filter)
|
||||
|
||||
config = CrawlerRunConfig(markdown_generator=generator)
|
||||
```
|
||||
|
||||
Whitelisted nodes skip scoring entirely. Default is empty sets — no behavior change for existing users. (#1900, thanks @hafezparast)
|
||||
|
||||
## Bug fixes
|
||||
|
||||
### Docker (6 fixes)
|
||||
|
||||
- **Auth gate UI**: Dashboard and playground now load when JWT auth is active. Token input bar added to both UIs; all fetch calls use `authFetch()` with Bearer token. API routes remain fail-closed. (#2037)
|
||||
- **Supervisord/Redis dirs**: Pidfile and RDB snapshot dirs moved to writable paths for read-only rootfs deployments. (#2047, thanks @TobiasWallura-xitaso)
|
||||
- **tmpfs writable**: Read-only tmpfs mounts made writable. (#2027, thanks @nightcityblade)
|
||||
- **FastAPI cap**: Pinned FastAPI below 0.137 to avoid compatibility breakage. (#2025, thanks @nightcityblade)
|
||||
- **Redis auth**: Rate-limit Redis storage now authenticates with the configured password. (#2040, thanks @harshmathurx)
|
||||
- **Posture tests updated**: Dashboard/playground moved to public UI assertions to match auth gate fix.
|
||||
|
||||
### Browser (2 fixes)
|
||||
|
||||
- **Windows channel crash**: `channel='chromium'` (the default) caused Playwright to look for a system Chrome install instead of the bundled binary, crashing on Windows with `TargetClosedError`. The default channel is no longer passed to Playwright. (#2051, thanks @fstark96)
|
||||
- **Context snapshot leak**: Browser contexts from snapshot are now properly closed. (#1999, thanks @nightcityblade)
|
||||
|
||||
### Core (2 fixes)
|
||||
|
||||
- **HTTP timeout**: `page_timeout` (60000ms) was passed directly to `aiohttp.ClientTimeout` which expects seconds, making the effective timeout 16.7 hours. Now correctly divided by 1000. (#1894, thanks @hafezparast)
|
||||
- **Best-first ordering**: Batch ordering in `BestFirstCrawlingStrategy` stabilized for deterministic crawl order. (#1998, thanks @nightcityblade)
|
||||
|
||||
### Extraction (1 fix)
|
||||
|
||||
- **Table attributes**: `html2text` now preserves all attributes on table tags when `bypass_tables` is enabled. (#2007)
|
||||
|
||||
### Dependencies (1 fix)
|
||||
|
||||
- **lxml 6.x**: Widened lxml ceiling from `<6` to `<7` so crawl4ai can co-install with packages requiring lxml 6.x (e.g. scrapling). (#2019)
|
||||
|
||||
### Housekeeping
|
||||
|
||||
- Removed dead `normalize_url` duplicates and accidental `adaptive_crawler` copy. (thanks @RajanChavada)
|
||||
- Sponsor logos hosted locally to fix broken GitHub rendering.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
crawl4ai-doctor # verify installation
|
||||
```
|
||||
|
||||
Docker users: pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thanks to the community contributors who made this release possible: @hafezparast (#1894, #1900), @nightcityblade (#1998, #1999, #2025, #2027), @fstark96 (#2051), @TobiasWallura-xitaso (#2047), @harshmathurx (#2040), @RajanChavada (#2042).
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- [Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Issues](https://github.com/unclecode/crawl4ai/issues)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
@@ -0,0 +1,51 @@
|
||||
### browser_manager.py
|
||||
|
||||
| Function | What it does |
|
||||
|---|---|
|
||||
| `ManagedBrowser.build_browser_flags` | Returns baseline Chromium CLI flags, disables GPU and sandbox, plugs locale, timezone, stealth tweaks, and any extras from `BrowserConfig`. |
|
||||
| `ManagedBrowser.__init__` | Stores config and logger, creates temp dir, preps internal state. |
|
||||
| `ManagedBrowser.start` | Spawns or connects to the Chromium process, returns its CDP endpoint plus the `subprocess.Popen` handle. |
|
||||
| `ManagedBrowser._initial_startup_check` | Pings the CDP endpoint once to be sure the browser is alive, raises if not. |
|
||||
| `ManagedBrowser._monitor_browser_process` | Async-loops on the subprocess, logs exits or crashes, restarts if policy allows. |
|
||||
| `ManagedBrowser._get_browser_path_WIP` | Old helper that maps OS + browser type to an executable path. |
|
||||
| `ManagedBrowser._get_browser_path` | Current helper, checks env vars, Playwright cache, and OS defaults for the real executable. |
|
||||
| `ManagedBrowser._get_browser_args` | Builds the final CLI arg list by merging user flags, stealth flags, and defaults. |
|
||||
| `ManagedBrowser.cleanup` | Terminates the browser, stops monitors, deletes the temp dir. |
|
||||
| `ManagedBrowser.create_profile` | Opens a visible browser so a human can log in, then zips the resulting user-data-dir to `~/.crawl4ai/profiles/<name>`. |
|
||||
| `ManagedBrowser.list_profiles` | Thin wrapper, now forwarded to `BrowserProfiler.list_profiles()`. |
|
||||
| `ManagedBrowser.delete_profile` | Thin wrapper, now forwarded to `BrowserProfiler.delete_profile()`. |
|
||||
| `BrowserManager.__init__` | Holds the global Playwright instance, browser handle, config signature cache, session map, and logger. |
|
||||
| `BrowserManager.start` | Boots the underlying `ManagedBrowser`, then spins up the default Playwright browser context with stealth patches. |
|
||||
| `BrowserManager._build_browser_args` | Translates `CrawlerRunConfig` (proxy, UA, timezone, headless flag, etc.) into Playwright `launch_args`. |
|
||||
| `BrowserManager.setup_context` | Applies locale, geolocation, permissions, cookies, and UA overrides on a fresh context. |
|
||||
| `BrowserManager.create_browser_context` | Internal helper that actually calls `browser.new_context(**options)` after running `setup_context`. |
|
||||
| `BrowserManager._make_config_signature` | Hashes the non-ephemeral parts of `CrawlerRunConfig` so contexts can be reused safely. |
|
||||
| `BrowserManager.get_page` | Returns a ready `Page` for a given session id, reusing an existing one or creating a new context/page, injects helper scripts, updates `last_used`. |
|
||||
| `BrowserManager.kill_session` | Force-closes a context/page for a session and removes it from the session map. |
|
||||
| `BrowserManager._cleanup_expired_sessions` | Periodic sweep that drops sessions idle longer than `ttl_seconds`. |
|
||||
| `BrowserManager.close` | Gracefully shuts down all contexts, the browser, Playwright, and background tasks. |
|
||||
|
||||
---
|
||||
|
||||
### browser_profiler.py
|
||||
|
||||
| Function | What it does |
|
||||
|---|---|
|
||||
| `BrowserProfiler.__init__` | Sets up profile folder paths, async logger, and signal handlers. |
|
||||
| `BrowserProfiler.create_profile` | Launches a visible browser with a new user-data-dir for manual login, on exit compresses and stores it as a named profile. |
|
||||
| `BrowserProfiler.cleanup_handler` | General SIGTERM/SIGINT cleanup wrapper that kills child processes. |
|
||||
| `BrowserProfiler.sigint_handler` | Handles Ctrl-C during an interactive session, makes sure the browser shuts down cleanly. |
|
||||
| `BrowserProfiler.listen_for_quit_command` | Async REPL that exits when the user types `q`. |
|
||||
| `BrowserProfiler.list_profiles` | Enumerates `~/.crawl4ai/profiles`, prints profile name, browser type, size, and last modified. |
|
||||
| `BrowserProfiler.get_profile_path` | Returns the absolute path of a profile given its name, or `None` if missing. |
|
||||
| `BrowserProfiler.delete_profile` | Removes a profile folder or a direct path from disk, with optional confirmation prompt. |
|
||||
| `BrowserProfiler.interactive_manager` | Text UI loop for listing, creating, deleting, or launching profiles. |
|
||||
| `BrowserProfiler.launch_standalone_browser` | Starts a non-headless Chromium with remote debugging enabled and keeps it alive for manual tests. |
|
||||
| `BrowserProfiler.get_cdp_json` | Pulls `/json/version` from a CDP endpoint and returns the parsed JSON. |
|
||||
| `BrowserProfiler.launch_builtin_browser` | Spawns a headless Chromium in the background, saves `{wsEndpoint, pid, started_at}` to `~/.crawl4ai/builtin_browser.json`. |
|
||||
| `BrowserProfiler.get_builtin_browser_info` | Reads that JSON file, verifies the PID, and returns browser status info. |
|
||||
| `BrowserProfiler._is_browser_running` | Cross-platform helper that checks if a PID is still alive. |
|
||||
| `BrowserProfiler.kill_builtin_browser` | Terminates the background builtin browser and removes its status file. |
|
||||
| `BrowserProfiler.get_builtin_browser_status` | Returns `{running: bool, wsEndpoint, pid, started_at}` for quick health checks. |
|
||||
|
||||
Let me know what you want to tweak or dive into next.
|
||||
@@ -0,0 +1,40 @@
|
||||
### `cli.py` command surface
|
||||
|
||||
| Command | Inputs / flags | What it does |
|
||||
|---|---|---|
|
||||
| **profiles** | *(none)* | Opens the interactive profile manager, lets you list, create, delete saved browser profiles that live in `~/.crawl4ai/profiles`. |
|
||||
| **browser status** | – | Prints whether the always-on *builtin* browser is running, shows its CDP URL, PID, start time. |
|
||||
| **browser stop** | – | Kills the builtin browser and deletes its status file. |
|
||||
| **browser view** | `--url, -u` URL *(optional)* | Pops a visible window of the builtin browser, navigates to `URL` or `about:blank`. |
|
||||
| **config list** | – | Dumps every global setting, showing current value, default, and description. |
|
||||
| **config get** | `key` | Prints the value of a single setting, falls back to default if unset. |
|
||||
| **config set** | `key value` | Persists a new value in the global config (stored under `~/.crawl4ai/config.yml`). |
|
||||
| **examples** | – | Just spits out real-world CLI usage samples. |
|
||||
| **crawl** | `url` *(positional)*<br>`--browser-config,-B` path<br>`--crawler-config,-C` path<br>`--filter-config,-f` path<br>`--extraction-config,-e` path<br>`--json-extract,-j` [desc]\*<br>`--schema,-s` path<br>`--browser,-b` k=v list<br>`--crawler,-c` k=v list<br>`--output,-o` all,json,markdown,md,markdown-fit,md-fit *(default all)*<br>`--output-file,-O` path<br>`--bypass-cache,-b` *(flag, default true — note flag reuse)*<br>`--question,-q` str<br>`--verbose,-v` *(flag)*<br>`--profile,-p` profile-name | One-shot crawl + extraction. Builds `BrowserConfig` and `CrawlerRunConfig` from inline flags or separate YAML/JSON files, runs `AsyncWebCrawler.run()`, can route through a named saved profile and pipe the result to stdout or a file. |
|
||||
| **(default)** | Same flags as **crawl**, plus `--example` | Shortcut so you can type just `crwl https://site.com`. When first arg is not a known sub-command, it falls through to *crawl*. |
|
||||
|
||||
\* `--json-extract/-j` with no value turns on LLM-based JSON extraction using an auto schema, supplying a string lets you prompt-engineer the field descriptions.
|
||||
|
||||
> Quick mental model
|
||||
> `profiles` = manage identities,
|
||||
> `browser ...` = control long-running headless Chrome that all crawls can piggy-back on,
|
||||
> `crawl` = do the actual work,
|
||||
> `config` = tweak global defaults,
|
||||
> everything else is sugar.
|
||||
|
||||
### Quick-fire “profile” usage cheatsheet
|
||||
|
||||
| Scenario | Command (copy-paste ready) | Notes |
|
||||
|---|---|---|
|
||||
| **Launch interactive Profile Manager UI** | `crwl profiles` | Opens TUI with options: 1 List, 2 Create, 3 Delete, 4 Use-to-crawl, 5 Exit. |
|
||||
| **Create a fresh profile** | `crwl profiles` → choose **2** → name it → browser opens → log in → press **q** in terminal | Saves to `~/.crawl4ai/profiles/<name>`. |
|
||||
| **List saved profiles** | `crwl profiles` → choose **1** | Shows name, browser type, size, last-modified. |
|
||||
| **Delete a profile** | `crwl profiles` → choose **3** → pick the profile index → confirm | Removes the folder. |
|
||||
| **Crawl with a profile (default alias)** | `crwl https://site.com/dashboard -p my-profile` | Keeps login cookies, sets `use_managed_browser=true` under the hood. |
|
||||
| **Crawl + verbose JSON output** | `crwl https://site.com -p my-profile -o json -v` | Any other `crawl` flags work the same. |
|
||||
| **Crawl with extra browser tweaks** | `crwl https://site.com -p my-profile -b "headless=true,viewport_width=1680"` | CLI overrides go on top of the profile. |
|
||||
| **Same but via explicit sub-command** | `crwl crawl https://site.com -p my-profile` | Identical to default alias. |
|
||||
| **Use profile from inside Profile Manager** | `crwl profiles` → choose **4** → pick profile → enter URL → follow prompts | Handy when demo-ing to non-CLI folks. |
|
||||
| **One-off crawl with a profile folder path (no name lookup)** | `crwl https://site.com -b "user_data_dir=$HOME/.crawl4ai/profiles/my-profile,use_managed_browser=true"` | Bypasses registry, useful for CI scripts. |
|
||||
| **Launch a dev browser on CDP port with the same identity** | `crwl cdp -d $HOME/.crawl4ai/profiles/my-profile -P 9223` | Lets Puppeteer/Playwright attach for debugging. |
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
# 🐳 Using Docker (Legacy)
|
||||
|
||||
Crawl4AI is available as Docker images for easy deployment. You can either pull directly from Docker Hub (recommended) or build from the repository.
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary>🐳 <strong>Option 1: Docker Hub (Recommended)</strong></summary>
|
||||
|
||||
Choose the appropriate image based on your platform and needs:
|
||||
|
||||
### For AMD64 (Regular Linux/Windows):
|
||||
```bash
|
||||
# Basic version (recommended)
|
||||
docker pull unclecode/crawl4ai:basic-amd64
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic-amd64
|
||||
|
||||
# Full ML/LLM support
|
||||
docker pull unclecode/crawl4ai:all-amd64
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:all-amd64
|
||||
|
||||
# With GPU support
|
||||
docker pull unclecode/crawl4ai:gpu-amd64
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:gpu-amd64
|
||||
```
|
||||
|
||||
### For ARM64 (M1/M2 Macs, ARM servers):
|
||||
```bash
|
||||
# Basic version (recommended)
|
||||
docker pull unclecode/crawl4ai:basic-arm64
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic-arm64
|
||||
|
||||
# Full ML/LLM support
|
||||
docker pull unclecode/crawl4ai:all-arm64
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:all-arm64
|
||||
|
||||
# With GPU support
|
||||
docker pull unclecode/crawl4ai:gpu-arm64
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:gpu-arm64
|
||||
```
|
||||
|
||||
Need more memory? Add `--shm-size`:
|
||||
```bash
|
||||
docker run --shm-size=2gb -p 11235:11235 unclecode/crawl4ai:basic-amd64
|
||||
```
|
||||
|
||||
Test the installation:
|
||||
```bash
|
||||
curl http://localhost:11235/health
|
||||
```
|
||||
|
||||
### For Raspberry Pi (32-bit) (coming soon):
|
||||
```bash
|
||||
# Pull and run basic version (recommended for Raspberry Pi)
|
||||
docker pull unclecode/crawl4ai:basic-armv7
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:basic-armv7
|
||||
|
||||
# With increased shared memory if needed
|
||||
docker run --shm-size=2gb -p 11235:11235 unclecode/crawl4ai:basic-armv7
|
||||
```
|
||||
|
||||
Note: Due to hardware constraints, only the basic version is recommended for Raspberry Pi.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>🐳 <strong>Option 2: Build from Repository</strong></summary>
|
||||
|
||||
Build the image locally based on your platform:
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/unclecode/crawl4ai.git
|
||||
cd crawl4ai
|
||||
|
||||
# For AMD64 (Regular Linux/Windows)
|
||||
docker build --platform linux/amd64 \
|
||||
--tag crawl4ai:local \
|
||||
--build-arg INSTALL_TYPE=basic \
|
||||
.
|
||||
|
||||
# For ARM64 (M1/M2 Macs, ARM servers)
|
||||
docker build --platform linux/arm64 \
|
||||
--tag crawl4ai:local \
|
||||
--build-arg INSTALL_TYPE=basic \
|
||||
.
|
||||
```
|
||||
|
||||
Build options:
|
||||
- INSTALL_TYPE=basic (default): Basic crawling features
|
||||
- INSTALL_TYPE=all: Full ML/LLM support
|
||||
- ENABLE_GPU=true: Add GPU support
|
||||
|
||||
Example with all options:
|
||||
```bash
|
||||
docker build --platform linux/amd64 \
|
||||
--tag crawl4ai:local \
|
||||
--build-arg INSTALL_TYPE=all \
|
||||
--build-arg ENABLE_GPU=true \
|
||||
.
|
||||
```
|
||||
|
||||
Run your local build:
|
||||
```bash
|
||||
# Regular run
|
||||
docker run -p 11235:11235 crawl4ai:local
|
||||
|
||||
# With increased shared memory
|
||||
docker run --shm-size=2gb -p 11235:11235 crawl4ai:local
|
||||
```
|
||||
|
||||
Test the installation:
|
||||
```bash
|
||||
curl http://localhost:11235/health
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>🐳 <strong>Option 3: Using Docker Compose</strong></summary>
|
||||
|
||||
Docker Compose provides a more structured way to run Crawl4AI, especially when dealing with environment variables and multiple configurations.
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/unclecode/crawl4ai.git
|
||||
cd crawl4ai
|
||||
```
|
||||
|
||||
### For AMD64 (Regular Linux/Windows):
|
||||
```bash
|
||||
# Build and run locally
|
||||
docker-compose --profile local-amd64 up
|
||||
|
||||
# Run from Docker Hub
|
||||
VERSION=basic docker-compose --profile hub-amd64 up # Basic version
|
||||
VERSION=all docker-compose --profile hub-amd64 up # Full ML/LLM support
|
||||
VERSION=gpu docker-compose --profile hub-amd64 up # GPU support
|
||||
```
|
||||
|
||||
### For ARM64 (M1/M2 Macs, ARM servers):
|
||||
```bash
|
||||
# Build and run locally
|
||||
docker-compose --profile local-arm64 up
|
||||
|
||||
# Run from Docker Hub
|
||||
VERSION=basic docker-compose --profile hub-arm64 up # Basic version
|
||||
VERSION=all docker-compose --profile hub-arm64 up # Full ML/LLM support
|
||||
VERSION=gpu docker-compose --profile hub-arm64 up # GPU support
|
||||
```
|
||||
|
||||
Environment variables (optional):
|
||||
```bash
|
||||
# Create a .env file
|
||||
CRAWL4AI_API_TOKEN=your_token
|
||||
OPENAI_API_KEY=your_openai_key
|
||||
CLAUDE_API_KEY=your_claude_key
|
||||
```
|
||||
|
||||
The compose file includes:
|
||||
- Memory management (4GB limit, 1GB reserved)
|
||||
- Shared memory volume for browser support
|
||||
- Health checks
|
||||
- Auto-restart policy
|
||||
- All necessary port mappings
|
||||
|
||||
Test the installation:
|
||||
```bash
|
||||
curl http://localhost:11235/health
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>🚀 <strong>One-Click Deployment</strong></summary>
|
||||
|
||||
Deploy your own instance of Crawl4AI with one click:
|
||||
|
||||
[](https://www.digitalocean.com/?repo=https://github.com/unclecode/crawl4ai/tree/0.3.74&refcode=a0780f1bdb3d&utm_campaign=Referral_Invite&utm_medium=Referral_Program&utm_source=badge)
|
||||
|
||||
> 💡 **Recommended specs**: 4GB RAM minimum. Select "professional-xs" or higher when deploying for stable operation.
|
||||
|
||||
The deploy will:
|
||||
- Set up a Docker container with Crawl4AI
|
||||
- Configure Playwright and all dependencies
|
||||
- Start the FastAPI server on port `11235`
|
||||
- Set up health checks and auto-deployment
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,123 @@
|
||||
# Builtin Browser in Crawl4AI
|
||||
|
||||
This document explains the builtin browser feature in Crawl4AI and how to use it effectively.
|
||||
|
||||
## What is the Builtin Browser?
|
||||
|
||||
The builtin browser is a persistent Chrome instance that Crawl4AI manages for you. It runs in the background and can be used by multiple crawling operations, eliminating the need to start and stop browsers for each crawl.
|
||||
|
||||
Benefits include:
|
||||
- **Faster startup times** - The browser is already running, so your scripts start faster
|
||||
- **Shared resources** - All your crawling scripts can use the same browser instance
|
||||
- **Simplified management** - No need to worry about CDP URLs or browser processes
|
||||
- **Persistent cookies and sessions** - Browser state persists between script runs
|
||||
- **Less resource usage** - Only one browser instance for multiple scripts
|
||||
|
||||
## Using the Builtin Browser
|
||||
|
||||
### In Python Code
|
||||
|
||||
Using the builtin browser in your code is simple:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
|
||||
# Create browser config with builtin mode
|
||||
browser_config = BrowserConfig(
|
||||
browser_mode="builtin", # This is the key setting!
|
||||
headless=True # Can be headless or not
|
||||
)
|
||||
|
||||
# Create the crawler
|
||||
crawler = AsyncWebCrawler(config=browser_config)
|
||||
|
||||
# Use it - no need to explicitly start()
|
||||
result = await crawler.arun("https://example.com")
|
||||
```
|
||||
|
||||
Key points:
|
||||
1. Set `browser_mode="builtin"` in your BrowserConfig
|
||||
2. No need for explicit `start()` call - the crawler will automatically connect to the builtin browser
|
||||
3. No need to use a context manager or call `close()` - the browser stays running
|
||||
|
||||
### Via CLI
|
||||
|
||||
The CLI provides commands to manage the builtin browser:
|
||||
|
||||
```bash
|
||||
# Start the builtin browser
|
||||
crwl browser start
|
||||
|
||||
# Check its status
|
||||
crwl browser status
|
||||
|
||||
# Open a visible window to see what the browser is doing
|
||||
crwl browser view --url https://example.com
|
||||
|
||||
# Stop it when no longer needed
|
||||
crwl browser stop
|
||||
|
||||
# Restart with different settings
|
||||
crwl browser restart --no-headless
|
||||
```
|
||||
|
||||
When crawling via CLI, simply add the builtin browser mode:
|
||||
|
||||
```bash
|
||||
crwl https://example.com -b "browser_mode=builtin"
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
1. When a crawler with `browser_mode="builtin"` is created:
|
||||
- It checks if a builtin browser is already running
|
||||
- If not, it automatically launches one
|
||||
- It connects to the browser via CDP (Chrome DevTools Protocol)
|
||||
|
||||
2. The browser process continues running after your script exits
|
||||
- This means it's ready for the next crawl
|
||||
- You can manage it via the CLI commands
|
||||
|
||||
3. During installation, Crawl4AI attempts to create a builtin browser automatically
|
||||
|
||||
## Example
|
||||
|
||||
See the [builtin_browser_example.py](builtin_browser_example.py) file for a complete example.
|
||||
|
||||
Run it with:
|
||||
|
||||
```bash
|
||||
python builtin_browser_example.py
|
||||
```
|
||||
|
||||
## When to Use
|
||||
|
||||
The builtin browser is ideal for:
|
||||
- Scripts that run frequently
|
||||
- Development and testing workflows
|
||||
- Applications that need to minimize startup time
|
||||
- Systems where you want to manage browser instances centrally
|
||||
|
||||
You might not want to use it when:
|
||||
- Running one-off scripts
|
||||
- When you need different browser configurations for different tasks
|
||||
- In environments where persistent processes are not allowed
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If you encounter issues:
|
||||
|
||||
1. Check the browser status:
|
||||
```
|
||||
crwl browser status
|
||||
```
|
||||
|
||||
2. Try restarting it:
|
||||
```
|
||||
crwl browser restart
|
||||
```
|
||||
|
||||
3. If problems persist, stop it and let Crawl4AI start a fresh one:
|
||||
```
|
||||
crwl browser stop
|
||||
```
|
||||
@@ -0,0 +1,85 @@
|
||||
# Adaptive Crawling Examples
|
||||
|
||||
This directory contains examples demonstrating various aspects of Crawl4AI's Adaptive Crawling feature.
|
||||
|
||||
## Examples Overview
|
||||
|
||||
### 1. `basic_usage.py`
|
||||
- Simple introduction to adaptive crawling
|
||||
- Uses default statistical strategy
|
||||
- Shows how to get crawl statistics and relevant content
|
||||
|
||||
### 2. `embedding_strategy.py` ⭐ NEW
|
||||
- Demonstrates the embedding-based strategy for semantic understanding
|
||||
- Shows query expansion and irrelevance detection
|
||||
- Includes configuration for both local and API-based embeddings
|
||||
|
||||
### 3. `embedding_vs_statistical.py` ⭐ NEW
|
||||
- Direct comparison between statistical and embedding strategies
|
||||
- Helps you choose the right strategy for your use case
|
||||
- Shows performance and accuracy trade-offs
|
||||
|
||||
### 4. `embedding_configuration.py` ⭐ NEW
|
||||
- Advanced configuration options for embedding strategy
|
||||
- Parameter tuning guide for different scenarios
|
||||
- Examples for research, exploration, and quality-focused crawling
|
||||
|
||||
### 5. `advanced_configuration.py`
|
||||
- Shows various configuration options for both strategies
|
||||
- Demonstrates threshold tuning and performance optimization
|
||||
|
||||
### 6. `custom_strategies.py`
|
||||
- How to implement your own crawling strategy
|
||||
- Extends the base CrawlStrategy class
|
||||
- Advanced use case for specialized requirements
|
||||
|
||||
### 7. `export_import_kb.py`
|
||||
- Export crawled knowledge base to JSONL
|
||||
- Import and continue crawling from saved state
|
||||
- Useful for building persistent knowledge bases
|
||||
|
||||
## Quick Start
|
||||
|
||||
For your first adaptive crawling experience, run:
|
||||
|
||||
```bash
|
||||
python basic_usage.py
|
||||
```
|
||||
|
||||
To try the new embedding strategy with semantic understanding:
|
||||
|
||||
```bash
|
||||
python embedding_strategy.py
|
||||
```
|
||||
|
||||
To compare strategies and see which works best for your use case:
|
||||
|
||||
```bash
|
||||
python embedding_vs_statistical.py
|
||||
```
|
||||
|
||||
## Strategy Selection Guide
|
||||
|
||||
### Use Statistical Strategy (Default) When:
|
||||
- Working with technical documentation
|
||||
- Queries contain specific terms or code
|
||||
- Speed is critical
|
||||
- No API access available
|
||||
|
||||
### Use Embedding Strategy When:
|
||||
- Queries are conceptual or ambiguous
|
||||
- Need semantic understanding beyond exact matches
|
||||
- Want to detect irrelevant content
|
||||
- Working with diverse content sources
|
||||
|
||||
## Requirements
|
||||
|
||||
- Crawl4AI installed
|
||||
- For embedding strategy with local models: `sentence-transformers`
|
||||
- For embedding strategy with OpenAI: Set `OPENAI_API_KEY` environment variable
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Adaptive Crawling Documentation](https://docs.crawl4ai.com/core/adaptive-crawling/)
|
||||
- [Mathematical Framework](https://github.com/unclecode/crawl4ai/blob/main/PROGRESSIVE_CRAWLING.md)
|
||||
- [Blog: The Adaptive Crawling Revolution](https://docs.crawl4ai.com/blog/adaptive-crawling-revolution/)
|
||||
@@ -0,0 +1,207 @@
|
||||
"""
|
||||
Advanced Adaptive Crawling Configuration
|
||||
|
||||
This example demonstrates all configuration options available for adaptive crawling,
|
||||
including threshold tuning, persistence, and custom parameters.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
|
||||
async def main():
|
||||
"""Demonstrate advanced configuration options"""
|
||||
|
||||
# Example 1: Custom thresholds for different use cases
|
||||
print("="*60)
|
||||
print("EXAMPLE 1: Custom Confidence Thresholds")
|
||||
print("="*60)
|
||||
|
||||
# High-precision configuration (exhaustive crawling)
|
||||
high_precision_config = AdaptiveConfig(
|
||||
confidence_threshold=0.9, # Very high confidence required
|
||||
max_pages=50, # Allow more pages
|
||||
top_k_links=5, # Follow more links per page
|
||||
min_gain_threshold=0.02 # Lower threshold to continue
|
||||
)
|
||||
|
||||
# Balanced configuration (default use case)
|
||||
balanced_config = AdaptiveConfig(
|
||||
confidence_threshold=0.7, # Moderate confidence
|
||||
max_pages=20, # Reasonable limit
|
||||
top_k_links=3, # Moderate branching
|
||||
min_gain_threshold=0.05 # Standard gain threshold
|
||||
)
|
||||
|
||||
# Quick exploration configuration
|
||||
quick_config = AdaptiveConfig(
|
||||
confidence_threshold=0.5, # Lower confidence acceptable
|
||||
max_pages=10, # Strict limit
|
||||
top_k_links=2, # Minimal branching
|
||||
min_gain_threshold=0.1 # High gain required
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
# Test different configurations
|
||||
for config_name, config in [
|
||||
("High Precision", high_precision_config),
|
||||
("Balanced", balanced_config),
|
||||
("Quick Exploration", quick_config)
|
||||
]:
|
||||
print(f"\nTesting {config_name} configuration...")
|
||||
adaptive = AdaptiveCrawler(crawler, config=config)
|
||||
|
||||
result = await adaptive.digest(
|
||||
start_url="https://httpbin.org",
|
||||
query="http headers authentication"
|
||||
)
|
||||
|
||||
print(f" - Pages crawled: {len(result.crawled_urls)}")
|
||||
print(f" - Confidence achieved: {adaptive.confidence:.2%}")
|
||||
print(f" - Coverage score: {adaptive.coverage_stats['coverage']:.2f}")
|
||||
|
||||
# Example 2: Persistence and state management
|
||||
print("\n" + "="*60)
|
||||
print("EXAMPLE 2: State Persistence")
|
||||
print("="*60)
|
||||
|
||||
state_file = "crawl_state_demo.json"
|
||||
|
||||
# Configuration with persistence
|
||||
persistent_config = AdaptiveConfig(
|
||||
confidence_threshold=0.8,
|
||||
max_pages=30,
|
||||
save_state=True, # Enable auto-save
|
||||
state_path=state_file # Specify save location
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
# First crawl - will be interrupted
|
||||
print("\nStarting initial crawl (will interrupt after 5 pages)...")
|
||||
|
||||
interrupt_config = AdaptiveConfig(
|
||||
confidence_threshold=0.8,
|
||||
max_pages=5, # Artificially low to simulate interruption
|
||||
save_state=True,
|
||||
state_path=state_file
|
||||
)
|
||||
|
||||
adaptive = AdaptiveCrawler(crawler, config=interrupt_config)
|
||||
result1 = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/",
|
||||
query="exception handling try except finally"
|
||||
)
|
||||
|
||||
print(f"First crawl completed: {len(result1.crawled_urls)} pages")
|
||||
print(f"Confidence reached: {adaptive.confidence:.2%}")
|
||||
|
||||
# Resume crawl with higher page limit
|
||||
print("\nResuming crawl from saved state...")
|
||||
|
||||
resume_config = AdaptiveConfig(
|
||||
confidence_threshold=0.8,
|
||||
max_pages=20, # Increase limit
|
||||
save_state=True,
|
||||
state_path=state_file
|
||||
)
|
||||
|
||||
adaptive2 = AdaptiveCrawler(crawler, config=resume_config)
|
||||
result2 = await adaptive2.digest(
|
||||
start_url="https://docs.python.org/3/",
|
||||
query="exception handling try except finally",
|
||||
resume_from=state_file
|
||||
)
|
||||
|
||||
print(f"Resumed crawl completed: {len(result2.crawled_urls)} total pages")
|
||||
print(f"Final confidence: {adaptive2.confidence:.2%}")
|
||||
|
||||
# Clean up
|
||||
Path(state_file).unlink(missing_ok=True)
|
||||
|
||||
# Example 3: Link selection strategies
|
||||
print("\n" + "="*60)
|
||||
print("EXAMPLE 3: Link Selection Strategies")
|
||||
print("="*60)
|
||||
|
||||
# Conservative link following
|
||||
conservative_config = AdaptiveConfig(
|
||||
confidence_threshold=0.7,
|
||||
max_pages=15,
|
||||
top_k_links=1, # Only follow best link
|
||||
min_gain_threshold=0.15 # High threshold
|
||||
)
|
||||
|
||||
# Aggressive link following
|
||||
aggressive_config = AdaptiveConfig(
|
||||
confidence_threshold=0.7,
|
||||
max_pages=15,
|
||||
top_k_links=10, # Follow many links
|
||||
min_gain_threshold=0.01 # Very low threshold
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
for strategy_name, config in [
|
||||
("Conservative", conservative_config),
|
||||
("Aggressive", aggressive_config)
|
||||
]:
|
||||
print(f"\n{strategy_name} link selection:")
|
||||
adaptive = AdaptiveCrawler(crawler, config=config)
|
||||
|
||||
result = await adaptive.digest(
|
||||
start_url="https://httpbin.org",
|
||||
query="api endpoints"
|
||||
)
|
||||
|
||||
# Analyze crawl pattern
|
||||
print(f" - Total pages: {len(result.crawled_urls)}")
|
||||
print(f" - Unique domains: {len(set(url.split('/')[2] for url in result.crawled_urls))}")
|
||||
print(f" - Max depth reached: {max(url.count('/') for url in result.crawled_urls) - 2}")
|
||||
|
||||
# Show saturation trend
|
||||
if hasattr(result, 'new_terms_history') and result.new_terms_history:
|
||||
print(f" - New terms discovered: {result.new_terms_history[:5]}...")
|
||||
print(f" - Saturation trend: {'decreasing' if result.new_terms_history[-1] < result.new_terms_history[0] else 'increasing'}")
|
||||
|
||||
# Example 4: Monitoring crawl progress
|
||||
print("\n" + "="*60)
|
||||
print("EXAMPLE 4: Progress Monitoring")
|
||||
print("="*60)
|
||||
|
||||
# Configuration with detailed monitoring
|
||||
monitor_config = AdaptiveConfig(
|
||||
confidence_threshold=0.75,
|
||||
max_pages=10,
|
||||
top_k_links=3
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config=monitor_config)
|
||||
|
||||
# Start crawl
|
||||
print("\nMonitoring crawl progress...")
|
||||
result = await adaptive.digest(
|
||||
start_url="https://httpbin.org",
|
||||
query="http methods headers"
|
||||
)
|
||||
|
||||
# Detailed statistics
|
||||
print("\nDetailed crawl analysis:")
|
||||
adaptive.print_stats(detailed=True)
|
||||
|
||||
# Export for analysis
|
||||
print("\nExporting knowledge base for external analysis...")
|
||||
adaptive.export_knowledge_base("knowledge_export_demo.jsonl")
|
||||
print("Knowledge base exported to: knowledge_export_demo.jsonl")
|
||||
|
||||
# Show sample of exported data
|
||||
with open("knowledge_export_demo.jsonl", 'r') as f:
|
||||
first_line = f.readline()
|
||||
print(f"Sample export: {first_line[:100]}...")
|
||||
|
||||
# Clean up
|
||||
Path("knowledge_export_demo.jsonl").unlink(missing_ok=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,76 @@
|
||||
"""
|
||||
Basic Adaptive Crawling Example
|
||||
|
||||
This example demonstrates the simplest use case of adaptive crawling:
|
||||
finding information about a specific topic and knowing when to stop.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler
|
||||
|
||||
|
||||
async def main():
|
||||
"""Basic adaptive crawling example"""
|
||||
|
||||
# Initialize the crawler
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Create an adaptive crawler with default settings (statistical strategy)
|
||||
adaptive = AdaptiveCrawler(crawler)
|
||||
|
||||
# Note: You can also use embedding strategy for semantic understanding:
|
||||
# from crawl4ai import AdaptiveConfig
|
||||
# config = AdaptiveConfig(strategy="embedding")
|
||||
# adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Start adaptive crawling
|
||||
print("Starting adaptive crawl for Python async programming information...")
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/library/asyncio.html",
|
||||
query="async await context managers coroutines"
|
||||
)
|
||||
|
||||
# Display crawl statistics
|
||||
print("\n" + "="*50)
|
||||
print("CRAWL STATISTICS")
|
||||
print("="*50)
|
||||
adaptive.print_stats(detailed=False)
|
||||
|
||||
# Get the most relevant content found
|
||||
print("\n" + "="*50)
|
||||
print("MOST RELEVANT PAGES")
|
||||
print("="*50)
|
||||
|
||||
relevant_pages = adaptive.get_relevant_content(top_k=5)
|
||||
for i, page in enumerate(relevant_pages, 1):
|
||||
print(f"\n{i}. {page['url']}")
|
||||
print(f" Relevance Score: {page['score']:.2%}")
|
||||
|
||||
# Show a snippet of the content
|
||||
content = page['content'] or ""
|
||||
if content:
|
||||
snippet = content[:200].replace('\n', ' ')
|
||||
if len(content) > 200:
|
||||
snippet += "..."
|
||||
print(f" Preview: {snippet}")
|
||||
|
||||
# Show final confidence
|
||||
print(f"\n{'='*50}")
|
||||
print(f"Final Confidence: {adaptive.confidence:.2%}")
|
||||
print(f"Total Pages Crawled: {len(result.crawled_urls)}")
|
||||
print(f"Knowledge Base Size: {len(adaptive.state.knowledge_base)} documents")
|
||||
|
||||
# Example: Check if we can answer specific questions
|
||||
print(f"\n{'='*50}")
|
||||
print("INFORMATION SUFFICIENCY CHECK")
|
||||
print(f"{'='*50}")
|
||||
|
||||
if adaptive.confidence >= 0.8:
|
||||
print("✓ High confidence - can answer detailed questions about async Python")
|
||||
elif adaptive.confidence >= 0.6:
|
||||
print("~ Moderate confidence - can answer basic questions")
|
||||
else:
|
||||
print("✗ Low confidence - need more information")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,373 @@
|
||||
"""
|
||||
Custom Adaptive Crawling Strategies
|
||||
|
||||
This example demonstrates how to implement custom scoring strategies
|
||||
for domain-specific crawling needs.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from typing import List, Dict, Set
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
from crawl4ai.adaptive_crawler import CrawlState, Link
|
||||
import math
|
||||
|
||||
|
||||
class APIDocumentationStrategy:
|
||||
"""
|
||||
Custom strategy optimized for API documentation crawling.
|
||||
Prioritizes endpoint references, code examples, and parameter descriptions.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
# Keywords that indicate high-value API documentation
|
||||
self.api_keywords = {
|
||||
'endpoint', 'request', 'response', 'parameter', 'authentication',
|
||||
'header', 'body', 'query', 'path', 'method', 'get', 'post', 'put',
|
||||
'delete', 'patch', 'status', 'code', 'example', 'curl', 'python'
|
||||
}
|
||||
|
||||
# URL patterns that typically contain API documentation
|
||||
self.valuable_patterns = [
|
||||
r'/api/',
|
||||
r'/reference/',
|
||||
r'/endpoints?/',
|
||||
r'/methods?/',
|
||||
r'/resources?/'
|
||||
]
|
||||
|
||||
# Patterns to avoid
|
||||
self.avoid_patterns = [
|
||||
r'/blog/',
|
||||
r'/news/',
|
||||
r'/about/',
|
||||
r'/contact/',
|
||||
r'/legal/'
|
||||
]
|
||||
|
||||
def score_link(self, link: Link, query: str, state: CrawlState) -> float:
|
||||
"""Custom link scoring for API documentation"""
|
||||
score = 1.0
|
||||
url = link.href.lower()
|
||||
|
||||
# Boost API-related URLs
|
||||
for pattern in self.valuable_patterns:
|
||||
if re.search(pattern, url):
|
||||
score *= 2.0
|
||||
break
|
||||
|
||||
# Reduce score for non-API content
|
||||
for pattern in self.avoid_patterns:
|
||||
if re.search(pattern, url):
|
||||
score *= 0.1
|
||||
break
|
||||
|
||||
# Boost if preview contains API keywords
|
||||
if link.text:
|
||||
preview_lower = link.text.lower()
|
||||
keyword_count = sum(1 for kw in self.api_keywords if kw in preview_lower)
|
||||
score *= (1 + keyword_count * 0.2)
|
||||
|
||||
# Prioritize shallow URLs (likely overview pages)
|
||||
depth = url.count('/') - 2 # Subtract protocol slashes
|
||||
if depth <= 3:
|
||||
score *= 1.5
|
||||
elif depth > 6:
|
||||
score *= 0.5
|
||||
|
||||
return score
|
||||
|
||||
def calculate_api_coverage(self, state: CrawlState, query: str) -> Dict[str, float]:
|
||||
"""Calculate specialized coverage metrics for API documentation"""
|
||||
metrics = {
|
||||
'endpoint_coverage': 0.0,
|
||||
'example_coverage': 0.0,
|
||||
'parameter_coverage': 0.0
|
||||
}
|
||||
|
||||
# Analyze knowledge base for API-specific content
|
||||
endpoint_patterns = [r'GET\s+/', r'POST\s+/', r'PUT\s+/', r'DELETE\s+/']
|
||||
example_patterns = [r'```\w+', r'curl\s+-', r'import\s+requests']
|
||||
param_patterns = [r'param(?:eter)?s?\s*:', r'required\s*:', r'optional\s*:']
|
||||
|
||||
total_docs = len(state.knowledge_base)
|
||||
if total_docs == 0:
|
||||
return metrics
|
||||
|
||||
docs_with_endpoints = 0
|
||||
docs_with_examples = 0
|
||||
docs_with_params = 0
|
||||
|
||||
for doc in state.knowledge_base:
|
||||
content = doc.markdown.raw_markdown if hasattr(doc, 'markdown') else str(doc)
|
||||
|
||||
# Check for endpoints
|
||||
if any(re.search(pattern, content, re.IGNORECASE) for pattern in endpoint_patterns):
|
||||
docs_with_endpoints += 1
|
||||
|
||||
# Check for examples
|
||||
if any(re.search(pattern, content, re.IGNORECASE) for pattern in example_patterns):
|
||||
docs_with_examples += 1
|
||||
|
||||
# Check for parameters
|
||||
if any(re.search(pattern, content, re.IGNORECASE) for pattern in param_patterns):
|
||||
docs_with_params += 1
|
||||
|
||||
metrics['endpoint_coverage'] = docs_with_endpoints / total_docs
|
||||
metrics['example_coverage'] = docs_with_examples / total_docs
|
||||
metrics['parameter_coverage'] = docs_with_params / total_docs
|
||||
|
||||
return metrics
|
||||
|
||||
|
||||
class ResearchPaperStrategy:
|
||||
"""
|
||||
Strategy optimized for crawling research papers and academic content.
|
||||
Prioritizes citations, abstracts, and methodology sections.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.academic_keywords = {
|
||||
'abstract', 'introduction', 'methodology', 'results', 'conclusion',
|
||||
'references', 'citation', 'paper', 'study', 'research', 'analysis',
|
||||
'hypothesis', 'experiment', 'findings', 'doi'
|
||||
}
|
||||
|
||||
self.citation_patterns = [
|
||||
r'\[\d+\]', # [1] style citations
|
||||
r'\(\w+\s+\d{4}\)', # (Author 2024) style
|
||||
r'doi:\s*\S+', # DOI references
|
||||
]
|
||||
|
||||
def calculate_academic_relevance(self, content: str, query: str) -> float:
|
||||
"""Calculate relevance score for academic content"""
|
||||
score = 0.0
|
||||
content_lower = content.lower()
|
||||
|
||||
# Check for academic keywords
|
||||
keyword_matches = sum(1 for kw in self.academic_keywords if kw in content_lower)
|
||||
score += keyword_matches * 0.1
|
||||
|
||||
# Check for citations
|
||||
citation_count = sum(
|
||||
len(re.findall(pattern, content))
|
||||
for pattern in self.citation_patterns
|
||||
)
|
||||
score += min(citation_count * 0.05, 1.0) # Cap at 1.0
|
||||
|
||||
# Check for query terms in academic context
|
||||
query_terms = query.lower().split()
|
||||
for term in query_terms:
|
||||
# Boost if term appears near academic keywords
|
||||
for keyword in ['abstract', 'conclusion', 'results']:
|
||||
if keyword in content_lower:
|
||||
section = content_lower[content_lower.find(keyword):content_lower.find(keyword) + 500]
|
||||
if term in section:
|
||||
score += 0.2
|
||||
|
||||
return min(score, 2.0) # Cap total score
|
||||
|
||||
|
||||
async def demo_custom_strategies():
|
||||
"""Demonstrate custom strategy usage"""
|
||||
|
||||
# Example 1: API Documentation Strategy
|
||||
print("="*60)
|
||||
print("EXAMPLE 1: Custom API Documentation Strategy")
|
||||
print("="*60)
|
||||
|
||||
api_strategy = APIDocumentationStrategy()
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Standard adaptive crawler
|
||||
config = AdaptiveConfig(
|
||||
confidence_threshold=0.8,
|
||||
max_pages=15
|
||||
)
|
||||
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Override link scoring with custom strategy
|
||||
original_rank_links = adaptive._rank_links
|
||||
|
||||
def custom_rank_links(links, query, state):
|
||||
# Apply custom scoring
|
||||
scored_links = []
|
||||
for link in links:
|
||||
base_score = api_strategy.score_link(link, query, state)
|
||||
scored_links.append((link, base_score))
|
||||
|
||||
# Sort by score
|
||||
scored_links.sort(key=lambda x: x[1], reverse=True)
|
||||
return [link for link, _ in scored_links[:config.top_k_links]]
|
||||
|
||||
adaptive._rank_links = custom_rank_links
|
||||
|
||||
# Crawl API documentation
|
||||
print("\nCrawling API documentation with custom strategy...")
|
||||
state = await adaptive.digest(
|
||||
start_url="https://httpbin.org",
|
||||
query="api endpoints authentication headers"
|
||||
)
|
||||
|
||||
# Calculate custom metrics
|
||||
api_metrics = api_strategy.calculate_api_coverage(state, "api endpoints")
|
||||
|
||||
print(f"\nResults:")
|
||||
print(f"Pages crawled: {len(state.crawled_urls)}")
|
||||
print(f"Confidence: {adaptive.confidence:.2%}")
|
||||
print(f"\nAPI-Specific Metrics:")
|
||||
print(f" - Endpoint coverage: {api_metrics['endpoint_coverage']:.2%}")
|
||||
print(f" - Example coverage: {api_metrics['example_coverage']:.2%}")
|
||||
print(f" - Parameter coverage: {api_metrics['parameter_coverage']:.2%}")
|
||||
|
||||
# Example 2: Combined Strategy
|
||||
print("\n" + "="*60)
|
||||
print("EXAMPLE 2: Hybrid Strategy Combining Multiple Approaches")
|
||||
print("="*60)
|
||||
|
||||
class HybridStrategy:
|
||||
"""Combines multiple strategies with weights"""
|
||||
|
||||
def __init__(self):
|
||||
self.api_strategy = APIDocumentationStrategy()
|
||||
self.research_strategy = ResearchPaperStrategy()
|
||||
self.weights = {
|
||||
'api': 0.7,
|
||||
'research': 0.3
|
||||
}
|
||||
|
||||
def score_content(self, content: str, query: str) -> float:
|
||||
# Get scores from each strategy
|
||||
api_score = self._calculate_api_score(content, query)
|
||||
research_score = self.research_strategy.calculate_academic_relevance(content, query)
|
||||
|
||||
# Weighted combination
|
||||
total_score = (
|
||||
api_score * self.weights['api'] +
|
||||
research_score * self.weights['research']
|
||||
)
|
||||
|
||||
return total_score
|
||||
|
||||
def _calculate_api_score(self, content: str, query: str) -> float:
|
||||
# Simplified API scoring based on keyword presence
|
||||
content_lower = content.lower()
|
||||
api_keywords = self.api_strategy.api_keywords
|
||||
|
||||
keyword_count = sum(1 for kw in api_keywords if kw in content_lower)
|
||||
return min(keyword_count * 0.1, 2.0)
|
||||
|
||||
hybrid_strategy = HybridStrategy()
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler)
|
||||
|
||||
# Crawl with hybrid scoring
|
||||
print("\nTesting hybrid strategy on technical documentation...")
|
||||
state = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/library/asyncio.html",
|
||||
query="async await coroutines api"
|
||||
)
|
||||
|
||||
# Analyze results with hybrid strategy
|
||||
print(f"\nHybrid Strategy Analysis:")
|
||||
total_score = 0
|
||||
for doc in adaptive.get_relevant_content(top_k=5):
|
||||
content = doc['content'] or ""
|
||||
score = hybrid_strategy.score_content(content, "async await api")
|
||||
total_score += score
|
||||
print(f" - {doc['url'][:50]}... Score: {score:.2f}")
|
||||
|
||||
print(f"\nAverage hybrid score: {total_score/5:.2f}")
|
||||
|
||||
|
||||
async def demo_performance_optimization():
|
||||
"""Demonstrate performance optimization with custom strategies"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("EXAMPLE 3: Performance-Optimized Strategy")
|
||||
print("="*60)
|
||||
|
||||
class PerformanceOptimizedStrategy:
|
||||
"""Strategy that balances thoroughness with speed"""
|
||||
|
||||
def __init__(self):
|
||||
self.url_cache: Set[str] = set()
|
||||
self.domain_scores: Dict[str, float] = {}
|
||||
|
||||
def should_crawl_domain(self, url: str) -> bool:
|
||||
"""Implement domain-level filtering"""
|
||||
domain = url.split('/')[2] if url.startswith('http') else url
|
||||
|
||||
# Skip if we've already crawled many pages from this domain
|
||||
domain_count = sum(1 for cached in self.url_cache if domain in cached)
|
||||
if domain_count > 5:
|
||||
return False
|
||||
|
||||
# Skip low-scoring domains
|
||||
if domain in self.domain_scores and self.domain_scores[domain] < 0.3:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def update_domain_score(self, url: str, relevance: float):
|
||||
"""Track domain-level performance"""
|
||||
domain = url.split('/')[2] if url.startswith('http') else url
|
||||
|
||||
if domain not in self.domain_scores:
|
||||
self.domain_scores[domain] = relevance
|
||||
else:
|
||||
# Moving average
|
||||
self.domain_scores[domain] = (
|
||||
0.7 * self.domain_scores[domain] + 0.3 * relevance
|
||||
)
|
||||
|
||||
perf_strategy = PerformanceOptimizedStrategy()
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
config = AdaptiveConfig(
|
||||
confidence_threshold=0.7,
|
||||
max_pages=10,
|
||||
top_k_links=2 # Fewer links for speed
|
||||
)
|
||||
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Track performance
|
||||
import time
|
||||
start_time = time.time()
|
||||
|
||||
state = await adaptive.digest(
|
||||
start_url="https://httpbin.org",
|
||||
query="http methods headers"
|
||||
)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
print(f"\nPerformance Results:")
|
||||
print(f" - Time elapsed: {elapsed:.2f} seconds")
|
||||
print(f" - Pages crawled: {len(state.crawled_urls)}")
|
||||
print(f" - Pages per second: {len(state.crawled_urls)/elapsed:.2f}")
|
||||
print(f" - Final confidence: {adaptive.confidence:.2%}")
|
||||
print(f" - Efficiency: {adaptive.confidence/len(state.crawled_urls):.2%} confidence per page")
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run all demonstrations"""
|
||||
try:
|
||||
await demo_custom_strategies()
|
||||
await demo_performance_optimization()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("All custom strategy examples completed!")
|
||||
print("="*60)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,206 @@
|
||||
"""
|
||||
Advanced Embedding Configuration Example
|
||||
|
||||
This example demonstrates all configuration options available for the
|
||||
embedding strategy, including fine-tuning parameters for different use cases.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
|
||||
async def test_configuration(name: str, config: AdaptiveConfig, url: str, query: str):
|
||||
"""Test a specific configuration"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Configuration: {name}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
result = await adaptive.digest(start_url=url, query=query)
|
||||
|
||||
print(f"Pages crawled: {len(result.crawled_urls)}")
|
||||
print(f"Final confidence: {adaptive.confidence:.1%}")
|
||||
print(f"Stopped reason: {result.metrics.get('stopped_reason', 'max_pages')}")
|
||||
|
||||
if result.metrics.get('is_irrelevant', False):
|
||||
print("⚠️ Query detected as irrelevant!")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def main():
|
||||
"""Demonstrate various embedding configurations"""
|
||||
|
||||
print("EMBEDDING STRATEGY CONFIGURATION EXAMPLES")
|
||||
print("=" * 60)
|
||||
|
||||
# Base URL and query for testing
|
||||
test_url = "https://docs.python.org/3/library/asyncio.html"
|
||||
|
||||
# 1. Default Configuration
|
||||
config_default = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=10
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"Default Settings",
|
||||
config_default,
|
||||
test_url,
|
||||
"async programming patterns"
|
||||
)
|
||||
|
||||
# 2. Strict Coverage Requirements
|
||||
config_strict = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=20,
|
||||
|
||||
# Stricter similarity requirements
|
||||
embedding_k_exp=5.0, # Default is 3.0, higher = stricter
|
||||
embedding_coverage_radius=0.15, # Default is 0.2, lower = stricter
|
||||
|
||||
# Higher validation threshold
|
||||
embedding_validation_min_score=0.6, # Default is 0.3
|
||||
|
||||
# More query variations for better coverage
|
||||
n_query_variations=15 # Default is 10
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"Strict Coverage (Research/Academic)",
|
||||
config_strict,
|
||||
test_url,
|
||||
"comprehensive guide async await"
|
||||
)
|
||||
|
||||
# 3. Fast Exploration
|
||||
config_fast = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=10,
|
||||
top_k_links=5, # Follow more links per page
|
||||
|
||||
# Relaxed requirements for faster convergence
|
||||
embedding_k_exp=1.0, # Lower = more lenient
|
||||
embedding_min_relative_improvement=0.05, # Stop earlier
|
||||
|
||||
# Lower quality thresholds
|
||||
embedding_quality_min_confidence=0.5, # Display lower confidence
|
||||
embedding_quality_max_confidence=0.85,
|
||||
|
||||
# Fewer query variations for speed
|
||||
n_query_variations=5
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"Fast Exploration (Quick Overview)",
|
||||
config_fast,
|
||||
test_url,
|
||||
"async basics"
|
||||
)
|
||||
|
||||
# 4. Irrelevance Detection Focus
|
||||
config_irrelevance = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=5,
|
||||
|
||||
# Aggressive irrelevance detection
|
||||
embedding_min_confidence_threshold=0.2, # Higher threshold (default 0.1)
|
||||
embedding_k_exp=5.0, # Strict similarity
|
||||
|
||||
# Quick stopping for irrelevant content
|
||||
embedding_min_relative_improvement=0.15
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"Irrelevance Detection",
|
||||
config_irrelevance,
|
||||
test_url,
|
||||
"recipe for chocolate cake" # Irrelevant query
|
||||
)
|
||||
|
||||
# 5. High-Quality Knowledge Base
|
||||
config_quality = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=30,
|
||||
|
||||
# Deduplication settings
|
||||
embedding_overlap_threshold=0.75, # More aggressive deduplication
|
||||
|
||||
# Quality focus
|
||||
embedding_validation_min_score=0.5,
|
||||
embedding_quality_scale_factor=1.0, # Linear quality mapping
|
||||
|
||||
# Balanced parameters
|
||||
embedding_k_exp=3.0,
|
||||
embedding_nearest_weight=0.8, # Focus on best matches
|
||||
embedding_top_k_weight=0.2
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"High-Quality Knowledge Base",
|
||||
config_quality,
|
||||
test_url,
|
||||
"asyncio advanced patterns best practices"
|
||||
)
|
||||
|
||||
# 6. Custom Embedding Provider
|
||||
if os.getenv('OPENAI_API_KEY'):
|
||||
config_openai = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=10,
|
||||
|
||||
# Use OpenAI embeddings
|
||||
embedding_llm_config={
|
||||
'provider': 'openai/text-embedding-3-small',
|
||||
'api_token': os.getenv('OPENAI_API_KEY')
|
||||
},
|
||||
|
||||
# OpenAI embeddings are high quality, can be stricter
|
||||
embedding_k_exp=4.0,
|
||||
n_query_variations=12
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"OpenAI Embeddings",
|
||||
config_openai,
|
||||
test_url,
|
||||
"event-driven architecture patterns"
|
||||
)
|
||||
|
||||
# Parameter Guide
|
||||
print("\n" + "="*60)
|
||||
print("PARAMETER TUNING GUIDE")
|
||||
print("="*60)
|
||||
|
||||
print("\n📊 Key Parameters and Their Effects:")
|
||||
print("\n1. embedding_k_exp (default: 3.0)")
|
||||
print(" - Lower (1-2): More lenient, faster convergence")
|
||||
print(" - Higher (4-5): Stricter, better precision")
|
||||
|
||||
print("\n2. embedding_coverage_radius (default: 0.2)")
|
||||
print(" - Lower (0.1-0.15): Requires closer matches")
|
||||
print(" - Higher (0.25-0.3): Accepts broader matches")
|
||||
|
||||
print("\n3. n_query_variations (default: 10)")
|
||||
print(" - Lower (5-7): Faster, less comprehensive")
|
||||
print(" - Higher (15-20): Better coverage, slower")
|
||||
|
||||
print("\n4. embedding_min_confidence_threshold (default: 0.1)")
|
||||
print(" - Set to 0.15-0.2 for aggressive irrelevance detection")
|
||||
print(" - Set to 0.05 to crawl even barely relevant content")
|
||||
|
||||
print("\n5. embedding_validation_min_score (default: 0.3)")
|
||||
print(" - Higher (0.5-0.6): Requires strong validation")
|
||||
print(" - Lower (0.2): More permissive stopping")
|
||||
|
||||
print("\n💡 Tips:")
|
||||
print("- For research: High k_exp, more variations, strict validation")
|
||||
print("- For exploration: Low k_exp, fewer variations, relaxed thresholds")
|
||||
print("- For quality: Focus on overlap_threshold and validation scores")
|
||||
print("- For speed: Reduce variations, increase min_relative_improvement")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,109 @@
|
||||
"""
|
||||
Embedding Strategy Example for Adaptive Crawling
|
||||
|
||||
This example demonstrates how to use the embedding-based strategy
|
||||
for semantic understanding and intelligent crawling.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
|
||||
async def main():
|
||||
"""Demonstrate embedding strategy for adaptive crawling"""
|
||||
|
||||
# Configure embedding strategy
|
||||
config = AdaptiveConfig(
|
||||
strategy="embedding", # Use embedding strategy
|
||||
embedding_model="sentence-transformers/all-MiniLM-L6-v2", # Default model
|
||||
n_query_variations=10, # Generate 10 semantic variations
|
||||
max_pages=15,
|
||||
top_k_links=3,
|
||||
min_gain_threshold=0.05,
|
||||
|
||||
# Embedding-specific parameters
|
||||
embedding_k_exp=3.0, # Higher = stricter similarity requirements
|
||||
embedding_min_confidence_threshold=0.1, # Stop if <10% relevant
|
||||
embedding_validation_min_score=0.4 # Validation threshold
|
||||
)
|
||||
|
||||
# Optional: Use OpenAI embeddings instead
|
||||
if os.getenv('OPENAI_API_KEY'):
|
||||
config.embedding_llm_config = {
|
||||
'provider': 'openai/text-embedding-3-small',
|
||||
'api_token': os.getenv('OPENAI_API_KEY')
|
||||
}
|
||||
print("Using OpenAI embeddings")
|
||||
else:
|
||||
print("Using sentence-transformers (local embeddings)")
|
||||
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Test 1: Relevant query with semantic understanding
|
||||
print("\n" + "="*50)
|
||||
print("TEST 1: Semantic Query Understanding")
|
||||
print("="*50)
|
||||
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/library/asyncio.html",
|
||||
query="concurrent programming event-driven architecture"
|
||||
)
|
||||
|
||||
print("\nQuery Expansion:")
|
||||
print(f"Original query expanded to {len(result.expanded_queries)} variations")
|
||||
for i, q in enumerate(result.expanded_queries[:3], 1):
|
||||
print(f" {i}. {q}")
|
||||
print(" ...")
|
||||
|
||||
print("\nResults:")
|
||||
adaptive.print_stats(detailed=False)
|
||||
|
||||
# Test 2: Detecting irrelevant queries
|
||||
print("\n" + "="*50)
|
||||
print("TEST 2: Irrelevant Query Detection")
|
||||
print("="*50)
|
||||
|
||||
# Reset crawler for new query
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/library/asyncio.html",
|
||||
query="how to bake chocolate chip cookies"
|
||||
)
|
||||
|
||||
if result.metrics.get('is_irrelevant', False):
|
||||
print("\n✅ Successfully detected irrelevant query!")
|
||||
print(f"Stopped after just {len(result.crawled_urls)} pages")
|
||||
print(f"Reason: {result.metrics.get('stopped_reason', 'unknown')}")
|
||||
else:
|
||||
print("\n❌ Failed to detect irrelevance")
|
||||
|
||||
print(f"Final confidence: {adaptive.confidence:.1%}")
|
||||
|
||||
# Test 3: Semantic gap analysis
|
||||
print("\n" + "="*50)
|
||||
print("TEST 3: Semantic Gap Analysis")
|
||||
print("="*50)
|
||||
|
||||
# Show how embedding strategy identifies gaps
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
result = await adaptive.digest(
|
||||
start_url="https://realpython.com",
|
||||
query="python decorators advanced patterns"
|
||||
)
|
||||
|
||||
print(f"\nSemantic gaps identified: {len(result.semantic_gaps)}")
|
||||
print(f"Knowledge base embeddings shape: {result.kb_embeddings.shape if result.kb_embeddings is not None else 'None'}")
|
||||
|
||||
# Show coverage metrics specific to embedding strategy
|
||||
print("\nEmbedding-specific metrics:")
|
||||
print(f" Average best similarity: {result.metrics.get('avg_best_similarity', 0):.3f}")
|
||||
print(f" Coverage score: {result.metrics.get('coverage_score', 0):.3f}")
|
||||
print(f" Validation confidence: {result.metrics.get('validation_confidence', 0):.2%}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,167 @@
|
||||
"""
|
||||
Comparison: Embedding vs Statistical Strategy
|
||||
|
||||
This example demonstrates the differences between statistical and embedding
|
||||
strategies for adaptive crawling, showing when to use each approach.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
|
||||
async def crawl_with_strategy(url: str, query: str, strategy: str, **kwargs):
|
||||
"""Helper function to crawl with a specific strategy"""
|
||||
config = AdaptiveConfig(
|
||||
strategy=strategy,
|
||||
max_pages=20,
|
||||
top_k_links=3,
|
||||
min_gain_threshold=0.05,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
start_time = time.time()
|
||||
result = await adaptive.digest(start_url=url, query=query)
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
return {
|
||||
'result': result,
|
||||
'crawler': adaptive,
|
||||
'elapsed': elapsed,
|
||||
'pages': len(result.crawled_urls),
|
||||
'confidence': adaptive.confidence
|
||||
}
|
||||
|
||||
|
||||
async def main():
|
||||
"""Compare embedding and statistical strategies"""
|
||||
|
||||
# Test scenarios
|
||||
test_cases = [
|
||||
{
|
||||
'name': 'Technical Documentation (Specific Terms)',
|
||||
'url': 'https://docs.python.org/3/library/asyncio.html',
|
||||
'query': 'asyncio.create_task event_loop.run_until_complete'
|
||||
},
|
||||
{
|
||||
'name': 'Conceptual Query (Semantic Understanding)',
|
||||
'url': 'https://docs.python.org/3/library/asyncio.html',
|
||||
'query': 'concurrent programming patterns'
|
||||
},
|
||||
{
|
||||
'name': 'Ambiguous Query',
|
||||
'url': 'https://realpython.com',
|
||||
'query': 'python performance optimization'
|
||||
}
|
||||
]
|
||||
|
||||
# Configure embedding strategy
|
||||
embedding_config = {}
|
||||
if os.getenv('OPENAI_API_KEY'):
|
||||
embedding_config['embedding_llm_config'] = {
|
||||
'provider': 'openai/text-embedding-3-small',
|
||||
'api_token': os.getenv('OPENAI_API_KEY')
|
||||
}
|
||||
|
||||
for test in test_cases:
|
||||
print("\n" + "="*70)
|
||||
print(f"TEST: {test['name']}")
|
||||
print(f"URL: {test['url']}")
|
||||
print(f"Query: '{test['query']}'")
|
||||
print("="*70)
|
||||
|
||||
# Run statistical strategy
|
||||
print("\n📊 Statistical Strategy:")
|
||||
stat_result = await crawl_with_strategy(
|
||||
test['url'],
|
||||
test['query'],
|
||||
'statistical'
|
||||
)
|
||||
|
||||
print(f" Pages crawled: {stat_result['pages']}")
|
||||
print(f" Time taken: {stat_result['elapsed']:.2f}s")
|
||||
print(f" Confidence: {stat_result['confidence']:.1%}")
|
||||
print(f" Sufficient: {'Yes' if stat_result['crawler'].is_sufficient else 'No'}")
|
||||
|
||||
# Show term coverage
|
||||
if hasattr(stat_result['result'], 'term_frequencies'):
|
||||
query_terms = test['query'].lower().split()
|
||||
covered = sum(1 for term in query_terms
|
||||
if term in stat_result['result'].term_frequencies)
|
||||
print(f" Term coverage: {covered}/{len(query_terms)} query terms found")
|
||||
|
||||
# Run embedding strategy
|
||||
print("\n🧠 Embedding Strategy:")
|
||||
emb_result = await crawl_with_strategy(
|
||||
test['url'],
|
||||
test['query'],
|
||||
'embedding',
|
||||
**embedding_config
|
||||
)
|
||||
|
||||
print(f" Pages crawled: {emb_result['pages']}")
|
||||
print(f" Time taken: {emb_result['elapsed']:.2f}s")
|
||||
print(f" Confidence: {emb_result['confidence']:.1%}")
|
||||
print(f" Sufficient: {'Yes' if emb_result['crawler'].is_sufficient else 'No'}")
|
||||
|
||||
# Show semantic understanding
|
||||
if emb_result['result'].expanded_queries:
|
||||
print(f" Query variations: {len(emb_result['result'].expanded_queries)}")
|
||||
print(f" Semantic gaps: {len(emb_result['result'].semantic_gaps)}")
|
||||
|
||||
# Compare results
|
||||
print("\n📈 Comparison:")
|
||||
efficiency_diff = ((stat_result['pages'] - emb_result['pages']) /
|
||||
stat_result['pages'] * 100) if stat_result['pages'] > 0 else 0
|
||||
|
||||
print(f" Efficiency: ", end="")
|
||||
if efficiency_diff > 0:
|
||||
print(f"Embedding used {efficiency_diff:.0f}% fewer pages")
|
||||
else:
|
||||
print(f"Statistical used {-efficiency_diff:.0f}% fewer pages")
|
||||
|
||||
print(f" Speed: ", end="")
|
||||
if stat_result['elapsed'] < emb_result['elapsed']:
|
||||
print(f"Statistical was {emb_result['elapsed']/stat_result['elapsed']:.1f}x faster")
|
||||
else:
|
||||
print(f"Embedding was {stat_result['elapsed']/emb_result['elapsed']:.1f}x faster")
|
||||
|
||||
print(f" Confidence difference: {abs(stat_result['confidence'] - emb_result['confidence'])*100:.0f} percentage points")
|
||||
|
||||
# Recommendation
|
||||
print("\n💡 Recommendation:")
|
||||
if 'specific' in test['name'].lower() or all(len(term) > 5 for term in test['query'].split()):
|
||||
print(" → Statistical strategy is likely better for this use case (specific terms)")
|
||||
elif 'conceptual' in test['name'].lower() or 'semantic' in test['name'].lower():
|
||||
print(" → Embedding strategy is likely better for this use case (semantic understanding)")
|
||||
else:
|
||||
if emb_result['confidence'] > stat_result['confidence'] + 0.1:
|
||||
print(" → Embedding strategy achieved significantly better understanding")
|
||||
elif stat_result['elapsed'] < emb_result['elapsed'] / 2:
|
||||
print(" → Statistical strategy is much faster with similar results")
|
||||
else:
|
||||
print(" → Both strategies performed similarly; choose based on your priorities")
|
||||
|
||||
# Summary recommendations
|
||||
print("\n" + "="*70)
|
||||
print("STRATEGY SELECTION GUIDE")
|
||||
print("="*70)
|
||||
print("\n✅ Use STATISTICAL strategy when:")
|
||||
print(" - Queries contain specific technical terms")
|
||||
print(" - Speed is critical")
|
||||
print(" - No API access available")
|
||||
print(" - Working with well-structured documentation")
|
||||
|
||||
print("\n✅ Use EMBEDDING strategy when:")
|
||||
print(" - Queries are conceptual or ambiguous")
|
||||
print(" - Semantic understanding is important")
|
||||
print(" - Need to detect irrelevant content")
|
||||
print(" - Working with diverse content sources")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,232 @@
|
||||
"""
|
||||
Knowledge Base Export and Import
|
||||
|
||||
This example demonstrates how to export crawled knowledge bases and
|
||||
import them for reuse, sharing, or analysis.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
|
||||
|
||||
async def build_knowledge_base():
|
||||
"""Build a knowledge base about web technologies"""
|
||||
print("="*60)
|
||||
print("PHASE 1: Building Knowledge Base")
|
||||
print("="*60)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler)
|
||||
|
||||
# Crawl information about HTTP
|
||||
print("\n1. Gathering HTTP protocol information...")
|
||||
await adaptive.digest(
|
||||
start_url="https://httpbin.org",
|
||||
query="http methods headers status codes"
|
||||
)
|
||||
print(f" - Pages crawled: {len(adaptive.state.crawled_urls)}")
|
||||
print(f" - Confidence: {adaptive.confidence:.2%}")
|
||||
|
||||
# Add more information about APIs
|
||||
print("\n2. Adding API documentation knowledge...")
|
||||
await adaptive.digest(
|
||||
start_url="https://httpbin.org/anything",
|
||||
query="rest api json response request"
|
||||
)
|
||||
print(f" - Total pages: {len(adaptive.state.crawled_urls)}")
|
||||
print(f" - Confidence: {adaptive.confidence:.2%}")
|
||||
|
||||
# Export the knowledge base
|
||||
export_path = "web_tech_knowledge.jsonl"
|
||||
print(f"\n3. Exporting knowledge base to {export_path}")
|
||||
adaptive.export_knowledge_base(export_path)
|
||||
|
||||
# Show export statistics
|
||||
export_size = Path(export_path).stat().st_size / 1024
|
||||
with open(export_path, 'r') as f:
|
||||
line_count = sum(1 for _ in f)
|
||||
|
||||
print(f" - Exported {line_count} documents")
|
||||
print(f" - File size: {export_size:.1f} KB")
|
||||
|
||||
return export_path
|
||||
|
||||
|
||||
async def analyze_knowledge_base(kb_path):
|
||||
"""Analyze the exported knowledge base"""
|
||||
print("\n" + "="*60)
|
||||
print("PHASE 2: Analyzing Exported Knowledge Base")
|
||||
print("="*60)
|
||||
|
||||
# Read and analyze JSONL
|
||||
documents = []
|
||||
with open(kb_path, 'r') as f:
|
||||
for line in f:
|
||||
documents.append(json.loads(line))
|
||||
|
||||
print(f"\nKnowledge base contains {len(documents)} documents:")
|
||||
|
||||
# Analyze document properties
|
||||
total_content_length = 0
|
||||
urls_by_domain = {}
|
||||
|
||||
for doc in documents:
|
||||
# Content analysis
|
||||
content_length = len(doc.get('content', ''))
|
||||
total_content_length += content_length
|
||||
|
||||
# URL analysis
|
||||
url = doc.get('url', '')
|
||||
domain = url.split('/')[2] if url.startswith('http') else 'unknown'
|
||||
urls_by_domain[domain] = urls_by_domain.get(domain, 0) + 1
|
||||
|
||||
# Show sample document
|
||||
if documents.index(doc) == 0:
|
||||
print(f"\nSample document structure:")
|
||||
print(f" - URL: {url}")
|
||||
print(f" - Content length: {content_length} chars")
|
||||
print(f" - Has metadata: {'metadata' in doc}")
|
||||
print(f" - Has links: {len(doc.get('links', []))} links")
|
||||
print(f" - Query: {doc.get('query', 'N/A')}")
|
||||
|
||||
print(f"\nContent statistics:")
|
||||
print(f" - Total content: {total_content_length:,} characters")
|
||||
print(f" - Average per document: {total_content_length/len(documents):,.0f} chars")
|
||||
|
||||
print(f"\nDomain distribution:")
|
||||
for domain, count in urls_by_domain.items():
|
||||
print(f" - {domain}: {count} pages")
|
||||
|
||||
|
||||
async def import_and_continue():
|
||||
"""Import a knowledge base and continue crawling"""
|
||||
print("\n" + "="*60)
|
||||
print("PHASE 3: Importing and Extending Knowledge Base")
|
||||
print("="*60)
|
||||
|
||||
kb_path = "web_tech_knowledge.jsonl"
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
# Create new adaptive crawler
|
||||
adaptive = AdaptiveCrawler(crawler)
|
||||
|
||||
# Import existing knowledge base
|
||||
print(f"\n1. Importing knowledge base from {kb_path}")
|
||||
await adaptive.import_knowledge_base(kb_path)
|
||||
|
||||
print(f" - Imported {len(adaptive.state.knowledge_base)} documents")
|
||||
print(f" - Existing URLs: {len(adaptive.state.crawled_urls)}")
|
||||
|
||||
# Check current state
|
||||
print("\n2. Checking imported knowledge state:")
|
||||
adaptive.print_stats(detailed=False)
|
||||
|
||||
# Continue crawling with new query
|
||||
print("\n3. Extending knowledge with new query...")
|
||||
await adaptive.digest(
|
||||
start_url="https://httpbin.org/status/200",
|
||||
query="error handling retry timeout"
|
||||
)
|
||||
|
||||
print("\n4. Final knowledge base state:")
|
||||
adaptive.print_stats(detailed=False)
|
||||
|
||||
# Export extended knowledge base
|
||||
extended_path = "web_tech_knowledge_extended.jsonl"
|
||||
adaptive.export_knowledge_base(extended_path)
|
||||
print(f"\n5. Extended knowledge base exported to {extended_path}")
|
||||
|
||||
|
||||
async def share_knowledge_bases():
|
||||
"""Demonstrate sharing knowledge bases between projects"""
|
||||
print("\n" + "="*60)
|
||||
print("PHASE 4: Sharing Knowledge Between Projects")
|
||||
print("="*60)
|
||||
|
||||
# Simulate two different projects
|
||||
project_a_kb = "project_a_knowledge.jsonl"
|
||||
project_b_kb = "project_b_knowledge.jsonl"
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
# Project A: Security documentation
|
||||
print("\n1. Project A: Building security knowledge...")
|
||||
crawler_a = AdaptiveCrawler(crawler)
|
||||
await crawler_a.digest(
|
||||
start_url="https://httpbin.org/basic-auth/user/pass",
|
||||
query="authentication security headers"
|
||||
)
|
||||
crawler_a.export_knowledge_base(project_a_kb)
|
||||
print(f" - Exported {len(crawler_a.state.knowledge_base)} documents")
|
||||
|
||||
# Project B: API testing
|
||||
print("\n2. Project B: Building testing knowledge...")
|
||||
crawler_b = AdaptiveCrawler(crawler)
|
||||
await crawler_b.digest(
|
||||
start_url="https://httpbin.org/anything",
|
||||
query="testing endpoints mocking"
|
||||
)
|
||||
crawler_b.export_knowledge_base(project_b_kb)
|
||||
print(f" - Exported {len(crawler_b.state.knowledge_base)} documents")
|
||||
|
||||
# Merge knowledge bases
|
||||
print("\n3. Merging knowledge bases...")
|
||||
merged_crawler = AdaptiveCrawler(crawler)
|
||||
|
||||
# Import both knowledge bases
|
||||
await merged_crawler.import_knowledge_base(project_a_kb)
|
||||
initial_size = len(merged_crawler.state.knowledge_base)
|
||||
|
||||
await merged_crawler.import_knowledge_base(project_b_kb)
|
||||
final_size = len(merged_crawler.state.knowledge_base)
|
||||
|
||||
print(f" - Project A documents: {initial_size}")
|
||||
print(f" - Additional from Project B: {final_size - initial_size}")
|
||||
print(f" - Total merged documents: {final_size}")
|
||||
|
||||
# Export merged knowledge
|
||||
merged_kb = "merged_knowledge.jsonl"
|
||||
merged_crawler.export_knowledge_base(merged_kb)
|
||||
print(f"\n4. Merged knowledge base exported to {merged_kb}")
|
||||
|
||||
# Show combined coverage
|
||||
print("\n5. Combined knowledge coverage:")
|
||||
merged_crawler.print_stats(detailed=False)
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run all examples"""
|
||||
try:
|
||||
# Build initial knowledge base
|
||||
kb_path = await build_knowledge_base()
|
||||
|
||||
# Analyze the export
|
||||
await analyze_knowledge_base(kb_path)
|
||||
|
||||
# Import and extend
|
||||
await import_and_continue()
|
||||
|
||||
# Demonstrate sharing
|
||||
await share_knowledge_bases()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("All examples completed successfully!")
|
||||
print("="*60)
|
||||
|
||||
finally:
|
||||
# Clean up generated files
|
||||
print("\nCleaning up generated files...")
|
||||
for file in [
|
||||
"web_tech_knowledge.jsonl",
|
||||
"web_tech_knowledge_extended.jsonl",
|
||||
"project_a_knowledge.jsonl",
|
||||
"project_b_knowledge.jsonl",
|
||||
"merged_knowledge.jsonl"
|
||||
]:
|
||||
Path(file).unlink(missing_ok=True)
|
||||
print("Cleanup complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,154 @@
|
||||
import asyncio
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig, LLMConfig
|
||||
|
||||
|
||||
async def test_configuration(name: str, config: AdaptiveConfig, url: str, query: str):
|
||||
"""Test a specific configuration"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Configuration: {name}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
result = await adaptive.digest(start_url=url, query=query)
|
||||
|
||||
print("\n" + "="*50)
|
||||
print("CRAWL STATISTICS")
|
||||
print("="*50)
|
||||
adaptive.print_stats(detailed=False)
|
||||
|
||||
# Get the most relevant content found
|
||||
print("\n" + "="*50)
|
||||
print("MOST RELEVANT PAGES")
|
||||
print("="*50)
|
||||
|
||||
relevant_pages = adaptive.get_relevant_content(top_k=5)
|
||||
for i, page in enumerate(relevant_pages, 1):
|
||||
print(f"\n{i}. {page['url']}")
|
||||
print(f" Relevance Score: {page['score']:.2%}")
|
||||
|
||||
# Show a snippet of the content
|
||||
content = page['content'] or ""
|
||||
if content:
|
||||
snippet = content[:200].replace('\n', ' ')
|
||||
if len(content) > 200:
|
||||
snippet += "..."
|
||||
print(f" Preview: {snippet}")
|
||||
|
||||
print(f"\n{'='*50}")
|
||||
print(f"Pages crawled: {len(result.crawled_urls)}")
|
||||
print(f"Final confidence: {adaptive.confidence:.1%}")
|
||||
print(f"Stopped reason: {result.metrics.get('stopped_reason', 'max_pages')}")
|
||||
|
||||
if result.metrics.get('is_irrelevant', False):
|
||||
print("⚠️ Query detected as irrelevant!")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def llm_embedding():
|
||||
"""Demonstrate various embedding configurations"""
|
||||
|
||||
print("EMBEDDING STRATEGY CONFIGURATION EXAMPLES")
|
||||
print("=" * 60)
|
||||
|
||||
# Base URL and query for testing
|
||||
test_url = "https://docs.python.org/3/library/asyncio.html"
|
||||
|
||||
openai_llm_config = LLMConfig(
|
||||
provider='openai/text-embedding-3-small',
|
||||
api_token=os.getenv('OPENAI_API_KEY'),
|
||||
temperature=0.7,
|
||||
max_tokens=2000
|
||||
)
|
||||
config_openai = AdaptiveConfig(
|
||||
strategy="embedding",
|
||||
max_pages=10,
|
||||
|
||||
# Use OpenAI embeddings
|
||||
embedding_llm_config=openai_llm_config,
|
||||
# embedding_llm_config={
|
||||
# 'provider': 'openai/text-embedding-3-small',
|
||||
# 'api_token': os.getenv('OPENAI_API_KEY')
|
||||
# },
|
||||
|
||||
# OpenAI embeddings are high quality, can be stricter
|
||||
embedding_k_exp=4.0,
|
||||
n_query_variations=12
|
||||
)
|
||||
|
||||
await test_configuration(
|
||||
"OpenAI Embeddings",
|
||||
config_openai,
|
||||
test_url,
|
||||
# "event-driven architecture patterns"
|
||||
"async await context managers coroutines"
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
|
||||
async def basic_adaptive_crawling():
|
||||
"""Basic adaptive crawling example"""
|
||||
|
||||
# Initialize the crawler
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# Create an adaptive crawler with default settings (statistical strategy)
|
||||
adaptive = AdaptiveCrawler(crawler)
|
||||
|
||||
# Note: You can also use embedding strategy for semantic understanding:
|
||||
# from crawl4ai import AdaptiveConfig
|
||||
# config = AdaptiveConfig(strategy="embedding")
|
||||
# adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
# Start adaptive crawling
|
||||
print("Starting adaptive crawl for Python async programming information...")
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/library/asyncio.html",
|
||||
query="async await context managers coroutines"
|
||||
)
|
||||
|
||||
# Display crawl statistics
|
||||
print("\n" + "="*50)
|
||||
print("CRAWL STATISTICS")
|
||||
print("="*50)
|
||||
adaptive.print_stats(detailed=False)
|
||||
|
||||
# Get the most relevant content found
|
||||
print("\n" + "="*50)
|
||||
print("MOST RELEVANT PAGES")
|
||||
print("="*50)
|
||||
|
||||
relevant_pages = adaptive.get_relevant_content(top_k=5)
|
||||
for i, page in enumerate(relevant_pages, 1):
|
||||
print(f"\n{i}. {page['url']}")
|
||||
print(f" Relevance Score: {page['score']:.2%}")
|
||||
|
||||
# Show a snippet of the content
|
||||
content = page['content'] or ""
|
||||
if content:
|
||||
snippet = content[:200].replace('\n', ' ')
|
||||
if len(content) > 200:
|
||||
snippet += "..."
|
||||
print(f" Preview: {snippet}")
|
||||
|
||||
# Show final confidence
|
||||
print(f"\n{'='*50}")
|
||||
print(f"Final Confidence: {adaptive.confidence:.2%}")
|
||||
print(f"Total Pages Crawled: {len(result.crawled_urls)}")
|
||||
print(f"Knowledge Base Size: {len(adaptive.state.knowledge_base)} documents")
|
||||
|
||||
|
||||
if adaptive.confidence >= 0.8:
|
||||
print("✓ High confidence - can answer detailed questions about async Python")
|
||||
elif adaptive.confidence >= 0.6:
|
||||
print("~ Moderate confidence - can answer basic questions")
|
||||
else:
|
||||
print("✗ Low confidence - need more information")
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(llm_embedding())
|
||||
# asyncio.run(basic_adaptive_crawling())
|
||||
@@ -0,0 +1,110 @@
|
||||
"""
|
||||
This example demonstrates how to use JSON CSS extraction to scrape product information
|
||||
from Amazon search results. It shows how to extract structured data like product titles,
|
||||
prices, ratings, and other details using CSS selectors.
|
||||
"""
|
||||
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
import json
|
||||
|
||||
|
||||
async def extract_amazon_products():
|
||||
# Initialize browser config
|
||||
browser_config = BrowserConfig(browser_type="chromium", headless=True)
|
||||
|
||||
# Initialize crawler config with JSON CSS extraction strategy
|
||||
crawler_config = CrawlerRunConfig(
|
||||
extraction_strategy=JsonCssExtractionStrategy(
|
||||
schema={
|
||||
"name": "Amazon Product Search Results",
|
||||
"baseSelector": "[data-component-type='s-search-result']",
|
||||
"fields": [
|
||||
{
|
||||
"name": "asin",
|
||||
"selector": "",
|
||||
"type": "attribute",
|
||||
"attribute": "data-asin",
|
||||
},
|
||||
{"name": "title", "selector": "h2 a span", "type": "text"},
|
||||
{
|
||||
"name": "url",
|
||||
"selector": "h2 a",
|
||||
"type": "attribute",
|
||||
"attribute": "href",
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"selector": ".s-image",
|
||||
"type": "attribute",
|
||||
"attribute": "src",
|
||||
},
|
||||
{
|
||||
"name": "rating",
|
||||
"selector": ".a-icon-star-small .a-icon-alt",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "reviews_count",
|
||||
"selector": "[data-csa-c-func-deps='aui-da-a-popover'] ~ span span",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": ".a-price .a-offscreen",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "original_price",
|
||||
"selector": ".a-price.a-text-price .a-offscreen",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "sponsored",
|
||||
"selector": ".puis-sponsored-label-text",
|
||||
"type": "exists",
|
||||
},
|
||||
{
|
||||
"name": "delivery_info",
|
||||
"selector": "[data-cy='delivery-recipe'] .a-color-base",
|
||||
"type": "text",
|
||||
"multiple": True,
|
||||
},
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# Example search URL (you should replace with your actual Amazon URL)
|
||||
url = "https://www.amazon.com/s?k=Samsung+Galaxy+Tab"
|
||||
|
||||
# Use context manager for proper resource handling
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Extract the data
|
||||
result = await crawler.arun(url=url, config=crawler_config)
|
||||
|
||||
# Process and print the results
|
||||
if result and result.extracted_content:
|
||||
# Parse the JSON string into a list of products
|
||||
products = json.loads(result.extracted_content)
|
||||
|
||||
# Process each product in the list
|
||||
for product in products:
|
||||
print("\nProduct Details:")
|
||||
print(f"ASIN: {product.get('asin')}")
|
||||
print(f"Title: {product.get('title')}")
|
||||
print(f"Price: {product.get('price')}")
|
||||
print(f"Original Price: {product.get('original_price')}")
|
||||
print(f"Rating: {product.get('rating')}")
|
||||
print(f"Reviews: {product.get('reviews_count')}")
|
||||
print(f"Sponsored: {'Yes' if product.get('sponsored') else 'No'}")
|
||||
if product.get("delivery_info"):
|
||||
print(f"Delivery: {' '.join(product['delivery_info'])}")
|
||||
print("-" * 80)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import asyncio
|
||||
|
||||
asyncio.run(extract_amazon_products())
|
||||
@@ -0,0 +1,150 @@
|
||||
"""
|
||||
This example demonstrates how to use JSON CSS extraction to scrape product information
|
||||
from Amazon search results. It shows how to extract structured data like product titles,
|
||||
prices, ratings, and other details using CSS selectors.
|
||||
"""
|
||||
|
||||
from crawl4ai import AsyncWebCrawler, CacheMode
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
import json
|
||||
from playwright.async_api import Page, BrowserContext
|
||||
|
||||
|
||||
async def extract_amazon_products():
|
||||
# Initialize browser config
|
||||
browser_config = BrowserConfig(
|
||||
# browser_type="chromium",
|
||||
headless=True
|
||||
)
|
||||
|
||||
# Initialize crawler config with JSON CSS extraction strategy nav-search-submit-button
|
||||
crawler_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
extraction_strategy=JsonCssExtractionStrategy(
|
||||
schema={
|
||||
"name": "Amazon Product Search Results",
|
||||
"baseSelector": "[data-component-type='s-search-result']",
|
||||
"fields": [
|
||||
{
|
||||
"name": "asin",
|
||||
"selector": "",
|
||||
"type": "attribute",
|
||||
"attribute": "data-asin",
|
||||
},
|
||||
{"name": "title", "selector": "h2 a span", "type": "text"},
|
||||
{
|
||||
"name": "url",
|
||||
"selector": "h2 a",
|
||||
"type": "attribute",
|
||||
"attribute": "href",
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"selector": ".s-image",
|
||||
"type": "attribute",
|
||||
"attribute": "src",
|
||||
},
|
||||
{
|
||||
"name": "rating",
|
||||
"selector": ".a-icon-star-small .a-icon-alt",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "reviews_count",
|
||||
"selector": "[data-csa-c-func-deps='aui-da-a-popover'] ~ span span",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": ".a-price .a-offscreen",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "original_price",
|
||||
"selector": ".a-price.a-text-price .a-offscreen",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "sponsored",
|
||||
"selector": ".puis-sponsored-label-text",
|
||||
"type": "exists",
|
||||
},
|
||||
{
|
||||
"name": "delivery_info",
|
||||
"selector": "[data-cy='delivery-recipe'] .a-color-base",
|
||||
"type": "text",
|
||||
"multiple": True,
|
||||
},
|
||||
],
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
url = "https://www.amazon.com/"
|
||||
|
||||
async def after_goto(
|
||||
page: Page, context: BrowserContext, url: str, response: dict, **kwargs
|
||||
):
|
||||
"""Hook called after navigating to each URL"""
|
||||
print(f"[HOOK] after_goto - Successfully loaded: {url}")
|
||||
|
||||
try:
|
||||
# Wait for search box to be available
|
||||
search_box = await page.wait_for_selector(
|
||||
"#twotabsearchtextbox", timeout=1000
|
||||
)
|
||||
|
||||
# Type the search query
|
||||
await search_box.fill("Samsung Galaxy Tab")
|
||||
|
||||
# Get the search button and prepare for navigation
|
||||
search_button = await page.wait_for_selector(
|
||||
"#nav-search-submit-button", timeout=1000
|
||||
)
|
||||
|
||||
# Click with navigation waiting
|
||||
await search_button.click()
|
||||
|
||||
# Wait for search results to load
|
||||
await page.wait_for_selector(
|
||||
'[data-component-type="s-search-result"]', timeout=10000
|
||||
)
|
||||
print("[HOOK] Search completed and results loaded!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"[HOOK] Error during search operation: {str(e)}")
|
||||
|
||||
return page
|
||||
|
||||
# Use context manager for proper resource handling
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
crawler.crawler_strategy.set_hook("after_goto", after_goto)
|
||||
|
||||
# Extract the data
|
||||
result = await crawler.arun(url=url, config=crawler_config)
|
||||
|
||||
# Process and print the results
|
||||
if result and result.extracted_content:
|
||||
# Parse the JSON string into a list of products
|
||||
products = json.loads(result.extracted_content)
|
||||
|
||||
# Process each product in the list
|
||||
for product in products:
|
||||
print("\nProduct Details:")
|
||||
print(f"ASIN: {product.get('asin')}")
|
||||
print(f"Title: {product.get('title')}")
|
||||
print(f"Price: {product.get('price')}")
|
||||
print(f"Original Price: {product.get('original_price')}")
|
||||
print(f"Rating: {product.get('rating')}")
|
||||
print(f"Reviews: {product.get('reviews_count')}")
|
||||
print(f"Sponsored: {'Yes' if product.get('sponsored') else 'No'}")
|
||||
if product.get("delivery_info"):
|
||||
print(f"Delivery: {' '.join(product['delivery_info'])}")
|
||||
print("-" * 80)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import asyncio
|
||||
|
||||
asyncio.run(extract_amazon_products())
|
||||
@@ -0,0 +1,126 @@
|
||||
"""
|
||||
This example demonstrates how to use JSON CSS extraction to scrape product information
|
||||
from Amazon search results. It shows how to extract structured data like product titles,
|
||||
prices, ratings, and other details using CSS selectors.
|
||||
"""
|
||||
|
||||
from crawl4ai import AsyncWebCrawler, CacheMode
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
import json
|
||||
|
||||
|
||||
async def extract_amazon_products():
|
||||
# Initialize browser config
|
||||
browser_config = BrowserConfig(
|
||||
# browser_type="chromium",
|
||||
headless=True
|
||||
)
|
||||
|
||||
js_code_to_search = """
|
||||
const task = async () => {
|
||||
document.querySelector('#twotabsearchtextbox').value = 'Samsung Galaxy Tab';
|
||||
document.querySelector('#nav-search-submit-button').click();
|
||||
}
|
||||
await task();
|
||||
"""
|
||||
js_code_to_search_sync = """
|
||||
document.querySelector('#twotabsearchtextbox').value = 'Samsung Galaxy Tab';
|
||||
document.querySelector('#nav-search-submit-button').click();
|
||||
"""
|
||||
crawler_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
js_code=js_code_to_search,
|
||||
wait_for='css:[data-component-type="s-search-result"]',
|
||||
extraction_strategy=JsonCssExtractionStrategy(
|
||||
schema={
|
||||
"name": "Amazon Product Search Results",
|
||||
"baseSelector": "[data-component-type='s-search-result']",
|
||||
"fields": [
|
||||
{
|
||||
"name": "asin",
|
||||
"selector": "",
|
||||
"type": "attribute",
|
||||
"attribute": "data-asin",
|
||||
},
|
||||
{"name": "title", "selector": "h2 a span", "type": "text"},
|
||||
{
|
||||
"name": "url",
|
||||
"selector": "h2 a",
|
||||
"type": "attribute",
|
||||
"attribute": "href",
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"selector": ".s-image",
|
||||
"type": "attribute",
|
||||
"attribute": "src",
|
||||
},
|
||||
{
|
||||
"name": "rating",
|
||||
"selector": ".a-icon-star-small .a-icon-alt",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "reviews_count",
|
||||
"selector": "[data-csa-c-func-deps='aui-da-a-popover'] ~ span span",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": ".a-price .a-offscreen",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "original_price",
|
||||
"selector": ".a-price.a-text-price .a-offscreen",
|
||||
"type": "text",
|
||||
},
|
||||
{
|
||||
"name": "sponsored",
|
||||
"selector": ".puis-sponsored-label-text",
|
||||
"type": "exists",
|
||||
},
|
||||
{
|
||||
"name": "delivery_info",
|
||||
"selector": "[data-cy='delivery-recipe'] .a-color-base",
|
||||
"type": "text",
|
||||
"multiple": True,
|
||||
},
|
||||
],
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
# Example search URL (you should replace with your actual Amazon URL)
|
||||
url = "https://www.amazon.com/"
|
||||
|
||||
# Use context manager for proper resource handling
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Extract the data
|
||||
result = await crawler.arun(url=url, config=crawler_config)
|
||||
|
||||
# Process and print the results
|
||||
if result and result.extracted_content:
|
||||
# Parse the JSON string into a list of products
|
||||
products = json.loads(result.extracted_content)
|
||||
|
||||
# Process each product in the list
|
||||
for product in products:
|
||||
print("\nProduct Details:")
|
||||
print(f"ASIN: {product.get('asin')}")
|
||||
print(f"Title: {product.get('title')}")
|
||||
print(f"Price: {product.get('price')}")
|
||||
print(f"Original Price: {product.get('original_price')}")
|
||||
print(f"Rating: {product.get('rating')}")
|
||||
print(f"Reviews: {product.get('reviews_count')}")
|
||||
print(f"Sponsored: {'Yes' if product.get('sponsored') else 'No'}")
|
||||
if product.get("delivery_info"):
|
||||
print(f"Delivery: {' '.join(product['delivery_info'])}")
|
||||
print("-" * 80)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import asyncio
|
||||
|
||||
asyncio.run(extract_amazon_products())
|
||||
@@ -0,0 +1,79 @@
|
||||
import asyncio
|
||||
import time
|
||||
from crawl4ai.async_webcrawler import AsyncWebCrawler, CacheMode
|
||||
from crawl4ai.async_configs import CrawlerRunConfig
|
||||
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher, RateLimiter
|
||||
|
||||
VERBOSE = False
|
||||
|
||||
async def crawl_sequential(urls):
|
||||
config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, verbose=VERBOSE)
|
||||
results = []
|
||||
start_time = time.perf_counter()
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
for url in urls:
|
||||
result_container = await crawler.arun(url=url, config=config)
|
||||
results.append(result_container[0])
|
||||
total_time = time.perf_counter() - start_time
|
||||
return total_time, results
|
||||
|
||||
async def crawl_parallel_dispatcher(urls):
|
||||
config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, verbose=VERBOSE)
|
||||
# Dispatcher with rate limiter enabled (default behavior)
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
rate_limiter=RateLimiter(base_delay=(1.0, 3.0), max_delay=60.0, max_retries=3),
|
||||
max_session_permit=50,
|
||||
)
|
||||
start_time = time.perf_counter()
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result_container = await crawler.arun_many(urls=urls, config=config, dispatcher=dispatcher)
|
||||
results = []
|
||||
if isinstance(result_container, list):
|
||||
results = result_container
|
||||
else:
|
||||
async for res in result_container:
|
||||
results.append(res)
|
||||
total_time = time.perf_counter() - start_time
|
||||
return total_time, results
|
||||
|
||||
async def crawl_parallel_no_rate_limit(urls):
|
||||
config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, verbose=VERBOSE)
|
||||
# Dispatcher with no rate limiter and a high session permit to avoid queuing
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
rate_limiter=None,
|
||||
max_session_permit=len(urls) # allow all URLs concurrently
|
||||
)
|
||||
start_time = time.perf_counter()
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result_container = await crawler.arun_many(urls=urls, config=config, dispatcher=dispatcher)
|
||||
results = []
|
||||
if isinstance(result_container, list):
|
||||
results = result_container
|
||||
else:
|
||||
async for res in result_container:
|
||||
results.append(res)
|
||||
total_time = time.perf_counter() - start_time
|
||||
return total_time, results
|
||||
|
||||
async def main():
|
||||
urls = ["https://example.com"] * 100
|
||||
print(f"Crawling {len(urls)} URLs sequentially...")
|
||||
seq_time, seq_results = await crawl_sequential(urls)
|
||||
print(f"Sequential crawling took: {seq_time:.2f} seconds\n")
|
||||
|
||||
print(f"Crawling {len(urls)} URLs in parallel using arun_many with dispatcher (with rate limit)...")
|
||||
disp_time, disp_results = await crawl_parallel_dispatcher(urls)
|
||||
print(f"Parallel (dispatcher with rate limiter) took: {disp_time:.2f} seconds\n")
|
||||
|
||||
print(f"Crawling {len(urls)} URLs in parallel using dispatcher with no rate limiter...")
|
||||
no_rl_time, no_rl_results = await crawl_parallel_no_rate_limit(urls)
|
||||
print(f"Parallel (dispatcher without rate limiter) took: {no_rl_time:.2f} seconds\n")
|
||||
|
||||
print("Crawl4ai - Crawling Comparison")
|
||||
print("--------------------------------------------------------")
|
||||
print(f"Sequential crawling took: {seq_time:.2f} seconds")
|
||||
print(f"Parallel (dispatcher with rate limiter) took: {disp_time:.2f} seconds")
|
||||
print(f"Parallel (dispatcher without rate limiter) took: {no_rl_time:.2f} seconds")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 372 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 403 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 537 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 375 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 469 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.6 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 477 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 419 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 485 KiB |
@@ -0,0 +1,132 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Append-Only Scroll (Traditional Infinite Scroll)</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
h1 {
|
||||
color: #333;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.posts-container {
|
||||
max-width: 800px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 5px;
|
||||
padding: 20px;
|
||||
}
|
||||
|
||||
.post {
|
||||
background: #f9f9f9;
|
||||
padding: 15px;
|
||||
margin-bottom: 15px;
|
||||
border-radius: 5px;
|
||||
border: 1px solid #eee;
|
||||
}
|
||||
|
||||
.post-title {
|
||||
font-size: 18px;
|
||||
font-weight: bold;
|
||||
color: #2c3e50;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.post-content {
|
||||
color: #555;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.loading {
|
||||
text-align: center;
|
||||
padding: 20px;
|
||||
color: #888;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Traditional Infinite Scroll Demo</h1>
|
||||
<p style="text-align: center; color: #666;">This appends new content without removing old content</p>
|
||||
<div class="posts-container"></div>
|
||||
|
||||
<script>
|
||||
// Traditional infinite scroll - APPENDS content
|
||||
const container = document.querySelector('.posts-container');
|
||||
const totalPosts = 200;
|
||||
const postsPerPage = 20;
|
||||
let loadedPosts = 0;
|
||||
let isLoading = false;
|
||||
|
||||
// Generate fake post data
|
||||
function generatePost(index) {
|
||||
return {
|
||||
id: index,
|
||||
title: `Post Title #${index + 1}`,
|
||||
content: `This is the content of post ${index + 1}. In traditional infinite scroll, new content is appended to existing content. The DOM keeps growing. Post ID: ${index}`
|
||||
};
|
||||
}
|
||||
|
||||
// Load more posts - APPENDS to existing content
|
||||
function loadMorePosts() {
|
||||
if (isLoading || loadedPosts >= totalPosts) return;
|
||||
|
||||
isLoading = true;
|
||||
|
||||
// Show loading indicator
|
||||
const loadingDiv = document.createElement('div');
|
||||
loadingDiv.className = 'loading';
|
||||
loadingDiv.textContent = 'Loading more posts...';
|
||||
container.appendChild(loadingDiv);
|
||||
|
||||
// Simulate network delay
|
||||
setTimeout(() => {
|
||||
// Remove loading indicator
|
||||
container.removeChild(loadingDiv);
|
||||
|
||||
// Add new posts
|
||||
const fragment = document.createDocumentFragment();
|
||||
const endIndex = Math.min(loadedPosts + postsPerPage, totalPosts);
|
||||
|
||||
for (let i = loadedPosts; i < endIndex; i++) {
|
||||
const post = generatePost(i);
|
||||
const postElement = document.createElement('div');
|
||||
postElement.className = 'post';
|
||||
postElement.setAttribute('data-post-id', post.id);
|
||||
postElement.innerHTML = `
|
||||
<div class="post-title">${post.title}</div>
|
||||
<div class="post-content">${post.content}</div>
|
||||
`;
|
||||
fragment.appendChild(postElement);
|
||||
}
|
||||
|
||||
// APPEND new posts to existing ones
|
||||
container.appendChild(fragment);
|
||||
loadedPosts = endIndex;
|
||||
isLoading = false;
|
||||
|
||||
console.log(`Loaded ${loadedPosts} of ${totalPosts} posts`);
|
||||
}, 300);
|
||||
}
|
||||
|
||||
// Initial load
|
||||
loadMorePosts();
|
||||
|
||||
// Load more on scroll
|
||||
window.addEventListener('scroll', () => {
|
||||
const scrollBottom = window.innerHeight + window.scrollY;
|
||||
const threshold = document.body.offsetHeight - 500;
|
||||
|
||||
if (scrollBottom >= threshold) {
|
||||
loadMorePosts();
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,158 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Instagram-like Grid Virtual Scroll</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background-color: #fafafa;
|
||||
}
|
||||
|
||||
h1 {
|
||||
text-align: center;
|
||||
color: #262626;
|
||||
font-weight: 300;
|
||||
}
|
||||
|
||||
.feed-container {
|
||||
max-width: 935px;
|
||||
margin: 0 auto;
|
||||
height: 800px;
|
||||
overflow-y: auto;
|
||||
background: white;
|
||||
border: 1px solid #dbdbdb;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
.grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
gap: 28px;
|
||||
padding: 28px;
|
||||
}
|
||||
|
||||
.post {
|
||||
aspect-ratio: 1;
|
||||
background: #f0f0f0;
|
||||
border-radius: 3px;
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.post:hover .overlay {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
.post img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
}
|
||||
|
||||
.overlay {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
bottom: 0;
|
||||
background: rgba(0, 0, 0, 0.3);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
color: white;
|
||||
font-size: 14px;
|
||||
opacity: 0;
|
||||
transition: opacity 0.2s;
|
||||
}
|
||||
|
||||
.stats {
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Instagram Grid Virtual Scroll</h1>
|
||||
<p style="text-align: center; color: #8e8e8e;">Grid layout with virtual scrolling - only visible rows are rendered</p>
|
||||
<div class="feed-container">
|
||||
<div class="grid" id="grid"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// Instagram-like grid virtual scroll
|
||||
const grid = document.getElementById('grid');
|
||||
const container = document.querySelector('.feed-container');
|
||||
const totalPosts = 999; // Instagram style count
|
||||
const postsPerRow = 3;
|
||||
const rowsPerPage = 4; // 12 posts per page
|
||||
const postsPerPage = postsPerRow * rowsPerPage;
|
||||
let currentStartIndex = 0;
|
||||
|
||||
// Generate fake Instagram post data
|
||||
const allPosts = [];
|
||||
for (let i = 0; i < totalPosts; i++) {
|
||||
allPosts.push({
|
||||
id: i,
|
||||
likes: Math.floor(Math.random() * 10000),
|
||||
comments: Math.floor(Math.random() * 500),
|
||||
imageNumber: (i % 10) + 1 // Cycle through 10 placeholder images
|
||||
});
|
||||
}
|
||||
|
||||
// Render grid - REPLACES content for performance
|
||||
function renderGrid(startIndex) {
|
||||
const posts = [];
|
||||
const endIndex = Math.min(startIndex + postsPerPage, totalPosts);
|
||||
|
||||
for (let i = startIndex; i < endIndex; i++) {
|
||||
const post = allPosts[i];
|
||||
posts.push(`
|
||||
<div class="post" data-post-id="${post.id}">
|
||||
<img src="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='400' height='400'%3E%3Crect width='400' height='400' fill='%23${Math.floor(Math.random()*16777215).toString(16)}'/%3E%3Ctext x='50%25' y='50%25' text-anchor='middle' dy='.3em' font-family='Arial' font-size='48' fill='white'%3E${post.id + 1}%3C/text%3E%3C/svg%3E" alt="Post ${post.id + 1}">
|
||||
<div class="overlay">
|
||||
<div class="stats">
|
||||
<span>❤️ ${post.likes.toLocaleString()}</span>
|
||||
<span>💬 ${post.comments}</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
// REPLACE grid content (virtual scroll)
|
||||
grid.innerHTML = posts.join('');
|
||||
currentStartIndex = startIndex;
|
||||
}
|
||||
|
||||
// Initial render
|
||||
renderGrid(0);
|
||||
|
||||
// Handle scroll
|
||||
let scrollTimeout;
|
||||
container.addEventListener('scroll', () => {
|
||||
clearTimeout(scrollTimeout);
|
||||
scrollTimeout = setTimeout(() => {
|
||||
const scrollTop = container.scrollTop;
|
||||
const scrollHeight = container.scrollHeight;
|
||||
const clientHeight = container.clientHeight;
|
||||
|
||||
// Calculate which "page" we should show
|
||||
const scrollPercentage = scrollTop / (scrollHeight - clientHeight);
|
||||
const targetIndex = Math.floor(scrollPercentage * (totalPosts - postsPerPage) / postsPerPage) * postsPerPage;
|
||||
|
||||
// When scrolled to bottom, show next page
|
||||
if (scrollTop + clientHeight >= scrollHeight - 100) {
|
||||
const nextIndex = currentStartIndex + postsPerPage;
|
||||
if (nextIndex < totalPosts) {
|
||||
renderGrid(nextIndex);
|
||||
container.scrollTop = 100; // Reset scroll for continuous experience
|
||||
}
|
||||
}
|
||||
}, 50);
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,210 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>News Feed with Mixed Scroll Behavior</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: Georgia, serif;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background-color: #f8f8f8;
|
||||
}
|
||||
|
||||
h1 {
|
||||
text-align: center;
|
||||
color: #1a1a1a;
|
||||
font-size: 32px;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.description {
|
||||
text-align: center;
|
||||
color: #666;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
#newsContainer {
|
||||
max-width: 900px;
|
||||
margin: 0 auto;
|
||||
height: 700px;
|
||||
overflow-y: auto;
|
||||
background: white;
|
||||
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
|
||||
padding: 20px;
|
||||
}
|
||||
|
||||
.article {
|
||||
margin-bottom: 30px;
|
||||
padding-bottom: 30px;
|
||||
border-bottom: 1px solid #e0e0e0;
|
||||
}
|
||||
|
||||
.article:last-child {
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
.article-header {
|
||||
margin-bottom: 15px;
|
||||
}
|
||||
|
||||
.category {
|
||||
display: inline-block;
|
||||
background: #ff6b6b;
|
||||
color: white;
|
||||
padding: 4px 12px;
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
border-radius: 3px;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.headline {
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #1a1a1a;
|
||||
margin: 10px 0;
|
||||
line-height: 1.3;
|
||||
}
|
||||
|
||||
.meta {
|
||||
color: #888;
|
||||
font-size: 14px;
|
||||
margin-bottom: 15px;
|
||||
}
|
||||
|
||||
.content {
|
||||
font-size: 16px;
|
||||
line-height: 1.8;
|
||||
color: #333;
|
||||
}
|
||||
|
||||
.featured {
|
||||
background: #fff9e6;
|
||||
padding: 20px;
|
||||
border-radius: 5px;
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.featured .category {
|
||||
background: #ffa500;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>📰 Dynamic News Feed</h1>
|
||||
<p class="description">Mixed behavior: Featured articles stay, regular articles use virtual scroll</p>
|
||||
<div id="newsContainer"></div>
|
||||
|
||||
<script>
|
||||
const container = document.getElementById('newsContainer');
|
||||
const totalArticles = 100;
|
||||
const articlesPerPage = 5;
|
||||
let currentRegularIndex = 0;
|
||||
|
||||
// Categories for variety
|
||||
const categories = ['Politics', 'Technology', 'Business', 'Science', 'Sports', 'Entertainment'];
|
||||
|
||||
// Generate article data
|
||||
const featuredArticles = [];
|
||||
const regularArticles = [];
|
||||
|
||||
// 3 featured articles that always stay
|
||||
for (let i = 0; i < 3; i++) {
|
||||
featuredArticles.push({
|
||||
id: `featured-${i}`,
|
||||
category: 'Featured',
|
||||
headline: `Breaking: Major Story ${i + 1} That Stays Visible`,
|
||||
date: new Date().toLocaleDateString(),
|
||||
content: `This is featured article ${i + 1}. Featured articles remain in the DOM and are not replaced during scrolling. They provide important persistent content.`
|
||||
});
|
||||
}
|
||||
|
||||
// Regular articles that get virtualized
|
||||
for (let i = 0; i < totalArticles; i++) {
|
||||
regularArticles.push({
|
||||
id: `article-${i}`,
|
||||
category: categories[i % categories.length],
|
||||
headline: `${categories[i % categories.length]} News: Article ${i + 1} of ${totalArticles}`,
|
||||
date: new Date(Date.now() - i * 86400000).toLocaleDateString(),
|
||||
content: `This is regular article ${i + 1}. These articles are replaced as you scroll to maintain performance. Only a subset is shown at any time. Article ID: ${i}`
|
||||
});
|
||||
}
|
||||
|
||||
// Render articles - Featured stay, regular ones are replaced
|
||||
function renderArticles(regularStartIndex) {
|
||||
const html = [];
|
||||
|
||||
// Always show featured articles
|
||||
featuredArticles.forEach(article => {
|
||||
html.push(`
|
||||
<div class="article featured" data-article-id="${article.id}">
|
||||
<div class="article-header">
|
||||
<span class="category">${article.category}</span>
|
||||
<h2 class="headline">${article.headline}</h2>
|
||||
<div class="meta">📅 ${article.date}</div>
|
||||
</div>
|
||||
<div class="content">${article.content}</div>
|
||||
</div>
|
||||
`);
|
||||
});
|
||||
|
||||
// Add divider
|
||||
html.push('<div style="text-align: center; color: #999; margin: 20px 0;">— Latest News —</div>');
|
||||
|
||||
// Show current page of regular articles (virtual scroll)
|
||||
const endIndex = Math.min(regularStartIndex + articlesPerPage, totalArticles);
|
||||
for (let i = regularStartIndex; i < endIndex; i++) {
|
||||
const article = regularArticles[i];
|
||||
html.push(`
|
||||
<div class="article" data-article-id="${article.id}">
|
||||
<div class="article-header">
|
||||
<span class="category" style="background: ${getCategoryColor(article.category)}">${article.category}</span>
|
||||
<h2 class="headline">${article.headline}</h2>
|
||||
<div class="meta">📅 ${article.date}</div>
|
||||
</div>
|
||||
<div class="content">${article.content}</div>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
container.innerHTML = html.join('');
|
||||
currentRegularIndex = regularStartIndex;
|
||||
}
|
||||
|
||||
function getCategoryColor(category) {
|
||||
const colors = {
|
||||
'Politics': '#e74c3c',
|
||||
'Technology': '#3498db',
|
||||
'Business': '#2ecc71',
|
||||
'Science': '#9b59b6',
|
||||
'Sports': '#f39c12',
|
||||
'Entertainment': '#e91e63'
|
||||
};
|
||||
return colors[category] || '#95a5a6';
|
||||
}
|
||||
|
||||
// Initial render
|
||||
renderArticles(0);
|
||||
|
||||
// Handle scroll
|
||||
container.addEventListener('scroll', () => {
|
||||
const scrollTop = container.scrollTop;
|
||||
const scrollHeight = container.scrollHeight;
|
||||
const clientHeight = container.clientHeight;
|
||||
|
||||
// When near bottom, load next page of regular articles
|
||||
if (scrollTop + clientHeight >= scrollHeight - 200) {
|
||||
const nextIndex = currentRegularIndex + articlesPerPage;
|
||||
if (nextIndex < totalArticles) {
|
||||
renderArticles(nextIndex);
|
||||
// Scroll to where regular articles start
|
||||
const regularStart = document.querySelector('.article:not(.featured)');
|
||||
if (regularStart) {
|
||||
container.scrollTop = regularStart.offsetTop - 100;
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,122 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Twitter-like Virtual Scroll</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: Arial, sans-serif;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background-color: #f0f2f5;
|
||||
}
|
||||
|
||||
h1 {
|
||||
color: #1da1f2;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
#timeline {
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
height: 600px;
|
||||
overflow-y: auto;
|
||||
background: white;
|
||||
border: 1px solid #e1e8ed;
|
||||
border-radius: 10px;
|
||||
}
|
||||
|
||||
.tweet {
|
||||
padding: 15px;
|
||||
border-bottom: 1px solid #e1e8ed;
|
||||
min-height: 80px;
|
||||
}
|
||||
|
||||
.tweet:hover {
|
||||
background-color: #f7f9fa;
|
||||
}
|
||||
|
||||
.author {
|
||||
font-weight: bold;
|
||||
color: #14171a;
|
||||
margin-bottom: 5px;
|
||||
}
|
||||
|
||||
.content {
|
||||
color: #14171a;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.stats {
|
||||
color: #657786;
|
||||
font-size: 14px;
|
||||
margin-top: 10px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Virtual Scroll Demo - Twitter Style</h1>
|
||||
<p style="text-align: center; color: #666;">This simulates Twitter's timeline where content is replaced as you scroll</p>
|
||||
<div id="timeline"></div>
|
||||
|
||||
<script>
|
||||
// Simulate Twitter-like virtual scrolling where DOM elements are replaced
|
||||
const timeline = document.getElementById('timeline');
|
||||
const totalTweets = 500;
|
||||
const tweetsPerPage = 10;
|
||||
let currentIndex = 0;
|
||||
|
||||
// Generate fake tweet data
|
||||
const allTweets = [];
|
||||
for (let i = 0; i < totalTweets; i++) {
|
||||
allTweets.push({
|
||||
id: i,
|
||||
author: `User_${i + 1}`,
|
||||
content: `This is tweet #${i + 1} of ${totalTweets}. Virtual scrolling replaces DOM elements to maintain performance. Unique content ID: ${i}`,
|
||||
likes: Math.floor(Math.random() * 1000),
|
||||
retweets: Math.floor(Math.random() * 500)
|
||||
});
|
||||
}
|
||||
|
||||
// Render tweets - REPLACES content
|
||||
function renderTweets(startIndex) {
|
||||
const tweets = [];
|
||||
const endIndex = Math.min(startIndex + tweetsPerPage, totalTweets);
|
||||
|
||||
for (let i = startIndex; i < endIndex; i++) {
|
||||
const tweet = allTweets[i];
|
||||
tweets.push(`
|
||||
<div class="tweet" data-tweet-id="${tweet.id}">
|
||||
<div class="author">@${tweet.author}</div>
|
||||
<div class="content">${tweet.content}</div>
|
||||
<div class="stats">❤️ ${tweet.likes} | 🔁 ${tweet.retweets}</div>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
// REPLACE entire content (virtual scroll behavior)
|
||||
timeline.innerHTML = tweets.join('');
|
||||
currentIndex = startIndex;
|
||||
}
|
||||
|
||||
// Initial render
|
||||
renderTweets(0);
|
||||
|
||||
// Handle scroll
|
||||
timeline.addEventListener('scroll', () => {
|
||||
const scrollTop = timeline.scrollTop;
|
||||
const scrollHeight = timeline.scrollHeight;
|
||||
const clientHeight = timeline.clientHeight;
|
||||
|
||||
// When near bottom, load next page
|
||||
if (scrollTop + clientHeight >= scrollHeight - 100) {
|
||||
const nextIndex = currentIndex + tweetsPerPage;
|
||||
if (nextIndex < totalTweets) {
|
||||
renderTweets(nextIndex);
|
||||
// Small scroll adjustment for continuous scrolling
|
||||
timeline.scrollTop = 50;
|
||||
}
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,55 @@
|
||||
# File: async_webcrawler_multiple_urls_example.py
|
||||
import os, sys
|
||||
|
||||
# append 2 parent directories to sys.path to import crawl4ai
|
||||
parent_dir = os.path.dirname(
|
||||
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
)
|
||||
sys.path.append(parent_dir)
|
||||
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
|
||||
async def main():
|
||||
# Initialize the AsyncWebCrawler
|
||||
async with AsyncWebCrawler(verbose=True) as crawler:
|
||||
# List of URLs to crawl
|
||||
urls = [
|
||||
"https://example.com",
|
||||
"https://python.org",
|
||||
"https://github.com",
|
||||
"https://stackoverflow.com",
|
||||
"https://news.ycombinator.com",
|
||||
]
|
||||
|
||||
# Set up crawling parameters
|
||||
word_count_threshold = 100
|
||||
|
||||
# Run the crawling process for multiple URLs
|
||||
results = await crawler.arun_many(
|
||||
urls=urls,
|
||||
word_count_threshold=word_count_threshold,
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
# Process the results
|
||||
for result in results:
|
||||
if result.success:
|
||||
print(f"Successfully crawled: {result.url}")
|
||||
print(f"Title: {result.metadata.get('title', 'N/A')}")
|
||||
print(f"Word count: {len(result.markdown.split())}")
|
||||
print(
|
||||
f"Number of links: {len(result.links.get('internal', [])) + len(result.links.get('external', []))}"
|
||||
)
|
||||
print(f"Number of images: {len(result.media.get('images', []))}")
|
||||
print("---")
|
||||
else:
|
||||
print(f"Failed to crawl: {result.url}")
|
||||
print(f"Error: {result.error_message}")
|
||||
print("---")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,126 @@
|
||||
"""
|
||||
This example demonstrates optimal browser usage patterns in Crawl4AI:
|
||||
1. Sequential crawling with session reuse
|
||||
2. Parallel crawling with browser instance reuse
|
||||
3. Performance optimization settings
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import List
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
|
||||
|
||||
|
||||
async def crawl_sequential(urls: List[str]):
|
||||
"""
|
||||
Sequential crawling using session reuse - most efficient for moderate workloads
|
||||
"""
|
||||
print("\n=== Sequential Crawling with Session Reuse ===")
|
||||
|
||||
# Configure browser with optimized settings
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
browser_args=[
|
||||
"--disable-gpu", # Disable GPU acceleration
|
||||
"--disable-dev-shm-usage", # Disable /dev/shm usage
|
||||
"--no-sandbox", # Required for Docker
|
||||
],
|
||||
viewport={
|
||||
"width": 800,
|
||||
"height": 600,
|
||||
}, # Smaller viewport for better performance
|
||||
)
|
||||
|
||||
# Configure crawl settings
|
||||
crawl_config = CrawlerRunConfig(
|
||||
markdown_generator=DefaultMarkdownGenerator(
|
||||
# content_filter=PruningContentFilter(), In case you need fit_markdown
|
||||
),
|
||||
)
|
||||
|
||||
# Create single crawler instance
|
||||
crawler = AsyncWebCrawler(config=browser_config)
|
||||
await crawler.start()
|
||||
|
||||
try:
|
||||
session_id = "session1" # Use same session for all URLs
|
||||
for url in urls:
|
||||
result = await crawler.arun(
|
||||
url=url,
|
||||
config=crawl_config,
|
||||
session_id=session_id, # Reuse same browser tab
|
||||
)
|
||||
if result.success:
|
||||
print(f"Successfully crawled {url}")
|
||||
print(f"Content length: {len(result.markdown.raw_markdown)}")
|
||||
finally:
|
||||
await crawler.close()
|
||||
|
||||
|
||||
async def crawl_parallel(urls: List[str], max_concurrent: int = 3):
|
||||
"""
|
||||
Parallel crawling while reusing browser instance - best for large workloads
|
||||
"""
|
||||
print("\n=== Parallel Crawling with Browser Reuse ===")
|
||||
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
browser_args=["--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox"],
|
||||
viewport={"width": 800, "height": 600},
|
||||
)
|
||||
|
||||
crawl_config = CrawlerRunConfig(
|
||||
markdown_generator=DefaultMarkdownGenerator(
|
||||
# content_filter=PruningContentFilter(), In case you need fit_markdown
|
||||
),
|
||||
)
|
||||
|
||||
# Create single crawler instance for all parallel tasks
|
||||
crawler = AsyncWebCrawler(config=browser_config)
|
||||
await crawler.start()
|
||||
|
||||
try:
|
||||
# Create tasks in batches to control concurrency
|
||||
for i in range(0, len(urls), max_concurrent):
|
||||
batch = urls[i : i + max_concurrent]
|
||||
tasks = []
|
||||
|
||||
for j, url in enumerate(batch):
|
||||
session_id = (
|
||||
f"parallel_session_{j}" # Different session per concurrent task
|
||||
)
|
||||
task = crawler.arun(url=url, config=crawl_config, session_id=session_id)
|
||||
tasks.append(task)
|
||||
|
||||
# Wait for batch to complete
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# Process results
|
||||
for url, result in zip(batch, results):
|
||||
if isinstance(result, Exception):
|
||||
print(f"Error crawling {url}: {str(result)}")
|
||||
elif result.success:
|
||||
print(f"Successfully crawled {url}")
|
||||
print(f"Content length: {len(result.markdown.raw_markdown)}")
|
||||
finally:
|
||||
await crawler.close()
|
||||
|
||||
|
||||
async def main():
|
||||
# Example URLs
|
||||
urls = [
|
||||
"https://example.com/page1",
|
||||
"https://example.com/page2",
|
||||
"https://example.com/page3",
|
||||
"https://example.com/page4",
|
||||
]
|
||||
|
||||
# Demo sequential crawling
|
||||
await crawl_sequential(urls)
|
||||
|
||||
# Demo parallel crawling
|
||||
await crawl_parallel(urls, max_concurrent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Builtin Browser Example
|
||||
|
||||
This example demonstrates how to use Crawl4AI's builtin browser feature,
|
||||
which simplifies the browser management process. With builtin mode:
|
||||
|
||||
- No need to manually start or connect to a browser
|
||||
- No need to manage CDP URLs or browser processes
|
||||
- Automatically connects to an existing browser or launches one if needed
|
||||
- Browser persists between script runs, reducing startup time
|
||||
- No explicit cleanup or close() calls needed
|
||||
|
||||
The example also demonstrates "auto-starting" where you don't need to explicitly
|
||||
call start() method on the crawler.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
|
||||
import time
|
||||
|
||||
async def crawl_with_builtin_browser():
|
||||
"""
|
||||
Simple example of crawling with the builtin browser.
|
||||
|
||||
Key features:
|
||||
1. browser_mode="builtin" in BrowserConfig
|
||||
2. No explicit start() call needed
|
||||
3. No explicit close() needed
|
||||
"""
|
||||
print("\n=== Crawl4AI Builtin Browser Example ===\n")
|
||||
|
||||
# Create a browser configuration with builtin mode
|
||||
browser_config = BrowserConfig(
|
||||
browser_mode="builtin", # This is the key setting!
|
||||
headless=True # Can run headless for background operation
|
||||
)
|
||||
|
||||
# Create crawler run configuration
|
||||
crawler_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS, # Skip cache for this demo
|
||||
screenshot=True, # Take a screenshot
|
||||
verbose=True # Show verbose logging
|
||||
)
|
||||
|
||||
# Create the crawler instance
|
||||
# Note: We don't need to use "async with" context manager
|
||||
crawler = AsyncWebCrawler(config=browser_config)
|
||||
|
||||
# Start crawling several URLs - no explicit start() needed!
|
||||
# The crawler will automatically connect to the builtin browser
|
||||
print("\n➡️ Crawling first URL...")
|
||||
t0 = time.time()
|
||||
result1 = await crawler.arun(
|
||||
url="https://crawl4ai.com",
|
||||
config=crawler_config
|
||||
)
|
||||
t1 = time.time()
|
||||
print(f"✅ First URL crawled in {t1-t0:.2f} seconds")
|
||||
print(f" Got {len(result1.markdown.raw_markdown)} characters of content")
|
||||
print(f" Title: {result1.metadata.get('title', 'No title')}")
|
||||
|
||||
# Try another URL - the browser is already running, so this should be faster
|
||||
print("\n➡️ Crawling second URL...")
|
||||
t0 = time.time()
|
||||
result2 = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=crawler_config
|
||||
)
|
||||
t1 = time.time()
|
||||
print(f"✅ Second URL crawled in {t1-t0:.2f} seconds")
|
||||
print(f" Got {len(result2.markdown.raw_markdown)} characters of content")
|
||||
print(f" Title: {result2.metadata.get('title', 'No title')}")
|
||||
|
||||
# The builtin browser continues running in the background
|
||||
# No need to explicitly close it
|
||||
print("\n🔄 The builtin browser remains running for future use")
|
||||
print(" You can use 'crwl browser status' to check its status")
|
||||
print(" or 'crwl browser stop' to stop it when completely done")
|
||||
|
||||
async def main():
|
||||
"""Run the example"""
|
||||
await crawl_with_builtin_browser()
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,171 @@
|
||||
# Amazon R2D2 Product Search Example
|
||||
|
||||
A real-world demonstration of Crawl4AI's multi-step crawling with LLM-generated automation scripts.
|
||||
|
||||
## 🎯 What This Example Shows
|
||||
|
||||
This example demonstrates advanced Crawl4AI features:
|
||||
- **LLM-Generated Scripts**: Automatically create C4A-Script from HTML snippets
|
||||
- **Multi-Step Crawling**: Navigate through multiple pages using session persistence
|
||||
- **Structured Data Extraction**: Extract product data using JSON CSS schemas
|
||||
- **Visual Automation**: Watch the browser perform the search (headless=False)
|
||||
|
||||
## 🚀 How It Works
|
||||
|
||||
### 1. **Script Generation Phase**
|
||||
The example uses `C4ACompiler.generate_script()` to analyze Amazon's HTML and create:
|
||||
- **Search Script**: Automates filling the search box and clicking search
|
||||
- **Extraction Schema**: Defines how to extract product information
|
||||
|
||||
### 2. **Crawling Workflow**
|
||||
```
|
||||
Homepage → Execute Search Script → Extract Products → Save Results
|
||||
```
|
||||
|
||||
All steps use the same `session_id` to maintain browser state.
|
||||
|
||||
### 3. **Data Extraction**
|
||||
Products are extracted with:
|
||||
- Title, price, rating, reviews
|
||||
- Delivery information
|
||||
- Sponsored/Small Business badges
|
||||
- Direct product URLs
|
||||
|
||||
## 📁 Files
|
||||
|
||||
- `amazon_r2d2_search.py` - Main example script
|
||||
- `header.html` - Amazon search bar HTML (provided)
|
||||
- `product.html` - Product card HTML (provided)
|
||||
- **Generated files:**
|
||||
- `generated_search_script.c4a` - Auto-generated search automation
|
||||
- `generated_product_schema.json` - Auto-generated extraction rules
|
||||
- `extracted_products.json` - Final scraped data
|
||||
- `search_results_screenshot.png` - Visual proof of results
|
||||
|
||||
## 🏃 Running the Example
|
||||
|
||||
1. **Prerequisites**
|
||||
```bash
|
||||
# Ensure Crawl4AI is installed
|
||||
pip install crawl4ai
|
||||
|
||||
# Set up LLM API key (for script generation)
|
||||
export OPENAI_API_KEY="your-key-here"
|
||||
```
|
||||
|
||||
2. **Run the scraper**
|
||||
```bash
|
||||
python amazon_r2d2_search.py
|
||||
```
|
||||
|
||||
3. **Watch the magic!**
|
||||
- Browser window opens (not headless)
|
||||
- Navigates to Amazon.com
|
||||
- Searches for "r2d2"
|
||||
- Extracts all products
|
||||
- Saves results to JSON
|
||||
|
||||
## 📊 Sample Output
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"title": "Death Star BB8 R2D2 Golf Balls with 20 Printed tees",
|
||||
"price": "29.95",
|
||||
"rating": "4.7",
|
||||
"reviews_count": "184",
|
||||
"delivery": "FREE delivery Thu, Jun 19",
|
||||
"url": "https://www.amazon.com/Death-Star-R2D2-Balls-Printed/dp/B081XSYZMS",
|
||||
"is_sponsored": true,
|
||||
"small_business": true
|
||||
},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
## 🔍 Key Features Demonstrated
|
||||
|
||||
### Session Persistence
|
||||
```python
|
||||
# Same session_id across multiple arun() calls
|
||||
config = CrawlerRunConfig(
|
||||
session_id="amazon_r2d2_session",
|
||||
# ... other settings
|
||||
)
|
||||
```
|
||||
|
||||
### LLM Script Generation
|
||||
```python
|
||||
# Generate automation from natural language + HTML
|
||||
script = C4ACompiler.generate_script(
|
||||
html=header_html,
|
||||
query="Find search box, type 'r2d2', click search",
|
||||
mode="c4a"
|
||||
)
|
||||
```
|
||||
|
||||
### JSON CSS Extraction
|
||||
```python
|
||||
# Structured data extraction with CSS selectors
|
||||
schema = {
|
||||
"baseSelector": "[data-component-type='s-search-result']",
|
||||
"fields": [
|
||||
{"name": "title", "selector": "h2 a span", "type": "text"},
|
||||
{"name": "price", "selector": ".a-price-whole", "type": "text"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 🛠️ Customization
|
||||
|
||||
### Search Different Products
|
||||
Change the search term in the script generation:
|
||||
```python
|
||||
search_goal = """
|
||||
...
|
||||
3. Type "star wars lego" into the search box
|
||||
...
|
||||
"""
|
||||
```
|
||||
|
||||
### Extract More Data
|
||||
Add fields to the extraction schema:
|
||||
```python
|
||||
"fields": [
|
||||
# ... existing fields
|
||||
{"name": "prime", "selector": ".s-prime", "type": "exists"},
|
||||
{"name": "image_url", "selector": "img.s-image", "type": "attribute", "attribute": "src"}
|
||||
]
|
||||
```
|
||||
|
||||
### Use Different Sites
|
||||
Adapt the approach for other e-commerce sites by:
|
||||
1. Providing their HTML snippets
|
||||
2. Adjusting the search goals
|
||||
3. Updating the extraction schema
|
||||
|
||||
## 🎓 Learning Points
|
||||
|
||||
1. **No Manual Scripting**: LLM generates all automation code
|
||||
2. **Session Management**: Maintain state across page navigations
|
||||
3. **Robust Extraction**: Handle dynamic content and multiple products
|
||||
4. **Error Handling**: Graceful fallbacks if generation fails
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
- **"No products found"**: Check if Amazon's HTML structure changed
|
||||
- **"Script generation failed"**: Ensure LLM API key is configured
|
||||
- **"Page timeout"**: Increase wait times in the config
|
||||
- **"Session lost"**: Ensure same session_id is used consistently
|
||||
|
||||
## 📚 Next Steps
|
||||
|
||||
- Try searching for different products
|
||||
- Add pagination to get more results
|
||||
- Extract product details pages
|
||||
- Compare prices across different sellers
|
||||
- Build a price monitoring system
|
||||
|
||||
---
|
||||
|
||||
This example shows the power of combining LLM intelligence with web automation. The scripts adapt to HTML changes and natural language instructions make automation accessible to everyone!
|
||||
@@ -0,0 +1,202 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Amazon R2D2 Product Search Example using Crawl4AI
|
||||
|
||||
This example demonstrates:
|
||||
1. Using LLM to generate C4A-Script from HTML snippets
|
||||
2. Multi-step crawling with session persistence
|
||||
3. JSON CSS extraction for structured product data
|
||||
4. Complete workflow: homepage → search → extract products
|
||||
|
||||
Requirements:
|
||||
- Crawl4AI with generate_script support
|
||||
- LLM API key (configured in environment)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any
|
||||
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai.script.c4a_compile import C4ACompiler
|
||||
|
||||
|
||||
class AmazonR2D2Scraper:
|
||||
def __init__(self):
|
||||
self.base_dir = Path(__file__).parent
|
||||
self.search_script_path = self.base_dir / "generated_search_script.js"
|
||||
self.schema_path = self.base_dir / "generated_product_schema.json"
|
||||
self.results_path = self.base_dir / "extracted_products.json"
|
||||
self.session_id = "amazon_r2d2_session"
|
||||
|
||||
async def generate_search_script(self) -> str:
|
||||
"""Generate JavaScript for Amazon search interaction"""
|
||||
print("🔧 Generating search script from header.html...")
|
||||
|
||||
# Check if already generated
|
||||
if self.search_script_path.exists():
|
||||
print("✅ Using cached search script")
|
||||
return self.search_script_path.read_text()
|
||||
|
||||
# Read the header HTML
|
||||
header_html = (self.base_dir / "header.html").read_text()
|
||||
|
||||
# Generate script using LLM
|
||||
search_goal = """
|
||||
Find the search box and search button, then:
|
||||
1. Wait for the search box to be visible
|
||||
2. Click on the search box to focus it
|
||||
3. Clear any existing text
|
||||
4. Type "r2d2" into the search box
|
||||
5. Click the search submit button
|
||||
6. Wait for navigation to complete and search results to appear
|
||||
"""
|
||||
|
||||
try:
|
||||
script = C4ACompiler.generate_script(
|
||||
html=header_html,
|
||||
query=search_goal,
|
||||
mode="js"
|
||||
)
|
||||
|
||||
# Save for future use
|
||||
self.search_script_path.write_text(script)
|
||||
print("✅ Search script generated and saved!")
|
||||
print(f"📄 Script:\n{script}")
|
||||
return script
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error generating search script: {e}")
|
||||
|
||||
|
||||
async def generate_product_schema(self) -> Dict[str, Any]:
|
||||
"""Generate JSON CSS extraction schema from product HTML"""
|
||||
print("\n🔧 Generating product extraction schema...")
|
||||
|
||||
# Check if already generated
|
||||
if self.schema_path.exists():
|
||||
print("✅ Using cached extraction schema")
|
||||
return json.loads(self.schema_path.read_text())
|
||||
|
||||
# Read the product HTML
|
||||
product_html = (self.base_dir / "product.html").read_text()
|
||||
|
||||
# Generate extraction schema using LLM
|
||||
schema_goal = """
|
||||
Create a JSON CSS extraction schema to extract:
|
||||
- Product title (from the h2 element)
|
||||
- Price (the dollar amount)
|
||||
- Rating (star rating value)
|
||||
- Number of reviews
|
||||
- Delivery information
|
||||
- Product URL (from the main product link)
|
||||
- Whether it's sponsored
|
||||
- Small business badge if present
|
||||
|
||||
The schema should handle multiple products on a search results page.
|
||||
"""
|
||||
|
||||
try:
|
||||
# Generate JavaScript that returns the schema
|
||||
schema = JsonCssExtractionStrategy.generate_schema(
|
||||
html=product_html,
|
||||
query=schema_goal,
|
||||
)
|
||||
|
||||
# Save for future use
|
||||
self.schema_path.write_text(json.dumps(schema, indent=2))
|
||||
print("✅ Extraction schema generated and saved!")
|
||||
print(f"📄 Schema fields: {[f['name'] for f in schema['fields']]}")
|
||||
return schema
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error generating schema: {e}")
|
||||
|
||||
async def crawl_amazon(self):
|
||||
"""Main crawling logic with 2 calls using same session"""
|
||||
print("\n🚀 Starting Amazon R2D2 product search...")
|
||||
|
||||
# Generate scripts and schemas
|
||||
search_script = await self.generate_search_script()
|
||||
product_schema = await self.generate_product_schema()
|
||||
|
||||
# Configure browser (headless=False to see the action)
|
||||
browser_config = BrowserConfig(
|
||||
headless=False,
|
||||
verbose=True,
|
||||
viewport_width=1920,
|
||||
viewport_height=1080
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
print("\n📍 Step 1: Navigate to Amazon and search for R2D2")
|
||||
|
||||
# FIRST CALL: Navigate to Amazon and execute search
|
||||
search_config = CrawlerRunConfig(
|
||||
session_id=self.session_id,
|
||||
js_code= f"(() => {{ {search_script} }})()", # Execute generated JS
|
||||
wait_for=".s-search-results", # Wait for search results
|
||||
extraction_strategy=JsonCssExtractionStrategy(schema=product_schema),
|
||||
delay_before_return_html=3.0 # Give time for results to load
|
||||
)
|
||||
|
||||
results = await crawler.arun(
|
||||
url="https://www.amazon.com",
|
||||
config=search_config
|
||||
)
|
||||
|
||||
if not results.success:
|
||||
print("❌ Failed to search Amazon")
|
||||
print(f"Error: {results.error_message}")
|
||||
return
|
||||
|
||||
print("✅ Search completed successfully!")
|
||||
print("✅ Product extraction completed!")
|
||||
|
||||
# Extract and save results
|
||||
print("\n📍 Extracting product data")
|
||||
|
||||
if results[0].extracted_content:
|
||||
products = json.loads(results[0].extracted_content)
|
||||
print(f"🔍 Found {len(products)} products in search results")
|
||||
|
||||
print(f"✅ Extracted {len(products)} R2D2 products")
|
||||
|
||||
# Save results
|
||||
self.results_path.write_text(
|
||||
json.dumps(products, indent=2)
|
||||
)
|
||||
print(f"💾 Results saved to: {self.results_path}")
|
||||
|
||||
# Print sample results
|
||||
print("\n📊 Sample Results:")
|
||||
for i, product in enumerate(products[:3], 1):
|
||||
print(f"\n{i}. {product['title'][:60]}...")
|
||||
print(f" Price: ${product['price']}")
|
||||
print(f" Rating: {product['rating']} ({product['number_of_reviews']} reviews)")
|
||||
print(f" {'🏪 Small Business' if product['small_business_badge'] else ''}")
|
||||
print(f" {'📢 Sponsored' if product['sponsored'] else ''}")
|
||||
|
||||
else:
|
||||
print("❌ No products extracted")
|
||||
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run the Amazon scraper"""
|
||||
scraper = AmazonR2D2Scraper()
|
||||
await scraper.crawl_amazon()
|
||||
|
||||
print("\n🎉 Amazon R2D2 search example completed!")
|
||||
print("Check the generated files:")
|
||||
print(" - generated_search_script.js")
|
||||
print(" - generated_product_schema.json")
|
||||
print(" - extracted_products.json")
|
||||
print(" - search_results_screenshot.png")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,114 @@
|
||||
[
|
||||
{
|
||||
"title": "Death Star BB8 R2D2 Golf Balls with 20 Printed tees \u2022 Great Gift IDEA from Moms, DADS and Kids -",
|
||||
"price": "$29.95",
|
||||
"rating": "4.7 out of 5 stars",
|
||||
"number_of_reviews": "184",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfYXRmOjIwMDA2NzY0ODgwMjc5ODo6MDo6&url=%2FDeath-Star-R2D2-Balls-Printed%2Fdp%2FB081XSYZMS%2Fref%3Dsr_1_1_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-1-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9hdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored",
|
||||
"small_business_badge": "Small Business"
|
||||
},
|
||||
{
|
||||
"title": "TEENKON French Press Insulated 304 Stainless Steel Coffee Maker, 32 Oz Robot R2D2 Hand Home Coffee Presser, with Filter Screen for Brew Coffee and Tea (White)",
|
||||
"price": "$49.99",
|
||||
"rating": "4.3 out of 5 stars",
|
||||
"number_of_reviews": "82",
|
||||
"delivery_info": "Delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjMwMDAzNzc4Njg4MDAwMjo6MDo6&url=%2FTEENKON-French-Insulated-Stainless-Presser%2Fdp%2FB0CD3HH5PN%2Fref%3Dsr_1_17_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-17-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "3D Illusion LED Night Light,7 Colors Gradual Changing Touch Switch USB Table Lamp for Holiday Gifts or Home Decorations (R2-D2)",
|
||||
"price": "$9.97",
|
||||
"rating": "4.3 out of 5 stars",
|
||||
"number_of_reviews": "235",
|
||||
"delivery_info": "Delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjIwMDA0NjMwMTQwODA4MTo6MDo6&url=%2FIllusion-Gradual-Changing-Holiday-Decorations%2Fdp%2FB089NMBKF2%2Fref%3Dsr_1_18_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-18-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "Paladone Star Wars R2-D2 Headlamp with Droid Sounds, Officially Licensed Disney Star Wars Head Lamp and Reading Light",
|
||||
"price": "$21.99",
|
||||
"rating": "4.1 out of 5 stars",
|
||||
"number_of_reviews": "66",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjMwMDI1NjA0MDQwMTUwMjo6MDo6&url=%2FSounds-Officially-Licensed-Headlamp-Flashlight%2Fdp%2FB09RTDZF8J%2Fref%3Dsr_1_19_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-19-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "4 Pcs Set Star Wars Kylo Ren BB8 Stormtrooper R2D2 Silicone Travel Luggage Baggage Identification Labels ID Tag for Bag Suitcase Plane Cruise Ships with Belt Strap",
|
||||
"price": "$16.99",
|
||||
"rating": "4.7 out of 5 stars",
|
||||
"number_of_reviews": "3,414",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjIwMDAyMzk3ODkwMzIxMTo6MDo6&url=%2FFinex-Set-Suitcase-Adjustable-Stormtrooper%2Fdp%2FB01D1CBFJS%2Fref%3Dsr_1_24_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-24-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored",
|
||||
"small_business_badge": "Small Business"
|
||||
},
|
||||
{
|
||||
"title": "Papyrus Star Wars Birthday Card Assortment, Darth Vader, Storm Trooper, and R2-D2 (3-Count)",
|
||||
"price": "$23.16",
|
||||
"rating": "4.8 out of 5 stars",
|
||||
"number_of_reviews": "328",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjMwMDcwNzI4MjA1MzcwMjo6MDo6&url=%2FPapyrus-Birthday-Assortment-Characters-3-Count%2Fdp%2FB07YT2ZPKX%2Fref%3Dsr_1_25_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-25-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "STAR WARS R2-D2 Artoo 3D Top Motion Lamp, Mood Light | 18 Inches",
|
||||
"price": "$69.99",
|
||||
"rating": "4.5 out of 5 stars",
|
||||
"number_of_reviews": "520",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjIwMDA5NDc3MzczMTQ0MTo6MDo6&url=%2FR2-D2-Artoo-Motion-Light-Inches%2Fdp%2FB08MCWPHQR%2Fref%3Dsr_1_26_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-26-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "Saturday Park Star Wars Droids Full Sheet Set - 4 Piece 100% Organic Cotton Sheets Features R2-D2 & BB-8 - GOTS & Oeko-TEX Certified (Star Wars Official)",
|
||||
"price": "$70.00",
|
||||
"rating": "4.5 out of 5 stars",
|
||||
"number_of_reviews": "388",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjMwMDAyMzI0NDI5MDQwMjo6MDo6&url=%2FSaturday-Park-Star-Droids-Sheet%2Fdp%2FB0BBSFX4J2%2Fref%3Dsr_1_27_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-27-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored",
|
||||
"small_business_badge": "1 sustainability feature"
|
||||
},
|
||||
{
|
||||
"title": "AQUARIUS Star Wars R2D2 Action Figure Funky Chunky Novelty Magnet for Refrigerator, Locker, Whiteboard & Game Room Officially Licensed Merchandise & Collectibles",
|
||||
"price": "$11.94",
|
||||
"rating": "4.3 out of 5 stars",
|
||||
"number_of_reviews": "10",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjMwMDA5MDMwMzY5NjEwMjo6MDo6&url=%2FAQUARIUS-Refrigerator-Whiteboard-Merchandise-Collectibles%2Fdp%2FB09W8VKXGC%2Fref%3Dsr_1_32_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-32-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "STAR WARS C-3PO and R2-D2 Men's Crew Socks 2 Pair Pack",
|
||||
"price": "$11.95",
|
||||
"rating": "4.7 out of 5 stars",
|
||||
"number_of_reviews": "1,272",
|
||||
"delivery_info": "Delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjIwMDAxMDk5NDkyMTg2MTo6MDo6&url=%2FStar-Wars-R2-D2-C-3PO-Socks%2Fdp%2FB0178IU1GY%2Fref%3Dsr_1_33_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-33-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored"
|
||||
},
|
||||
{
|
||||
"title": "Buckle-Down Belt Women's Cinch Star Wars R2D2 Bounding Parts3 White Black Blue Gray Available In Adjustable Sizes",
|
||||
"price": "$24.95",
|
||||
"rating": "4.3 out of 5 stars",
|
||||
"number_of_reviews": "32",
|
||||
"delivery_info": "FREE delivery",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjMwMDY1OTQ5NTQ4MzkwMjo6MDo6&url=%2FWomens-Cinch-Bounding-Parts3-Inches%2Fdp%2FB07WK7RG4D%2Fref%3Dsr_1_34_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-34-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored",
|
||||
"small_business_badge": "Small Business"
|
||||
},
|
||||
{
|
||||
"title": "Star Wars R2D2 Metal Head Vintage Disney+ T-Shirt",
|
||||
"price": "$22.99",
|
||||
"rating": "4.8 out of 5 stars",
|
||||
"number_of_reviews": "869",
|
||||
"product_url": "/sspa/click?ie=UTF8&spc=MToxNDMzMjA0MzA4MzEzMjAxOjE3NDkzMDI3NDY6c3BfbXRmOjIwMDA1OTUyMzgzNDMyMTo6MDo6&url=%2FStar-Wars-Vintage-Graphic-T-Shirt%2Fdp%2FB07H9PSNXS%2Fref%3Dsr_1_35_sspa%3Fdib%3DeyJ2IjoiMSJ9.iiJYY01upNMdD4BNNt8CYLZEIMXulNkcBlKEMJlr_U_h9eSGqChxwcIiCKUbJeEO_plLkXZvB7Yx-v4UDOCdiUFI-sHFgcTznXrP7tdD8xHpRaMKmaBDWMCAFwzPmVcgK_6Q9qIRoN4sp8tunKX26j5EC_8LiK-D5QximGkE8i8f-R5GhSUo__DaSkAP1cnzxUtSESfA8fYfewsZ1iSol9_zohE6r1ZZeawnWHPmDTkLqzCW3uK44EnvJbPFvzMlpiKcs9p9Eh9w5Rc5rrumMihdaWkC63B0cz5jU-S2Ieg._D8d5nv3hOExHPbZ04L-vaC7YwJjEZM-vu5AED5sz0U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749302746%26sr%3D8-35-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9tdGY%26psc%3D1",
|
||||
"sponsored": "Sponsored",
|
||||
"small_business_badge": "1 sustainability feature"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"name": "Amazon Product Search Results",
|
||||
"baseSelector": "div[data-component-type='s-impression-counter']",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h2.a-size-base-plus.a-spacing-none.a-color-base.a-text-normal span",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "price",
|
||||
"selector": "span.a-price > span.a-offscreen",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "rating",
|
||||
"selector": "i.a-icon-star-small span.a-icon-alt",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "number_of_reviews",
|
||||
"selector": "a.a-link-normal.s-underline-text span.a-size-base",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "delivery_info",
|
||||
"selector": "div[data-cy='delivery-recipe'] span.a-color-base",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "product_url",
|
||||
"selector": "a.a-link-normal.s-no-outline",
|
||||
"type": "attribute",
|
||||
"attribute": "href"
|
||||
},
|
||||
{
|
||||
"name": "sponsored",
|
||||
"selector": "span.puis-label-popover-default span.a-color-secondary",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "small_business_badge",
|
||||
"selector": "span.a-size-base.a-color-base",
|
||||
"type": "text"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
const searchBox = document.querySelector('#twotabsearchtextbox');
|
||||
const searchButton = document.querySelector('#nav-search-submit-button');
|
||||
|
||||
if (searchBox && searchButton) {
|
||||
searchBox.focus();
|
||||
searchBox.value = '';
|
||||
searchBox.value = 'r2d2';
|
||||
searchButton.click();
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
<div id="nav-belt" style="width: 100%;">
|
||||
<div class="nav-left">
|
||||
<script type="text/javascript">window.navmet.tmp = +new Date();</script>
|
||||
<div id="nav-logo">
|
||||
<a href="/ref=nav_logo" id="nav-logo-sprites" class="nav-logo-link nav-progressive-attribute"
|
||||
aria-label="Amazon" lang="en">
|
||||
<span class="nav-sprite nav-logo-base"></span>
|
||||
<span id="logo-ext" class="nav-sprite nav-logo-ext nav-progressive-content"></span>
|
||||
<span class="nav-logo-locale">.us</span>
|
||||
</a>
|
||||
</div>
|
||||
<script
|
||||
type="text/javascript">window.navmet.push({ key: 'Logo', end: +new Date(), begin: window.navmet.tmp });</script>
|
||||
|
||||
<div id="nav-global-location-slot">
|
||||
<span id="nav-global-location-data-modal-action" class="a-declarative nav-progressive-attribute"
|
||||
data-a-modal="{"width":375, "closeButton":"true","popoverLabel":"Choose your location", "ajaxHeaders":{"anti-csrftoken-a2z":"hHBwllskaYQrylaW9ifYQIdmqBZOtGdKro0TWb5kDoPKAAAAAGhEMhsAAAAB"}, "name":"glow-modal", "url":"/portal-migration/hz/glow/get-rendered-address-selections?deviceType=desktop&pageType=Gateway&storeContext=NoStoreName&actionSource=desktop-modal", "footer":"<span class=\"a-declarative\" data-action=\"a-popover-close\" data-a-popover-close=\"{}\"><span class=\"a-button a-button-primary\"><span class=\"a-button-inner\"><button name=\"glowDoneButton\" class=\"a-button-text\" type=\"button\">Done</button></span></span></span>","header":"Choose your location"}"
|
||||
data-action="a-modal">
|
||||
<a id="nav-global-location-popover-link" role="button" tabindex="0"
|
||||
class="nav-a nav-a-2 a-popover-trigger a-declarative nav-progressive-attribute" href="">
|
||||
<div class="nav-sprite nav-progressive-attribute" id="nav-packard-glow-loc-icon"></div>
|
||||
<div id="glow-ingress-block">
|
||||
<span class="nav-line-1 nav-progressive-content" id="glow-ingress-line1">
|
||||
Deliver to
|
||||
</span>
|
||||
<span class="nav-line-2 nav-progressive-content" id="glow-ingress-line2">
|
||||
Malaysia
|
||||
</span>
|
||||
</div>
|
||||
</a>
|
||||
</span>
|
||||
<input data-addnewaddress="add-new" id="unifiedLocation1ClickAddress" name="dropdown-selection"
|
||||
type="hidden" value="add-new" class="nav-progressive-attribute">
|
||||
<input data-addnewaddress="add-new" id="ubbShipTo" name="dropdown-selection-ubb" type="hidden"
|
||||
value="add-new" class="nav-progressive-attribute">
|
||||
<input id="glowValidationToken" name="glow-validation-token" type="hidden"
|
||||
value="hHBwllskaYQrylaW9ifYQIdmqBZOtGdKro0TWb5kDoPKAAAAAGhEMhsAAAAB" class="nav-progressive-attribute">
|
||||
<input id="glowDestinationType" name="glow-destination-type" type="hidden" value="COUNTRY"
|
||||
class="nav-progressive-attribute">
|
||||
</div>
|
||||
|
||||
<div id="nav-global-location-toaster-script-container" class="nav-progressive-content">
|
||||
<!-- NAVYAAN-GLOW-NAV-TOASTER -->
|
||||
<script>
|
||||
P.when('glow-toaster-strings').execute(function (S) {
|
||||
S.load({ "glow-toaster-address-change-error": "An error has occurred and the address has not been updated. Please try again.", "glow-toaster-unknown-error": "An error has occurred. Please try again." });
|
||||
});
|
||||
</script>
|
||||
<script>
|
||||
P.when('glow-toaster-manager').execute(function (M) {
|
||||
M.create({ "pageType": "Gateway", "aisTransitionState": null, "rancorLocationSource": "REALM_DEFAULT" })
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="nav-fill" id="nav-fill-search">
|
||||
<script type="text/javascript">window.navmet.tmp = +new Date();</script>
|
||||
<div id="nav-search">
|
||||
<div id="nav-bar-left"></div>
|
||||
<form id="nav-search-bar-form" accept-charset="utf-8" action="/s/ref=nb_sb_noss_1"
|
||||
class="nav-searchbar nav-progressive-attribute" method="GET" name="site-search" role="search">
|
||||
|
||||
<div class="nav-left">
|
||||
<div id="nav-search-dropdown-card">
|
||||
|
||||
<div class="nav-search-scope nav-sprite">
|
||||
<div class="nav-search-facade" data-value="search-alias=aps">
|
||||
<span id="nav-search-label-id" class="nav-search-label nav-progressive-content"
|
||||
style="width: auto;">All</span>
|
||||
<i class="nav-icon"></i>
|
||||
</div>
|
||||
<label id="searchDropdownDescription" for="searchDropdownBox"
|
||||
class="nav-progressive-attribute" style="display:none">Select the department you want to
|
||||
search in</label>
|
||||
<select aria-describedby="searchDropdownDescription"
|
||||
class="nav-search-dropdown searchSelect nav-progressive-attrubute nav-progressive-search-dropdown"
|
||||
data-nav-digest="k+fyIAyB82R9jVEmroQ0OWwSW3A=" data-nav-selected="0"
|
||||
id="searchDropdownBox" name="url" style="display: block; top: 2.5px;" tabindex="0"
|
||||
title="Search in">
|
||||
<option selected="selected" value="search-alias=aps">All Departments</option>
|
||||
<option value="search-alias=arts-crafts-intl-ship">Arts & Crafts</option>
|
||||
<option value="search-alias=automotive-intl-ship">Automotive</option>
|
||||
<option value="search-alias=baby-products-intl-ship">Baby</option>
|
||||
<option value="search-alias=beauty-intl-ship">Beauty & Personal Care</option>
|
||||
<option value="search-alias=stripbooks-intl-ship">Books</option>
|
||||
<option value="search-alias=fashion-boys-intl-ship">Boys' Fashion</option>
|
||||
<option value="search-alias=computers-intl-ship">Computers</option>
|
||||
<option value="search-alias=deals-intl-ship">Deals</option>
|
||||
<option value="search-alias=digital-music">Digital Music</option>
|
||||
<option value="search-alias=electronics-intl-ship">Electronics</option>
|
||||
<option value="search-alias=fashion-girls-intl-ship">Girls' Fashion</option>
|
||||
<option value="search-alias=hpc-intl-ship">Health & Household</option>
|
||||
<option value="search-alias=kitchen-intl-ship">Home & Kitchen</option>
|
||||
<option value="search-alias=industrial-intl-ship">Industrial & Scientific</option>
|
||||
<option value="search-alias=digital-text">Kindle Store</option>
|
||||
<option value="search-alias=luggage-intl-ship">Luggage</option>
|
||||
<option value="search-alias=fashion-mens-intl-ship">Men's Fashion</option>
|
||||
<option value="search-alias=movies-tv-intl-ship">Movies & TV</option>
|
||||
<option value="search-alias=music-intl-ship">Music, CDs & Vinyl</option>
|
||||
<option value="search-alias=pets-intl-ship">Pet Supplies</option>
|
||||
<option value="search-alias=instant-video">Prime Video</option>
|
||||
<option value="search-alias=software-intl-ship">Software</option>
|
||||
<option value="search-alias=sporting-intl-ship">Sports & Outdoors</option>
|
||||
<option value="search-alias=tools-intl-ship">Tools & Home Improvement</option>
|
||||
<option value="search-alias=toys-and-games-intl-ship">Toys & Games</option>
|
||||
<option value="search-alias=videogames-intl-ship">Video Games</option>
|
||||
<option value="search-alias=fashion-womens-intl-ship">Women's Fashion</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="nav-fill">
|
||||
<div class="nav-search-field ">
|
||||
<label for="twotabsearchtextbox" style="display: none;">Search Amazon</label>
|
||||
<input type="text" id="twotabsearchtextbox" value="" name="field-keywords" autocomplete="off"
|
||||
placeholder="Search Amazon" class="nav-input nav-progressive-attribute" dir="auto"
|
||||
tabindex="0" aria-label="Search Amazon" role="searchbox" aria-autocomplete="list"
|
||||
aria-controls="sac-autocomplete-results-container" aria-expanded="false"
|
||||
aria-haspopup="grid" spellcheck="false">
|
||||
</div>
|
||||
<div id="nav-iss-attach"></div>
|
||||
</div>
|
||||
<div class="nav-right">
|
||||
<div class="nav-search-submit nav-sprite">
|
||||
<span id="nav-search-submit-text"
|
||||
class="nav-search-submit-text nav-sprite nav-progressive-attribute" aria-label="Go">
|
||||
<input id="nav-search-submit-button" type="submit"
|
||||
class="nav-input nav-progressive-attribute" value="Go" tabindex="0">
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
<input type="hidden" id="isscrid" name="crid" value="15O5T5OCG5OZE"><input type="hidden" id="issprefix"
|
||||
name="sprefix" value="r2d2,aps,588">
|
||||
</form>
|
||||
</div>
|
||||
<script
|
||||
type="text/javascript">window.navmet.push({ key: 'Search', end: +new Date(), begin: window.navmet.tmp });</script>
|
||||
</div>
|
||||
<div class="nav-right">
|
||||
<script type="text/javascript">window.navmet.tmp = +new Date();</script>
|
||||
<div id="nav-tools" class="layoutToolbarPadding">
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="nav-div" id="icp-nav-flyout">
|
||||
<a href="/customer-preferences/edit?ie=UTF8&preferencesReturnUrl=%2F&ref_=topnav_lang_ais"
|
||||
class="nav-a nav-a-2 icp-link-style-2" aria-label="Choose a language for shopping in Amazon United States. The current selection is English (EN).
|
||||
">
|
||||
<span class="icp-nav-link-inner">
|
||||
<span class="nav-line-1">
|
||||
</span>
|
||||
<span class="nav-line-2">
|
||||
<span class="icp-nav-flag icp-nav-flag-us icp-nav-flag-lop" role="img"
|
||||
aria-label="United States"></span>
|
||||
<div>EN</div>
|
||||
</span>
|
||||
</span>
|
||||
</a>
|
||||
<button class="nav-flyout-button nav-icon nav-arrow" aria-label="Expand to Change Language or Country"
|
||||
tabindex="0" style="visibility: visible;"></button>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="nav-div" id="nav-link-accountList">
|
||||
<a href="https://www.amazon.com/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2F%3Fref_%3Dnav_ya_signin&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0"
|
||||
class="nav-a nav-a-2 nav-progressive-attribute" data-nav-ref="nav_ya_signin"
|
||||
data-nav-role="signin" data-ux-jq-mouseenter="true" tabindex="0" data-csa-c-type="link"
|
||||
data-csa-c-slot-id="nav-link-accountList" data-csa-c-content-id="nav_ya_signin"
|
||||
aria-controls="nav-flyout-accountList" data-csa-c-id="37vs0l-z575id-52hnw3-x34ncp">
|
||||
<div class="nav-line-1-container"><span id="nav-link-accountList-nav-line-1"
|
||||
class="nav-line-1 nav-progressive-content">Hello, sign in</span></div>
|
||||
<span class="nav-line-2 ">Account & Lists
|
||||
</span>
|
||||
</a>
|
||||
<button class="nav-flyout-button nav-icon nav-arrow" aria-label="Expand Account and Lists" tabindex="0"
|
||||
style="visibility: visible;"></button>
|
||||
</div>
|
||||
|
||||
|
||||
<a href="/gp/css/order-history?ref_=nav_orders_first" class="nav-a nav-a-2 nav-progressive-attribute"
|
||||
id="nav-orders" tabindex="0">
|
||||
<span class="nav-line-1">Returns</span>
|
||||
<span class="nav-line-2">& Orders<span class="nav-icon nav-arrow"></span></span>
|
||||
</a>
|
||||
|
||||
|
||||
|
||||
<a href="/gp/cart/view.html?ref_=nav_cart" aria-label="0 items in cart"
|
||||
class="nav-a nav-a-2 nav-progressive-attribute" id="nav-cart">
|
||||
<div id="nav-cart-count-container">
|
||||
<span id="nav-cart-count" aria-hidden="true"
|
||||
class="nav-cart-count nav-cart-0 nav-progressive-attribute nav-progressive-content">0</span>
|
||||
<span class="nav-cart-icon nav-sprite"></span>
|
||||
</div>
|
||||
<div id="nav-cart-text-container" class=" nav-progressive-attribute">
|
||||
<span aria-hidden="true" class="nav-line-1">
|
||||
|
||||
</span>
|
||||
<span aria-hidden="true" class="nav-line-2">
|
||||
Cart
|
||||
<span class="nav-icon nav-arrow"></span>
|
||||
</span>
|
||||
</div>
|
||||
</a>
|
||||
|
||||
</div>
|
||||
<script
|
||||
type="text/javascript">window.navmet.push({ key: 'Tools', end: +new Date(), begin: window.navmet.tmp });</script>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1,206 @@
|
||||
<div class="sg-col-inner">
|
||||
<div cel_widget_id="MAIN-SEARCH_RESULTS-2"
|
||||
class="s-widget-container s-spacing-small s-widget-container-height-small celwidget slot=MAIN template=SEARCH_RESULTS widgetId=search-results_1"
|
||||
data-csa-c-pos="1" data-csa-c-item-id="amzn1.asin.1.B081XSYZMS" data-csa-op-log-render="" data-csa-c-type="item"
|
||||
data-csa-c-id="dp9zuy-vyww1v-brlmmq-fmgitb" data-cel-widget="MAIN-SEARCH_RESULTS-2">
|
||||
|
||||
|
||||
<div data-component-type="s-impression-logger"
|
||||
data-component-props="{"percentageShownToFire":"50","batchable":true,"requiredElementSelector":".s-image:visible","url":"https://unagi-na.amazon.com/1/events/com.amazon.eel.SponsoredProductsEventTracking.prod?qualifier=1749299833&id=1740514893473797&widgetName=sp_atf&adId=200067648802798&eventType=1&adIndex=0"}"
|
||||
class="rush-component s-expand-height" data-component-id="6">
|
||||
|
||||
|
||||
|
||||
<div data-component-type="s-impression-counter"
|
||||
data-component-props="{"presenceCounterName":"sp_delivered","testElementSelector":".s-image","hiddenCounterName":"sp_hidden"}"
|
||||
class="rush-component s-featured-result-item s-expand-height" data-component-id="7">
|
||||
<span class="a-declarative" data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw" data-action="puis-card-container-declarative"
|
||||
data-csa-c-func-deps="aui-da-puis-card-container-declarative"
|
||||
data-csa-c-item-id="amzn1.asin.B081XSYZMS" data-csa-c-posx="1" data-csa-c-type="item"
|
||||
data-csa-c-owner="puis" data-csa-c-id="88w0j1-kcbf5g-80v4i9-96cv88">
|
||||
<div class="puis-card-container s-card-container s-overflow-hidden aok-relative puis-expand-height puis-include-content-margin puis puis-v2dwi5hq8xzthf26x0gg1mcl2oj s-latency-cf-section puis-card-border"
|
||||
data-cy="asin-faceout-container">
|
||||
<div class="a-section a-spacing-base">
|
||||
<div class="s-product-image-container aok-relative s-text-center s-image-overlay-grey puis-image-overlay-grey s-padding-left-small s-padding-right-small puis-spacing-small s-height-equalized puis puis-v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-cy="image-container" style="padding-top: 0px !important;"><span
|
||||
data-component-type="s-product-image" class="rush-component"
|
||||
data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw"><a aria-hidden="true"
|
||||
class="a-link-normal s-no-outline" tabindex="-1"
|
||||
href="/sspa/click?ie=UTF8&spc=MToxNzQwNTE0ODkzNDczNzk3OjE3NDkyOTk4MzM6c3BfYXRmOjIwMDA2NzY0ODgwMjc5ODo6MDo6&url=%2FDeath-Star-R2D2-Balls-Printed%2Fdp%2FB081XSYZMS%2Fref%3Dsr_1_1_sspa%3Fcrid%3D3C1EXMXN59Q9G%26dib%3DeyJ2IjoiMSJ9.7tBl5bhZh59L9qIPZUe9SLa2fy_HvzboxuQxvrRcAc0VUXayi9fxQFsMLyFplDE9vMkIJbP76AVpa-5-fxhNza3DqhX4tss4NlB49WPi_dA00Hw6O8qK5pDzdetYlhGgOyXOLBe7mTG9oJ5W0wcvQhEVoX9mpJk_SGeqRLWGA0dBSjYCZtiyrY8_B-DP53S7fbYwiSYtq-g7sQDXKVadRpGvUyKq7yxA0SLsU42uvoqSGb0qcd6udL1wbnTEkKmwNjNSb7xIUb-8PyE7DTPMt1ScJksn70sFQMJNkM2aK5M.x9_jYvKPnSibV1d0umUStZBxlSTSXrzVIFKqFzS8c-U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749299833%26sprefix%3Dr2d2%252Caps%252C548%26sr%3D8-1-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9hdGY%26psc%3D1">
|
||||
<div class="a-section aok-relative s-image-square-aspect"><img class="s-image"
|
||||
src="https://m.media-amazon.com/images/I/61kAC69zQUL._AC_UL320_.jpg"
|
||||
srcset="https://m.media-amazon.com/images/I/61kAC69zQUL._AC_UL320_.jpg 1x, https://m.media-amazon.com/images/I/61kAC69zQUL._AC_UL480_FMwebp_QL65_.jpg 1.5x, https://m.media-amazon.com/images/I/61kAC69zQUL._AC_UL640_FMwebp_QL65_.jpg 2x, https://m.media-amazon.com/images/I/61kAC69zQUL._AC_UL800_FMwebp_QL65_.jpg 2.5x, https://m.media-amazon.com/images/I/61kAC69zQUL._AC_UL960_FMwebp_QL65_.jpg 3x"
|
||||
alt="Sponsored Ad - Death Star BB8 R2D2 Golf Balls with 20 Printed tees • Great Gift IDEA from Moms, DADS and Kids -"
|
||||
aria-hidden="true" data-image-index="1" data-image-load=""
|
||||
data-image-latency="s-product-image" data-image-source-density="1">
|
||||
</div>
|
||||
</a></span></div>
|
||||
<div class="a-section a-spacing-small puis-padding-left-small puis-padding-right-small">
|
||||
<div data-cy="title-recipe"
|
||||
class="a-section a-spacing-none a-spacing-top-small s-title-instructions-style">
|
||||
<div class="a-row a-spacing-micro"><span class="a-declarative"
|
||||
data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw" data-action="a-popover"
|
||||
data-csa-c-func-deps="aui-da-a-popover"
|
||||
data-a-popover="{"name":"sp-info-popover-B081XSYZMS","position":"triggerVertical","popoverLabel":"View Sponsored information or leave ad feedback","closeButtonLabel":"Close popup","closeButton":"true","dataStrategy":"preload"}"
|
||||
data-csa-c-type="widget" data-csa-c-id="wqddan-z1l67e-lissct-rciw65"><a
|
||||
href="javascript:void(0)" role="button" style="text-decoration: none;"
|
||||
class="puis-label-popover puis-sponsored-label-text"><span
|
||||
class="puis-label-popover-default"><span
|
||||
aria-label="View Sponsored information or leave ad feedback"
|
||||
class="a-color-secondary">Sponsored</span></span><span
|
||||
class="puis-label-popover-hover"><span aria-hidden="true"
|
||||
class="a-color-base">Sponsored</span></span> <span
|
||||
class="aok-inline-block puis-sponsored-label-info-icon"></span></a></span>
|
||||
<div class="a-popover-preload" id="a-popover-sp-info-popover-B081XSYZMS">
|
||||
<div class="puis puis-v2dwi5hq8xzthf26x0gg1mcl2oj"><span>You’re seeing this
|
||||
ad based on the product’s relevance to your search query.</span>
|
||||
<div class="a-row"><span class="a-declarative"
|
||||
data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw"
|
||||
data-action="s-safe-ajax-modal-trigger"
|
||||
data-csa-c-func-deps="aui-da-s-safe-ajax-modal-trigger"
|
||||
data-s-safe-ajax-modal-trigger="{"header":"Leave feedback","dataStrategy":"ajax","ajaxUrl":"/af/sp-loom/feedback-form?pl=%7B%22adPlacementMetaData%22%3A%7B%22searchTerms%22%3A%22cjJkMg%3D%3D%22%2C%22pageType%22%3A%22Search%22%2C%22feedbackType%22%3A%22sponsoredProductsLoom%22%2C%22slotName%22%3A%22TOP%22%7D%2C%22adCreativeMetaData%22%3A%7B%22adProgramId%22%3A1024%2C%22adCreativeDetails%22%3A%5B%7B%22asin%22%3A%22B081XSYZMS%22%2C%22title%22%3A%22Death+Star+BB8+R2D2+Golf+Balls+with+20+Printed+tees+%E2%80%A2+Great+Gift+IDEA+from+Moms%2C+DADS+and+Kids+-%22%2C%22priceInfo%22%3A%7B%22amount%22%3A29.95%2C%22currencyCode%22%3A%22USD%22%7D%2C%22sku%22%3A%22starwars3pk20tees%22%2C%22adId%22%3A%22A03790291PREH7M3Q3SVS%22%2C%22campaignId%22%3A%22A01050612Q0SQZ2PTMGO9%22%2C%22advertiserIdNS%22%3Anull%2C%22selectionSignals%22%3Anull%7D%5D%7D%7D"}"
|
||||
data-csa-c-type="widget"
|
||||
data-csa-c-id="ygslsp-ir23ei-7k9x6z-73l1tp"><a
|
||||
class="a-link-normal s-underline-text s-underline-link-text s-link-style"
|
||||
href="#"><span>Leave ad feedback</span> </a> </span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div><a class="a-link-normal s-line-clamp-4 s-link-style a-text-normal"
|
||||
href="/sspa/click?ie=UTF8&spc=MToxNzQwNTE0ODkzNDczNzk3OjE3NDkyOTk4MzM6c3BfYXRmOjIwMDA2NzY0ODgwMjc5ODo6MDo6&url=%2FDeath-Star-R2D2-Balls-Printed%2Fdp%2FB081XSYZMS%2Fref%3Dsr_1_1_sspa%3Fcrid%3D3C1EXMXN59Q9G%26dib%3DeyJ2IjoiMSJ9.7tBl5bhZh59L9qIPZUe9SLa2fy_HvzboxuQxvrRcAc0VUXayi9fxQFsMLyFplDE9vMkIJbP76AVpa-5-fxhNza3DqhX4tss4NlB49WPi_dA00Hw6O8qK5pDzdetYlhGgOyXOLBe7mTG9oJ5W0wcvQhEVoX9mpJk_SGeqRLWGA0dBSjYCZtiyrY8_B-DP53S7fbYwiSYtq-g7sQDXKVadRpGvUyKq7yxA0SLsU42uvoqSGb0qcd6udL1wbnTEkKmwNjNSb7xIUb-8PyE7DTPMt1ScJksn70sFQMJNkM2aK5M.x9_jYvKPnSibV1d0umUStZBxlSTSXrzVIFKqFzS8c-U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749299833%26sprefix%3Dr2d2%252Caps%252C548%26sr%3D8-1-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9hdGY%26psc%3D1">
|
||||
<h2 aria-label="Sponsored Ad - Death Star BB8 R2D2 Golf Balls with 20 Printed tees • Great Gift IDEA from Moms, DADS and Kids -"
|
||||
class="a-size-base-plus a-spacing-none a-color-base a-text-normal">
|
||||
<span>Death Star BB8 R2D2 Golf Balls with 20 Printed tees • Great Gift IDEA
|
||||
from Moms, DADS and Kids -</span></h2>
|
||||
</a>
|
||||
</div>
|
||||
<div data-cy="reviews-block" class="a-section a-spacing-none a-spacing-top-micro">
|
||||
<div class="a-row a-size-small"><span class="a-declarative"
|
||||
data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw" data-action="a-popover"
|
||||
data-csa-c-func-deps="aui-da-a-popover"
|
||||
data-a-popover="{"position":"triggerBottom","popoverLabel":"4.7 out of 5 stars, rating details","url":"/review/widgets/average-customer-review/popover/ref=acr_search__popover?ie=UTF8&asin=B081XSYZMS&ref_=acr_search__popover&contextId=search","closeButton":true,"closeButtonLabel":""}"
|
||||
data-csa-c-type="widget" data-csa-c-id="oykdvt-8s1ebj-2kegf2-7ii7tp"><a
|
||||
aria-label="4.7 out of 5 stars, rating details"
|
||||
href="javascript:void(0)" role="button"
|
||||
class="a-popover-trigger a-declarative"><i
|
||||
data-cy="reviews-ratings-slot" aria-hidden="true"
|
||||
class="a-icon a-icon-star-small a-star-small-4-5"><span
|
||||
class="a-icon-alt">4.7 out of 5 stars</span></i><i
|
||||
class="a-icon a-icon-popover"></i></a></span> <span
|
||||
data-component-type="s-client-side-analytics" class="rush-component"
|
||||
data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw" data-component-id="8">
|
||||
<div style="display: inline-block"
|
||||
class="s-csa-instrumentation-wrapper alf-search-csa-instrumentation-wrapper"
|
||||
data-csa-c-type="alf-af-component"
|
||||
data-csa-c-content-id="alf-customer-ratings-count-component"
|
||||
data-csa-c-slot-id="alf-reviews" data-csa-op-log-render=""
|
||||
data-csa-c-layout="GRID" data-csa-c-asin="B081XSYZMS"
|
||||
data-csa-c-id="6l5wc4-ngelan-hd9x4t-d4a2k7"><a aria-label="184 ratings"
|
||||
class="a-link-normal s-underline-text s-underline-link-text s-link-style"
|
||||
href="/sspa/click?ie=UTF8&spc=MToxNzQwNTE0ODkzNDczNzk3OjE3NDkyOTk4MzM6c3BfYXRmOjIwMDA2NzY0ODgwMjc5ODo6MDo6&url=%2FDeath-Star-R2D2-Balls-Printed%2Fdp%2FB081XSYZMS%2Fref%3Dsr_1_1_sspa%3Fcrid%3D3C1EXMXN59Q9G%26dib%3DeyJ2IjoiMSJ9.7tBl5bhZh59L9qIPZUe9SLa2fy_HvzboxuQxvrRcAc0VUXayi9fxQFsMLyFplDE9vMkIJbP76AVpa-5-fxhNza3DqhX4tss4NlB49WPi_dA00Hw6O8qK5pDzdetYlhGgOyXOLBe7mTG9oJ5W0wcvQhEVoX9mpJk_SGeqRLWGA0dBSjYCZtiyrY8_B-DP53S7fbYwiSYtq-g7sQDXKVadRpGvUyKq7yxA0SLsU42uvoqSGb0qcd6udL1wbnTEkKmwNjNSb7xIUb-8PyE7DTPMt1ScJksn70sFQMJNkM2aK5M.x9_jYvKPnSibV1d0umUStZBxlSTSXrzVIFKqFzS8c-U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749299833%26sprefix%3Dr2d2%252Caps%252C548%26sr%3D8-1-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9hdGY%26psc%3D1#customerReviews"><span
|
||||
aria-hidden="true"
|
||||
class="a-size-base s-underline-text">184</span> </a> </div>
|
||||
</span></div>
|
||||
<div class="a-row a-size-base"><span class="a-size-base a-color-secondary">50+
|
||||
bought in past month</span></div>
|
||||
</div>
|
||||
<div data-cy="price-recipe"
|
||||
class="a-section a-spacing-none a-spacing-top-small s-price-instructions-style">
|
||||
<div class="a-row a-size-base a-color-base">
|
||||
<div class="a-row"><span id="price-link" class="aok-offscreen">Price, product
|
||||
page</span><a aria-describedby="price-link"
|
||||
class="a-link-normal s-no-hover s-underline-text s-underline-link-text s-link-style a-text-normal"
|
||||
href="/sspa/click?ie=UTF8&spc=MToxNzQwNTE0ODkzNDczNzk3OjE3NDkyOTk4MzM6c3BfYXRmOjIwMDA2NzY0ODgwMjc5ODo6MDo6&url=%2FDeath-Star-R2D2-Balls-Printed%2Fdp%2FB081XSYZMS%2Fref%3Dsr_1_1_sspa%3Fcrid%3D3C1EXMXN59Q9G%26dib%3DeyJ2IjoiMSJ9.7tBl5bhZh59L9qIPZUe9SLa2fy_HvzboxuQxvrRcAc0VUXayi9fxQFsMLyFplDE9vMkIJbP76AVpa-5-fxhNza3DqhX4tss4NlB49WPi_dA00Hw6O8qK5pDzdetYlhGgOyXOLBe7mTG9oJ5W0wcvQhEVoX9mpJk_SGeqRLWGA0dBSjYCZtiyrY8_B-DP53S7fbYwiSYtq-g7sQDXKVadRpGvUyKq7yxA0SLsU42uvoqSGb0qcd6udL1wbnTEkKmwNjNSb7xIUb-8PyE7DTPMt1ScJksn70sFQMJNkM2aK5M.x9_jYvKPnSibV1d0umUStZBxlSTSXrzVIFKqFzS8c-U%26dib_tag%3Dse%26keywords%3Dr2d2%26qid%3D1749299833%26sprefix%3Dr2d2%252Caps%252C548%26sr%3D8-1-spons%26sp_csd%3Dd2lkZ2V0TmFtZT1zcF9hdGY%26psc%3D1"><span
|
||||
class="a-price" data-a-size="xl" data-a-color="base"><span
|
||||
class="a-offscreen">$29.95</span><span aria-hidden="true"><span
|
||||
class="a-price-symbol">$</span><span
|
||||
class="a-price-whole">29<span
|
||||
class="a-price-decimal">.</span></span><span
|
||||
class="a-price-fraction">95</span></span></span></a></div>
|
||||
<div class="a-row"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div data-cy="delivery-recipe" class="a-section a-spacing-none a-spacing-top-micro">
|
||||
<div class="a-row a-size-base a-color-secondary s-align-children-center"><span
|
||||
aria-label="FREE delivery Thu, Jun 19 to Malaysia on $49 of eligible items"><span
|
||||
class="a-color-base">FREE delivery </span><span
|
||||
class="a-color-base a-text-bold">Thu, Jun 19 </span><span
|
||||
class="a-color-base">to Malaysia on $49 of eligible items</span></span>
|
||||
</div>
|
||||
</div>
|
||||
<div data-cy="certification-recipe"
|
||||
class="a-section a-spacing-none a-spacing-top-micro">
|
||||
<div class="a-row">
|
||||
<div class="a-section a-spacing-none s-align-children-center">
|
||||
<div class="a-section a-spacing-none s-pc-faceout-container">
|
||||
<div>
|
||||
<div class="s-align-children-center"><span class="a-declarative"
|
||||
data-version-id="v2dwi5hq8xzthf26x0gg1mcl2oj"
|
||||
data-render-id="r3o8bgr5zt3kmy2jv4su6fn4kyw"
|
||||
data-action="s-pc-sidesheet-open"
|
||||
data-csa-c-func-deps="aui-da-s-pc-sidesheet-open"
|
||||
data-s-pc-sidesheet-open="{"preloadDomId":"pc-side-sheet-B081XSYZMS","popoverLabel":"Product certifications","interactLoggingMetricsList":["provenanceCertifications_desktop_sbe_badge"],"closeButtonLabel":"Close popup","dwellMetric":"provenanceCertifications_desktop_sbe_badge_t"}"
|
||||
data-csa-c-type="widget"
|
||||
data-csa-c-id="hdfxi6-bjlgup-5dql15-88t9ao"><a
|
||||
data-cy="s-pc-faceout-badge"
|
||||
class="a-link-normal s-no-underline s-pc-badge s-align-children-center aok-block"
|
||||
href="javascript:void(0)" role="button">
|
||||
<div
|
||||
class="a-section s-pc-attribute-pill-text s-margin-bottom-none s-margin-bottom-none aok-block s-pc-certification-faceout">
|
||||
<span class="faceout-image-view"></span><img alt=""
|
||||
src="https://m.media-amazon.com/images/I/111mHoVK0kL._SS200_.png"
|
||||
class="s-image" height="18px" width="18px">
|
||||
<span class="a-size-base a-color-base">Small
|
||||
Business</span>
|
||||
<div
|
||||
class="s-margin-bottom-none s-pc-sidesheet-chevron aok-nowrap">
|
||||
<i class="a-icon a-icon-popover aok-align-center"
|
||||
role="presentation"></i></div>
|
||||
</div>
|
||||
</a></span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="pc-side-sheet-B081XSYZMS"
|
||||
class="a-section puis puis-v2dwi5hq8xzthf26x0gg1mcl2oj aok-hidden">
|
||||
<div class="a-section s-pc-container-side-sheet">
|
||||
<div class="s-align-children-center a-spacing-small">
|
||||
<div class="s-align-children-center s-pc-certification"
|
||||
role="heading" aria-level="2"><span
|
||||
class="faceout-image-view"></span>
|
||||
<div alt="" style="height: 24px; width: 24px;"
|
||||
class="a-image-wrapper a-lazy-loaded a-manually-loaded s-image"
|
||||
data-a-image-source="https://m.media-amazon.com/images/I/111mHoVK0kL._SS200_.png">
|
||||
<noscript><img alt=""
|
||||
src="https://m.media-amazon.com/images/I/111mHoVK0kL._SS200_.png"
|
||||
height="24px" width="24px" /></noscript></div> <span
|
||||
class="a-size-medium-plus a-color-base a-text-bold">Small
|
||||
Business</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="a-spacing-medium s-pc-link-container"><span
|
||||
class="a-size-base a-color-secondary">Shop products from small
|
||||
business brands sold in Amazon’s store. Discover more about the
|
||||
small businesses partnering with Amazon and Amazon’s commitment
|
||||
to empowering them.</span> <a
|
||||
class="a-size-base a-link-normal s-link-style"
|
||||
href="https://www.amazon.com/b/ref=s9_acss_bw_cg_sbp22c_1e1_w/ref=SBE_navbar_5?pf_rd_r=6W5X52VNZRB7GK1E1VX2&pf_rd_p=56621c3d-cff4-45e1-9bf4-79bbeb8006fc&pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-top-3&pf_rd_t=30901&pf_rd_i=17879387011&node=18018208011">Learn
|
||||
more</a> </div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1,217 @@
|
||||
"""
|
||||
C4A-Script API Usage Examples
|
||||
Shows how to use the new Result-based API in various scenarios
|
||||
"""
|
||||
|
||||
from crawl4ai.script.c4a_compile import compile, validate, compile_file
|
||||
from crawl4ai.script.c4a_result import CompilationResult, ValidationResult
|
||||
import json
|
||||
|
||||
|
||||
print("C4A-Script API Usage Examples")
|
||||
print("=" * 80)
|
||||
|
||||
# Example 1: Basic compilation
|
||||
print("\n1. Basic Compilation")
|
||||
print("-" * 40)
|
||||
|
||||
script = """
|
||||
GO https://example.com
|
||||
WAIT 2
|
||||
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept`
|
||||
REPEAT (SCROLL DOWN 300, 3)
|
||||
"""
|
||||
|
||||
result = compile(script)
|
||||
print(f"Success: {result.success}")
|
||||
print(f"Statements generated: {len(result.js_code) if result.js_code else 0}")
|
||||
|
||||
# Example 2: Error handling
|
||||
print("\n\n2. Error Handling")
|
||||
print("-" * 40)
|
||||
|
||||
error_script = """
|
||||
GO https://example.com
|
||||
IF (EXISTS `.modal`) CLICK `.close`
|
||||
undefined_procedure
|
||||
"""
|
||||
|
||||
result = compile(error_script)
|
||||
if not result.success:
|
||||
# Access error details
|
||||
error = result.first_error
|
||||
print(f"Error on line {error.line}: {error.message}")
|
||||
print(f"Error code: {error.code}")
|
||||
|
||||
# Show suggestions if available
|
||||
if error.suggestions:
|
||||
print("Suggestions:")
|
||||
for suggestion in error.suggestions:
|
||||
print(f" - {suggestion.message}")
|
||||
|
||||
# Example 3: Validation only
|
||||
print("\n\n3. Validation (no code generation)")
|
||||
print("-" * 40)
|
||||
|
||||
validation_script = """
|
||||
PROC validate_form
|
||||
IF (EXISTS `#email`) THEN TYPE "test@example.com"
|
||||
PRESS Tab
|
||||
ENDPROC
|
||||
|
||||
validate_form
|
||||
"""
|
||||
|
||||
validation = validate(validation_script)
|
||||
print(f"Valid: {validation.valid}")
|
||||
if validation.errors:
|
||||
print(f"Errors found: {len(validation.errors)}")
|
||||
|
||||
# Example 4: JSON output for UI
|
||||
print("\n\n4. JSON Output for UI Integration")
|
||||
print("-" * 40)
|
||||
|
||||
ui_script = """
|
||||
CLICK button.submit
|
||||
"""
|
||||
|
||||
result = compile(ui_script)
|
||||
if not result.success:
|
||||
# Get JSON for UI
|
||||
error_json = result.to_dict()
|
||||
print("Error data for UI:")
|
||||
print(json.dumps(error_json["errors"][0], indent=2))
|
||||
|
||||
# Example 5: File compilation
|
||||
print("\n\n5. File Compilation")
|
||||
print("-" * 40)
|
||||
|
||||
# Create a test file
|
||||
test_file = "test_script.c4a"
|
||||
with open(test_file, "w") as f:
|
||||
f.write("""
|
||||
GO https://example.com
|
||||
WAIT `.content` 5
|
||||
CLICK `.main-button`
|
||||
""")
|
||||
|
||||
result = compile_file(test_file)
|
||||
print(f"File compilation: {'Success' if result.success else 'Failed'}")
|
||||
if result.success:
|
||||
print(f"Generated {len(result.js_code)} JavaScript statements")
|
||||
|
||||
# Clean up
|
||||
import os
|
||||
os.remove(test_file)
|
||||
|
||||
# Example 6: Batch processing
|
||||
print("\n\n6. Batch Processing Multiple Scripts")
|
||||
print("-" * 40)
|
||||
|
||||
scripts = [
|
||||
"GO https://example1.com\nCLICK `.button`",
|
||||
"GO https://example2.com\nWAIT 2",
|
||||
"GO https://example3.com\nINVALID_CMD"
|
||||
]
|
||||
|
||||
results = []
|
||||
for i, script in enumerate(scripts, 1):
|
||||
result = compile(script)
|
||||
results.append(result)
|
||||
status = "✓" if result.success else "✗"
|
||||
print(f"Script {i}: {status}")
|
||||
|
||||
# Summary
|
||||
successful = sum(1 for r in results if r.success)
|
||||
print(f"\nBatch result: {successful}/{len(scripts)} successful")
|
||||
|
||||
# Example 7: Custom error formatting
|
||||
print("\n\n7. Custom Error Formatting")
|
||||
print("-" * 40)
|
||||
|
||||
def format_error_for_ide(error):
|
||||
"""Format error for IDE integration"""
|
||||
return f"{error.source_line}:{error.line}:{error.column}: {error.type.value}: {error.message} [{error.code}]"
|
||||
|
||||
error_script = "IF EXISTS `.button` THEN CLICK `.button`"
|
||||
result = compile(error_script)
|
||||
|
||||
if not result.success:
|
||||
error = result.first_error
|
||||
print("IDE format:", format_error_for_ide(error))
|
||||
print("Simple format:", error.simple_message)
|
||||
print("Full format:", error.formatted_message)
|
||||
|
||||
# Example 8: Working with warnings (future feature)
|
||||
print("\n\n8. Handling Warnings")
|
||||
print("-" * 40)
|
||||
|
||||
# In the future, we might have warnings
|
||||
result = compile("GO https://example.com\nWAIT 100") # Very long wait
|
||||
print(f"Success: {result.success}")
|
||||
print(f"Warnings: {len(result.warnings)}")
|
||||
|
||||
# Example 9: Metadata usage
|
||||
print("\n\n9. Using Metadata")
|
||||
print("-" * 40)
|
||||
|
||||
complex_script = """
|
||||
PROC helper1
|
||||
CLICK `.btn1`
|
||||
ENDPROC
|
||||
|
||||
PROC helper2
|
||||
CLICK `.btn2`
|
||||
ENDPROC
|
||||
|
||||
GO https://example.com
|
||||
helper1
|
||||
helper2
|
||||
"""
|
||||
|
||||
result = compile(complex_script)
|
||||
if result.success:
|
||||
print(f"Script metadata:")
|
||||
for key, value in result.metadata.items():
|
||||
print(f" {key}: {value}")
|
||||
|
||||
# Example 10: Integration patterns
|
||||
print("\n\n10. Integration Patterns")
|
||||
print("-" * 40)
|
||||
|
||||
# Web API endpoint simulation
|
||||
def api_compile(request_body):
|
||||
"""Simulate API endpoint"""
|
||||
script = request_body.get("script", "")
|
||||
result = compile(script)
|
||||
|
||||
response = {
|
||||
"status": "success" if result.success else "error",
|
||||
"data": result.to_dict()
|
||||
}
|
||||
return response
|
||||
|
||||
# CLI tool simulation
|
||||
def cli_compile(script, output_format="text"):
|
||||
"""Simulate CLI tool"""
|
||||
result = compile(script)
|
||||
|
||||
if output_format == "json":
|
||||
return result.to_json()
|
||||
elif output_format == "simple":
|
||||
if result.success:
|
||||
return f"OK: {len(result.js_code)} statements"
|
||||
else:
|
||||
return f"ERROR: {result.first_error.simple_message}"
|
||||
else:
|
||||
return str(result)
|
||||
|
||||
# Test the patterns
|
||||
api_response = api_compile({"script": "GO https://example.com"})
|
||||
print(f"API response status: {api_response['status']}")
|
||||
|
||||
cli_output = cli_compile("WAIT 2", "simple")
|
||||
print(f"CLI output: {cli_output}")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("All examples completed successfully!")
|
||||
@@ -0,0 +1,53 @@
|
||||
"""
|
||||
C4A-Script Hello World
|
||||
A concise example showing how to use the C4A-Script compiler
|
||||
"""
|
||||
|
||||
from crawl4ai.script.c4a_compile import compile
|
||||
|
||||
# Define your C4A-Script
|
||||
script = """
|
||||
GO https://example.com
|
||||
WAIT `#content` 5
|
||||
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept`
|
||||
CLICK `button.submit`
|
||||
"""
|
||||
|
||||
# Compile the script
|
||||
result = compile(script)
|
||||
|
||||
# Check if compilation was successful
|
||||
if result.success:
|
||||
# Success! Use the generated JavaScript
|
||||
print("✅ Compilation successful!")
|
||||
print(f"Generated {len(result.js_code)} JavaScript statements:\n")
|
||||
|
||||
for i, js in enumerate(result.js_code, 1):
|
||||
print(f"{i}. {js}\n")
|
||||
|
||||
# In real usage, you'd pass result.js_code to Crawl4AI:
|
||||
# config = CrawlerRunConfig(js_code=result.js_code)
|
||||
|
||||
else:
|
||||
# Error! Handle the compilation error
|
||||
print("❌ Compilation failed!")
|
||||
|
||||
# Get the first error (there might be multiple)
|
||||
error = result.first_error
|
||||
|
||||
# Show error details
|
||||
print(f"Error at line {error.line}, column {error.column}")
|
||||
print(f"Message: {error.message}")
|
||||
|
||||
# Show the problematic code
|
||||
print(f"\nCode: {error.source_line}")
|
||||
print(" " * (6 + error.column) + "^")
|
||||
|
||||
# Show suggestions if available
|
||||
if error.suggestions:
|
||||
print("\n💡 How to fix:")
|
||||
for suggestion in error.suggestions:
|
||||
print(f" {suggestion.message}")
|
||||
|
||||
# For debugging or logging, you can also get JSON
|
||||
# error_json = result.to_json()
|
||||
@@ -0,0 +1,53 @@
|
||||
"""
|
||||
C4A-Script Hello World - Error Example
|
||||
Shows how error handling works
|
||||
"""
|
||||
|
||||
from crawl4ai.script.c4a_compile import compile
|
||||
|
||||
# Define a script with an error (missing THEN)
|
||||
script = """
|
||||
GO https://example.com
|
||||
WAIT `#content` 5
|
||||
IF (EXISTS `.cookie-banner`) CLICK `.accept`
|
||||
CLICK `button.submit`
|
||||
"""
|
||||
|
||||
# Compile the script
|
||||
result = compile(script)
|
||||
|
||||
# Check if compilation was successful
|
||||
if result.success:
|
||||
# Success! Use the generated JavaScript
|
||||
print("✅ Compilation successful!")
|
||||
print(f"Generated {len(result.js_code)} JavaScript statements:\n")
|
||||
|
||||
for i, js in enumerate(result.js_code, 1):
|
||||
print(f"{i}. {js}\n")
|
||||
|
||||
# In real usage, you'd pass result.js_code to Crawl4AI:
|
||||
# config = CrawlerRunConfig(js_code=result.js_code)
|
||||
|
||||
else:
|
||||
# Error! Handle the compilation error
|
||||
print("❌ Compilation failed!")
|
||||
|
||||
# Get the first error (there might be multiple)
|
||||
error = result.first_error
|
||||
|
||||
# Show error details
|
||||
print(f"Error at line {error.line}, column {error.column}")
|
||||
print(f"Message: {error.message}")
|
||||
|
||||
# Show the problematic code
|
||||
print(f"\nCode: {error.source_line}")
|
||||
print(" " * (6 + error.column) + "^")
|
||||
|
||||
# Show suggestions if available
|
||||
if error.suggestions:
|
||||
print("\n💡 How to fix:")
|
||||
for suggestion in error.suggestions:
|
||||
print(f" {suggestion.message}")
|
||||
|
||||
# For debugging or logging, you can also get JSON
|
||||
# error_json = result.to_json()
|
||||
@@ -0,0 +1,285 @@
|
||||
"""
|
||||
Demonstration of C4A-Script integration with Crawl4AI
|
||||
Shows various use cases and features
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai import c4a_compile, CompilationResult
|
||||
|
||||
async def example_basic_usage():
|
||||
"""Basic C4A-Script usage with Crawl4AI"""
|
||||
print("\n" + "="*60)
|
||||
print("Example 1: Basic C4A-Script Usage")
|
||||
print("="*60)
|
||||
|
||||
# Define your automation script
|
||||
c4a_script = """
|
||||
# Wait for page to load
|
||||
WAIT `body` 2
|
||||
|
||||
# Handle cookie banner if present
|
||||
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept-btn`
|
||||
|
||||
# Scroll down to load more content
|
||||
SCROLL DOWN 500
|
||||
WAIT 1
|
||||
|
||||
# Click load more button if exists
|
||||
IF (EXISTS `.load-more`) THEN CLICK `.load-more`
|
||||
"""
|
||||
|
||||
# Create crawler config with C4A script
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com",
|
||||
c4a_script=c4a_script,
|
||||
wait_for="css:.content",
|
||||
verbose=False
|
||||
)
|
||||
|
||||
print("✅ C4A Script compiled successfully!")
|
||||
print(f"Generated {len(config.js_code)} JavaScript commands")
|
||||
|
||||
# In production, you would run:
|
||||
# async with AsyncWebCrawler() as crawler:
|
||||
# result = await crawler.arun(config=config)
|
||||
|
||||
|
||||
async def example_form_filling():
|
||||
"""Form filling with C4A-Script"""
|
||||
print("\n" + "="*60)
|
||||
print("Example 2: Form Filling with C4A-Script")
|
||||
print("="*60)
|
||||
|
||||
# Form automation script
|
||||
form_script = """
|
||||
# Set form values
|
||||
SET email = "test@example.com"
|
||||
SET message = "This is a test message"
|
||||
|
||||
# Fill the form
|
||||
CLICK `#email-input`
|
||||
TYPE $email
|
||||
|
||||
CLICK `#message-textarea`
|
||||
TYPE $message
|
||||
|
||||
# Submit the form
|
||||
CLICK `button[type="submit"]`
|
||||
|
||||
# Wait for success message
|
||||
WAIT `.success-message` 10
|
||||
"""
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com/contact",
|
||||
c4a_script=form_script
|
||||
)
|
||||
|
||||
print("✅ Form filling script ready")
|
||||
print("Script will:")
|
||||
print(" - Fill email field")
|
||||
print(" - Fill message textarea")
|
||||
print(" - Submit form")
|
||||
print(" - Wait for confirmation")
|
||||
|
||||
|
||||
async def example_dynamic_loading():
|
||||
"""Handle dynamic content loading"""
|
||||
print("\n" + "="*60)
|
||||
print("Example 3: Dynamic Content Loading")
|
||||
print("="*60)
|
||||
|
||||
# Script for infinite scroll or pagination
|
||||
pagination_script = """
|
||||
# Initial wait
|
||||
WAIT `.product-list` 5
|
||||
|
||||
# Load all products by clicking "Load More" repeatedly
|
||||
REPEAT (CLICK `.load-more`, `document.querySelector('.load-more') !== null`)
|
||||
|
||||
# Alternative: Scroll to load (infinite scroll)
|
||||
# REPEAT (SCROLL DOWN 1000, `document.querySelectorAll('.product').length < 100`)
|
||||
|
||||
# Extract count
|
||||
EVAL `console.log('Products loaded: ' + document.querySelectorAll('.product').length)`
|
||||
"""
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com/products",
|
||||
c4a_script=pagination_script,
|
||||
screenshot=True # Capture final state
|
||||
)
|
||||
|
||||
print("✅ Dynamic loading script ready")
|
||||
print("Script will load all products by repeatedly clicking 'Load More'")
|
||||
|
||||
|
||||
async def example_multi_step_workflow():
|
||||
"""Complex multi-step workflow with procedures"""
|
||||
print("\n" + "="*60)
|
||||
print("Example 4: Multi-Step Workflow with Procedures")
|
||||
print("="*60)
|
||||
|
||||
# Complex workflow with reusable procedures
|
||||
workflow_script = """
|
||||
# Define login procedure
|
||||
PROC login
|
||||
CLICK `#username`
|
||||
TYPE "demo_user"
|
||||
CLICK `#password`
|
||||
TYPE "demo_pass"
|
||||
CLICK `#login-btn`
|
||||
WAIT `.dashboard` 10
|
||||
ENDPROC
|
||||
|
||||
# Define search procedure
|
||||
PROC search_product
|
||||
CLICK `.search-box`
|
||||
TYPE "laptop"
|
||||
PRESS Enter
|
||||
WAIT `.search-results` 5
|
||||
ENDPROC
|
||||
|
||||
# Main workflow
|
||||
GO https://example.com
|
||||
login
|
||||
search_product
|
||||
|
||||
# Process results
|
||||
IF (EXISTS `.no-results`) THEN EVAL `console.log('No products found')`
|
||||
ELSE REPEAT (CLICK `.add-to-cart`, 3)
|
||||
"""
|
||||
|
||||
# Compile to check for errors
|
||||
result = c4a_compile(workflow_script)
|
||||
|
||||
if result.success:
|
||||
print("✅ Complex workflow compiled successfully!")
|
||||
print("Workflow includes:")
|
||||
print(" - Login procedure")
|
||||
print(" - Product search")
|
||||
print(" - Conditional cart additions")
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com",
|
||||
c4a_script=workflow_script
|
||||
)
|
||||
else:
|
||||
print("❌ Compilation error:")
|
||||
error = result.first_error
|
||||
print(f" Line {error.line}: {error.message}")
|
||||
|
||||
|
||||
async def example_error_handling():
|
||||
"""Demonstrate error handling"""
|
||||
print("\n" + "="*60)
|
||||
print("Example 5: Error Handling")
|
||||
print("="*60)
|
||||
|
||||
# Script with intentional error
|
||||
bad_script = """
|
||||
WAIT body 2
|
||||
CLICK button
|
||||
IF (EXISTS .modal) CLICK .close
|
||||
"""
|
||||
|
||||
try:
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com",
|
||||
c4a_script=bad_script
|
||||
)
|
||||
except ValueError as e:
|
||||
print("✅ Error caught as expected:")
|
||||
print(f" {e}")
|
||||
|
||||
# Fixed version
|
||||
good_script = """
|
||||
WAIT `body` 2
|
||||
CLICK `button`
|
||||
IF (EXISTS `.modal`) THEN CLICK `.close`
|
||||
"""
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com",
|
||||
c4a_script=good_script
|
||||
)
|
||||
|
||||
print("\n✅ Fixed script compiled successfully!")
|
||||
|
||||
|
||||
async def example_combining_with_extraction():
|
||||
"""Combine C4A-Script with extraction strategies"""
|
||||
print("\n" + "="*60)
|
||||
print("Example 6: C4A-Script + Extraction Strategies")
|
||||
print("="*60)
|
||||
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
|
||||
# Script to prepare page for extraction
|
||||
prep_script = """
|
||||
# Expand all collapsed sections
|
||||
REPEAT (CLICK `.expand-btn`, `document.querySelectorAll('.expand-btn:not(.expanded)').length > 0`)
|
||||
|
||||
# Load all comments
|
||||
IF (EXISTS `.load-comments`) THEN CLICK `.load-comments`
|
||||
WAIT `.comments-section` 5
|
||||
|
||||
# Close any popups
|
||||
IF (EXISTS `.popup-close`) THEN CLICK `.popup-close`
|
||||
"""
|
||||
|
||||
# Define extraction schema
|
||||
schema = {
|
||||
"name": "article",
|
||||
"selector": "article.main",
|
||||
"fields": {
|
||||
"title": {"selector": "h1", "type": "text"},
|
||||
"content": {"selector": ".content", "type": "text"},
|
||||
"comments": {
|
||||
"selector": ".comment",
|
||||
"type": "list",
|
||||
"fields": {
|
||||
"author": {"selector": ".author", "type": "text"},
|
||||
"text": {"selector": ".text", "type": "text"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
url="https://example.com/article",
|
||||
c4a_script=prep_script,
|
||||
extraction_strategy=JsonCssExtractionStrategy(schema),
|
||||
wait_for="css:.comments-section"
|
||||
)
|
||||
|
||||
print("✅ Combined C4A + Extraction ready")
|
||||
print("Workflow:")
|
||||
print(" 1. Expand collapsed sections")
|
||||
print(" 2. Load comments")
|
||||
print(" 3. Extract structured data")
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run all examples"""
|
||||
print("\n🚀 C4A-Script + Crawl4AI Integration Demo\n")
|
||||
|
||||
# Run all examples
|
||||
await example_basic_usage()
|
||||
await example_form_filling()
|
||||
await example_dynamic_loading()
|
||||
await example_multi_step_workflow()
|
||||
await example_error_handling()
|
||||
await example_combining_with_extraction()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("✅ All examples completed successfully!")
|
||||
print("="*60)
|
||||
|
||||
print("\nTo run actual crawls, uncomment the AsyncWebCrawler sections")
|
||||
print("or create your own scripts using these examples as templates.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Hello World Example: LLM-Generated C4A-Script
|
||||
|
||||
This example shows how to use the new generate_script() function to automatically
|
||||
create C4A-Script automation from natural language descriptions and HTML.
|
||||
"""
|
||||
|
||||
from crawl4ai.script.c4a_compile import C4ACompiler
|
||||
|
||||
def main():
|
||||
print("🤖 C4A-Script Generation Hello World")
|
||||
print("=" * 50)
|
||||
|
||||
# Example 1: Simple login form
|
||||
html = """
|
||||
<html>
|
||||
<body>
|
||||
<form id="login">
|
||||
<input id="email" type="email" placeholder="Email">
|
||||
<input id="password" type="password" placeholder="Password">
|
||||
<button id="submit">Login</button>
|
||||
</form>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
goal = "Fill in email 'user@example.com', password 'secret123', and submit the form"
|
||||
|
||||
print("📝 Goal:", goal)
|
||||
print("🌐 HTML: Simple login form")
|
||||
print()
|
||||
|
||||
# Generate C4A-Script
|
||||
print("🔧 Generated C4A-Script:")
|
||||
print("-" * 30)
|
||||
c4a_script = C4ACompiler.generate_script(
|
||||
html=html,
|
||||
query=goal,
|
||||
mode="c4a"
|
||||
)
|
||||
print(c4a_script)
|
||||
print()
|
||||
|
||||
# Generate JavaScript
|
||||
print("🔧 Generated JavaScript:")
|
||||
print("-" * 30)
|
||||
js_script = C4ACompiler.generate_script(
|
||||
html=html,
|
||||
query=goal,
|
||||
mode="js"
|
||||
)
|
||||
print(js_script)
|
||||
print()
|
||||
|
||||
# Example 2: Simple button click
|
||||
html2 = """
|
||||
<html>
|
||||
<body>
|
||||
<div class="content">
|
||||
<h1>Welcome!</h1>
|
||||
<button id="start-btn" class="primary">Get Started</button>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
goal2 = "Click the 'Get Started' button"
|
||||
|
||||
print("=" * 50)
|
||||
print("📝 Goal:", goal2)
|
||||
print("🌐 HTML: Simple button")
|
||||
print()
|
||||
|
||||
print("🔧 Generated C4A-Script:")
|
||||
print("-" * 30)
|
||||
c4a_script2 = C4ACompiler.generate_script(
|
||||
html=html2,
|
||||
query=goal2,
|
||||
mode="c4a"
|
||||
)
|
||||
print(c4a_script2)
|
||||
print()
|
||||
|
||||
print("✅ Done! The LLM automatically converted natural language goals")
|
||||
print(" into executable automation scripts.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,111 @@
|
||||
[
|
||||
{
|
||||
"repository_name": "unclecode/crawl4ai",
|
||||
"repository_owner": "unclecode/crawl4ai",
|
||||
"repository_url": "/unclecode/crawl4ai",
|
||||
"description": "\ud83d\ude80\ud83e\udd16Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here:https://discord.gg/jP8KfhDhyN",
|
||||
"primary_language": "Python",
|
||||
"star_count": "45.1k",
|
||||
"topics": [],
|
||||
"last_updated": "23 hours ago"
|
||||
},
|
||||
{
|
||||
"repository_name": "coleam00/mcp-crawl4ai-rag",
|
||||
"repository_owner": "coleam00/mcp-crawl4ai-rag",
|
||||
"repository_url": "/coleam00/mcp-crawl4ai-rag",
|
||||
"description": "Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants",
|
||||
"primary_language": "Python",
|
||||
"star_count": "748",
|
||||
"topics": [],
|
||||
"last_updated": "yesterday"
|
||||
},
|
||||
{
|
||||
"repository_name": "pdichone/crawl4ai-rag-system",
|
||||
"repository_owner": "pdichone/crawl4ai-rag-system",
|
||||
"repository_url": "/pdichone/crawl4ai-rag-system",
|
||||
"primary_language": "Python",
|
||||
"star_count": "44",
|
||||
"topics": [],
|
||||
"last_updated": "on 21 Jan"
|
||||
},
|
||||
{
|
||||
"repository_name": "weidwonder/crawl4ai-mcp-server",
|
||||
"repository_owner": "weidwonder/crawl4ai-mcp-server",
|
||||
"repository_url": "/weidwonder/crawl4ai-mcp-server",
|
||||
"description": "\u7528\u4e8e\u63d0\u4f9b\u7ed9\u672c\u5730\u5f00\u53d1\u8005\u7684 LLM\u7684\u9ad8\u6548\u4e92\u8054\u7f51\u641c\u7d22&\u5185\u5bb9\u83b7\u53d6\u7684MCP Server\uff0c \u8282\u7701\u4f60\u7684token",
|
||||
"primary_language": "Python",
|
||||
"star_count": "87",
|
||||
"topics": [],
|
||||
"last_updated": "24 days ago"
|
||||
},
|
||||
{
|
||||
"repository_name": "leonardogrig/crawl4ai-deepseek-example",
|
||||
"repository_owner": "leonardogrig/crawl4ai-deepseek-example",
|
||||
"repository_url": "/leonardogrig/crawl4ai-deepseek-example",
|
||||
"primary_language": "Python",
|
||||
"star_count": "29",
|
||||
"topics": [],
|
||||
"last_updated": "on 18 Jan"
|
||||
},
|
||||
{
|
||||
"repository_name": "laurentvv/crawl4ai-mcp",
|
||||
"repository_owner": "laurentvv/crawl4ai-mcp",
|
||||
"repository_url": "/laurentvv/crawl4ai-mcp",
|
||||
"description": "Web crawling tool that integrates with AI assistants via the MCP",
|
||||
"primary_language": "Python",
|
||||
"star_count": "10",
|
||||
"topics": [
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
{}
|
||||
],
|
||||
"last_updated": "on 16 Mar"
|
||||
},
|
||||
{
|
||||
"repository_name": "kaymen99/ai-web-scraper",
|
||||
"repository_owner": "kaymen99/ai-web-scraper",
|
||||
"repository_url": "/kaymen99/ai-web-scraper",
|
||||
"description": "AI web scraper built withCrawl4AIfor extracting structured leads data from websites.",
|
||||
"primary_language": "Python",
|
||||
"star_count": "30",
|
||||
"topics": [
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
{}
|
||||
],
|
||||
"last_updated": "on 13 Feb"
|
||||
},
|
||||
{
|
||||
"repository_name": "atakkant/ai_web_crawler",
|
||||
"repository_owner": "atakkant/ai_web_crawler",
|
||||
"repository_url": "/atakkant/ai_web_crawler",
|
||||
"description": "crawl4ai, DeepSeek, Groq",
|
||||
"primary_language": "Python",
|
||||
"star_count": "9",
|
||||
"topics": [],
|
||||
"last_updated": "on 19 Feb"
|
||||
},
|
||||
{
|
||||
"repository_name": "Croups/auto-scraper-with-llms",
|
||||
"repository_owner": "Croups/auto-scraper-with-llms",
|
||||
"repository_url": "/Croups/auto-scraper-with-llms",
|
||||
"description": "Web scraping AI that leverages thecrawl4ailibrary to extract structured data from web pages using various large language models (LLMs).",
|
||||
"primary_language": "Python",
|
||||
"star_count": "49",
|
||||
"topics": [],
|
||||
"last_updated": "on 8 Apr"
|
||||
},
|
||||
{
|
||||
"repository_name": "leonardogrig/crawl4ai_llm_examples",
|
||||
"repository_owner": "leonardogrig/crawl4ai_llm_examples",
|
||||
"repository_url": "/leonardogrig/crawl4ai_llm_examples",
|
||||
"primary_language": "Python",
|
||||
"star_count": "8",
|
||||
"topics": [],
|
||||
"last_updated": "on 29 Jan"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"name": "GitHub Repository Cards",
|
||||
"baseSelector": "div.Box-sc-g0xbh4-0.iwUbcA",
|
||||
"fields": [
|
||||
{
|
||||
"name": "repository_name",
|
||||
"selector": "div.search-title a span",
|
||||
"type": "text",
|
||||
"transform": "strip"
|
||||
},
|
||||
{
|
||||
"name": "repository_owner",
|
||||
"selector": "div.search-title a span",
|
||||
"type": "text",
|
||||
"transform": "split",
|
||||
"pattern": "/"
|
||||
},
|
||||
{
|
||||
"name": "repository_url",
|
||||
"selector": "div.search-title a",
|
||||
"type": "attribute",
|
||||
"attribute": "href",
|
||||
"transform": "prepend",
|
||||
"pattern": "https://github.com"
|
||||
},
|
||||
{
|
||||
"name": "description",
|
||||
"selector": "div.dcdlju span",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "primary_language",
|
||||
"selector": "ul.bZkODq li span[aria-label]",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "star_count",
|
||||
"selector": "ul.bZkODq li a[href*='stargazers'] span",
|
||||
"type": "text",
|
||||
"transform": "strip"
|
||||
},
|
||||
{
|
||||
"name": "topics",
|
||||
"type": "list",
|
||||
"selector": "div.jgRnBg div a",
|
||||
"fields": [
|
||||
{
|
||||
"name": "topic_name",
|
||||
"selector": "a",
|
||||
"type": "text"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "last_updated",
|
||||
"selector": "ul.bZkODq li span[title]",
|
||||
"type": "text"
|
||||
},
|
||||
{
|
||||
"name": "has_sponsor_button",
|
||||
"selector": "button[aria-label*='Sponsor']",
|
||||
"type": "text",
|
||||
"transform": "exists"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
(async () => {
|
||||
const waitForElement = (selector, timeout = 10000) => new Promise((resolve, reject) => {
|
||||
const el = document.querySelector(selector);
|
||||
if (el) return resolve(el);
|
||||
const observer = new MutationObserver(() => {
|
||||
const el = document.querySelector(selector);
|
||||
if (el) {
|
||||
observer.disconnect();
|
||||
resolve(el);
|
||||
}
|
||||
});
|
||||
observer.observe(document.body, { childList: true, subtree: true });
|
||||
setTimeout(() => {
|
||||
observer.disconnect();
|
||||
reject(new Error(`Timeout waiting for ${selector}`));
|
||||
}, timeout);
|
||||
});
|
||||
|
||||
try {
|
||||
const searchInput = await waitForElement('#adv_code_search input[type="text"]');
|
||||
searchInput.value = 'crawl4AI';
|
||||
searchInput.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
|
||||
const languageSelect = await waitForElement('#search_language');
|
||||
languageSelect.value = 'Python';
|
||||
languageSelect.dispatchEvent(new Event('change', { bubbles: true }));
|
||||
|
||||
const starsInput = await waitForElement('#search_stars');
|
||||
starsInput.value = '>10000';
|
||||
starsInput.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
|
||||
const searchButton = await waitForElement('#adv_code_search button[type="submit"]');
|
||||
searchButton.click();
|
||||
|
||||
await waitForElement('.codesearch-results, #search-results');
|
||||
} catch (e) {
|
||||
console.error('Search script failed:', e.message);
|
||||
}
|
||||
})();
|
||||
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
GitHub Advanced Search Example using Crawl4AI
|
||||
|
||||
This example demonstrates:
|
||||
1. Using LLM to generate C4A-Script from HTML snippets
|
||||
2. Single arun() call with navigation, search form filling, and extraction
|
||||
3. JSON CSS extraction for structured repository data
|
||||
4. Complete workflow: navigate → fill form → submit → extract results
|
||||
|
||||
Requirements:
|
||||
- Crawl4AI with generate_script support
|
||||
- LLM API key (configured in environment)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any
|
||||
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai.script.c4a_compile import C4ACompiler
|
||||
|
||||
|
||||
class GitHubSearchScraper:
|
||||
def __init__(self):
|
||||
self.base_dir = Path(__file__).parent
|
||||
self.search_script_path = self.base_dir / "generated_search_script.js"
|
||||
self.schema_path = self.base_dir / "generated_result_schema.json"
|
||||
self.results_path = self.base_dir / "extracted_repositories.json"
|
||||
self.session_id = "github_search_session"
|
||||
|
||||
async def generate_search_script(self) -> str:
|
||||
"""Generate JavaScript for GitHub advanced search interaction"""
|
||||
print("🔧 Generating search script from search_form.html...")
|
||||
|
||||
# Check if already generated
|
||||
if self.search_script_path.exists():
|
||||
print("✅ Using cached search script")
|
||||
return self.search_script_path.read_text()
|
||||
|
||||
# Read the search form HTML
|
||||
search_form_html = (self.base_dir / "search_form.html").read_text()
|
||||
|
||||
# Generate script using LLM
|
||||
search_goal = """
|
||||
Search for crawl4AI repositories written in Python with more than 10000 stars:
|
||||
1. Wait for the main search input to be visible
|
||||
2. Type "crawl4AI" into the main search box
|
||||
3. Select "Python" from the language dropdown (#search_language)
|
||||
4. Type ">10000" into the stars input field (#search_stars)
|
||||
5. Click the search button to submit the form
|
||||
6. Wait for the search results to appear
|
||||
"""
|
||||
|
||||
try:
|
||||
script = C4ACompiler.generate_script(
|
||||
html=search_form_html,
|
||||
query=search_goal,
|
||||
mode="js"
|
||||
)
|
||||
|
||||
# Save for future use
|
||||
self.search_script_path.write_text(script)
|
||||
print("✅ Search script generated and saved!")
|
||||
print(f"📄 Script preview:\n{script[:500]}...")
|
||||
return script
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error generating search script: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def generate_result_schema(self) -> Dict[str, Any]:
|
||||
"""Generate JSON CSS extraction schema from result HTML"""
|
||||
print("\n🔧 Generating result extraction schema...")
|
||||
|
||||
# Check if already generated
|
||||
if self.schema_path.exists():
|
||||
print("✅ Using cached extraction schema")
|
||||
return json.loads(self.schema_path.read_text())
|
||||
|
||||
# Read the result HTML
|
||||
result_html = (self.base_dir / "result.html").read_text()
|
||||
|
||||
# Generate extraction schema using LLM
|
||||
schema_goal = """
|
||||
Create a JSON CSS extraction schema to extract from each repository card:
|
||||
- Repository name (the repository name only, not including owner)
|
||||
- Repository owner (organization or username)
|
||||
- Repository URL (full GitHub URL)
|
||||
- Description
|
||||
- Primary programming language
|
||||
- Star count (numeric value)
|
||||
- Topics/tags (array of topic names)
|
||||
- Last updated (time ago string)
|
||||
- Whether it has a sponsor button
|
||||
|
||||
The schema should handle multiple repository results on the search results page.
|
||||
"""
|
||||
|
||||
try:
|
||||
# Generate schema
|
||||
schema = JsonCssExtractionStrategy.generate_schema(
|
||||
html=result_html,
|
||||
query=schema_goal,
|
||||
)
|
||||
|
||||
# Save for future use
|
||||
self.schema_path.write_text(json.dumps(schema, indent=2))
|
||||
print("✅ Extraction schema generated and saved!")
|
||||
print(f"📄 Schema fields: {[f['name'] for f in schema['fields']]}")
|
||||
return schema
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error generating schema: {e}")
|
||||
raise
|
||||
|
||||
async def crawl_github(self):
|
||||
"""Main crawling logic with single arun() call"""
|
||||
print("\n🚀 Starting GitHub repository search...")
|
||||
|
||||
# Generate scripts and schemas
|
||||
search_script = await self.generate_search_script()
|
||||
result_schema = await self.generate_result_schema()
|
||||
|
||||
# Configure browser (headless=False to see the action)
|
||||
browser_config = BrowserConfig(
|
||||
headless=False,
|
||||
verbose=True,
|
||||
viewport_width=1920,
|
||||
viewport_height=1080
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
print("\n📍 Navigating to GitHub advanced search and executing search...")
|
||||
|
||||
# Single call: Navigate, execute search, and extract results
|
||||
search_config = CrawlerRunConfig(
|
||||
session_id=self.session_id,
|
||||
js_code=search_script, # Execute generated JS
|
||||
# wait_for="[data-testid='results-list']", # Wait for search results
|
||||
wait_for=".Box-sc-g0xbh4-0.iwUbcA", # Wait for search results
|
||||
extraction_strategy=JsonCssExtractionStrategy(schema=result_schema),
|
||||
delay_before_return_html=3.0, # Give time for results to fully load
|
||||
cache_mode=CacheMode.BYPASS # Don't cache for fresh results
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
url="https://github.com/search/advanced",
|
||||
config=search_config
|
||||
)
|
||||
|
||||
if not result.success:
|
||||
print("❌ Failed to search GitHub")
|
||||
print(f"Error: {result.error_message}")
|
||||
return
|
||||
|
||||
print("✅ Search and extraction completed successfully!")
|
||||
|
||||
# Extract and save results
|
||||
if result.extracted_content:
|
||||
repositories = json.loads(result.extracted_content)
|
||||
print(f"\n🔍 Found {len(repositories)} repositories matching criteria")
|
||||
|
||||
# Save results
|
||||
self.results_path.write_text(
|
||||
json.dumps(repositories, indent=2)
|
||||
)
|
||||
print(f"💾 Results saved to: {self.results_path}")
|
||||
|
||||
# Print sample results
|
||||
print("\n📊 Sample Results:")
|
||||
for i, repo in enumerate(repositories[:5], 1):
|
||||
print(f"\n{i}. {repo.get('owner', 'Unknown')}/{repo.get('name', 'Unknown')}")
|
||||
print(f" Description: {repo.get('description', 'No description')[:80]}...")
|
||||
print(f" Language: {repo.get('language', 'Unknown')}")
|
||||
print(f" Stars: {repo.get('stars', 'Unknown')}")
|
||||
print(f" Updated: {repo.get('last_updated', 'Unknown')}")
|
||||
if repo.get('topics'):
|
||||
print(f" Topics: {', '.join(repo['topics'][:5])}")
|
||||
print(f" URL: {repo.get('url', 'Unknown')}")
|
||||
|
||||
else:
|
||||
print("❌ No repositories extracted")
|
||||
|
||||
# Save screenshot for reference
|
||||
if result.screenshot:
|
||||
screenshot_path = self.base_dir / "search_results_screenshot.png"
|
||||
with open(screenshot_path, "wb") as f:
|
||||
f.write(result.screenshot)
|
||||
print(f"\n📸 Screenshot saved to: {screenshot_path}")
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run the GitHub search scraper"""
|
||||
scraper = GitHubSearchScraper()
|
||||
await scraper.crawl_github()
|
||||
|
||||
print("\n🎉 GitHub search example completed!")
|
||||
print("Check the generated files:")
|
||||
print(" - generated_search_script.js")
|
||||
print(" - generated_result_schema.json")
|
||||
print(" - extracted_repositories.json")
|
||||
print(" - search_results_screenshot.png")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,54 @@
|
||||
<div class="Box-sc-g0xbh4-0 iwUbcA"><div class="Box-sc-g0xbh4-0 cSURfY"><div class="Box-sc-g0xbh4-0 gPrlij"><h3 class="Box-sc-g0xbh4-0 cvnppv"><div class="Box-sc-g0xbh4-0 kYLlPM"><div class="Box-sc-g0xbh4-0 eurdCD"><img data-component="Avatar" class="prc-Avatar-Avatar-ZRS-m" alt="" data-square="" width="20" height="20" src="https://github.com/TheAlgorithms.png?size=40" data-testid="github-avatar" style="--avatarSize-regular: 20px;"></div><div class="Box-sc-g0xbh4-0 MHoGG search-title"><a class="prc-Link-Link-85e08" href="/TheAlgorithms/Python"><span class="Box-sc-g0xbh4-0 kzfhBO search-match prc-Text-Text-0ima0">TheAlgorithms/<em>Python</em></span></a></div></div></h3><div class="Box-sc-g0xbh4-0 dcdlju"><span class="Box-sc-g0xbh4-0 gKFdvh search-match prc-Text-Text-0ima0">All Algorithms implemented in <em>Python</em></span></div><div class="Box-sc-g0xbh4-0 jgRnBg"><div><a class="Box-sc-g0xbh4-0 hIVEGR prc-Link-Link-85e08" href="/topics/python">python</a></div><div><a class="Box-sc-g0xbh4-0 hIVEGR prc-Link-Link-85e08" href="/topics/education">education</a></div><div><a class="Box-sc-g0xbh4-0 hIVEGR prc-Link-Link-85e08" href="/topics/algorithm">algorithm</a></div><div><a class="Box-sc-g0xbh4-0 hIVEGR prc-Link-Link-85e08" href="/topics/practice">practice</a></div><div><a class="Box-sc-g0xbh4-0 hIVEGR prc-Link-Link-85e08" href="/topics/interview">interview</a></div></div><ul class="Box-sc-g0xbh4-0 bZkODq"><li class="Box-sc-g0xbh4-0 eCfCAC"><div class="Box-sc-g0xbh4-0 hjDqIa"><div class="Box-sc-g0xbh4-0 fwSYsx"></div></div><span aria-label="Python language">Python</span></li><span class="Box-sc-g0xbh4-0 eXQoFa prc-Text-Text-0ima0" aria-hidden="true">·</span><li class="Box-sc-g0xbh4-0 eCfCAC"><a class="Box-sc-g0xbh4-0 iPuHRc prc-Link-Link-85e08" href="/TheAlgorithms/Python/stargazers" aria-label="201161 stars"><svg aria-hidden="true" focusable="false" class="octicon octicon-star Octicon-sc-9kayk9-0 kHVtWu" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M8 .25a.75.75 0 0 1 .673.418l1.882 3.815 4.21.612a.75.75 0 0 1 .416 1.279l-3.046 2.97.719 4.192a.751.751 0 0 1-1.088.791L8 12.347l-3.766 1.98a.75.75 0 0 1-1.088-.79l.72-4.194L.818 6.374a.75.75 0 0 1 .416-1.28l4.21-.611L7.327.668A.75.75 0 0 1 8 .25Zm0 2.445L6.615 5.5a.75.75 0 0 1-.564.41l-3.097.45 2.24 2.184a.75.75 0 0 1 .216.664l-.528 3.084 2.769-1.456a.75.75 0 0 1 .698 0l2.77 1.456-.53-3.084a.75.75 0 0 1 .216-.664l2.24-2.183-3.096-.45a.75.75 0 0 1-.564-.41L8 2.694Z"></path></svg><span class="prc-Text-Text-0ima0">201k</span></a></li><span class="Box-sc-g0xbh4-0 eXQoFa prc-Text-Text-0ima0" aria-hidden="true">·</span><li class="Box-sc-g0xbh4-0 eCfCAC"><span>Updated <div title="3 Jun 2025, 01:57 GMT+8" class="Truncate__StyledTruncate-sc-23o1d2-0 liVpTx"><span class="prc-Text-Text-0ima0" title="3 Jun 2025, 01:57 GMT+8">4 days ago</span></div></span></li></ul></div><div class="Box-sc-g0xbh4-0 gtlRHe"><div class="Box-sc-g0xbh4-0 fvaNTI"><button type="button" class="prc-Button-ButtonBase-c50BI" data-loading="false" data-size="small" data-variant="default" aria-describedby=":r1c:-loading-announcement"><span data-component="buttonContent" data-align="center" class="prc-Button-ButtonContent-HKbr-"><span data-component="leadingVisual" class="prc-Button-Visual-2epfX prc-Button-VisualWrap-Db-eB"><svg aria-hidden="true" focusable="false" class="octicon octicon-star" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"><path d="M8 .25a.75.75 0 0 1 .673.418l1.882 3.815 4.21.612a.75.75 0 0 1 .416 1.279l-3.046 2.97.719 4.192a.751.751 0 0 1-1.088.791L8 12.347l-3.766 1.98a.75.75 0 0 1-1.088-.79l.72-4.194L.818 6.374a.75.75 0 0 1 .416-1.28l4.21-.611L7.327.668A.75.75 0 0 1 8 .25Zm0 2.445L6.615 5.5a.75.75 0 0 1-.564.41l-3.097.45 2.24 2.184a.75.75 0 0 1 .216.664l-.528 3.084 2.769-1.456a.75.75 0 0 1 .698 0l2.77 1.456-.53-3.084a.75.75 0 0 1 .216-.664l2.24-2.183-3.096-.45a.75.75 0 0 1-.564-.41L8 2.694Z"></path></svg></span><span data-component="text" class="prc-Button-Label-pTQ3x">Star</span></span></button></div><div class="Box-sc-g0xbh4-0 llZEgI"><div class="Box-sc-g0xbh4-0"> <button id="dialog-show-funding-links-modal-TheAlgorithms-Python" aria-label="Sponsor TheAlgorithms/Python" data-show-dialog-id="funding-links-modal-TheAlgorithms-Python" type="button" data-view-component="true" class="Button--secondary Button--small Button"> <span class="Button-content">
|
||||
<span class="Button-label"><svg aria-hidden="true" height="16" viewBox="0 0 16 16" version="1.1" width="16" data-view-component="true" class="octicon octicon-heart icon-sponsor mr-1 color-fg-sponsors">
|
||||
<path d="m8 14.25.345.666a.75.75 0 0 1-.69 0l-.008-.004-.018-.01a7.152 7.152 0 0 1-.31-.17 22.055 22.055 0 0 1-3.434-2.414C2.045 10.731 0 8.35 0 5.5 0 2.836 2.086 1 4.25 1 5.797 1 7.153 1.802 8 3.02 8.847 1.802 10.203 1 11.75 1 13.914 1 16 2.836 16 5.5c0 2.85-2.045 5.231-3.885 6.818a22.066 22.066 0 0 1-3.744 2.584l-.018.01-.006.003h-.002ZM4.25 2.5c-1.336 0-2.75 1.164-2.75 3 0 2.15 1.58 4.144 3.365 5.682A20.58 20.58 0 0 0 8 13.393a20.58 20.58 0 0 0 3.135-2.211C12.92 9.644 14.5 7.65 14.5 5.5c0-1.836-1.414-3-2.75-3-1.373 0-2.609.986-3.029 2.456a.749.749 0 0 1-1.442 0C6.859 3.486 5.623 2.5 4.25 2.5Z"></path>
|
||||
</svg> <span data-view-component="true">Sponsor</span></span>
|
||||
</span>
|
||||
</button>
|
||||
|
||||
<dialog-helper>
|
||||
<dialog id="funding-links-modal-TheAlgorithms-Python" aria-modal="true" aria-labelledby="funding-links-modal-TheAlgorithms-Python-title" aria-describedby="funding-links-modal-TheAlgorithms-Python-description" data-view-component="true" class="Overlay Overlay-whenNarrow Overlay--size-medium Overlay--motion-scaleFade Overlay--disableScroll">
|
||||
<div data-view-component="true" class="Overlay-header">
|
||||
<div class="Overlay-headerContentWrap">
|
||||
<div class="Overlay-titleWrap">
|
||||
<h1 class="Overlay-title " id="funding-links-modal-TheAlgorithms-Python-title">
|
||||
Sponsor TheAlgorithms/Python
|
||||
</h1>
|
||||
|
||||
</div>
|
||||
<div class="Overlay-actionWrap">
|
||||
<button data-close-dialog-id="funding-links-modal-TheAlgorithms-Python" aria-label="Close" type="button" data-view-component="true" class="close-button Overlay-closeButton"><svg aria-hidden="true" height="16" viewBox="0 0 16 16" version="1.1" width="16" data-view-component="true" class="octicon octicon-x">
|
||||
<path d="M3.72 3.72a.75.75 0 0 1 1.06 0L8 6.94l3.22-3.22a.749.749 0 0 1 1.275.326.749.749 0 0 1-.215.734L9.06 8l3.22 3.22a.749.749 0 0 1-.326 1.275.749.749 0 0 1-.734-.215L8 9.06l-3.22 3.22a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042L6.94 8 3.72 4.78a.75.75 0 0 1 0-1.06Z"></path>
|
||||
</svg></button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<scrollable-region data-labelled-by="funding-links-modal-TheAlgorithms-Python-title" data-catalyst="" style="overflow: auto;">
|
||||
<div data-view-component="true" class="Overlay-body"> <div class="text-left f5">
|
||||
<div class="pt-3 color-bg-overlay">
|
||||
<h5 class="flex-auto mb-3 mt-0">External links</h5>
|
||||
<div class="d-flex mb-3">
|
||||
<div class="circle mr-2 border d-flex flex-justify-center flex-items-center flex-shrink-0" style="width:24px;height:24px;">
|
||||
<img width="16" height="16" class="octicon rounded-2 d-block" alt="liberapay" src="https://github.githubassets.com/assets/liberapay-48108ded7267.svg">
|
||||
</div>
|
||||
<div class="flex-auto min-width-0">
|
||||
<a target="_blank" data-ga-click="Dashboard, click, Nav menu - item:org-profile context:organization" data-hydro-click="{"event_type":"sponsors.repo_funding_links_link_click","payload":{"platform":{"platform_type":"LIBERAPAY","platform_url":"https://liberapay.com/TheAlgorithms"},"platforms":[{"platform_type":"LIBERAPAY","platform_url":"https://liberapay.com/TheAlgorithms"}],"repo_id":63476337,"owner_id":20487725,"user_id":12494079,"originating_url":"https://github.com/TheAlgorithms/Python/funding_links?fragment=1"}}" data-hydro-click-hmac="123b5aa7d5ffff5ef0530f8e7fbaebcb564e8de1af26f1b858a19b0e1d4f9e5f" href="https://liberapay.com/TheAlgorithms"><span>liberapay.com/<strong>TheAlgorithms</strong></span></a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="text-small p-3 border-top">
|
||||
<p class="my-0">
|
||||
<a class="Link--inTextBlock" href="https://docs.github.com/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository">Learn more about funding links in repositories</a>.
|
||||
</p>
|
||||
<p class="my-0">
|
||||
<a class="Link--secondary" href="/contact/report-abuse?report=TheAlgorithms%2FPython+%28Repository+Funding+Links%29">Report abuse</a>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</scrollable-region>
|
||||
|
||||
</dialog></dialog-helper>
|
||||
</div></div></div></div></div>
|
||||
@@ -0,0 +1,336 @@
|
||||
<form id="search_form" class="search_repos" data-turbo="false" action="/search" accept-charset="UTF-8" method="get">
|
||||
|
||||
<div class="pagehead codesearch-head color-border-muted">
|
||||
<div class="container-lg p-responsive d-flex flex-column flex-md-row">
|
||||
<h1 class="flex-shrink-0" id="search-title">Advanced search</h1>
|
||||
<div class="search-form-fluid flex-auto d-flex flex-column flex-md-row pt-2 pt-md-0" id="adv_code_search">
|
||||
<div class="flex-auto pr-md-2">
|
||||
<label class="form-control search-page-label js-advanced-search-label">
|
||||
<input aria-labelledby="search-title" class="form-control input-block search-page-input js-advanced-search-input js-advanced-search-prefix" data-search-prefix="" type="text" value="">
|
||||
<p class="completed-query js-advanced-query top-0 right-0 left-0"><span></span> </p>
|
||||
</label>
|
||||
<input class="js-search-query" type="hidden" name="q" value="">
|
||||
<input class="js-type-value" type="hidden" name="type" value="Repositories">
|
||||
<input type="hidden" name="ref" value="advsearch">
|
||||
</div>
|
||||
<div class="d-flex d-md-block flex-shrink-0 pt-2 pt-md-0">
|
||||
<button type="submit" data-view-component="true" class="btn flex-auto"> Search
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="container-lg p-responsive advanced-search-form">
|
||||
<fieldset class="pb-3 mb-4 border-bottom color-border-muted min-width-0">
|
||||
<h3>Advanced options</h3>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_from">From these owners</label></dt>
|
||||
<dd><input id="search_from" type="text" class="form-control js-advanced-search-prefix" placeholder="github, atom, electron, octokit" data-search-prefix="user:"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_repos">In these repositories</label></dt>
|
||||
<dd><input id="search_repos" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="twbs/bootstrap, rails/rails" data-search-prefix="repo:"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_date">Created on the dates</label></dt>
|
||||
<dd><input id="search_date" type="text" class="form-control js-advanced-search-prefix" value="" placeholder=">YYYY-MM-DD, YYYY-MM-DD" data-search-prefix="created:"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_language">Written in this language</label></dt>
|
||||
<dd>
|
||||
<select id="search_language" name="l" class="form-select js-advanced-search-prefix" data-search-prefix="language:">
|
||||
<option value="">Any language</option>
|
||||
<optgroup label="Popular">
|
||||
<option value="C">C</option>
|
||||
<option value="C#">C#</option>
|
||||
<option value="C++">C++</option>
|
||||
<option value="CoffeeScript">CoffeeScript</option>
|
||||
<option value="CSS">CSS</option>
|
||||
<option value="Dart">Dart</option>
|
||||
<option value="DM">DM</option>
|
||||
<option value="Elixir">Elixir</option>
|
||||
<option value="Go">Go</option>
|
||||
<option value="Groovy">Groovy</option>
|
||||
<option value="HTML">HTML</option>
|
||||
<option value="Java">Java</option>
|
||||
<option value="JavaScript">JavaScript</option>
|
||||
<option value="Kotlin">Kotlin</option>
|
||||
<option value="Objective-C">Objective-C</option>
|
||||
<option value="Perl">Perl</option>
|
||||
<option value="PHP">PHP</option>
|
||||
<option value="PowerShell">PowerShell</option>
|
||||
<option value="Python">Python</option>
|
||||
<option value="Ruby">Ruby</option>
|
||||
<option value="Rust">Rust</option>
|
||||
<option value="Scala">Scala</option>
|
||||
<option value="Shell">Shell</option>
|
||||
<option value="Swift">Swift</option>
|
||||
<option value="TypeScript">TypeScript</option>
|
||||
</optgroup>
|
||||
<optgroup label="Everything else">
|
||||
<option value="1C Enterprise">1C Enterprise</option>
|
||||
<option value="2-Dimensional Array">2-Dimensional Array</option>
|
||||
<option value="4D">4D</option>
|
||||
<option value="ABAP">ABAP</option>
|
||||
<option value="ABAP CDS">ABAP CDS</option>
|
||||
<option value="ABNF">ABNF</option>
|
||||
<option value="ActionScript">ActionScript</option>
|
||||
<option value="Ada">Ada</option>
|
||||
<option value="Adblock Filter List">Adblock Filter List</option>
|
||||
<option value="Adobe Font Metrics">Adobe Font Metrics</option>
|
||||
<option value="Agda">Agda</option>
|
||||
<option value="AGS Script">AGS Script</option>
|
||||
<option value="AIDL">AIDL</option>
|
||||
<option value="Aiken">Aiken</option>
|
||||
</optgroup>
|
||||
</select>
|
||||
</dd>
|
||||
</dl>
|
||||
</fieldset>
|
||||
<fieldset class="pb-3 mb-4 border-bottom color-border-muted min-width-0">
|
||||
<h3>Repositories options</h3>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_stars">With this many stars</label></dt>
|
||||
<dd><input id="search_stars" type="text" class="form-control js-advanced-search-prefix" placeholder="0..100, 200, >1000" data-search-prefix="stars:" data-search-type="Repositories"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_forks">With this many forks</label></dt>
|
||||
<dd><input id="search_forks" type="text" class="form-control js-advanced-search-prefix" placeholder="50..100, 200, <5" data-search-prefix="forks:" data-search-type="Repositories"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_size">Of this size</label></dt>
|
||||
<dd><input id="search_size" type="text" class="form-control js-advanced-search-prefix" placeholder="Repository size in KB" data-search-prefix="size:" data-search-type="Repositories"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_push">Pushed to</label></dt>
|
||||
<dd><input id="search_push" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="<YYYY-MM-DD" data-search-prefix="pushed:" data-search-type="Repositories"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_license">With this license</label></dt>
|
||||
<dd>
|
||||
<select id="search_license" class="form-select js-advanced-search-prefix" data-search-prefix="license:" data-search-type="Repositories">
|
||||
<option value="">Any license</option>
|
||||
<optgroup label="Licenses">
|
||||
<option value="0bsd">BSD Zero Clause License</option>
|
||||
<option value="afl-3.0">Academic Free License v3.0</option>
|
||||
<option value="agpl-3.0">GNU Affero General Public License v3.0</option>
|
||||
<option value="apache-2.0">Apache License 2.0</option>
|
||||
<option value="artistic-2.0">Artistic License 2.0</option>
|
||||
<option value="blueoak-1.0.0">Blue Oak Model License 1.0.0</option>
|
||||
<option value="bsd-2-clause">BSD 2-Clause "Simplified" License</option>
|
||||
<option value="bsd-2-clause-patent">BSD-2-Clause Plus Patent License</option>
|
||||
<option value="bsd-3-clause">BSD 3-Clause "New" or "Revised" License</option>
|
||||
<option value="bsd-3-clause-clear">BSD 3-Clause Clear License</option>
|
||||
<option value="bsd-4-clause">BSD 4-Clause "Original" or "Old" License</option>
|
||||
<option value="bsl-1.0">Boost Software License 1.0</option>
|
||||
<option value="cc-by-4.0">Creative Commons Attribution 4.0 International</option>
|
||||
<option value="cc-by-sa-4.0">Creative Commons Attribution Share Alike 4.0 International</option>
|
||||
<option value="cc0-1.0">Creative Commons Zero v1.0 Universal</option>
|
||||
<option value="cecill-2.1">CeCILL Free Software License Agreement v2.1</option>
|
||||
<option value="cern-ohl-p-2.0">CERN Open Hardware Licence Version 2 - Permissive</option>
|
||||
<option value="cern-ohl-s-2.0">CERN Open Hardware Licence Version 2 - Strongly Reciprocal</option>
|
||||
<option value="cern-ohl-w-2.0">CERN Open Hardware Licence Version 2 - Weakly Reciprocal</option>
|
||||
<option value="ecl-2.0">Educational Community License v2.0</option>
|
||||
<option value="epl-1.0">Eclipse Public License 1.0</option>
|
||||
<option value="epl-2.0">Eclipse Public License 2.0</option>
|
||||
<option value="eupl-1.1">European Union Public License 1.1</option>
|
||||
<option value="eupl-1.2">European Union Public License 1.2</option>
|
||||
<option value="gfdl-1.3">GNU Free Documentation License v1.3</option>
|
||||
<option value="gpl-2.0">GNU General Public License v2.0</option>
|
||||
<option value="gpl-3.0">GNU General Public License v3.0</option>
|
||||
<option value="isc">ISC License</option>
|
||||
<option value="lgpl-2.1">GNU Lesser General Public License v2.1</option>
|
||||
<option value="lgpl-3.0">GNU Lesser General Public License v3.0</option>
|
||||
<option value="lppl-1.3c">LaTeX Project Public License v1.3c</option>
|
||||
<option value="mit">MIT License</option>
|
||||
<option value="mit-0">MIT No Attribution</option>
|
||||
<option value="mpl-2.0">Mozilla Public License 2.0</option>
|
||||
<option value="ms-pl">Microsoft Public License</option>
|
||||
<option value="ms-rl">Microsoft Reciprocal License</option>
|
||||
<option value="mulanpsl-2.0">Mulan Permissive Software License, Version 2</option>
|
||||
<option value="ncsa">University of Illinois/NCSA Open Source License</option>
|
||||
<option value="odbl-1.0">Open Data Commons Open Database License v1.0</option>
|
||||
<option value="ofl-1.1">SIL Open Font License 1.1</option>
|
||||
<option value="osl-3.0">Open Software License 3.0</option>
|
||||
<option value="postgresql">PostgreSQL License</option>
|
||||
<option value="unlicense">The Unlicense</option>
|
||||
<option value="upl-1.0">Universal Permissive License v1.0</option>
|
||||
<option value="vim">Vim License</option>
|
||||
<option value="wtfpl">Do What The F*ck You Want To Public License</option>
|
||||
<option value="zlib">zlib License</option>
|
||||
</optgroup>
|
||||
<optgroup label="License families">
|
||||
<option value="cc">Creative Commons</option>
|
||||
<option value="gpl">GNU General Public License</option>
|
||||
<option value="lgpl">GNU Lesser General Public License</option>
|
||||
</optgroup>
|
||||
</select>
|
||||
</dd>
|
||||
</dl>
|
||||
<label>
|
||||
Return repositories <select class="form-select js-advanced-search-prefix" data-search-prefix="fork:" data-search-type="Repositories">
|
||||
<option value="">not</option>
|
||||
<option value="true">and</option>
|
||||
<option value="only">only</option>
|
||||
</select> including forks.
|
||||
</label>
|
||||
</fieldset>
|
||||
<fieldset class="pb-3 mb-4 border-bottom color-border-muted min-width-0">
|
||||
<h3>Code options</h3>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_extension">With this extension</label></dt>
|
||||
<dd>
|
||||
<input id="search_extension" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="rb, py, jpg" data-search-type="Code" data-search-prefix="path:" data-glob-pattern="*.$0" data-regex-pattern="/.$0$/" data-use-or="true">
|
||||
</dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_path">In this path</label></dt>
|
||||
<dd><input id="search_path" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="/foo/bar/baz/qux" data-search-prefix="path:" data-search-type="Code" data-use-or=""></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_filename">With this file name</label></dt>
|
||||
<dd>
|
||||
<input id="search_filename" type="text" class="form-control js-advanced-search-prefix" placeholder="app.rb, footer.erb" data-search-type="code:" data-search-prefix="path:" data-glob-pattern="**/$0" data-regex-pattern="/(^|/)$0$/" data-use-or="true">
|
||||
</dd>
|
||||
</dl>
|
||||
<label>
|
||||
Return code <select class="form-select js-advanced-search-prefix" data-search-prefix="fork:" data-search-type="Code">
|
||||
<option value="">not</option>
|
||||
<option value="true">and</option>
|
||||
<option value="only">only</option>
|
||||
</select> including forks.
|
||||
</label>
|
||||
</fieldset>
|
||||
<fieldset class="pb-3 mb-4 border-bottom color-border-muted min-width-0">
|
||||
<h3>Issues options</h3>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_state">In the state</label></dt>
|
||||
<dd><select id="search_state" class="form-select js-advanced-search-prefix" data-search-prefix="state:" data-search-type="Issues">
|
||||
<option value="">open/closed</option>
|
||||
<option value="open">open</option>
|
||||
<option value="closed">closed</option>
|
||||
</select></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_state_reason">With the reason</label></dt>
|
||||
<dd><select id="search_state_reason" class="form-select js-advanced-search-prefix" data-search-prefix="reason:" data-search-type="Issues">
|
||||
<option value="">any reason</option>
|
||||
<option value="completed">completed</option>
|
||||
<option value="not planned">not planned</option>
|
||||
<option value="reopened">reopened</option>
|
||||
</select></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_comments">With this many comments</label></dt>
|
||||
<dd><input id="search_comments" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="0..100, >442" data-search-prefix="comments:" data-search-type="Issues"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_labels">With the labels</label></dt>
|
||||
<dd><input id="search_labels" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="bug, ie6" data-search-prefix="label:" data-search-type="Issues"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_author">Opened by the author</label></dt>
|
||||
<dd><input id="search_author" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="hubot, octocat" data-search-prefix="author:" data-search-type="Issues"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_mention">Mentioning the users</label></dt>
|
||||
<dd><input id="search_mention" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="tpope, mattt" data-search-prefix="mentions:" data-search-type="Issues"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_assignment">Assigned to the users</label></dt>
|
||||
<dd><input id="search_assignment" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="twp, jim" data-search-prefix="assignee:" data-search-type="Issues"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_updated_date">Updated before the date</label></dt>
|
||||
<dd><input id="search_updated_date" type="text" class="form-control js-advanced-search-prefix" value="" placeholder="<YYYY-MM-DD" data-search-prefix="updated:" data-search-type="Issues"></dd>
|
||||
</dl>
|
||||
</fieldset>
|
||||
<fieldset class="pb-3 mb-4 border-bottom color-border-muted min-width-0">
|
||||
<h3>Users options</h3>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_full_name">With this full name</label></dt>
|
||||
<dd><input id="search_full_name" type="text" class="form-control js-advanced-search-prefix" placeholder="Grace Hopper" data-search-prefix="fullname:" data-search-type="Users"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_location">From this location</label></dt>
|
||||
<dd><input id="search_location" type="text" class="form-control js-advanced-search-prefix" placeholder="San Francisco, CA" data-search-prefix="location:" data-search-type="Users"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_followers">With this many followers</label></dt>
|
||||
<dd><input id="search_followers" type="text" class="form-control js-advanced-search-prefix" placeholder="20..50, >200, <2" data-search-prefix="followers:" data-search-type="Users"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_public_repos">With this many public repositories</label></dt>
|
||||
<dd><input id="search_public_repos" type="text" class="form-control js-advanced-search-prefix" placeholder="0, <42, >5" data-search-prefix="repos:" data-search-type="Users"></dd>
|
||||
</dl>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_user_language">Working in this language</label></dt>
|
||||
<dd>
|
||||
<select id="search_user_language" name="l" class="form-select js-advanced-search-prefix" data-search-prefix="language:">
|
||||
<option value="">Any language</option>
|
||||
<optgroup label="Popular">
|
||||
<option value="C">C</option>
|
||||
<option value="C#">C#</option>
|
||||
<option value="C++">C++</option>
|
||||
<option value="CoffeeScript">CoffeeScript</option>
|
||||
<option value="CSS">CSS</option>
|
||||
<option value="Dart">Dart</option>
|
||||
<option value="DM">DM</option>
|
||||
<option value="Elixir">Elixir</option>
|
||||
<option value="Go">Go</option>
|
||||
<option value="Groovy">Groovy</option>
|
||||
<option value="HTML">HTML</option>
|
||||
<option value="Java">Java</option>
|
||||
<option value="JavaScript">JavaScript</option>
|
||||
<option value="Kotlin">Kotlin</option>
|
||||
<option value="Objective-C">Objective-C</option>
|
||||
<option value="Perl">Perl</option>
|
||||
<option value="PHP">PHP</option>
|
||||
<option value="PowerShell">PowerShell</option>
|
||||
<option value="Python">Python</option>
|
||||
<option value="Ruby">Ruby</option>
|
||||
<option value="Rust">Rust</option>
|
||||
<option value="Scala">Scala</option>
|
||||
<option value="Shell">Shell</option>
|
||||
<option value="Swift">Swift</option>
|
||||
<option value="TypeScript">TypeScript</option>
|
||||
</optgroup>
|
||||
<optgroup label="Everything else">
|
||||
<option value="1C Enterprise">1C Enterprise</option>
|
||||
<option value="2-Dimensional Array">2-Dimensional Array</option>
|
||||
<option value="4D">4D</option>
|
||||
<option value="ABAP">ABAP</option>
|
||||
<option value="ABAP CDS">ABAP CDS</option>
|
||||
<option value="ABNF">ABNF</option>
|
||||
<option value="ActionScript">ActionScript</option>
|
||||
<option value="Ada">Ada</option>
|
||||
|
||||
<option value="Yul">Yul</option>
|
||||
<option value="ZAP">ZAP</option>
|
||||
<option value="Zeek">Zeek</option>
|
||||
<option value="ZenScript">ZenScript</option>
|
||||
<option value="Zephir">Zephir</option>
|
||||
<option value="Zig">Zig</option>
|
||||
<option value="ZIL">ZIL</option>
|
||||
<option value="Zimpl">Zimpl</option>
|
||||
<option value="Zmodel">Zmodel</option>
|
||||
</optgroup>
|
||||
</select>
|
||||
</dd>
|
||||
</dl>
|
||||
</fieldset>
|
||||
<fieldset class="pb-3 mb-4 border-bottom color-border-muted min-width-0">
|
||||
<h3>Wiki options</h3>
|
||||
<dl class="form-group flattened d-flex d-md-block flex-column">
|
||||
<dt><label for="search_wiki_updated_date">Updated before the date</label></dt>
|
||||
<dd><input id="search_wiki_updated_date" type="text" class="form-control js-advanced-search-prefix" placeholder="<YYYY-MM-DD" data-search-prefix="updated:" data-search-type="Wiki"></dd>
|
||||
</dl>
|
||||
</fieldset>
|
||||
<div class="form-group flattened">
|
||||
<div class="d-flex d-md-block"> <button type="submit" data-view-component="true" class="btn flex-auto"> Search
|
||||
</button></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</form>
|
||||
@@ -0,0 +1,7 @@
|
||||
GO https://store.example.com/product/laptop
|
||||
WAIT `.product-details` 8
|
||||
CLICK `button.add-to-cart`
|
||||
WAIT `.cart-notification` 3
|
||||
CLICK `.cart-icon`
|
||||
WAIT `.checkout-btn` 5
|
||||
CLICK `.checkout-btn`
|
||||
@@ -0,0 +1,43 @@
|
||||
# Advanced control flow with IF, EXISTS, and REPEAT
|
||||
|
||||
# Define reusable procedures
|
||||
PROC handle_cookie_banner
|
||||
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept-cookies`
|
||||
IF (EXISTS `.privacy-notice`) THEN CLICK `.dismiss-privacy`
|
||||
ENDPROC
|
||||
|
||||
PROC scroll_to_load
|
||||
SCROLL DOWN 500
|
||||
WAIT 0.5
|
||||
ENDPROC
|
||||
|
||||
PROC try_login
|
||||
CLICK `#email`
|
||||
TYPE "user@example.com"
|
||||
CLICK `#password`
|
||||
TYPE "secure123"
|
||||
CLICK `button[type="submit"]`
|
||||
WAIT 2
|
||||
ENDPROC
|
||||
|
||||
# Main script
|
||||
GO https://example.com
|
||||
WAIT 2
|
||||
|
||||
# Handle popups
|
||||
handle_cookie_banner
|
||||
|
||||
# Conditional navigation based on login state
|
||||
IF (EXISTS `.user-menu`) THEN CLICK `.dashboard-link` ELSE try_login
|
||||
|
||||
# Repeat scrolling based on content count
|
||||
REPEAT (scroll_to_load, 5)
|
||||
|
||||
# Load more content while button exists
|
||||
REPEAT (CLICK `.load-more`, `document.querySelector('.load-more') && !document.querySelector('.no-more-content')`)
|
||||
|
||||
# Process items conditionally
|
||||
IF (`document.querySelectorAll('.item').length > 10`) THEN EVAL `console.log('Found ' + document.querySelectorAll('.item').length + ' items')`
|
||||
|
||||
# Complex condition with viewport check
|
||||
IF (`window.innerWidth < 768 && document.querySelector('.mobile-menu')`) THEN CLICK `.mobile-menu-toggle`
|
||||
@@ -0,0 +1,8 @@
|
||||
GO https://myapp.com
|
||||
WAIT 2
|
||||
IF (EXISTS `.user-avatar`) THEN CLICK `.logout` ELSE CLICK `.login`
|
||||
WAIT `#auth-form` 5
|
||||
IF (EXISTS `#auth-form`) THEN TYPE "user@example.com"
|
||||
IF (EXISTS `#auth-form`) THEN PRESS Tab
|
||||
IF (EXISTS `#auth-form`) THEN TYPE "password123"
|
||||
IF (EXISTS `#auth-form`) THEN CLICK `button[type="submit"]`
|
||||
@@ -0,0 +1,56 @@
|
||||
# Data extraction example
|
||||
# Scrapes product information from an e-commerce site
|
||||
|
||||
# Navigate to products page
|
||||
GO https://shop.example.com/products
|
||||
WAIT `.product-list` 10
|
||||
|
||||
# Scroll to load lazy-loaded content
|
||||
SCROLL DOWN 500
|
||||
WAIT 1
|
||||
SCROLL DOWN 500
|
||||
WAIT 1
|
||||
SCROLL DOWN 500
|
||||
WAIT 2
|
||||
|
||||
# Extract product data
|
||||
EVAL `
|
||||
// Extract all product information
|
||||
const products = Array.from(document.querySelectorAll('.product-card')).map((card, index) => {
|
||||
return {
|
||||
id: index + 1,
|
||||
name: card.querySelector('.product-title')?.textContent?.trim() || 'N/A',
|
||||
price: card.querySelector('.price')?.textContent?.trim() || 'N/A',
|
||||
rating: card.querySelector('.rating')?.textContent?.trim() || 'N/A',
|
||||
availability: card.querySelector('.in-stock') ? 'In Stock' : 'Out of Stock',
|
||||
image: card.querySelector('img')?.src || 'N/A'
|
||||
};
|
||||
});
|
||||
|
||||
// Log results
|
||||
console.log('=== Product Extraction Results ===');
|
||||
console.log('Total products found:', products.length);
|
||||
console.log(JSON.stringify(products, null, 2));
|
||||
|
||||
// Save to localStorage for retrieval
|
||||
localStorage.setItem('scraped_products', JSON.stringify(products));
|
||||
`
|
||||
|
||||
# Optional: Click on first product for details
|
||||
CLICK `.product-card:first-child`
|
||||
WAIT `.product-details` 5
|
||||
|
||||
# Extract detailed information
|
||||
EVAL `
|
||||
const details = {
|
||||
description: document.querySelector('.product-description')?.textContent?.trim(),
|
||||
specifications: Array.from(document.querySelectorAll('.spec-item')).map(spec => ({
|
||||
label: spec.querySelector('.spec-label')?.textContent,
|
||||
value: spec.querySelector('.spec-value')?.textContent
|
||||
})),
|
||||
reviews: document.querySelector('.review-count')?.textContent
|
||||
};
|
||||
|
||||
console.log('=== Product Details ===');
|
||||
console.log(JSON.stringify(details, null, 2));
|
||||
`
|
||||
@@ -0,0 +1,8 @@
|
||||
GO https://company.com/contact
|
||||
WAIT `form#contact` 10
|
||||
TYPE "John Smith"
|
||||
PRESS Tab
|
||||
TYPE "john@email.com"
|
||||
PRESS Tab
|
||||
TYPE "Need help with my order"
|
||||
CLICK `button[type="submit"]`
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user