wehub-resource-sync
c889a57b6b
Test Suites / Build CI Environment (push) Has been cancelled
Test Suites / Basic Tests (push) Has been cancelled
Test Suites / End-to-End Tests (push) Has been cancelled
Test Suites / CLI Tests (push) Has been cancelled
Test Suites / Slow End-to-End Tests (push) Has been cancelled
Test Suites / Graph Database Tests (push) Has been cancelled
Test Suites / Vector DB Tests (push) Has been cancelled
Test Suites / Temporal Graph Test (push) Has been cancelled
Test Suites / Search Test on Different DBs (push) Has been cancelled
Test Suites / Example Tests (push) Has been cancelled
Test Suites / Notebook Tests (push) Has been cancelled
Test Suites / OS and Python Tests Ubuntu (push) Has been cancelled
Test Suites / OS and Python Tests Extended (push) Has been cancelled
Test Suites / LLM Test Suite (push) Has been cancelled
Test Suites / S3 File Storage Test (push) Has been cancelled
Test Suites / Run Integration Tests (push) Has been cancelled
Test Suites / MCP Tests (push) Has been cancelled
Test Suites / Docker Compose Test (push) Has been cancelled
Test Suites / Docker CI test (push) Has been cancelled
Test Suites / Relational DB Migration Tests (push) Has been cancelled
Test Suites / Distributed Cognee Test (push) Has been cancelled
Test Suites / DB Examples Tests (push) Has been cancelled
Test Suites / Test Completion Status (push) Has been cancelled
Test Suites / Claude Code Review (push) Has been cancelled
Test Suites / basic checks (push) Has been cancelled
build | Build and Push Cognee MCP Docker Image to dockerhub / docker-build-and-push (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
build | Build and Push Docker Image to dockerhub / docker-build-and-push (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges Core Functionality (3.11) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges Core Functionality (3.12) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges with Different Graph Databases (kuzu, kuzu) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges with Different Graph Databases (neo4j, neo4j) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges Examples (push) Has been cancelled
Weighted Edges Tests / Code Quality for Weighted Edges (push) Has been cancelled
27 lines
886 B
Python
27 lines
886 B
Python
from pydantic import BaseModel, Field
|
|
from typing import Any, Dict, Optional, Literal
|
|
import os
|
|
|
|
|
|
class TavilyConfig(BaseModel):
|
|
api_key: Optional[str] = os.getenv("TAVILY_API_KEY")
|
|
extract_depth: Literal["basic", "advanced"] = "basic"
|
|
proxies: Optional[Dict[str, str]] = None
|
|
timeout: Optional[int] = Field(default=10, ge=1, le=60)
|
|
|
|
|
|
class DefaultCrawlerConfig(BaseModel):
|
|
concurrency: int = 5
|
|
crawl_delay: float = 0.5
|
|
max_crawl_delay: Optional[float] = (
|
|
10.0 # Maximum crawl delay to respect from robots.txt (None = no limit)
|
|
)
|
|
timeout: float = float(os.getenv("WEB_SCRAPER_TIMEOUT", 15.0))
|
|
max_retries: int = 2
|
|
retry_delay_factor: float = 0.5
|
|
headers: Optional[Dict[str, str]] = None
|
|
use_playwright: bool = False
|
|
playwright_js_wait: float = 0.8
|
|
robots_cache_ttl: float = 3600.0
|
|
join_all_matches: bool = False
|