wehub-resource-sync
c889a57b6b
Test Suites / Build CI Environment (push) Has been cancelled
Test Suites / Basic Tests (push) Has been cancelled
Test Suites / End-to-End Tests (push) Has been cancelled
Test Suites / CLI Tests (push) Has been cancelled
Test Suites / Slow End-to-End Tests (push) Has been cancelled
Test Suites / Graph Database Tests (push) Has been cancelled
Test Suites / Vector DB Tests (push) Has been cancelled
Test Suites / Temporal Graph Test (push) Has been cancelled
Test Suites / Search Test on Different DBs (push) Has been cancelled
Test Suites / Example Tests (push) Has been cancelled
Test Suites / Notebook Tests (push) Has been cancelled
Test Suites / OS and Python Tests Ubuntu (push) Has been cancelled
Test Suites / OS and Python Tests Extended (push) Has been cancelled
Test Suites / LLM Test Suite (push) Has been cancelled
Test Suites / S3 File Storage Test (push) Has been cancelled
Test Suites / Run Integration Tests (push) Has been cancelled
Test Suites / MCP Tests (push) Has been cancelled
Test Suites / Docker Compose Test (push) Has been cancelled
Test Suites / Docker CI test (push) Has been cancelled
Test Suites / Relational DB Migration Tests (push) Has been cancelled
Test Suites / Distributed Cognee Test (push) Has been cancelled
Test Suites / DB Examples Tests (push) Has been cancelled
Test Suites / Test Completion Status (push) Has been cancelled
Test Suites / Claude Code Review (push) Has been cancelled
Test Suites / basic checks (push) Has been cancelled
build | Build and Push Cognee MCP Docker Image to dockerhub / docker-build-and-push (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
build | Build and Push Docker Image to dockerhub / docker-build-and-push (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges Core Functionality (3.11) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges Core Functionality (3.12) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges with Different Graph Databases (kuzu, kuzu) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges with Different Graph Databases (neo4j, neo4j) (push) Has been cancelled
Weighted Edges Tests / Test Weighted Edges Examples (push) Has been cancelled
Weighted Edges Tests / Code Quality for Weighted Edges (push) Has been cancelled
35 lines
1.1 KiB
Python
35 lines
1.1 KiB
Python
"""Web scraping module for cognee.
|
|
|
|
This module provides tools for scraping web content, managing scraping jobs, and storing
|
|
data in a graph database. It includes classes and functions for crawling web pages using
|
|
BeautifulSoup or Tavily, defining data models, and handling scraping configurations.
|
|
"""
|
|
|
|
from .utils import fetch_page_content
|
|
from .default_url_crawler import DefaultUrlCrawler
|
|
|
|
# Lazy import for web_scraper_task to avoid requiring apscheduler
|
|
# Import these directly if needed: from cognee.tasks.web_scraper.web_scraper_task import ...
|
|
|
|
|
|
def __getattr__(name):
|
|
"""Lazy load web scraper task functions that require apscheduler."""
|
|
if name == "cron_web_scraper_task":
|
|
from .web_scraper_task import cron_web_scraper_task
|
|
|
|
return cron_web_scraper_task
|
|
elif name == "web_scraper_task":
|
|
from .web_scraper_task import web_scraper_task
|
|
|
|
return web_scraper_task
|
|
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
|
|
|
|
|
__all__ = [
|
|
"BeautifulSoupCrawler",
|
|
"fetch_page_content",
|
|
"cron_web_scraper_task",
|
|
"web_scraper_task",
|
|
"DefaultUrlCrawler",
|
|
]
|