wehub-resource-sync
2cab53bc94
Test Vector Database Adaptors / Test MCP Vector DB Tools (push) Has been cancelled
Tests / Code Quality (Ruff & Mypy) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (macos-latest, 3.11) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (macos-latest, 3.12) (push) Has been cancelled
Tests / Tests (push) Has been cancelled
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers CLI - Convert documentation to AI skills dockerfile:Dockerfile name:skill-seekers]) (push) Has been cancelled
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers MCP Server - 25 tools for AI assistants dockerfile:Dockerfile.mcp name:skill-seekers-mcp]) (push) Has been cancelled
Docker Publish / Test Docker Images (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.10) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.11) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.12) (push) Has been cancelled
Tests / Serial / Integration / E2E Tests (push) Has been cancelled
Tests / MCP Server Tests (push) Has been cancelled
Test Vector Database Adaptors / Test chroma Adaptor (push) Has been cancelled
Test Vector Database Adaptors / Test faiss Adaptor (push) Has been cancelled
Test Vector Database Adaptors / Test qdrant Adaptor (push) Has been cancelled
Test Vector Database Adaptors / Test weaviate Adaptor (push) Has been cancelled
39 lines
1.3 KiB
Python
39 lines
1.3 KiB
Python
#!/usr/bin/env python3
|
|
"""The pre-crawl sitemap probe must fail fast on unreachable hosts.
|
|
|
|
`_try_sitemap` runs up to two blocking probes before the crawl starts. With a
|
|
single scalar timeout an unreachable host blocked the full window each time,
|
|
making `create <url>` look hung before any output. A (connect, read) timeout
|
|
bounds the connect phase tightly while still allowing a slow real sitemap to
|
|
download.
|
|
"""
|
|
|
|
from skill_seekers.cli import doc_scraper
|
|
|
|
|
|
def test_sitemap_probe_uses_connect_read_timeout(monkeypatch):
|
|
seen_timeouts = []
|
|
|
|
class _Resp:
|
|
status_code = 404
|
|
headers = {"content-type": "text/html"}
|
|
text = ""
|
|
|
|
def fake_get(_url, **kwargs):
|
|
seen_timeouts.append(kwargs.get("timeout"))
|
|
return _Resp()
|
|
|
|
monkeypatch.setattr(doc_scraper.requests, "get", fake_get)
|
|
converter = doc_scraper.DocToSkillConverter(
|
|
{"name": "t", "base_url": "https://example.com/"}, dry_run=True
|
|
)
|
|
converter._try_sitemap()
|
|
|
|
assert seen_timeouts, "sitemap probe made no request"
|
|
for timeout in seen_timeouts:
|
|
assert isinstance(timeout, tuple) and len(timeout) == 2, (
|
|
f"expected a (connect, read) timeout tuple, got {timeout!r}"
|
|
)
|
|
connect, read = timeout
|
|
assert connect <= read
|