wehub-resource-sync
2cab53bc94
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers CLI - Convert documentation to AI skills dockerfile:Dockerfile name:skill-seekers]) (push) Waiting to run
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers MCP Server - 25 tools for AI assistants dockerfile:Dockerfile.mcp name:skill-seekers-mcp]) (push) Waiting to run
Docker Publish / Test Docker Images (push) Blocked by required conditions
Test Vector Database Adaptors / Test chroma Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test faiss Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test qdrant Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test weaviate Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test MCP Vector DB Tools (push) Waiting to run
Tests / Code Quality (Ruff & Mypy) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (macos-latest, 3.11) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (macos-latest, 3.12) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.10) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.11) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.12) (push) Waiting to run
Tests / Tests (push) Blocked by required conditions
Tests / Serial / Integration / E2E Tests (push) Blocked by required conditions
Tests / MCP Server Tests (push) Blocked by required conditions
179 lines
18 KiB
Markdown
179 lines
18 KiB
Markdown
# Skill Seekers Architecture
|
|
|
|
> Updated 2026-06-11 | StarUML project: `docs/UML/skill_seekers.mdj`
|
|
>
|
|
> ⚠️ The PNG exports under `docs/UML/exports/` predate the Grand Unification
|
|
> refactor (see `docs/UNIFICATION_PLAN.md`) and are stale where noted below
|
|
> (CLICore, Scrapers, Enhancement, MCP Server, Parsers). The text in this file
|
|
> is current; regenerate the exports from StarUML when possible.
|
|
|
|
## Overview
|
|
|
|
Skill Seekers converts documentation from 18 source types into production-ready formats for 24+ AI platforms. The architecture follows a layered module design with 9 core modules and 5 utility modules. Source-type ingestion is routed through a single `skill-seekers create` command via the `SkillConverter` base class + factory pattern. A separate `skill-seekers scan` command (added in #327) is the AI-driven project knowledge-base bootstrapper that emits one config per detected framework — these configs feed back into `create`.
|
|
|
|
## Package Diagram
|
|
|
|
|
|
|
|
**Core Modules** (upper area):
|
|
- **CLICore** -- Git-style command dispatcher, entry point for all `skill-seekers` commands
|
|
- **Scan** -- AI-driven project knowledge-base bootstrapper (`scan_command.py` + `signal_collectors.py`); emits one config per detected framework + a `<project>-codebase.json`
|
|
- **Scrapers** -- 17 source-type extractors (web, GitHub, PDF, Word, EPUB, video, etc.)
|
|
- **Adaptors** -- Strategy+Factory pattern for 20+ output platforms (Claude, Gemini, OpenAI, RAG frameworks)
|
|
- **Analysis** -- C3.x codebase analysis pipeline (AST parsing, 10 GoF pattern detectors, guide builders)
|
|
- **Enhancement** -- AI-powered skill improvement via `AgentClient` (API mode: Anthropic/Kimi/Gemini/OpenAI + LOCAL mode: Claude Code/Kimi/Codex/Copilot/OpenCode/custom, --enhance-level 0-3)
|
|
- **Packaging** -- Package, upload, and install skills to AI agent directories
|
|
- **MCP** -- FastMCP server exposing 40 tools via stdio/HTTP transport (includes marketplace and config publishing)
|
|
- **Sync** -- Documentation change detection and re-scraping triggers
|
|
|
|
**Utility Modules** (lower area):
|
|
- **Parsers** -- CLI argument parsers (30+ SubcommandParser subclasses)
|
|
- **Storage** -- Cloud storage abstraction (S3, GCS, Azure)
|
|
- **Embedding** -- Multi-provider vector embedding generation
|
|
- **Benchmark** -- Performance measurement framework
|
|
- **Utilities** -- Shared helpers (LanguageDetector, RAGChunker, MarkdownCleaner, etc.)
|
|
|
|
## Core Module Diagrams
|
|
|
|
### CLICore
|
|
|
|
|
|
Entry point: `skill-seekers` CLI. `CLIDispatcher` maps subcommands to modules via `COMMAND_MODULES` dict (`scan`/`doctor` use the newer `COMMAND_CLASSES` table — `Cls(args).execute()`). Every command's flags are defined exactly once, in the central `SubcommandParser` classes (`cli/parsers/`); module `main(args=None)` standalone paths build their parser FROM the central class, and a drift-guard test (`tests/test_cli_parsers.py`) pins dests/defaults/option-strings equality. Exit codes are standardized in `cli/exit_codes.py` (0/1/2/130). `CreateCommand` auto-detects source type via `SourceDetector`, initializes `ExecutionContext` singleton (Pydantic model, single source of truth for all config; `override()` is contextvars-based, so concurrent threads/async tasks can't clobber each other), then calls `get_converter()` → `converter.run()`. Enhancement runs centrally in CreateCommand after the converter completes. *(PNG export predates the single-definition parsers + exit codes.)*
|
|
|
|
### Scrapers
|
|
|
|
|
|
18 converter classes inheriting `SkillConverter` base class (Template Method: `run()` → `extract()` → `build_skill()`). Factory: `get_converter(source_type, config)` via `CONVERTER_REGISTRY`. No `main()` entry points — all routing through `CreateCommand`. The 9 document scrapers (pdf, word, epub, html, pptx, jupyter, man, rss, chat) inherit the intermediate `DocumentSkillBuilder` base (`cli/document_skill_builder.py`), which owns the shared build-side machinery (categorization, reference/index/SKILL.md generation) with class-attr + hook-method variation points; ports are byte-identical, pinned by golden trees (`tests/golden/phase2/`). `UnifiedScraper` (multi-source orchestrator) routes via a class-level `SOURCE_DISPATCH` table with a shared `_scrape_with_converter()` engine for the 13 mechanical source types, and is factory-constructible: `get_converter("config", {"config_path": ...})`. Notable: `GitHubScraper` (3-stream fetcher) + `GitHubToSkillConverter` and `UnifiedSkillBuilder` (builder strategies, deliberately outside the converter hierarchy). *(PNG export predates DocumentSkillBuilder and SOURCE_DISPATCH.)*
|
|
|
|
### Scan
|
|
|
|
No UML export yet — pipeline is ~1100 lines across `scan_command.py` and `signal_collectors.py`. Dispatched via the new `COMMAND_CLASSES` table (`main.py`) — `ScanCommand(args).execute()` consumes the parsed argparse namespace directly, no duplicate argparse. Flow:
|
|
|
|
1. `collect_signals(root)` (`signal_collectors.py`) → `SignalBundle` with **per-kind byte budgets** (24 KB manifest / 6 KB README / 6 KB CI / 28 KB source samples, total 64 KB). Manifests cover ~50 file types; source samples are whole first-2-KB chunks (the AI parses imports — replaced the brittle regex approach in WS4). Source dirs cover web/JS monorepos + Go `cmd/` + Rust `crates/` + plus root-walk for Django/flat-Python.
|
|
2. `detect_with_ai(bundle, AgentClient)` (`scan_command.py`) → `list[Detection]`. Single LLM call with a canonical-slug-demanding prompt. JSON extracted via raw parse → markdown fence → bracket-substring fallback. Wraps `client.call` in try/except so auth/network errors don't crash the scan.
|
|
3. `_archive_removed(out_dir, removed_slugs)` — when a config disappears from current detections (per `diff_against_existing`), MOVE (not delete) to `out_dir/.archived/<UTC-timestamp>/`. Runs after diff, before fresh writes, so a re-emit with the same name doesn't race the move.
|
|
4. `resolve_or_generate_with_status(detection, probe_urls=…)` for each detection (capped at `--max-ai-generations`):
|
|
- **Cache hit:** `out_dir/<slug>.json` exists from a prior scan → re-stamp `metadata.detected_version`, return.
|
|
- **Resolve:** try each candidate from `_canonical_name_candidates` (original → lowercase → hyphenated → suffix-stripped, where suffixes include CJK + European-language terms for engine/framework/library/core → npm-scope-unwrap) via `resolve_config_path` (local repo → user dir → API). Always appends `.json` to the lookup name.
|
|
- **Generate:** `generate_config_with_ai` produces a fresh unified config, validated by `UniSkillConfigValidator` and re-checked against the registry name regex `^[a-zA-Z0-9_-]+$`. With `probe_urls=True`: HEAD-probes `base_url` + GitHub repos; on 4xx/5xx re-prompts the AI with feedback; stamps `metadata._url_unverified` on confirmed-bad URLs.
|
|
5. `emit_codebase_config(root, out_dir)` — always writes `<project>-codebase.json` wrapping a `type: local` source.
|
|
6. `diff_against_existing(out_dir, detections)` — keyed by filename slug (not internal config name) so re-scans don't churn when the AI returns a display name and the registry has the canonical slug. Reads `metadata.detected_version` with backwards-compat fallback to legacy top-level placement.
|
|
7. `maybe_publish(generated, skip_prompt)` — **async-native** (WS11). Opt-in submission via the existing MCP `submit_config_tool`. Pre-checks `GITHUB_TOKEN`. `_find_existing_issue` queries GitHub Search API for an existing open issue with the same config name (idempotency). `_submit_config` retries transient failures (rate limit, 5xx) with 0s/5s/15s backoff and a 30s per-attempt timeout. `_prompt_async` wraps `input()` via `asyncio.to_thread` so the event loop isn't blocked.
|
|
|
|
The whole pipeline runs inside a single `asyncio.run` at `ScanCommand.execute()` — sync core (file IO, AgentClient, signal collection) inside, async publish at the edge. `--dry-run` previews steps 1-6 (without writing) and skips publish entirely.
|
|
|
|
All JSON writes use `_atomic_write_json` (temp file + `os.replace`) so SIGINT mid-write can't corrupt a config and silently flip it to "removed" on the next scan. `_safe_size` guards `stat()` so broken symlinks in `src/` don't crash signal collection. `logging.basicConfig` ensures `logger.warning`/`error` reaches the user (silenced by default without it). Exit code 1 when nothing was emitted, so CI shell pipelines detect total-failure scans.
|
|
|
|
### Adaptors
|
|
|
|
|
|
`SkillAdaptor` ABC with 3 abstract methods: `format_skill_md()`, `package()`, `upload()`. Two-level hierarchy: direct subclasses (Claude, Gemini, OpenAI, Markdown, OpenCode, RAG adaptors) and `OpenAICompatibleAdaptor` intermediate (MiniMax, Kimi, DeepSeek, Qwen, OpenRouter, Together, Fireworks).
|
|
|
|
### Analysis (C3.x Pipeline)
|
|
|
|
|
|
`UnifiedCodebaseAnalyzer` controller orchestrates: `CodeAnalyzer` (AST, 9 languages), `PatternRecognizer` (10 GoF detectors via `BasePatternDetector`), `TestExampleExtractor`, `HowToGuideBuilder`, `ConfigExtractor`, `SignalFlowAnalyzer`, `DependencyAnalyzer`, `ArchitecturalPatternDetector`.
|
|
|
|
### Enhancement
|
|
|
|
|
|
`AgentClient` (`cli/agent_client.py`) is the single AI transport: every API-mode enhancement call routes through it (provider/base_url/model overrides, system prompts, temperature, central truncation gate, timeout policy, error classification). The ordered `API_PROVIDERS` registry in agent_client is the one home for provider/env-var/priority data. SKILL.md enhancement flows through `SkillAdaptor._enhance_skill_md_via_client` (adaptors/base.py) with atomic backup saves — the claude/openai/gemini/openai_compatible adaptors' `enhance()` are thin routing declarations. Two enhancer hierarchies remain for C3.x content: `AIEnhancer` (ai_enhancer.py) and `UnifiedEnhancer` (their duplicated thread pools now share `cli/parallel_batches.py`). `WorkflowEngine` orchestrates multi-stage `EnhancementWorkflow`. LOCAL mode (Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents) uses `build_local_agent_command()` and a recursion guard (`SKILL_SEEKER_ENHANCE_ACTIVE`) on every spawn path. *(PNG export predates the AgentClient consolidation.)*
|
|
|
|
### Packaging
|
|
|
|
|
|
`PackageSkill` delegates to adaptors for format-specific packaging. `UploadSkill` handles platform API uploads. `InstallSkill`/`InstallAgent` install to AI agent directories. `OpenCodeSkillSplitter` handles large file splitting.
|
|
|
|
### MCP Server
|
|
|
|
|
|
`SkillSeekerMCPServer` (FastMCP) with 40 tools in 10 categories. The MCP layer is a thin adapter over the `skill_seekers.services` package, which now owns the shared domain classes: `SourceManager` (config CRUD), `GitConfigRepo` (community configs), `MarketplacePublisher` (publish skills to marketplace repos), `MarketplaceManager` (marketplace registry CRUD), `ConfigPublisher` (push configs to registered source repos + the only `detect_category` implementation). Back-compat shims remain at the old `mcp.*` paths. `AgentDetector` (environment detection) stays in mcp. Nine former subprocess tools (estimate_pages, detect_patterns, extract_test_examples, extract_config_patterns, build_how_to_guides, split_config, generate_router, package_skill, upload_skill) now run in-process via `run_cli_main()` in `mcp/tools/_common.py`; only LOCAL-agent enhancement stays subprocess by design. *(PNG export predates the services layer.)*
|
|
|
|
### Sync
|
|
|
|
|
|
`SyncMonitor` controller schedules periodic checks via `ChangeDetector` (SHA-256 hashing, HTTP headers, content diffing). `Notifier` sends alerts when changes are found. Pydantic models: `PageChange`, `ChangeReport`, `SyncConfig`, `SyncState`.
|
|
|
|
## Utility Module Diagrams
|
|
|
|
### Parsers
|
|
|
|
|
|
`SubcommandParser` ABC with 18 subclasses — individual scraper parsers removed after Grand Unification (all source types route through `CreateParser`). Remaining: Create, Doctor, Config, Enhance, EnhanceStatus, Package, Upload, Estimate, Install, InstallAgent, TestExamples, Resume, Quality, Workflows, SyncConfig, Stream, Update, Multilang. These central classes are the ONLY place a command's flags are defined — each module's standalone `main(args=None)` builds its parser from the central class, and `TestCentralParserSingleSource` pins the equality.
|
|
|
|
### Storage
|
|
|
|
|
|
`BaseStorageAdaptor` ABC with `S3StorageAdaptor`, `GCSStorageAdaptor`, `AzureStorageAdaptor`. `StorageObject` dataclass for file metadata.
|
|
|
|
### Embedding
|
|
|
|
|
|
`EmbeddingGenerator` (multi-provider: OpenAI, Sentence Transformers, Voyage AI). `EmbeddingPipeline` coordinates provider, caching, and cost tracking. `EmbeddingProvider` ABC with OpenAI and Local implementations.
|
|
|
|
### Benchmark
|
|
|
|
|
|
`BenchmarkRunner` orchestrates `Benchmark` instances. `BenchmarkResult` collects timings/memory/metrics and produces `BenchmarkReport`. Supporting data types: `Metric`, `TimingResult`, `MemoryUsage`, `ComparisonReport`.
|
|
|
|
### Utilities
|
|
|
|
|
|
16 shared helper classes: `LanguageDetector`, `MarkdownCleaner`, `RAGChunker`, `RateLimitHandler`, `ConfigManager`, `ConfigValidator`, `SkillQualityChecker`, `QualityAnalyzer`, `LlmsTxtDetector`/`Downloader`/`Parser`, `ConfigSplitter`, `ConflictDetector`, `IncrementalUpdater`, `MultiLanguageManager`, `StreamingIngester`.
|
|
|
|
## Key Design Patterns
|
|
|
|
| Pattern | Where | Classes |
|
|
|---------|-------|---------|
|
|
| Strategy + Factory | Adaptors | `SkillAdaptor` ABC + `get_adaptor()` factory + 20+ implementations |
|
|
| Strategy + Factory | Storage | `BaseStorageAdaptor` ABC + S3/GCS/Azure |
|
|
| Strategy + Factory | Embedding | `EmbeddingProvider` ABC + OpenAI/Local |
|
|
| Template Method + Factory | Scrapers | `SkillConverter` base (+ `DocumentSkillBuilder` intermediate for the 9 document scrapers) + `get_converter()` factory + 18 converter subclasses |
|
|
| Singleton | Configuration | `ExecutionContext` Pydantic model — single source of truth for all config; `override()` layers a contextvars-based override over the base singleton (thread/async safe) |
|
|
| Command | CLI | `CLIDispatcher` + `COMMAND_MODULES` lazy dispatch (+ `COMMAND_CLASSES` for scan/doctor) |
|
|
| Template Method | Pattern Detection | `BasePatternDetector` + 10 GoF detectors |
|
|
| Template Method | Parsers | `SubcommandParser` + 18 subclasses |
|
|
|
|
## Behavioral Diagrams
|
|
|
|
### Create Pipeline Sequence
|
|
|
|
|
|
`CreateCommand` is now the pipeline orchestrator. Flow: User → `execute()` → `SourceDetector.detect(source)` → `validate_source()` → `ExecutionContext.initialize()` → `_validate_arguments()` → `get_converter(type, config)` → `converter.run()` (extract + build_skill) → `_run_enhancement(ctx)` → `_run_workflows()`. Enhancement is centralized in CreateCommand, not inside each converter.
|
|
|
|
### GitHub Unified Flow + C3.x
|
|
|
|
|
|
`UnifiedScraper` orchestrates GitHub scraping (3-stream fetch) then delegates to `analyze_codebase(enhance_level)` for C3.x analysis. Shows all 5 C3.x stages: `PatternRecognizer` (C3.1), `TestExampleExtractor` (C3.2), `HowToGuideBuilder` with examples from C3.2 (C3.3), `ConfigExtractor` (C3.4), and `ArchitecturalPatternDetector` (C3.5). Note: `enhance_level` is the sole AI control parameter — `enhance_with_ai`/`ai_mode` are internal to C3.x classes only.
|
|
|
|
### Source Auto-Detection
|
|
|
|
|
|
Activity diagram showing `source_detector.py` decision tree in correct code order: file extension first (.json config, .pdf/.docx/.epub/.ipynb/.html/.pptx/etc) → video URL → `os.path.isdir()` (Codebase) → GitHub pattern (owner/repo or github.com URL) → http/https URL (Web) → bare domain inference → error.
|
|
|
|
### MCP Tool Invocation
|
|
|
|
|
|
MCP Client (Claude Code/Cursor) → FastMCPServer (stdio/HTTP) with two invocation paths: **Path A** (scraping tools) uses `get_converter(type, config).run()` in-process via `_run_converter()` helper, **Path B** (packaging/analysis/config tools) runs in-process via direct Python imports or `run_cli_main()` (`mcp/tools/_common.py`), which parses argv with the command's real parser. Both return TextContent → JSON-RPC. Only LOCAL-agent enhancement (`enhance_skill`, `install_skill`'s enhance step) spawns subprocesses, by design.
|
|
|
|
### Enhancement Pipeline
|
|
|
|
|
|
`--enhance-level` decision flow with precise internal variable mapping: Level 0 sets `ai_mode=none`, skips all AI. Level >= 1 selects `ai_mode=api` (if any supported API key set: Anthropic, Moonshot/Kimi, Gemini, OpenAI) or `ai_mode=local` (via `AgentClient` with configurable agent: Claude Code, Kimi, Codex, Copilot, OpenCode, or custom), then SKILL.md enhancement happens post-build via `enhance_command`. Level >= 2 enables `enhance_config=True`, `enhance_architecture=True` inside `analyze_codebase()`. Level 3 adds `enhance_patterns=True`, `enhance_tests=True`.
|
|
|
|
### Runtime Components
|
|
|
|
|
|
Component diagram with runtime dependencies. Key flows: `CLI Core` dispatches to `Scrapers` via `get_converter()` → `converter.run()` (in-process, no subprocess). `Scrapers` call `Codebase Analysis` via `analyze_codebase(enhance_level)`. `Codebase Analysis` uses `C3.x Classes` internally and `Enhancement` when level ≥ 2. `MCP Server` reaches `Scrapers` via `get_converter()` in-process and `Adaptors` via direct import. `Scrapers` optionally use `Browser Renderer (Playwright)` via `render_page()` when `--browser` flag is set for JavaScript SPA sites.
|
|
|
|
### Browser Rendering Flow
|
|
|
|
|
|
When `--browser` flag is set, `DocScraper.scrape_page()` delegates to `BrowserRenderer.render_page(url)` instead of `requests.get()`. The renderer auto-installs Chromium on first use, navigates with `wait_until='networkidle'` to let JavaScript execute, then returns the fully-rendered HTML. The rest of the pipeline (BeautifulSoup → `extract_content()` → `save_page()`) remains unchanged. Optional dependency: `pip install "skill-seekers[browser]"`.
|
|
|
|
## File Locations
|
|
|
|
- **StarUML project**: `docs/UML/skill_seekers.mdj`
|
|
- **Diagram exports**: `docs/UML/exports/*.png`
|
|
- **Source code**: `src/skill_seekers/`
|