yusufkaraaslan--skill_seekers/docs/UNIFICATION_PLAN.md

Unification Plan — 5 Phases

Created 2026-06-11 from the full-repo architecture review (see PR for fix/code-review-findings). Goal: finish the Grand Unification — one dispatch, one config source, one build pipeline, one AI client. Each phase is independently shippable; do them in order.

Status

Phase	State	Branch / PR
1 — CLI drift & preset bugs	done (2026-06-11)	refactor/phase-1-quick-fixes
2 — DocumentSkillBuilder	done (2026-06-11)	refactor/phase-2-document-skill-builder
3 — Enhancement consolidation	done (2026-06-11; 3.1 deferred)	refactor/phase-3-enhancement-consolidation
4 — UnifiedScraper conformance	done (2026-06-11; 4.1 deferred)	refactor/phase-3-enhancement-consolidation (same branch)
5 — Config / dispatch / MCP platform	done (2026-06-11)	refactor/phase-3-enhancement-consolidation (same branch)

Phase 1 notes: all 5 parser drifts fixed + a programmatic drift-guard test (tests/test_cli_parsers.py::TestCentralModuleParserSync) that fails if any module flag is ever missing from its central parser again. stream --output and multilang --languages were central-parser fictions — now implemented in the modules. AGENT_PRESETS and agent-name normalization now live ONLY in agent_client.py (claude's skip-permissions flag became preset metadata so interactive callers can omit it). unified_enhancer's zero-importer "backward compat" subclasses (which shadowed the real PatternEnhancer / TestExampleEnhancer / GuideEnhancer / ConfigEnhancer) were deleted; a guard test prevents reintroduction.

Phase 1 — Stop the bleeding: shipping bugs from drift (small)

User-facing flags broken TODAY because central parsers (parsers/*.py) drifted from module parsers (each command's main(args=None)):

1. estimate central parser missing --unlimited, --timeout → CLI rejects them.
2. update central parser missing --generate-package, --apply-update.
3. quality central parser missing --output.
4. stream central/module drift (--overlap-chars, --batch-size, --checkpoint; verify both directions).
5. multilang bidirectional drift (--report/--export missing centrally; --languages central-only).

Also: 6. AGENT_PRESETS duplicated between agent_client.py and enhance_skill_local.py — the kimi preset has already diverged ({cwd} vs {skill_dir}, missing parse_output). Make agent_client.AGENT_PRESETS the single source; enhance_skill_local layers its extra metadata (supports_skip_permissions) on top. 7. GuideEnhancer name collision: guide_enhancer.py:GuideEnhancer (standalone) vs unified_enhancer.py:GuideEnhancer (subclass). Rename the unified_enhancer one (it's the newer, less-referenced alias).

Each fix gets a regression test. The durable fix for the parser-drift class is Phase 5 (single argument registry); Phase 1 just makes the flags work.

Phase 2 results (2026-06-11)

All 9 document scrapers (epub, word, pptx, html, pdf, man, rss, chat, jupyter) now inherit cli/document_skill_builder.py:DocumentSkillBuilder. Net −1,859/+424 lines across the scrapers; the shared machinery lives once. Every port is byte-identical, proven by golden trees captured from the pre-refactor code (tests/golden/phase2/*, harness in tests/phase2_golden_utils.py, protocol: UPDATE_GOLDENS=1 only on purpose). Dedup depth varies by tier: epub/word/pptx/html share nearly everything; pdf/jupyter share orchestration + several generators; man/rss/chat share the build_skill spine and helpers, keeping their domain-shaped reference/SKILL writers as overrides.

Follow-up hook candidates surfaced by the ports (do as a small Phase 2.1 — each removes a full-method override that exists only for wording/keys):

• DOC_NOUN ("documentation"/"presentation"/"chat") + index title hook
• PATTERN_KEYWORDS class attr on _format_patterns_from_content
• LOAD_TOTAL_KEY on load_extracted_data (man/pdf use total_pages)
• category_stem(cat_key) hook in _reference_filename (chat)
• sectioned _generate_skill_md (metadata block / when-to-use bullets / extra-stats hooks) — would let html/pptx/chat drop their full overrides
• footer "Skill Seeker" vs "Skill Seekers" inconsistency (man)

Phase 2 — Unify the build side of scrapers (LARGE, highest ROI)

The 8-9 document scrapers (epub, word, pptx, html, man, rss, chat, jupyter, pdf) carry ~90%-identical build machinery (~6k+ avoidable LOC) with user-visible inconsistencies (truncation: 500 vs 1000 vs none; frontmatter keys; code-example ordering).

1. Normalize extracted-data schema: one page/section dict shape (section_number everywhere; pdf's page_number mapped). Consider a shared ExtractedDocument model.
2. Extract DocumentSkillBuilder (base class between SkillConverter and the document scrapers, or composable builder owned by the base) owning: categorize_content, reference-file writing (incl. table rendering, truncation policy), index.md generation, SKILL.md generation (frontmatter incl. json.dumps quoting + version keys). Per-scraper hooks: source_type_label, base stem, metadata keys.
3. Port scrapers one at a time, golden-file tests comparing pre/post output.

Phase 3 results (2026-06-11)

AgentClient is now the single AI transport for every text-based call:

• AgentClient gained provider/base_url/model overrides, system prompts, temperature, and a ThinkingBlock-safe anthropic response reader.
• SkillAdaptor._enhance_skill_md_via_client (adaptors/base.py) is the one SKILL.md enhancement flow; claude/openai/gemini/openai_compatible enhance() are now ~10-line routing declarations. This also FIXED two latent bugs: gemini and openai adaptors had no truncation gate and used the destructive rename-then-write save. One canonical _read_reference_files (was 4 copies).
• enhance_skill.SkillEnhancer routed through AgentClient (import-time sys.exit removed); unified_scraper Phase-6 API save is atomic; doc_scraper's path already routed via adaptors.
• video_scraper's reference cleaner goes through AgentClient. video_visual's frame classification is the DOCUMENTED EXCEPTION (multimodal; AgentClient is text-only today).
• build_local_agent_command() in agent_client is the single preset-template/permissions-flag handler; LocalSkillEnhancer delegates.

Deferred to Phase 3.1 (consumers depend on per-feature prompts/output contracts — blind merges would silently change AI quality):

• Collapse AIEnhancer (ai_enhancer.py) / UnifiedEnhancer hierarchies and merge guide_enhancer/config_enhancer; extract their duplicated parallel-batching first.
• LocalSkillEnhancer terminal/background/daemon orchestration is kept; the remaining overlap with AgentClient._call_local is the prompt-file + subprocess loop (small).
• AgentClient multimodal support (would absorb video_visual).

Phase 3 — Enhancement consolidation (medium-large)

Vision: AgentClient is the ONLY AI transport.

1. SkillMarkdownEnhancer: one implementation of read-references → build-prompt → call AgentClient → validate → atomic save. Replaces the 5 copies in enhance_skill.py + adaptors (claude/openai/gemini/ openai_compatible enhance()).
2. Route video_scraper._clean_reference_with_ai and video_visual frame classification through AgentClient.
3. LocalSkillEnhancer → thin orchestration (terminal/background/daemon UI) over AgentClient._call_local; delete its duplicated presets/command building.
4. Collapse hierarchies: unified_enhancer.py is canonical; delete ai_enhancer.py duplicates; merge guide_enhancer.py.
5. All 6 SKILL.md-enhancement entry points call SkillMarkdownEnhancer.

Phase 4 results (2026-06-11)

• scrape_all_sources() dispatches via a class-level SOURCE_DISPATCH table (was a 17-branch if/elif).
• _scrape_with_converter() is the shared engine for the 13 mechanical source types (pdf, word, epub, jupyter, html, openapi, asciidoc, pptx, confluence, notion, rss, manpage, chat): get_converter() + PUBLIC converter.extract() + data-file load + cache copy + record append + standalone sub-skill build (kept — the unified build consumes sub-skill references from the cache). Thin per-type wrappers own only id resolution, config keys, and record keys. unified_scraper.py −280 lines.
• _scrape_video uses the public extract() (which now returns the result).
• documentation / github / local stay bespoke WITH comments explaining why (function-based scrape_documentation + ExecutionContext override; clone/C3.x/dual-write; function-based analyze_codebase).

Deferred to Phase 4.1:

• run() template-method conformance (move phases into extract()/build_skill() and use base run()) — run()'s orchestration, error strings, and arg handling are pinned by tests; needs its own careful pass.
• Config-dict constructor so get_converter() can build UnifiedScraper (removes the create_command/scraping_tools special-cases).
• Bring GitHubToSkillConverter / UnifiedSkillBuilder into the hierarchy.
• Wrap documentation/local in converter-shaped adapters so they join the engine.

Phase 4 — UnifiedScraper conformance (medium)

1. Replace 17 _scrape_* methods with a data-driven loop: get_converter(source["type"], sub_config) + public converter.extract()
   • standard data_file_for() retrieval. New converter types then work in unified configs automatically.
2. Move phase logic (scrape-all → conflicts → merge) into extract(), build into build_skill(); stop overriding run() (base template provides skip_scrape/dry_run/error handling).
3. Constructor takes a config dict (config_path inside it) so the factory special-case in create_command dies.
4. Bring GitHubToSkillConverter and UnifiedSkillBuilder into the SkillConverter hierarchy (or make them explicit builder strategies the base invokes).

Phase 5 results so far (2026-06-11)

5a — MCP services layer + import hygiene (done):

• New src/skill_seekers/services/ package: marketplace_manager, marketplace_publisher, config_publisher (the ONLY category-detection logic), source_manager, git_repo — importable by CLI and MCP alike; a regression test proves importing services never loads skill_seekers.mcp.
• Back-compat shims at the old mcp/ paths (named re-exports, identity tests); 29 import sites updated; patch targets retargeted.
• All 7 sys.path.insert hacks in mcp/ removed (incl. server_legacy); server_fastmcp's 100-line import fallback collapsed to one absolute import block. CLI_DIR remains only where subprocess tools still need script paths (goes away with 5d).

5b — contextvars ExecutionContext.override (done):

• override() activates via a ContextVar layered over the unchanged base singleton — concurrent threads/asyncio tasks no longer clobber each other's overrides; nesting stacks; exceptions restore; is_initialized() semantics unchanged. Thread propagation contract documented (copy_context, same pattern as the MCP log capture). +6 tests.

5c — single-definition parsers + exit codes (done 2026-06-11):

• 13 legacy command modules (config, enhance-status, upload, install, install-agent, estimate, extract-test-examples, resume, quality, workflows, stream, update, multilang) no longer define their own add_argument blocks: main(args=None)'s standalone path builds its parser FROM the central SubcommandParser class, so every flag is defined exactly once (kills the Phase-1 drift class permanently). enhance, package, and sync-config were already single-source via shared arguments/* modules and were left as-is.
• backfill_parser_defaults else-branches removed from all 7 modules that had them (the central dispatch namespace has every dest by construction); the helper stays in arguments/common.py.
• Two real drift bugs fixed by completing the central parsers: install --target and extract-test-examples --recursive were module-only and REJECTED by the unified CLI. update --force remains a central-parser no-op (accepted, unread) — now accepted identically in both paths; implementing it is a separate feature.
• workflows' standalone subparser dest unified to workflows_action (was action); main() reads it via getattr in both call styles.
• New cli/exit_codes.py (EXIT_SUCCESS/ERROR/VALIDATION/INTERRUPT); test_example_extractor.main now returns 0 instead of None; constants applied across the touched modules.
• Drift guard upgraded: tests/test_cli_parsers.py:: TestCentralParserSingleSource asserts module-built and central parsers have IDENTICAL option dests, defaults, and option strings for all 13.

5d — MCP subprocess→in-process (done 2026-06-11):

• 9 tools migrated (estimate_pages, detect_patterns, extract_test_examples, extract_config_patterns, build_how_to_guides, split_config, generate_router, package_skill, upload_skill) via the shared run_cli_main() helper in mcp/tools/_common.py: real-parser argv parsing (sys.argv patch under a lock), redirect_stdout/stderr + the contextvar log-capture pattern, SystemExit/KeyboardInterrupt containment, and an identical (stdout, stderr, returncode) contract so output formats (incl. the "❌ Error:" sniffing in install_skill's workflow) stay byte-identical. Bonus fix: extract_config_patterns_tool had been passing flags the CLI parser rejects — it ALWAYS failed; now mapped to the real flags and pinned by a regression test.
• enhance_skill_tool (LOCAL agent) and install_skill_tool's enhancement step stay subprocess BY DESIGN (long-running real agent; fork-bomb-guard env semantics) — documented at the call sites.

4.1 follow-ups (done 2026-06-11, run() conformance deliberately dropped):

• UnifiedScraper accepts the factory-shaped dict ({"config_path": ..., "merge_mode", "output_dir", "dry_run"}); get_converter("config", {...}) works; the special-cases in create_command._route_to_scraper and scrape_docs_tool are GONE; legacy str-positional construction still supported (+4 tests).
• GitHubToSkillConverter / UnifiedSkillBuilder documented as builder strategies (deliberately not SkillConverters — no extract() phase).
• run() template conformance DROPPED deliberately: TestRunOrchestration pins that run() itself (not build_skill) triggers workflows; conformance would require weakening those tests.

2.1 hook polish (done 2026-06-11): LOAD_TOTAL_KEY / DOC_NOUN / PATTERN_KEYWORDS / RANGE_LABEL / category_stem / sectioned-SKILL.md hooks added to DocumentSkillBuilder; full-method overrides collapsed in pdf (−61), html (−202), pptx (−106), jupyter (−48), man (−16), chat (−12 structural), all under byte-identical golden protection (24/24 trees). Overrides that survive are genuinely domain-shaped (documented per scraper).

3.1 safe slice (done 2026-06-11): cli/parallel_batches.py run_batches_parallel() replaced the 3 duplicated ThreadPoolExecutor blocks in ai_enhancer/unified_enhancer (contextvars propagation preserved; net −75). The full hierarchy merge remains deliberately deferred — consumers depend on per-feature prompts; a blind merge would silently change AI quality.

Remaining (small, cosmetic):

• Move the ~14 legacy commands from COMMAND_MODULES to COMMAND_CLASSES dispatch (Cls(args).execute()) — aesthetic now that parsers are single-definition and dispatch passes namespaces.
• Declarative per-source-type config models replacing create_command._build_config's if/elif chain.
• Per-request ExecutionContext initialization at the MCP boundary — lower urgency since the converters' raw-config fallback (fixed earlier) plus the contextvars override (5b) cover the known divergences.
• AgentClient multimodal support (absorbs video_visual).

Phase 5 — Platform unification: config, dispatch, MCP (medium)

1. ExecutionContext = the single runtime config source. Initialized once per entry point (CLI AND MCP server request layer), merging: args/request → config file → ConfigManager (~/.config) → defaults.json. Nothing else reads os.environ directly (env reads live in the context/agent_client provider registry only).
2. override() reimplemented on contextvars (thread/async-safe); forbid global singleton mutation.
3. Replace create_command._build_config's 227-line if/elif with per-source- type config models (Pydantic) derived from the context.
4. Finish COMMAND_CLASSES migration for the ~14 legacy commands; single argument registry so every flag is defined exactly once (kills the Phase-1 drift class permanently); standardize exit codes.
5. MCP: services/ layer (MarketplaceManager, MarketplacePublisher, ConfigPublisher + detect_category, SourceManager) importable by CLI and MCP; replace 11 subprocess tools with in-process dispatch; remove all 6 sys.path.insert hacks; scraping_tools uses SourceDetector instead of its hardcoded 4-case detection.

Invariants for every phase

• pytest tests/ fast subset green before each commit; full suite before PR.
• uvx ruff check + format clean.
• No behavior change without a regression test pinning the new behavior.
• Update docs/UML_ARCHITECTURE.md when a phase changes the real structure.