chore: import upstream snapshot with attribution
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers CLI - Convert documentation to AI skills dockerfile:Dockerfile name:skill-seekers]) (push) Waiting to run
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers MCP Server - 25 tools for AI assistants dockerfile:Dockerfile.mcp name:skill-seekers-mcp]) (push) Waiting to run
Docker Publish / Test Docker Images (push) Blocked by required conditions
Test Vector Database Adaptors / Test chroma Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test faiss Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test qdrant Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test weaviate Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test MCP Vector DB Tools (push) Waiting to run
Tests / Code Quality (Ruff & Mypy) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (macos-latest, 3.11) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (macos-latest, 3.12) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.10) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.11) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.12) (push) Waiting to run
Tests / Tests (push) Blocked by required conditions
Tests / Serial / Integration / E2E Tests (push) Blocked by required conditions
Tests / MCP Server Tests (push) Blocked by required conditions

This commit is contained in:
wehub-resource-sync
2026-07-13 12:46:28 +08:00
commit 2cab53bc94
2985 changed files with 1288070 additions and 0 deletions
+14
View File
@@ -0,0 +1,14 @@
{
"mcpServers": {
"skill-seeker": {
"type": "stdio",
"command": "/path/to/your/Skill_Seekers/.venv/bin/python3",
"args": [
"-m",
"skill_seekers.mcp.server_fastmcp"
],
"cwd": "/path/to/your/Skill_Seekers",
"env": {}
}
}
}
+21
View File
@@ -0,0 +1,21 @@
{
"name": "skill-seekers",
"version": "3.5.0",
"description": "Convert docs and repos into AI skills via MCP",
"author": {
"name": "yusufkaraaslan",
"url": "https://github.com/yusufkaraaslan/Skill_Seekers"
},
"homepage": "https://skillseekersweb.com/",
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
"keywords": ["mcp", "codex", "skills", "documentation"],
"mcpServers": "./.mcp.json",
"skills": "./skills/",
"interface": {
"displayName": "Skill Seekers",
"shortDescription": "Convert docs and repos into AI skills via MCP",
"longDescription": "Convert documentation websites, GitHub repositories, and PDFs into AI skills with automatic conflict detection.",
"category": "Development",
"websiteURL": "https://skillseekersweb.com/"
}
}
+83
View File
@@ -0,0 +1,83 @@
# Python artifacts
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Virtual environments
venv/
env/
ENV/
.venv
# Testing
.pytest_cache/
.coverage
.coverage.*
htmlcov/
.tox/
.hypothesis/
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
.DS_Store
# Git
.git/
.gitignore
.gitattributes
# Documentation
docs/
*.md
!README.md
# CI/CD
.github/
.gitlab-ci.yml
.travis.yml
# Output directories
output/
data/
*.zip
*.tar.gz
# Logs
*.log
logs/
# Environment files
.env
.env.*
!.env.example
# Test files
tests/
test_*.py
*_test.py
# Docker
Dockerfile*
docker-compose*.yml
.dockerignore
+46
View File
@@ -0,0 +1,46 @@
# Skill Seekers Docker Environment Configuration
# Copy this file to .env and fill in your API keys
# Claude AI / Anthropic API
# Required for AI enhancement features
# Get your key from: https://console.anthropic.com/
ANTHROPIC_API_KEY=sk-ant-your-key-here
# Google Gemini API (Optional)
# Required for Gemini platform support
# Get your key from: https://makersuite.google.com/app/apikey
GOOGLE_API_KEY=
# OpenAI API (Optional)
# Required for OpenAI/ChatGPT platform support
# Get your key from: https://platform.openai.com/api-keys
OPENAI_API_KEY=
# GitHub Token (Optional, but recommended)
# Increases rate limits from 60/hour to 5000/hour
# Create token at: https://github.com/settings/tokens
# Required scopes: public_repo (for public repos)
GITHUB_TOKEN=
# MCP Server Configuration
MCP_TRANSPORT=http
MCP_PORT=8765
# Docker Resource Limits (Optional)
# Uncomment to set custom limits
# DOCKER_CPU_LIMIT=2.0
# DOCKER_MEMORY_LIMIT=4g
# Pinecone API (Optional)
# Required for Pinecone vector database upload
# Get your key from: https://app.pinecone.io/
PINECONE_API_KEY=
# Vector Database Ports (Optional - change if needed)
# WEAVIATE_PORT=8080
# QDRANT_PORT=6333
# CHROMA_PORT=8000
# Logging (Optional)
# SKILL_SEEKERS_LOG_LEVEL=INFO
# SKILL_SEEKERS_LOG_FILE=/data/logs/skill-seekers.log
+4
View File
@@ -0,0 +1,4 @@
# GitHub Sponsors configuration
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository
buy_me_a_coffee: yusufkaraaslan
+52
View File
@@ -0,0 +1,52 @@
---
name: Bug Report
about: Report a bug or issue with Skill Seekers
title: '[BUG] '
labels: 'type: bug'
assignees: ''
---
## 🐛 Bug Description
A clear and concise description of what the bug is.
## 🔄 Steps to Reproduce
1. Go to '...'
2. Run command '...'
3. See error
## ✅ Expected Behavior
What you expected to happen.
## ❌ Actual Behavior
What actually happened.
## 📋 Environment
- **OS:** [e.g., macOS 14.0, Ubuntu 22.04, Windows 11]
- **Python Version:** [e.g., 3.10, 3.11]
- **Skill Seekers Version:** [e.g., v1.0.0]
- **Installation Method:** [pip, git clone, etc.]
## 📊 Error Output
```
Paste the full error message or traceback here
```
## 📸 Screenshots
If applicable, add screenshots to help explain the problem.
## 🔍 Additional Context
- Config file used (if applicable)
- Documentation URL being scraped
- Any custom modifications made
## 🎯 Possible Solution
If you have an idea of how to fix this, please share!
+41
View File
@@ -0,0 +1,41 @@
---
name: Documentation Improvement
about: Suggest improvements to documentation
title: '[DOCS] '
labels: 'type: documentation'
assignees: ''
---
## 📚 Documentation Issue
What documentation needs to be improved, added, or fixed?
## 📍 Location
- **File:** [e.g., README.md, docs/CLAUDE.md]
- **Section:** [e.g., Installation, Configuration]
- **URL:** [if applicable]
## ❌ Current State
Describe what's currently unclear, missing, or incorrect.
## ✅ Proposed Improvement
How should the documentation be changed?
## 🎯 Target Audience
Who would benefit from this documentation improvement?
- [ ] New users
- [ ] Advanced users
- [ ] Contributors
- [ ] API users
## 📝 Additional Context
Any additional context or examples that would help.
## 🔗 Related Issues
Link to any related issues or PRs.
+39
View File
@@ -0,0 +1,39 @@
---
name: Feature Request
about: Suggest a new feature for Skill Seekers
title: '[FEATURE] '
labels: 'type: feature'
assignees: ''
---
## 🚀 Feature Description
A clear and concise description of the feature you'd like to see.
## 💡 Use Case
Describe the problem this feature would solve. What is the user trying to accomplish?
## 📋 Proposed Solution
Describe how you envision this feature working.
## 🔄 Alternatives Considered
Have you considered any alternative solutions or workarounds?
## 📊 Expected Impact
- **Priority:** Low / Medium / High / Critical
- **Effort:** XS / S / M / L / XL
- **Users Affected:** Describe who would benefit
## 📝 Additional Context
Add any other context, screenshots, or examples about the feature request.
## ✅ Acceptance Criteria
- [ ] Criteria 1
- [ ] Criteria 2
- [ ] Criteria 3
+42
View File
@@ -0,0 +1,42 @@
---
name: MCP Tool Request
about: Suggest a new tool for the MCP server
title: '[MCP] Add tool: '
labels: mcp, enhancement
assignees: ''
---
## Tool Name
<!-- e.g., auto_detect_selectors -->
## Tool Description
<!-- What does this tool do? -->
## Input Parameters
```json
{
"param1": {
"type": "string",
"description": "...",
"required": true
}
}
```
## Expected Output
<!-- What should the tool return? -->
## Use Case Example
<!-- How would users interact with this tool? -->
```
User: "Auto-detect selectors for https://docs.example.com"
Tool: Analyzes page structure and suggests optimal selectors
```
## CLI Integration
<!-- Which CLI tool does this wrap? Or is it new logic? -->
- [ ] Wraps existing CLI tool: `cli/tool_name.py`
- [ ] New functionality
## Implementation Notes
<!-- Technical details, dependencies, etc. -->
+49
View File
@@ -0,0 +1,49 @@
# Pull Request
## 📋 Description
Brief description of changes made.
## 🔗 Related Issues
Closes #(issue number)
Relates to #(issue number)
## 🎯 Type of Change
- [ ] 🐛 Bug fix (non-breaking change which fixes an issue)
- [ ] ✨ New feature (non-breaking change which adds functionality)
- [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] 📚 Documentation update
- [ ] ♻️ Code refactoring
- [ ] ⚡ Performance improvement
- [ ] 🧪 Test update
## ✅ Checklist
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] Documentation updated (README, examples, AGENTS.md if CLI changed)
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published
## 🧪 Testing
Describe the tests you ran to verify your changes.
**Test Configuration:**
- Python version:
- OS:
- Dependencies installed:
## 📸 Screenshots (if applicable)
Add screenshots to demonstrate visual changes.
## 📝 Additional Notes
Any additional information reviewers should know.
+43
View File
@@ -0,0 +1,43 @@
#!/bin/bash
# Script to create GitHub issues via web browser
# Since gh CLI is not available, we'll open browser to create issues
REPO="yusufkaraaslan/Skill_Seekers"
BASE_URL="https://github.com/${REPO}/issues/new"
echo "🚀 Creating GitHub Issues for Skill Seeker MCP Development"
echo "=========================================================="
echo ""
echo "Opening browser to create issues..."
echo "Please copy the content from .github/ISSUES_TO_CREATE.md"
echo ""
# Issue 1: Fix test failures
echo "📝 Issue 1: Fix 3 test failures"
echo "URL: ${BASE_URL}?labels=bug,tests,good+first+issue&title=Fix+3+test+failures+(warnings+vs+errors+handling)"
echo ""
# Issue 2: MCP setup guide
echo "📝 Issue 2: Create MCP setup guide"
echo "URL: ${BASE_URL}?labels=documentation,mcp,enhancement&title=Create+comprehensive+MCP+setup+guide+for+Claude+Code"
echo ""
# Issue 3: Test MCP server
echo "📝 Issue 3: Test MCP server"
echo "URL: ${BASE_URL}?labels=testing,mcp,priority-high&title=Test+MCP+server+with+actual+Claude+Code+instance"
echo ""
# Issue 4: Update documentation
echo "📝 Issue 4: Update documentation"
echo "URL: ${BASE_URL}?labels=documentation,breaking-change&title=Update+all+documentation+for+new+monorepo+structure"
echo ""
echo "=========================================================="
echo "📋 Instructions:"
echo "1. Click each URL above (or copy to browser)"
echo "2. Copy the issue body from .github/ISSUES_TO_CREATE.md"
echo "3. Paste into the issue description"
echo "4. Click 'Submit new issue'"
echo ""
echo "Or use this quick link to view all templates:"
echo "cat .github/ISSUES_TO_CREATE.md"
+139
View File
@@ -0,0 +1,139 @@
# Docker Image Publishing - Automated builds and pushes to Docker Hub
# Security Note: Uses secrets for Docker Hub credentials. Matrix values are hardcoded.
# Triggers: push/pull_request/workflow_dispatch only. No untrusted input.
name: Docker Publish
on:
push:
branches: [ main ]
tags:
- 'v*'
pull_request:
branches: [ main ]
paths:
- 'Dockerfile*'
- 'docker-compose.yml'
- 'src/**'
- 'pyproject.toml'
workflow_dispatch:
env:
DOCKER_REGISTRY: docker.io
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
jobs:
build-and-push:
name: Build and Push Docker Images
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
image:
- name: skill-seekers
dockerfile: Dockerfile
description: "Skill Seekers CLI - Convert documentation to AI skills"
- name: skill-seekers-mcp
dockerfile: Dockerfile.mcp
description: "Skill Seekers MCP Server - 25 tools for AI assistants"
env:
IMAGE_NAME: ${{ matrix.image.name }}
IMAGE_DOCKERFILE: ${{ matrix.image.dockerfile }}
IMAGE_DESCRIPTION: ${{ matrix.image.description }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Log in to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=raw,value=latest,enable={{is_default_branch}}
- name: Build and push Docker image
uses: docker/build-push-action@v4
with:
context: .
file: ${{ env.IMAGE_DOCKERFILE }}
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
platforms: linux/amd64,linux/arm64
- name: Create image summary
run: |
echo "## 🐳 Docker Image: $IMAGE_NAME" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Description:** $IMAGE_DESCRIPTION" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Tags:**" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
echo "${{ steps.meta.outputs.tags }}" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
test-images:
name: Test Docker Images
needs: build-and-push
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build CLI image
run: |
docker build -t skill-seekers:test -f Dockerfile .
- name: Test CLI image
run: |
echo "🧪 Testing CLI image..."
docker run --rm skill-seekers:test skill-seekers --version
docker run --rm skill-seekers:test skill-seekers --help
- name: Build MCP image
run: |
docker build -t skill-seekers-mcp:test -f Dockerfile.mcp .
- name: Test MCP image
run: |
echo "🧪 Testing MCP server image..."
# Start MCP server in background
docker run -d --name mcp-test -p 8765:8765 skill-seekers-mcp:test
# Wait for server to start
sleep 10
# Check health
curl -f http://localhost:8765/health || exit 1
# Stop container
docker stop mcp-test
docker rm mcp-test
- name: Test Docker Compose
run: |
echo "🧪 Testing Docker Compose..."
docker compose config
echo "✅ Docker Compose configuration valid"
+171
View File
@@ -0,0 +1,171 @@
# Security Note: This workflow uses workflow_dispatch inputs and pull_request events.
# All untrusted inputs are accessed via environment variables (env:) as recommended.
# No direct usage of github.event.issue/comment/review content in run: commands.
name: Quality Metrics Dashboard
on:
workflow_dispatch:
inputs:
skill_dir:
description: 'Path to skill directory to analyze (e.g., output/react)'
required: true
type: string
fail_threshold:
description: 'Minimum quality score to pass (default: 70)'
required: false
default: '70'
type: string
pull_request:
paths:
- 'output/**'
- 'configs/**'
jobs:
analyze:
name: Quality Metrics Analysis
runs-on: ubuntu-latest
env:
SKILL_DIR_INPUT: ${{ github.event.inputs.skill_dir }}
FAIL_THRESHOLD_INPUT: ${{ github.event.inputs.fail_threshold }}
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
- name: Find skill directories
id: find_skills
run: |
if [ -n "$SKILL_DIR_INPUT" ]; then
# Manual trigger with specific directory
echo "dirs=$SKILL_DIR_INPUT" >> $GITHUB_OUTPUT
else
# PR trigger - find all skill directories
DIRS=$(find output -maxdepth 1 -type d -name "*" ! -name "output" | tr '\n' ' ' || echo "")
if [ -z "$DIRS" ]; then
echo "No skill directories found"
echo "dirs=" >> $GITHUB_OUTPUT
else
echo "dirs=$DIRS" >> $GITHUB_OUTPUT
fi
fi
- name: Analyze quality metrics
id: quality
run: |
DIRS="${{ steps.find_skills.outputs.dirs }}"
THRESHOLD="${FAIL_THRESHOLD_INPUT:-70}"
if [ -z "$DIRS" ]; then
echo "No directories to analyze"
exit 0
fi
ALL_PASSED=true
SUMMARY_FILE="quality_summary.md"
echo "# 📊 Quality Metrics Dashboard" > $SUMMARY_FILE
echo "" >> $SUMMARY_FILE
echo "**Threshold:** $THRESHOLD/100" >> $SUMMARY_FILE
echo "" >> $SUMMARY_FILE
for skill_dir in $DIRS; do
if [ ! -d "$skill_dir" ]; then
continue
fi
SKILL_NAME=$(basename "$skill_dir")
echo "🔍 Analyzing $SKILL_NAME..."
# Run quality analysis
python3 -c "
import sys
from pathlib import Path
from skill_seekers.cli.quality_metrics import QualityAnalyzer
skill_dir = Path('$skill_dir')
threshold = float('$THRESHOLD')
skill_name = '$SKILL_NAME'
analyzer = QualityAnalyzer(skill_dir)
report = analyzer.generate_report()
formatted = analyzer.format_report(report)
print(formatted)
with open(f'quality_{skill_name}.txt', 'w') as f:
f.write(formatted)
score = report.overall_score.total_score
grade = report.overall_score.grade
status = 'PASS' if score >= threshold else 'FAIL'
summary_line = f'{status} **{skill_name}**: {grade} ({score:.1f}/100)'
print(f'\n{summary_line}')
with open('quality_summary.md', 'a') as f:
f.write(f'{summary_line}\n')
if score < threshold:
print(f'::error file={skill_dir}/SKILL.md::Quality score {score:.1f} is below threshold {threshold}')
sys.exit(1)
elif score < 80:
print(f'::warning file={skill_dir}/SKILL.md::Quality score {score:.1f} could be improved')
else:
print(f'::notice file={skill_dir}/SKILL.md::Quality score {score:.1f} - Excellent!')
"
if [ $? -ne 0 ]; then
ALL_PASSED=false
fi
echo "" >> $SUMMARY_FILE
done
if [ "$ALL_PASSED" = false ]; then
echo "❌ Some skills failed quality thresholds"
exit 1
else
echo "✅ All skills passed quality thresholds"
fi
- name: Upload quality reports
uses: actions/upload-artifact@v4
with:
name: quality-metrics-reports
path: quality_*.txt
retention-days: 30
continue-on-error: true
- name: Post summary to PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const summary = fs.readFileSync('quality_summary.md', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: summary
});
continue-on-error: true
- name: Create dashboard summary
run: |
if [ -f "quality_summary.md" ]; then
cat quality_summary.md >> $GITHUB_STEP_SUMMARY
fi
+125
View File
@@ -0,0 +1,125 @@
name: Release
on:
push:
tags:
- 'v*'
permissions:
contents: write
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: 'recursive'
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
if [ -f skill_seeker_mcp/requirements.txt ]; then pip install -r skill_seeker_mcp/requirements.txt; fi
# Install package in editable mode for tests (required for src/ layout)
pip install -e .
- name: Run tests
timeout-minutes: 30
run: |
pip install pytest-timeout
python -m pytest tests/ -v \
-m "not slow and not integration and not e2e and not network and not serial and not mcp_only" \
--timeout=120
- name: Extract version from tag
id: get_version
run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
- name: Verify Python version
run: |
python --version
python -c "import sys; assert sys.version_info >= (3, 10), f'Python {sys.version} is not >= 3.10'"
- name: Verify version consistency
run: |
TAG_VERSION="${{ steps.get_version.outputs.VERSION }}"
PKG_VERSION=$(python -c "import skill_seekers; print(skill_seekers.__version__)")
TOML_VERSION=$(grep -m1 '^version' pyproject.toml | sed 's/version *= *"\(.*\)"/\1/')
echo "Tag version: $TAG_VERSION"
echo "Package version: $PKG_VERSION"
echo "TOML version: $TOML_VERSION"
if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
echo "::error::Version mismatch! Tag=$TAG_VERSION but package reports=$PKG_VERSION"
exit 1
fi
if [ "$TAG_VERSION" != "$TOML_VERSION" ]; then
echo "::error::Version mismatch! Tag=$TAG_VERSION but pyproject.toml has=$TOML_VERSION"
exit 1
fi
echo "✅ All versions match: $TAG_VERSION"
- name: Create Release Notes
id: release_notes
run: |
if [ -f CHANGELOG.md ]; then
# Extract changelog for this version (escape dots for exact match)
VERSION="${{ steps.get_version.outputs.VERSION }}"
ESCAPED_VERSION=$(echo "$VERSION" | sed 's/\./\\./g')
sed -n "/## \[${ESCAPED_VERSION}\]/,/## \[/p" CHANGELOG.md | sed '$d' > release_notes.md
fi
# Fallback if extraction produced empty file or CHANGELOG.md missing
if [ ! -s release_notes.md ]; then
echo "Release v${{ steps.get_version.outputs.VERSION }}" > release_notes.md
fi
- name: Check if release exists
id: check_release
run: |
if gh release view ${{ github.ref_name }} > /dev/null 2>&1; then
echo "exists=true" >> $GITHUB_OUTPUT
else
echo "exists=false" >> $GITHUB_OUTPUT
fi
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Create GitHub Release
if: steps.check_release.outputs.exists == 'false'
uses: softprops/action-gh-release@v1
with:
name: v${{ steps.get_version.outputs.VERSION }}
tag_name: ${{ github.ref_name }}
body_path: release_notes.md
draft: false
prerelease: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Skip Release Creation
if: steps.check_release.outputs.exists == 'true'
run: |
echo "️ Release ${{ github.ref_name }} already exists, skipping creation"
echo "View at: https://github.com/${{ github.repository }}/releases/tag/${{ github.ref_name }}"
- name: Build package
run: |
uv build
- name: Publish to PyPI
env:
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
run: |
uv publish --token $UV_PUBLISH_TOKEN
+197
View File
@@ -0,0 +1,197 @@
# Automated Skill Updates - Runs weekly to refresh documentation
# Security Note: Schedule triggers with hardcoded constants. Workflow_dispatch input
# accessed via FRAMEWORKS_INPUT env variable (safe pattern).
name: Scheduled Skill Updates
on:
schedule:
# Run every Sunday at 3 AM UTC
- cron: '0 3 * * 0'
workflow_dispatch:
inputs:
frameworks:
description: 'Frameworks to update (comma-separated or "all")'
required: false
default: 'all'
type: string
jobs:
update-skills:
name: Update ${{ matrix.framework }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
# Popular frameworks to keep updated
framework:
- react
- django
- fastapi
- godot
- vue
- flask
env:
FRAMEWORK: ${{ matrix.framework }}
FRAMEWORKS_INPUT: ${{ github.event.inputs.frameworks }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
- name: Check if framework should be updated
id: should_update
run: |
FRAMEWORKS_INPUT="${FRAMEWORKS_INPUT:-all}"
if [ "$FRAMEWORKS_INPUT" = "all" ] || [ -z "$FRAMEWORKS_INPUT" ]; then
echo "update=true" >> $GITHUB_OUTPUT
elif echo "$FRAMEWORKS_INPUT" | grep -q "$FRAMEWORK"; then
echo "update=true" >> $GITHUB_OUTPUT
else
echo "update=false" >> $GITHUB_OUTPUT
echo "⏭️ Skipping $FRAMEWORK (not in update list)"
fi
- name: Check for existing skill
if: steps.should_update.outputs.update == 'true'
id: check_existing
run: |
SKILL_DIR="output/$FRAMEWORK"
if [ -d "$SKILL_DIR" ]; then
echo "exists=true" >> $GITHUB_OUTPUT
echo "📦 Found existing skill at $SKILL_DIR"
else
echo "exists=false" >> $GITHUB_OUTPUT
echo "🆕 No existing skill found"
fi
- name: Incremental update (if exists)
if: steps.should_update.outputs.update == 'true' && steps.check_existing.outputs.exists == 'true'
run: |
echo "⚡ Performing incremental update for $FRAMEWORK..."
SKILL_DIR="output/$FRAMEWORK"
# Detect changes using incremental updater
python3 -c "
import sys, os
from pathlib import Path
from skill_seekers.cli.incremental_updater import IncrementalUpdater
framework = os.environ['FRAMEWORK']
skill_dir = Path(f'output/{framework}')
updater = IncrementalUpdater(skill_dir)
changes = updater.detect_changes()
if changes.has_changes:
print(f'Changes detected:')
print(f' Added: {len(changes.added)}')
print(f' Modified: {len(changes.modified)}')
print(f' Deleted: {len(changes.deleted)}')
updater.current_versions = updater._scan_documents()
updater.save_current_versions()
else:
print('No changes detected, skill is up to date')
"
- name: Full scrape (if new or manual)
if: steps.should_update.outputs.update == 'true' && steps.check_existing.outputs.exists == 'false'
run: |
echo "📥 Performing full scrape for $FRAMEWORK..."
CONFIG_FILE="configs/${FRAMEWORK}.json"
if [ ! -f "$CONFIG_FILE" ]; then
echo "⚠️ Config not found: $CONFIG_FILE"
exit 0
fi
# Use streaming ingestion for large docs
skill-seekers create "$CONFIG_FILE" --max-pages 200
- name: Generate quality report
if: steps.should_update.outputs.update == 'true'
run: |
SKILL_DIR="output/$FRAMEWORK"
if [ ! -d "$SKILL_DIR" ]; then
echo "⚠️ Skill directory not found"
exit 0
fi
echo "📊 Generating quality metrics..."
python3 -c "
import sys, os
from pathlib import Path
from skill_seekers.cli.quality_metrics import QualityAnalyzer
framework = os.environ['FRAMEWORK']
skill_dir = Path(f'output/{framework}')
analyzer = QualityAnalyzer(skill_dir)
report = analyzer.generate_report()
print(f'Quality Score: {report.overall_score.grade} ({report.overall_score.total_score:.1f}/100)')
print(f' Completeness: {report.overall_score.completeness:.1f}%')
print(f' Accuracy: {report.overall_score.accuracy:.1f}%')
print(f' Coverage: {report.overall_score.coverage:.1f}%')
print(f' Health: {report.overall_score.health:.1f}%')
"
- name: Package for Claude
if: steps.should_update.outputs.update == 'true'
run: |
SKILL_DIR="output/$FRAMEWORK"
if [ -d "$SKILL_DIR" ]; then
echo "📦 Packaging $FRAMEWORK for Claude AI..."
skill-seekers package "$SKILL_DIR" --target claude
fi
- name: Upload updated skill
if: steps.should_update.outputs.update == 'true'
uses: actions/upload-artifact@v4
with:
name: ${{ env.FRAMEWORK }}-skill-updated
path: output/${{ env.FRAMEWORK }}.zip
retention-days: 90
summary:
name: Update Summary
needs: update-skills
runs-on: ubuntu-latest
if: always()
steps:
- name: Create summary
run: |
echo "## 🔄 Scheduled Skills Update" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Date:** $(date -u '+%Y-%m-%d %H:%M UTC')" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "### Updated Frameworks" >> $GITHUB_STEP_SUMMARY
echo "- React" >> $GITHUB_STEP_SUMMARY
echo "- Django" >> $GITHUB_STEP_SUMMARY
echo "- FastAPI" >> $GITHUB_STEP_SUMMARY
echo "- Godot" >> $GITHUB_STEP_SUMMARY
echo "- Vue" >> $GITHUB_STEP_SUMMARY
echo "- Flask" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "Updated skills available in workflow artifacts." >> $GITHUB_STEP_SUMMARY
+112
View File
@@ -0,0 +1,112 @@
# Security Note: This workflow uses only push/pull_request/workflow_dispatch triggers.
# Matrix values are hardcoded constants. No untrusted input is used in run: commands.
name: Test Vector Database Adaptors
on:
push:
branches: [ main, development ]
paths:
- 'src/skill_seekers/cli/adaptors/**'
- 'src/skill_seekers/mcp/tools/vector_db_tools.py'
- 'tests/test_*adaptor*.py'
- 'tests/test_mcp_vector_dbs.py'
- 'tests/test_docker_smoke.py'
- 'tests/test_upload_integration.py'
pull_request:
branches: [ main, development ]
paths:
- 'src/skill_seekers/cli/adaptors/**'
- 'src/skill_seekers/mcp/tools/vector_db_tools.py'
- 'tests/test_*adaptor*.py'
- 'tests/test_mcp_vector_dbs.py'
- 'tests/test_docker_smoke.py'
- 'tests/test_upload_integration.py'
workflow_dispatch:
jobs:
test-adaptors:
name: Test ${{ matrix.adaptor }} Adaptor
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
adaptor: [weaviate, chroma, faiss, qdrant]
python-version: ['3.10', '3.12']
env:
ADAPTOR_NAME: ${{ matrix.adaptor }}
PYTHON_VERSION: ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest
pip install -e .
- name: Run adaptor tests
run: |
echo "🧪 Testing $ADAPTOR_NAME adaptor..."
python -m pytest "tests/test_adaptors/test_${ADAPTOR_NAME}_adaptor.py" -v --tb=short
- name: Test adaptor integration
run: |
echo "🔗 Testing $ADAPTOR_NAME integration..."
# Create test skill
mkdir -p test_skill/references
echo "# Test Skill" > test_skill/SKILL.md
echo "Test content" >> test_skill/SKILL.md
echo "# Reference" > test_skill/references/ref.md
# Test adaptor packaging
python3 -c "
import os
from pathlib import Path
from skill_seekers.cli.adaptors import get_adaptor
adaptor_name = os.environ['ADAPTOR_NAME']
adaptor = get_adaptor(adaptor_name)
package_path = adaptor.package(Path('test_skill'), Path('.'))
print(f'Package created: {package_path}')
assert package_path.exists(), 'Package file not created'
print(f'Package size: {package_path.stat().st_size} bytes')
"
- name: Upload test package
uses: actions/upload-artifact@v4
with:
name: test-package-${{ env.ADAPTOR_NAME }}-py${{ env.PYTHON_VERSION }}
path: test_skill-${{ env.ADAPTOR_NAME }}.json
retention-days: 7
test-mcp-tools:
name: Test MCP Vector DB Tools
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest
pip install -e .
- name: Run MCP vector DB tests
run: |
echo "🧪 Testing MCP vector database tools..."
python -m pytest tests/test_mcp_vector_dbs.py -v --tb=short
+155
View File
@@ -0,0 +1,155 @@
name: Tests
on:
push:
branches: [ main, development ]
pull_request:
branches: [ main, development ]
jobs:
lint:
name: Code Quality (Ruff & Mypy)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install "ruff==0.15.8" mypy
pip install -e .
- name: Run ruff linter
run: ruff check src/ tests/ --output-format=github
- name: Run ruff formatter check
run: ruff format --check src/ tests/
- name: Run mypy type checker
run: mypy src/skill_seekers --show-error-codes --pretty
continue-on-error: true
test-fast:
name: Fast Unit Tests (parallel)
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python-version: ['3.10', '3.11', '3.12']
exclude:
- os: macos-latest
python-version: '3.10'
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip packages
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest-xdist pytest-timeout
pip install -e .
- name: Run fast tests (xdist parallel)
run: |
python -m pytest tests/ -n auto --dist=loadfile \
-m "not slow and not integration and not e2e and not network and not serial and not mcp_only" \
-q --timeout=120 --cov=src/skill_seekers --cov-report=xml --cov-report=term
timeout-minutes: 15
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
file: ./coverage.xml
flags: unittests
name: codecov-umbrella
fail_ci_if_error: false
test-serial:
name: Serial / Integration / E2E Tests
runs-on: ubuntu-latest
needs: [test-fast]
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest-timeout
pip install -e .
- name: Run serial/integration/E2E tests
run: |
python -m pytest tests/ \
--ignore=tests/test_bootstrap_skill_e2e.py \
--ignore=tests/test_bootstrap_skill.py \
-m "(integration or e2e or slow or network or serial) and not mcp_only" \
-v --timeout=300
timeout-minutes: 25
test-mcp:
name: MCP Server Tests
runs-on: ubuntu-latest
needs: [test-fast]
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest-timeout
pip install -e ".[mcp]"
pip install -e .
- name: Run MCP tests
run: |
python -m pytest tests/ \
-m "mcp_only and not network" \
-v --timeout=180
timeout-minutes: 10
tests-complete:
name: Tests
needs: [lint, test-fast, test-serial, test-mcp]
runs-on: ubuntu-latest
if: always()
steps:
- name: Check all results
run: |
if [ "${{ needs.lint.result }}" != "success" ]; then
echo "❌ Code quality checks failed!"
exit 1
fi
if [ "${{ needs.test-fast.result }}" != "success" ]; then
echo "❌ Fast tests failed!"
exit 1
fi
if [ "${{ needs.test-serial.result }}" == "failure" ]; then
echo "❌ Serial/integration tests failed!"
exit 1
fi
if [ "${{ needs.test-mcp.result }}" == "failure" ]; then
echo "❌ MCP tests failed!"
exit 1
fi
echo "✅ All checks passed!"
+188
View File
@@ -0,0 +1,188 @@
name: Vector Database Export
on:
workflow_dispatch:
inputs:
skill_name:
description: 'Skill name to export (e.g., react, django, godot)'
required: true
type: string
targets:
description: 'Vector databases to export (comma-separated: weaviate,chroma,faiss,qdrant or "all")'
required: true
default: 'all'
type: string
config_path:
description: 'Path to config file (optional, auto-detected from skill_name if not provided)'
required: false
type: string
schedule:
# Run weekly on Sunday at 2 AM UTC for popular frameworks
- cron: '0 2 * * 0'
jobs:
export:
name: Export to Vector Databases
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
# For scheduled runs, export popular frameworks
skill: ${{ github.event_name == 'schedule' && fromJson('["react", "django", "godot", "fastapi"]') || fromJson(format('["{0}"]', github.event.inputs.skill_name)) }}
env:
SKILL_NAME: ${{ matrix.skill }}
TARGETS_INPUT: ${{ github.event.inputs.targets }}
CONFIG_PATH_INPUT: ${{ github.event.inputs.config_path }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
- name: Determine config path
id: config
run: |
if [ -n "$CONFIG_PATH_INPUT" ]; then
echo "path=$CONFIG_PATH_INPUT" >> $GITHUB_OUTPUT
else
echo "path=configs/$SKILL_NAME.json" >> $GITHUB_OUTPUT
fi
- name: Check if config exists
id: check_config
run: |
CONFIG_FILE="${{ steps.config.outputs.path }}"
if [ -f "$CONFIG_FILE" ]; then
echo "exists=true" >> $GITHUB_OUTPUT
else
echo "exists=false" >> $GITHUB_OUTPUT
echo "⚠️ Config not found: $CONFIG_FILE"
fi
- name: Scrape documentation
if: steps.check_config.outputs.exists == 'true'
run: |
echo "📥 Scraping documentation for $SKILL_NAME..."
skill-seekers create "${{ steps.config.outputs.path }}" --max-pages 100
continue-on-error: true
- name: Determine export targets
id: targets
run: |
TARGETS="${TARGETS_INPUT:-all}"
if [ "$TARGETS" = "all" ]; then
echo "list=weaviate chroma faiss qdrant" >> $GITHUB_OUTPUT
else
echo "list=$(echo "$TARGETS" | tr ',' ' ')" >> $GITHUB_OUTPUT
fi
- name: Export to vector databases
if: steps.check_config.outputs.exists == 'true'
env:
EXPORT_TARGETS: ${{ steps.targets.outputs.list }}
run: |
SKILL_DIR="output/$SKILL_NAME"
if [ ! -d "$SKILL_DIR" ]; then
echo "❌ Skill directory not found: $SKILL_DIR"
exit 1
fi
echo "📦 Exporting $SKILL_NAME to vector databases..."
for target in $EXPORT_TARGETS; do
echo ""
echo "🔹 Exporting to $target..."
# Use adaptor directly via CLI
python3 -c "
from pathlib import Path
from skill_seekers.cli.adaptors import get_adaptor
adaptor = get_adaptor('$target')
package_path = adaptor.package(Path('$SKILL_DIR'), Path('output'))
print(f'Exported to {package_path}')
"
if [ $? -eq 0 ]; then
echo "✅ $target export complete"
else
echo "❌ $target export failed"
fi
done
- name: Generate quality report
if: steps.check_config.outputs.exists == 'true'
run: |
SKILL_DIR="output/$SKILL_NAME"
if [ -d "$SKILL_DIR" ]; then
echo "📊 Generating quality metrics..."
python3 -c "
from pathlib import Path
from skill_seekers.cli.quality_metrics import QualityAnalyzer
analyzer = QualityAnalyzer(Path('$SKILL_DIR'))
report = analyzer.generate_report()
formatted = analyzer.format_report(report)
print(formatted)
with open('quality_report_${SKILL_NAME}.txt', 'w') as f:
f.write(formatted)
"
fi
continue-on-error: true
- name: Upload vector database exports
if: steps.check_config.outputs.exists == 'true'
uses: actions/upload-artifact@v4
with:
name: ${{ env.SKILL_NAME }}-vector-exports
path: |
output/${{ env.SKILL_NAME }}-*.json
retention-days: 30
- name: Upload quality report
if: steps.check_config.outputs.exists == 'true'
uses: actions/upload-artifact@v4
with:
name: ${{ env.SKILL_NAME }}-quality-report
path: quality_report_${{ env.SKILL_NAME }}.txt
retention-days: 30
continue-on-error: true
- name: Create export summary
if: steps.check_config.outputs.exists == 'true'
env:
EXPORT_TARGETS: ${{ steps.targets.outputs.list }}
run: |
echo "## 📦 Vector Database Export Summary: $SKILL_NAME" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
for target in $EXPORT_TARGETS; do
FILE="output/${SKILL_NAME}-${target}.json"
if [ -f "$FILE" ]; then
SIZE=$(du -h "$FILE" | cut -f1)
echo "✅ **$target**: $SIZE" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **$target**: Export failed" >> $GITHUB_STEP_SUMMARY
fi
done
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f "quality_report_${SKILL_NAME}.txt" ]; then
echo "### 📊 Quality Metrics" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
head -30 "quality_report_${SKILL_NAME}.txt" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
+79
View File
@@ -0,0 +1,79 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Environment variables (secrets)
.env
.env.local
.env.*.local
# Virtual Environment
.venv/
venv/
ENV/
env/
# Output directory
output/
*.zip
# Skill Seekers cache (intermediate files)
.skillseeker-cache/
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Backups
*.backup
# Testing artifacts
.pytest_cache/
.coverage
htmlcov/
.tox/
*.cover
.hypothesis/
.mypy_cache/
.ruff_cache/
# Build artifacts
.build/
skill-seekers-configs/
.claude/skills
.mcp.json
!/.mcp.json
!distribution/claude-plugin/.mcp.json
settings.json
USER_GUIDE.md
# Local tooling artifacts (per-developer state, not project state)
# chromadb's default persistence directory
chroma_db/
# Superpowers SDK output
docs/superpowers/
+3
View File
@@ -0,0 +1,3 @@
[submodule "api/configs_repo"]
path = api/configs_repo
url = https://github.com/yusufkaraaslan/skill-seekers-configs.git
+8
View File
@@ -0,0 +1,8 @@
{
"mcpServers": {
"skill-seekers": {
"command": "python",
"args": ["-m", "skill_seekers.mcp.server_fastmcp"]
}
}
}
+302
View File
@@ -0,0 +1,302 @@
# AGENTS.md - Skill Seekers
Comprehensive reference for AI coding agents. Skill Seekers is a Python CLI tool (v3.6.0) that converts documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more into AI-ready skills for 21+ LLM platforms and RAG pipelines.
## Project Overview
**Skill Seekers** is a universal preprocessing layer that transforms raw documentation and code into structured knowledge assets. It supports 17+ source types and exports to 21+ AI platforms including Claude, Gemini, OpenAI, LangChain, LlamaIndex, and various vector databases.
### Key Capabilities
- **Source Types (17):** Documentation websites, GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML, OpenAPI specs, AsciiDoc, PowerPoint, Confluence, Notion, RSS feeds, man pages, chat exports
- **Export Targets (21):** Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown, LangChain, LlamaIndex, Haystack, Weaviate, ChromaDB, FAISS, Qdrant, Pinecone
- **MCP Server:** FastMCP-based Model Context Protocol server for AI assistant integration
## Setup
```bash
# REQUIRED before running tests (src/ layout — tests hard-exit if package not installed)
pip install -e .
# With dev tools (pytest, ruff, mypy, coverage)
pip install -e ".[dev]"
# With specific LLM platform support
pip install -e ".[gemini]" # Google Gemini
pip install -e ".[openai]" # OpenAI ChatGPT
pip install -e ".[all-llms]" # All LLM platforms
# With all optional dependencies (except video-full)
pip install -e ".[all]"
# Full video processing (heavy dependencies)
pip install -e ".[video-full]"
```
Note: `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not. Always install in editable mode first.
### Environment Variables
Create a `.env` file or export these variables:
```bash
ANTHROPIC_API_KEY # For Claude AI enhancement
GOOGLE_API_KEY # For Gemini support
OPENAI_API_KEY # For OpenAI support
GITHUB_TOKEN # For GitHub repo scraping (higher rate limits)
```
## Build / Test / Lint
```bash
# Full suite (never skip — all must pass)
pytest tests/ -v
# Fast iteration (skip slow, integration, E2E, network, MCP)
pytest tests/ -m "not slow and not integration and not e2e and not network and not serial and not mcp_only" -q
# Fast parallel (install pytest-xdist first)
pytest tests/ -n auto --dist=loadfile -m "not slow and not integration and not e2e and not network and not serial and not mcp_only" -q
# 3-phase runner script (recommended for local dev)
bash scripts/run_tests_fast.sh
# Single test
pytest tests/test_scraper_features.py::test_detect_language -v
# Skip slow/integration
pytest tests/ -v -m "not slow and not integration"
# With coverage
pytest tests/ --cov=src/skill_seekers --cov-report=term
# Lint + format check (matches CI)
ruff check src/ tests/
ruff format --check src/ tests/
# Type check (non-blocking — mypy is continue-on-error in CI)
mypy src/skill_seekers --show-error-codes --pretty
```
**Pytest config:** `asyncio_mode = "auto"`, so `@pytest.mark.asyncio` is implicit. Test markers: `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`, `serial`, `network`, `mcp_only`.
**CI note:** CI pins `ruff==0.15.8` (not the `>=0.14.13` dev dep). If formatting behaves differently locally, check the CI version.
**CI test phases:** Tests are split into 3 parallel jobs:
- `test-fast` — 3386 unit tests with xdist across OS/Python matrix
- `test-serial` — 69 serial/integration/E2E/network tests
- `test-mcp` — 193 MCP tests (requires `[mcp]` extras)
## Code Style
### Formatting Rules (ruff — from pyproject.toml)
- **Line length:** 100 characters
- **Target Python:** 3.10+
- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM
- **Ignored rules:** E501 (line length handled by formatter), F541 (f-string style), ARG002 (unused method args for interface compliance), B007 (intentional unused loop vars), I001 (formatter handles imports), SIM114 (readability preference)
### Imports
- Sort with isort (via ruff); `skill_seekers` is first-party
- Standard library → third-party → first-party, separated by blank lines
- Use `from __future__ import annotations` only if needed for forward refs
- Guard optional imports with try/except ImportError (see `adaptors/__init__.py` pattern):
```python
try:
from .claude import ClaudeAdaptor
from .minimax import MiniMaxAdaptor
except ImportError:
ClaudeAdaptor = None
MiniMaxAdaptor = None
```
### Naming Conventions
- **Files:** `snake_case.py` (e.g., `source_detector.py`, `config_validator.py`)
- **Classes:** `PascalCase` (e.g., `SkillAdaptor`, `ClaudeAdaptor`, `SourceDetector`)
- **Functions/methods:** `snake_case` (e.g., `get_adaptor()`, `detect_language()`)
- **Constants:** `UPPER_CASE` (e.g., `ADAPTORS`, `DEFAULT_CHUNK_TOKENS`, `VALID_SOURCE_TYPES`)
- **Private:** prefix with `_` (e.g., `_read_existing_content()`, `_validate_unified()`)
### Type Hints
- Gradual typing — add hints where practical, not enforced everywhere
- Use modern syntax: `str | None` not `Optional[str]`, `list[str]` not `List[str]`
- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`, `ignore_missing_imports = true`
- Tests are excluded from strict type checking (`disallow_untyped_defs = false`, `check_untyped_defs = false` for `tests.*`)
### Docstrings
- Module-level docstring on every file (triple-quoted, describes purpose)
- Google-style docstrings for public functions/classes
- Include `Args:`, `Returns:`, `Raises:` sections where useful
### Error Handling
- Use specific exceptions, never bare `except:`
- Provide helpful error messages with context
- Use `raise ValueError(...)` for invalid arguments, `raise RuntimeError(...)` for state errors
- Guard optional dependency imports with try/except and give clear install instructions on failure
- Chain exceptions with `raise ... from e` when wrapping
### Suppressing Lint Warnings
- Use inline `# noqa: XXXX` comments (e.g., `# noqa: F401` for re-exports, `# noqa: ARG001` for required but unused params)
## Project Layout
```
src/skill_seekers/ # Main package (src/ layout)
cli/ # CLI commands and entry points (100+ files)
adaptors/ # Platform adaptors (Strategy pattern, inherit SkillAdaptor)
arguments/ # CLI argument definitions (one per source type)
parsers/ # Subcommand parsers (one per source type)
storage/ # Cloud storage (inherit BaseStorageAdaptor)
main.py # Unified CLI entry point (COMMAND_MODULES dict)
source_detector.py # Auto-detects source type from user input
create_command.py # Unified `create` command routing
config_validator.py # VALID_SOURCE_TYPES set + per-type validation
unified_scraper.py # Multi-source orchestrator (scraped_data + dispatch)
unified_skill_builder.py # Pairwise synthesis + generic merge
mcp/ # MCP server (FastMCP + legacy)
tools/ # MCP tool implementations by category (10 files)
server_fastmcp.py # FastMCP server implementation
server_legacy.py # Legacy MCP server
sync/ # Sync monitoring (Pydantic models)
benchmark/ # Benchmarking framework
embedding/ # FastAPI embedding server
workflows/ # 67 YAML workflow presets
_version.py # Reads version from pyproject.toml
tests/ # 160 test files (pytest)
test_adaptors/ # 22 adaptor-specific test files
conftest.py # Test configuration with package check
configs/ # Preset JSON scraping configs
docs/ # Documentation (guides, integrations, architecture)
```
## Key Patterns
**Adaptor (Strategy) pattern** — all platform logic in `cli/adaptors/`. Inherit `SkillAdaptor`, implement `format_skill_md()`, `package()`, `upload()`. Register in `adaptors/__init__.py` ADAPTORS dict.
**Scraper pattern** — each source type has: `cli/<type>_scraper.py` (with `<Type>ToSkillConverter` class + `main()`), `arguments/<type>.py`, `parsers/<type>_parser.py`. Register in `parsers/__init__.py` PARSERS list, `main.py` COMMAND_MODULES dict, `config_validator.py` VALID_SOURCE_TYPES set.
**Unified pipeline** — `unified_scraper.py` dispatches to per-type `_scrape_<type>()` methods. `unified_skill_builder.py` uses pairwise synthesis for docs+github+pdf combos and `_generic_merge()` for all other combinations.
**MCP tools** — grouped in `mcp/tools/` by category. `scrape_generic_tool` handles all new source types.
**CLI subcommands** — git-style in `cli/main.py`. Each delegates to a module's `main()` function.
**Supported source types (17):** documentation (web), github, pdf, local, word, video, epub, jupyter, html, openapi, asciidoc, pptx, confluence, notion, rss, manpage, chat. Each detected automatically by `source_detector.py`.
**Supported platforms (21):** claude, gemini, openai, minimax, opencode, kimi, deepseek, qwen, openrouter, together, fireworks, markdown, langchain, llama-index, haystack, weaviate, chroma, faiss, qdrant, pinecone.
## CLI Commands
```bash
# Core commands
skill-seekers create <source> # Create skill from any source (auto-detects type)
skill-seekers scan <dir> # AI-detect a project's tech stack and emit per-framework configs
skill-seekers enhance <directory> # AI-powered enhancement
skill-seekers package <directory> # Package skill for target platform
skill-seekers upload <file> # Upload skill to target platform
skill-seekers install <source> # One-command workflow (scrape + enhance + package + upload)
# Utilities
skill-seekers estimate <source> # Estimate page count before scraping
skill-seekers doctor # Health check for dependencies
skill-seekers config # Configure API keys and settings
skill-seekers workflows # List and apply workflow presets
skill-seekers resume <job_id> # Resume interrupted scraping
# Advanced
skill-seekers stream <source> # Streaming ingestion
skill-seekers update <directory> # Incremental update
skill-seekers multilang <directory> # Multi-language support
```
## Testing Instructions
### Test Structure
- Unit tests: `tests/test_*.py` — test individual modules
- Adaptor tests: `tests/test_adaptors/test_*_adaptor.py` — test platform adaptors
- E2E tests: `tests/test_*_e2e.py` — end-to-end integration tests
### Running Tests
```bash
# Fast test run (skip slow/integration tests)
pytest tests/ -v -m "not slow and not integration"
# Full test suite
pytest tests/ -v
# With coverage report
pytest tests/ --cov=src/skill_seekers --cov-report=term-missing
# Specific test categories
pytest tests/ -v -m "slow" # Only slow tests
pytest tests/ -v -m "integration" # Only integration tests
pytest tests/ -v -m "e2e" # Only E2E tests
```
### Test Fixtures
Test fixtures are located in `tests/fixtures/` and include sample configs, HTML files, and mock data.
## Git Workflow
- **`main`** — production, protected
- **`development`** — default PR target, active dev
- Feature branches created from `development`
## Pre-commit Checklist
```bash
ruff check src/ tests/
ruff format --check src/ tests/
pytest tests/ -v -x # stop on first failure
```
Never commit API keys. Use env vars: `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `GITHUB_TOKEN`.
## CI/CD
GitHub Actions (7 workflows in `.github/workflows/`):
- **tests.yml** — ruff + mypy lint job, then pytest matrix (Ubuntu + macOS, Python 3.10-3.12) with Codecov upload
- **release.yml** — tag-triggered: tests → version verification → PyPI publish via `uv build`
- **test-vector-dbs.yml** — tests vector DB adaptors (weaviate, chroma, faiss, qdrant)
- **docker-publish.yml** — multi-platform Docker builds (amd64, arm64) for CLI + MCP images
- **quality-metrics.yml** — quality analysis with configurable threshold
- **scheduled-updates.yml** — weekly skill updates for popular frameworks
- **vector-db-export.yml** — weekly vector DB exports
## Deployment
### Docker
Multi-stage Dockerfile with Python 3.12 slim base:
```bash
# Build image
docker build -t skill-seekers .
# Run CLI
docker run -v $(pwd)/output:/output skill-seekers create https://docs.example.com
# Run MCP server
docker run -p 8765:8765 skill-seekers skill-seekers-mcp
```
### MCP Server
The MCP server provides Model Context Protocol integration:
```bash
# Start FastMCP server
skill-seekers-mcp
# Or use the Python module
python -m skill_seekers.mcp.server_fastmcp
```
## Security Considerations
- **API Keys:** Never commit API keys to version control. Use environment variables or `.env` files (already in `.gitignore`)
- **Docker:** Runs as non-root user (`skillseeker`, UID 1000)
- **Dependencies:** Regular security updates via `pip audit` or `safety check`
- **Sandboxing:** Video processing uses optional dependencies that can be heavy; install `[video-full]` only when needed
## Additional Resources
- **Website:** https://skillseekersweb.com/
- **Documentation:** https://skillseekersweb.com/
- **PyPI:** https://pypi.org/project/skill-seekers/
- **Repository:** https://github.com/yusufkaraaslan/Skill_Seekers
- **Config Browser:** https://skillseekersweb.com/
- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2
+617
View File
@@ -0,0 +1,617 @@
# Bulletproof Quick Start Guide
**Target Audience:** Complete beginners | Never used Python/git before? Start here!
**Time:** 15-30 minutes total (including all installations)
**Result:** Working Skill Seeker installation + your first Claude skill created
---
## 📋 What You'll Need
Before starting, you need:
- A computer (macOS, Linux, or Windows with WSL)
- Internet connection
- 30 minutes of time
That's it! We'll install everything else together.
---
## Step 1: Install Python (5 minutes)
### Check if You Already Have Python
Open Terminal (macOS/Linux) or Command Prompt (Windows) and type:
```bash
python3 --version
```
**✅ If you see:** `Python 3.10.x` or `Python 3.11.x` or higher → **Skip to Step 2!**
**❌ If you see:** `command not found` or version less than 3.10 → **Continue below**
### Install Python
#### macOS:
```bash
# Install Homebrew (if not installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install Python
brew install python3
```
**Verify:**
```bash
python3 --version
# Should show: Python 3.11.x or similar
```
#### Linux (Ubuntu/Debian):
```bash
sudo apt update
sudo apt install python3 python3-pip
```
**Verify:**
```bash
python3 --version
pip3 --version
```
#### Windows:
1. Download Python from: https://www.python.org/downloads/
2. Run installer
3. **IMPORTANT:** Check "Add Python to PATH" during installation
4. Open Command Prompt and verify:
```bash
python --version
```
**✅ Success looks like:**
```
Python 3.11.5
```
---
## Step 2: Install Git (3 minutes)
### Check if You Have Git
```bash
git --version
```
**✅ If you see:** `git version 2.x.x`**Skip to Step 3!**
**❌ If not installed:**
#### macOS:
```bash
brew install git
```
#### Linux:
```bash
sudo apt install git
```
#### Windows:
Download from: https://git-scm.com/download/win
**Verify:**
```bash
git --version
# Should show: git version 2.x.x
```
---
## Step 3: Get Skill Seeker (2 minutes)
### Choose Where to Put It
Pick a location for the project. Good choices:
- macOS/Linux: `~/Projects/` or `~/Documents/`
- Note: `~` means your home directory (`$HOME` or `/Users/yourname` on macOS, `/home/yourname` on Linux)
- Windows: `C:\Users\YourName\Projects\`
### Clone the Repository
```bash
# Create Projects directory (if it doesn't exist)
mkdir -p ~/Projects
cd ~/Projects
# Clone Skill Seeker
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git
# Enter the directory
cd Skill_Seekers
```
**✅ Success looks like:**
```
Cloning into 'Skill_Seekers'...
remote: Enumerating objects: 245, done.
remote: Counting objects: 100% (245/245), done.
```
**Verify you're in the right place:**
```bash
pwd
# Should show something like:
# macOS: /Users/yourname/Projects/Skill_Seekers
# Linux: /home/yourname/Projects/Skill_Seekers
# (Replace 'yourname' with YOUR actual username)
ls
# Should show: README.md, cli/, mcp/, configs/, etc.
```
**❌ If `git clone` fails:**
```bash
# Check internet connection
ping google.com
# Or download ZIP manually:
# https://github.com/yusufkaraaslan/Skill_Seekers/archive/refs/heads/main.zip
# Then unzip and cd into it
```
---
## Step 4: Setup Virtual Environment & Install Skill Seekers (3 minutes)
A virtual environment keeps Skill Seeker's dependencies isolated and prevents conflicts.
```bash
# Make sure you're in the Skill_Seekers directory
cd ~/Projects/Skill_Seekers # ~ means your home directory ($HOME)
# Adjust if you chose a different location
# Create virtual environment
python3 -m venv venv
# Activate it
source venv/bin/activate # macOS/Linux
# Windows users: venv\Scripts\activate
```
**✅ Success looks like:**
```
(venv) username@computer Skill_Seekers %
```
Notice `(venv)` appears in your prompt - this means the virtual environment is active!
```bash
# Now install Skill Seekers package (this installs all dependencies automatically)
pip install -e .
```
**✅ Success looks like:**
```
Successfully installed skill-seekers-2.7.4 requests-2.32.5 beautifulsoup4-4.14.2 anthropic-0.76.0 ...
Obtaining file:///path/to/Skill_Seekers
Installing collected packages: skill-seekers
Successfully installed skill-seekers
```
**What just happened?**
- `pip install -e .` installs the package in "editable" mode
- The `.` means "current directory" (where pyproject.toml is)
- This automatically installs ALL required dependencies
- This registers the `skill-seekers` command so you can use it from anywhere
- The `-e` flag means changes to the code take effect immediately (useful for development)
**Important Notes:**
- **Every time** you open a new terminal to use Skill Seeker, run `source venv/bin/activate` first (Windows: `venv\Scripts\activate`)
- You'll know it's active when you see `(venv)` in your terminal prompt
- To deactivate later: just type `deactivate`
**❌ If python3 not found:**
```bash
# Try without the 3
python -m venv venv
```
**❌ If permission denied:**
```bash
# Virtual environment approach doesn't need sudo - you might have the wrong path
# Make sure you're in the Skill_Seekers directory:
pwd
# Should show something like:
# macOS: /Users/yourname/Projects/Skill_Seekers
# Linux: /home/yourname/Projects/Skill_Seekers
# (Replace 'yourname' with YOUR actual username)
```
**❌ If "pip: command not found":**
```bash
# Try with python -m pip instead
python3 -m pip install -e .
```
---
## Step 5: Test Your Installation (1 minute)
Let's make sure everything works:
```bash
# Test the CLI is installed correctly
skill-seekers create --help
```
**✅ Success looks like:**
```
usage: skill-seekers create [-h] ...
```
**❌ If you see "command not found":**
```bash
# Ensure the package is installed
pip install -e .
# Verify installation
skill-seekers --version
```
---
## Step 6: Create Your First Skill! (5-10 minutes)
Let's create a simple skill using a preset configuration.
### Option A: Small Test (Recommended First Time)
```bash
# Create a config for a small site first
cat > configs/test.json << 'EOF'
{
"name": "test-skill",
"description": "Test skill creation",
"base_url": "https://tailwindcss.com/docs/installation",
"selectors": {
"main_content": "#content-wrapper",
"title": "h1, h2, h3",
"code_blocks": "pre code"
},
"max_pages": 5,
"rate_limit": 0.5
}
EOF
# Run the scraper
skill-seekers create --config configs/test.json
```
**Note for Windows users:** The `cat > file << 'EOF'` syntax doesn't work in PowerShell. Instead, create the file manually:
```powershell
# In PowerShell, create configs/test.json with this content:
@"
{
"name": "test-skill",
"description": "Test skill creation",
"base_url": "https://tailwindcss.com/docs/installation",
"selectors": {
"main_content": "#content-wrapper",
"title": "h1, h2, h3",
"code_blocks": "pre code"
},
"max_pages": 5,
"rate_limit": 0.5
}
"@ | Out-File -FilePath configs/test.json -Encoding utf8
# Then run the scraper
skill-seekers create --config configs/test.json
```
**What happens:**
1. Scrapes 5 pages from Tailwind CSS docs
2. Creates `output/test-skill/` directory
3. Generates SKILL.md and reference files
**⏱️ Time:** ~30 seconds
**✅ Success looks like:**
```
Scraping: https://tailwindcss.com/docs/installation
Page 1/5: Installation
Page 2/5: Editor Setup
...
✅ Skill created at: output/test-skill/
```
### Option B: Full Example (React Docs)
```bash
# Use the React preset
skill-seekers create --config configs/react.json --max-pages 50
```
**⏱️ Time:** ~5 minutes
**What you get:**
- `output/react/SKILL.md` - Main skill file
- `output/react/references/` - Organized documentation
### Verify It Worked
```bash
# Check the output
ls output/test-skill/
# Should show: SKILL.md, references/, scripts/, assets/
# Look at the generated skill
head output/test-skill/SKILL.md
```
---
## Step 7: Package for Claude (30 seconds)
```bash
# Package the skill
skill-seekers package output/test-skill/
```
**✅ Success looks like:**
```
✅ Skill packaged successfully!
📦 Created: output/test-skill.zip
📏 Size: 45.2 KB
Ready to upload to Claude AI!
```
**Now you have:** `output/test-skill.zip` ready to upload to Claude!
---
## Step 8: Upload to Claude (2 minutes)
1. Go to https://claude.ai
2. Click your profile → Settings
3. Click "Knowledge" or "Skills"
4. Click "Upload Skill"
5. Select `output/test-skill.zip`
6. Done! Claude can now use this skill
---
## 🎉 Success! What's Next?
You now have a working Skill Seeker installation! Here's what you can do:
### Try Other Presets
```bash
# See all available presets
ls configs/
# Try Vue.js
skill-seekers create --config configs/vue.json --max-pages 50
# Try Django
skill-seekers create --config configs/django.json --max-pages 50
```
### Try Other Source Types (17 Supported!)
```bash
# Auto-detect source type with the `create` command
skill-seekers create https://docs.example.com # Documentation
skill-seekers create facebook/react # GitHub repo
skill-seekers create manual.pdf # PDF
skill-seekers create report.docx # Word document
skill-seekers create book.epub # EPUB book
skill-seekers create analysis.ipynb # Jupyter Notebook
skill-seekers create spec.yaml # OpenAPI/Swagger spec
skill-seekers create slides.pptx # PowerPoint
# More source types
skill-seekers create --video-url https://youtube.com/watch?v=abc # Video
skill-seekers create --space-key DOCS # Confluence wiki
skill-seekers create --database-id DB_ID # Notion
skill-seekers create feed.rss # RSS feed
skill-seekers create grep.1 # Man page
skill-seekers create --chat-export-path ./slack-export # Slack/Discord
```
### Create Custom Skills
```bash
# Interactive mode - answer questions
skill-seekers create --interactive
# Or create config for any website
skill-seekers scrape \
--name myframework \
--url https://docs.myframework.com/ \
--description "My favorite framework"
```
### Where to Save Custom Configs
You have three options for placing your custom config files:
**Option 1: User Config Directory (Recommended)**
```bash
# Create config in your home directory
mkdir -p ~/.config/skill-seekers/configs
cat > ~/.config/skill-seekers/configs/myproject.json << 'EOF'
{
"name": "myproject",
"base_url": "https://docs.myproject.com/",
"max_pages": 50
}
EOF
# Use it
skill-seekers create --config myproject.json
```
**Option 2: Current Directory (Project-Specific)**
```bash
# Create config in your project
mkdir -p configs
nano configs/myproject.json
# Use it
skill-seekers create --config configs/myproject.json
```
**Option 3: Absolute Path**
```bash
# Use any file path
skill-seekers create --config /full/path/to/config.json
```
The tool searches in this order: exact path → `./configs/``~/.config/skill-seekers/configs/` → API presets
### Use with Claude Code (Advanced)
If you have Claude Code installed:
```bash
# One-time setup
./setup_mcp.sh
# Then use natural language in Claude Code:
# "Generate a skill for Svelte docs"
# "Package the skill at output/svelte/"
```
**See:** [docs/guides/MCP_SETUP.md](docs/guides/MCP_SETUP.md) for full MCP setup
---
## 🔧 Troubleshooting
### "Command not found" errors
**Problem:** `python3: command not found`
**Solution:** Python not installed or not in PATH
- macOS/Linux: Reinstall Python with brew/apt
- Windows: Reinstall Python, check "Add to PATH"
- Try `python` instead of `python3`
### "Permission denied" errors
**Problem:** Can't install packages
**Solution:**
```bash
# Use --user flag for pip
pip3 install --user skill-seekers
# Or use a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
```
### "No such file or directory"
**Problem:** CLI not found or config file path incorrect
**Solution:** Ensure you're using the CLI command correctly
```bash
# Verify installation
skill-seekers --version
# Should show version 3.6.0+
# Verify config file exists at expected path
ls configs/
```
### "ModuleNotFoundError" or "command not found: skill-seekers"
**Problem:** Package not installed or virtual environment not activated
**Solution:**
```bash
# Install the package in editable mode
pip install -e .
# Or if you need dev tools
pip install -e ".[dev]"
```
### Scraping is slow or fails
**Problem:** Takes forever or gets errors
**Solution:**
```bash
# Use smaller max_pages for testing
skill-seekers create --config configs/react.json --max-pages 10
# Check internet connection
ping google.com
# Check the website is accessible
curl -I https://docs.yoursite.com
```
### Still stuck?
1. **Check our detailed troubleshooting guide:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md)
2. **Open an issue:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
3. **Include this info:**
- Operating system (macOS 13, Ubuntu 22.04, Windows 11, etc.)
- Python version (`python3 --version`)
- Full error message
- What command you ran
---
## 📚 Next Steps
- **Read the full README:** [README.md](README.md)
- **Learn about presets:** [configs/](configs/)
- **Try MCP integration:** [docs/guides/MCP_SETUP.md](docs/guides/MCP_SETUP.md)
- **Advanced usage:** [docs/](docs/)
---
## ✅ Quick Reference
```bash
# Your typical workflow:
# 1. Create/use a config
skill-seekers create --config configs/react.json --max-pages 50
# 2. Package it
skill-seekers package output/react/
# 3. Upload output/react.zip to Claude
# Done! 🎉
```
**Common locations:**
- **Configs:** `configs/*.json`
- **Output:** `output/skill-name/`
- **Packaged skills:** `output/skill-name.zip`
**Time estimates:**
- Small skill (5-10 pages): 30 seconds
- Medium skill (50-100 pages): 3-5 minutes
- Large skill (500+ pages): 15-30 minutes
---
**Still confused?** That's okay! Open an issue and we'll help you get started: https://github.com/yusufkaraaslan/Skill_Seekers/issues/new
+3293
View File
File diff suppressed because it is too large Load Diff
+285
View File
@@ -0,0 +1,285 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
**Skill Seekers** converts documentation from 18 source types into production-ready formats for 21+ AI platforms (LLM platforms, RAG frameworks, vector databases, AI coding assistants). Published on PyPI as `skill-seekers`.
**Version:** 3.7.0 | **Python:** 3.10+ | **Website:** https://skillseekersweb.com/
**Architecture:** See `docs/UML_ARCHITECTURE.md` for UML diagrams and module overview. StarUML project at `docs/UML/skill_seekers.mdj`. Refactor state/history: `docs/UNIFICATION_PLAN.md` (Grand Unification — all 5 phases done; remaining cosmetic items listed there).
## Essential Commands
```bash
# REQUIRED before running tests or CLI (src/ layout)
pip install -e .
# Run all tests (NEVER skip - all must pass before commits)
pytest tests/ -v
# Fast iteration (skip slow MCP tests ~20min)
pytest tests/ --ignore=tests/test_mcp_fastmcp.py --ignore=tests/test_mcp_server.py --ignore=tests/test_install_skill_e2e.py -q
# Single test
pytest tests/test_scraper_features.py::test_detect_language -vv -s
# Code quality (must pass before push - matches CI)
uvx ruff check src/ tests/
uvx ruff format --check src/ tests/
mypy src/skill_seekers # continue-on-error in CI
# Auto-fix lint/format issues
uvx ruff check --fix --unsafe-fixes src/ tests/
uvx ruff format src/ tests/
# Build & publish
uv build
uv publish
```
## CI Matrix
Runs on push/PR to `main` or `development`. Lint job (Python 3.12, Ubuntu) + Test job (Ubuntu + macOS, Python 3.10/3.11/3.12, excludes macOS+3.10). Both must pass for merge.
## Git Workflow
- **Main branch:** `main` (requires tests + 1 review)
- **Development branch:** `development` (default PR target, requires tests)
- **Feature branches:** `feature/{task-id}-{description}` from `development`
- PRs always target `development`, never `main` directly
## Architecture
### CLI: Unified create command
Entry point `src/skill_seekers/cli/main.py`. The `create` command is the **primary** entry point for skill creation — it auto-detects source type and routes to the appropriate `SkillConverter`. The `scan` command (added in #327) is a separate discovery step for projects with multiple frameworks; it emits one config file per detected framework and you then run `create` on each.
```
skill-seekers create <source> # Auto-detect: URL, owner/repo, ./path, file.pdf, etc.
skill-seekers scan <dir> # AI-driven discovery → emits one config per detected framework + <project>-codebase.json
skill-seekers package <dir> # Package for platform (--target claude/gemini/openai/markdown/minimax/opencode/kimi/deepseek/qwen/openrouter/together/fireworks/langchain/llama-index/haystack/chroma/faiss/weaviate/qdrant/pinecone/ibm-bob)
```
### Scan command (issue #327)
`skill-seekers scan <dir>` is an AI-driven project knowledge-base bootstrapper. Pipeline in `src/skill_seekers/cli/scan_command.py`:
1. `collect_signals()` in `signal_collectors.py` — deterministic, bounded gathering of manifests + README + Dockerfile/CI + sampled source files + git remote. **Per-kind byte budgets** (24 KB manifest / 6 KB README / 6 KB CI / 28 KB samples, total 64 KB) so a fat package.json can't crowd out other kinds. `_SOURCE_DIRS` covers ~14 layouts (Go `cmd/`, Rust `crates/`, JS monorepo `apps/packages/`, Maven `source/`, Django at root); also walks root one level deep for flat-layout Python.
2. `detect_with_ai(bundle, AgentClient)` — one LLM call, structured JSON output. **Source signals are first-2-KB of each file** (whole-file sampling, no regex parsing — added in WS4 because regex missed Go multi-line imports + Rust `mod`/`extern crate`). Canonical-slug prompt + the canonical-name resolver are coupled — change one, update the other.
3. `resolve_or_generate_with_status()` — for each detection: try `out_dir/<slug>.json` (cache from prior run), then `resolve_config_path` from `config_fetcher` with multiple canonical name candidates (`_canonical_name_candidates` handles `"Godot Engine"``"godot"`, plus CJK / European suffixes like `"Godot 引擎"`, `"React フレームワーク"`, `"Lodash Bibliothek"`), then `generate_config_with_ai` as the last resort. Always appends `.json` to lookup names so local-disk and user-dir resolution actually finds files. Always stamps `metadata.detected_version` (nested, not top-level — `metadata.version` already exists and means config-schema version).
4. `emit_codebase_config()` — always writes `<project>-codebase.json` (a `type: local` source pointed at the project root).
5. `diff_against_existing()` — keyed by **filename slug** (not internal `data["name"]`) so re-scans don't churn when the AI returns a display name vs the registry canonical slug.
6. `_archive_removed()` — when a config disappears from detections, MOVE (not delete — user may have hand-edited) to `out_dir/.archived/<UTC-timestamp>/`. Runs after diff, before fresh writes.
7. `maybe_publish()`**native async** (WS11). Opt-in submission of freshly AI-generated configs to the community registry. Pre-checks `GITHUB_TOKEN`. Idempotency guard: `_find_existing_issue` queries GitHub Search API for an existing open issue with the same config name before submitting. Retries transient failures (rate limit, 5xx) with 0s/5s/15s backoff. `_prompt_async` wraps `input()` via `asyncio.to_thread` so the event loop isn't blocked.
**CLI dispatch** uses the `COMMAND_CLASSES` table in `main.py` (added in WS1). `scan` and `doctor` are dispatched as `Cls(args).execute()` consuming the parsed argparse namespace directly — no `_reconstruct_argv` hack, no duplicate argparse. `ScanCommand.execute()` is the single `asyncio.run` boundary wrapping `run_scan` (sync) + `maybe_publish` (async). Remaining ~14 commands still use the legacy `COMMAND_MODULES` dispatch; they're flagged for migration.
**Cost guardrails**: `--max-ai-generations N` (default 10) caps unbounded AI generation; `--dry-run` previews without writing or invoking AI; `--probe-urls` HEAD-checks AI-generated URLs with retry-on-404 and stamps `metadata._url_unverified` on confirmed-bad URLs.
**Safety**: All writes use `_atomic_write_json` (`os.replace` after writing to `.tmp`) so a `KeyboardInterrupt` mid-write can't corrupt configs. `_safe_size` guards `stat()` so broken symlinks don't crash the scan. `ScanCommand.execute` calls `logging.basicConfig` so `logger.warning`/`error` is visible; exit code is non-zero when no configs and no codebase config were emitted.
**Public constant**: `SourceDetector.CODE_PROJECT_MARKERS` (was `_CODE_PROJECT_MARKERS`) — shared between source_detector + signal_collectors. ~50 manifest types now (Pipfile, environment.yml, deno.json, flake.nix, Chart.yaml, deps.edn, dune-project, BUILD.bazel, …). Public so cross-module access doesn't reach into a private attribute.
### SkillConverter Pattern (Template Method + Factory)
All 18 source types implement the `SkillConverter` base class (`skill_converter.py`):
```python
converter = get_converter("web", config) # Factory lookup
converter.run() # Template: extract() → build_skill()
```
Registry in `CONVERTER_REGISTRY` maps source type → (module, class). `create_command.py` builds config from `ExecutionContext`, calls `get_converter()`, then runs centralized enhancement. `get_converter("config", {...})` constructs `UnifiedScraper` from the same factory-shaped dict (no special cases in create_command/MCP). The base resolves `skill_dir` once (strips trailing separators) and derives `data_file` via `data_file_for()` — subclasses must not re-derive paths.
### DocumentSkillBuilder (build side of 9 document scrapers)
`cli/document_skill_builder.py:DocumentSkillBuilder` sits between `SkillConverter` and the 9 document scrapers (epub, word, pptx, html, pdf, jupyter, man, rss, chat). It owns `categorize_content`, reference-file writing (tables, truncation, image guard), `index.md` + `SKILL.md` generation, and `load_extracted_data`. Variation points are class attrs (`DOC_NOUN`, `SOURCE_LABEL`, `LOAD_TOTAL_KEY`, `PATTERN_KEYWORDS`, `RANGE_LABEL`, …) and small hook methods (`category_stem`, `_write_reference_section`, `_write_skill_md_metadata`). Output is pinned **byte-identical** by golden trees in `tests/golden/phase2/``UPDATE_GOLDENS=1` rewrites them, only do that deliberately. Surviving full-method overrides are domain-shaped and commented per scraper.
### UnifiedScraper (multi-source configs)
`unified_scraper.py` dispatches via the class-level `SOURCE_DISPATCH` table; `_scrape_with_converter()` is the shared engine for the 13 mechanical source types (`get_converter()` + public `converter.extract()` + cache copy + sub-skill build), so **new types registered in `CONVERTER_REGISTRY` work in unified configs automatically**. documentation/github/local stay bespoke (commented why). `run()` deliberately does NOT follow the base template (TestRunOrchestration pins that run() triggers workflows).
### Data Flow (5 phases)
1. **Scrape** - Source-specific scraper extracts content to `output/{name}_data/pages/*.json`
2. **Build** - `build_skill()` categorizes pages, extracts patterns, generates `output/{name}/SKILL.md`
3. **Enhance** (optional) - LLM rewrites SKILL.md (`--enhance-level 0-3`, auto-detects API vs LOCAL mode)
4. **Package** - Platform adaptor formats output (`.zip`, `.tar.gz`, JSON, vector index)
5. **Upload** (optional) - Platform API upload
### Platform Adaptor Pattern (Strategy + Factory)
Factory: `get_adaptor(platform, config)` in `adaptors/__init__.py` returns a `SkillAdaptor` instance. Base class `SkillAdaptor` + `SkillMetadata` in `adaptors/base.py`.
```
src/skill_seekers/cli/adaptors/
├── __init__.py # Factory: get_adaptor(platform, config), ADAPTORS registry
├── base.py # Abstract base: SkillAdaptor, SkillMetadata
├── openai_compatible.py # Shared base for OpenAI-compatible platforms
├── claude.py # --target claude
├── gemini.py # --target gemini
├── openai.py # --target openai
├── markdown.py # --target markdown
├── minimax.py # --target minimax
├── opencode.py # --target opencode
├── kimi.py # --target kimi
├── deepseek.py # --target deepseek
├── qwen.py # --target qwen
├── openrouter.py # --target openrouter
├── together.py # --target together
├── fireworks.py # --target fireworks
├── langchain.py # --target langchain
├── llama_index.py # --target llama-index
├── haystack.py # --target haystack
├── chroma.py # --target chroma
├── faiss_helpers.py # --target faiss
├── qdrant.py # --target qdrant
├── weaviate.py # --target weaviate
├── pinecone_adaptor.py # --target pinecone
└── streaming_adaptor.py # --target streaming
```
All adaptors use `--target`. All adaptors are imported with `try/except ImportError` so missing optional deps don't break the registry.
### 18 Source Type Converters
Each in `src/skill_seekers/cli/{type}_scraper.py` as a `SkillConverter` subclass (no `main()`). The `create_command.py` uses `source_detector.py` to auto-detect, then calls `get_converter()`. Converters: web (doc_scraper), github, pdf, word, epub, video, local (codebase_scraper), jupyter, html, openapi, asciidoc, pptx, rss, manpage, confluence, notion, chat, config (unified_scraper).
### CLI Argument System (single-definition parsers)
```
src/skill_seekers/cli/
├── parsers/ # Central SubcommandParser classes — the ONLY definition of each command's flags
│ └── create_parser.py # Progressive help disclosure (--help-web, --help-github, etc.)
├── arguments/ # Argument definitions
│ ├── common.py # add_all_standard_arguments() - shared across all scrapers
│ └── create.py # UNIVERSAL_ARGUMENTS, WEB_ARGUMENTS, GITHUB_ARGUMENTS, etc.
├── exit_codes.py # EXIT_SUCCESS/ERROR/VALIDATION/INTERRUPT
└── source_detector.py # Auto-detect source type from input string
```
Command modules' standalone `main(args=None)` paths build their parser FROM the central `SubcommandParser` class — **add/change a flag in `parsers/*.py` only**. Drift guards (`tests/test_cli_parsers.py::TestCentralModuleParserSync` and `TestCentralParserSingleSource`) fail CI on any divergence of dests/defaults/option strings.
`ExecutionContext.override()` is context-local (a `ContextVar` layered over the unchanged base singleton) — thread/async safe for the MCP server; propagate to worker threads via `copy_context`.
### C3.x Codebase Analysis Pipeline
Local codebase analysis features, all opt-out (`--skip-*` flags):
- C3.1 `pattern_recognizer.py` - Design pattern detection (10 GoF patterns, 9 languages)
- C3.2 `test_example_extractor.py` - Usage examples from tests
- C3.3 `how_to_guide_builder.py` - AI-enhanced educational guides
- C3.4 `config_extractor.py` - Configuration pattern extraction
- C3.5 `generate_router.py` - Architecture overview generation
- C3.10 `signal_flow_analyzer.py` - Godot signal flow analysis
### MCP Server
`src/skill_seekers/mcp/server_fastmcp.py` - 40 tools via FastMCP. Transport: stdio (Claude Code) or HTTP (Cursor/Windsurf). Optional dependency: `pip install -e ".[mcp]"`
- **Tools run in-process** via `run_cli_main()` in `mcp/tools/_common.py`: same argv parsed by the command's REAL parser (sys.argv patch under a lock), stdout/stderr capture + contextvar log capture, identical `(stdout, stderr, returncode)` contract. No subprocess startup; old hard timeouts are advisory.
- **Exceptions BY DESIGN**: `enhance_skill` (LOCAL agent) and `install_skill`'s enhancement step stay subprocess — the agent must be a real child process for the fork-bomb-guard env semantics (`SKILL_SEEKER_ENHANCE_ACTIVE`). Never make these in-process.
- **Domain logic lives in `skill_seekers.services/`** (marketplace_manager, marketplace_publisher, config_publisher, source_manager, git_repo) — importable by CLI without the `[mcp]` extra; old `skill_seekers.mcp.*` paths are back-compat shims. No `sys.path` hacks anywhere in `mcp/`.
### Enhancement (AgentClient is the single AI transport)
Every text-based AI call goes through `AgentClient` (`src/skill_seekers/cli/agent_client.py`): central truncation gate, timeout policy, error classification. `API_PROVIDERS` (provider registry) and `AGENT_PRESETS` (local-agent command templates) live ONLY there. Adaptors declare provider/endpoint/model/prompt and route through `SkillAdaptor._enhance_skill_md_via_client` (atomic save with backup). `video_visual` frame classification is the documented multimodal exception (AgentClient is text-only).
- **API mode** (if API key set): Anthropic, Google Gemini, OpenAI, Moonshot/Kimi — detected in registry order; `SKILL_SEEKER_PROVIDER` forces one. Models: `SKILL_SEEKER_MODEL` (global) or `ANTHROPIC_MODEL`/`GOOGLE_MODEL`/`OPENAI_MODEL`/`MOONSHOT_MODEL`; `ANTHROPIC_BASE_URL` for compatible endpoints.
- **LOCAL mode** (fallback): Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — command built by `build_local_agent_command()`.
- Control: `--enhance-level 0` (off) / `1` (SKILL.md only) / `2` (default, balanced) / `3` (full)
- Agent selection: `--agent claude|codex|copilot|opencode|kimi|custom`
## Key Implementation Details
### Smart Categorization (`doc_scraper.py:smart_categorize()`)
Scores pages against category keywords: 3 points for URL match, 2 for title, 1 for content. Threshold of 2+ required. Falls back to "other".
### Content Extraction (`doc_scraper.py`)
`FALLBACK_MAIN_SELECTORS` constant + `_find_main_content()` helper handle CSS selector fallback. Links are extracted from the full page before early return (not just main content). `body` is deliberately excluded from fallbacks.
### Three-Stream GitHub Architecture (`unified_codebase_analyzer.py`)
Stream 1: Code Analysis (AST, patterns, tests, guides). Stream 2: Documentation (README, docs/, wiki). Stream 3: Community (issues, PRs, metadata). Depth control: `basic` (1-2 min) or `c3x` (20-60 min).
## Testing
### Test markers (pytest.ini)
```bash
pytest tests/ -v # Default: fast tests only
pytest tests/ -v -m slow # Include slow tests (>5s)
pytest tests/ -v -m integration # External services required
pytest tests/ -v -m e2e # Resource-intensive
pytest tests/ -v -m "not slow and not integration" # Fastest subset
```
### Known legitimate skips (~11)
- 2: chromadb incompatible with Python 3.14 (pydantic v1)
- 2: weaviate-client not installed
- 2: Qdrant not running (requires docker)
- 2: langchain/llama_index not installed
- 3: GITHUB_TOKEN not set
### sys.modules gotcha
`test_swift_detection.py` deletes `skill_seekers.cli` modules from `sys.modules`. It must save and restore both `sys.modules` entries AND parent package attributes (`setattr`). See the test file for the pattern.
## Dependencies
Core deps include `langchain`, `llama-index`, `anthropic`, `httpx`, `PyMuPDF`, `pydantic`. Platform-specific deps are optional:
```bash
pip install -e ".[mcp]" # MCP server
pip install -e ".[gemini]" # Google Gemini
pip install -e ".[openai]" # OpenAI
pip install -e ".[docx]" # Word documents
pip install -e ".[epub]" # EPUB books
pip install -e ".[video]" # Video (lightweight)
pip install -e ".[video-full]"# Video (Whisper + visual)
pip install -e ".[jupyter]" # Jupyter notebooks
pip install -e ".[pptx]" # PowerPoint
pip install -e ".[rss]" # RSS/Atom feeds
pip install -e ".[confluence]"# Confluence wiki
pip install -e ".[notion]" # Notion pages
pip install -e ".[chroma]" # ChromaDB
pip install -e ".[all]" # Everything (except video-full)
```
Dev dependencies use PEP 735 `[dependency-groups]` in pyproject.toml.
## Environment Variables
```bash
ANTHROPIC_API_KEY=sk-ant-... # Claude AI (or compatible endpoint)
ANTHROPIC_BASE_URL=https://... # Optional: Claude-compatible API endpoint
GOOGLE_API_KEY=AIza... # Google Gemini (optional)
OPENAI_API_KEY=sk-... # OpenAI (optional)
GITHUB_TOKEN=ghp_... # Higher GitHub rate limits
```
## Adding New Features
### New platform adaptor
1. Create `src/skill_seekers/cli/adaptors/{platform}.py` inheriting `SkillAdaptor` from `base.py`
2. Register in `adaptors/__init__.py` (add try/except import + add to `ADAPTORS` dict)
3. Add optional dep to `pyproject.toml`
4. Add tests in `tests/`
### New source type converter
1. Create `src/skill_seekers/cli/{type}_scraper.py` — for document-shaped sources inherit `DocumentSkillBuilder` (categorization/references/index/SKILL.md come free; implement `extract()` + hooks), otherwise inherit `SkillConverter` and implement `extract()` and `build_skill()`. Set `SOURCE_TYPE`.
2. Register in `CONVERTER_REGISTRY` in `skill_converter.py` — this also makes the type work in unified configs automatically (UnifiedScraper engine)
3. Add source type config building in `create_command.py:_build_config()`
4. Add auto-detection in `source_detector.py`
5. Add optional dep if needed
6. Add tests
### New CLI argument
- Subcommand flag: define ONLY in the central parser class (`parsers/{cmd}_parser.py`) — module `main()` builds from it; the drift-guard test fails otherwise
- Universal: `UNIVERSAL_ARGUMENTS` in `arguments/create.py`
- Source-specific: appropriate dict (`WEB_ARGUMENTS`, `GITHUB_ARGUMENTS`, etc.)
- Shared across scrapers: `add_all_standard_arguments()` in `arguments/common.py`
+550
View File
@@ -0,0 +1,550 @@
# Contributing to Skill Seeker
First off, thank you for considering contributing to Skill Seeker! It's people like you that make Skill Seeker such a great tool.
## Table of Contents
- [Branch Workflow](#branch-workflow)
- [Code of Conduct](#code-of-conduct)
- [How Can I Contribute?](#how-can-i-contribute)
- [Development Setup](#development-setup)
- [Pull Request Process](#pull-request-process)
- [Coding Standards](#coding-standards)
- [Testing](#testing)
- [Documentation](#documentation)
---
## Branch Workflow
**⚠️ IMPORTANT:** Skill Seekers uses a two-branch workflow.
### Branch Structure
```
main (production)
│ (only maintainer merges)
development (integration) ← default branch for PRs
│ (all contributor PRs go here)
feature branches
```
### Branches
- **`main`** - Production branch
- Always stable
- Only receives merges from `development` by maintainers
- Protected: requires tests + 1 review
- **`development`** - Integration branch
- **Default branch for all PRs**
- Active development happens here
- Protected: requires tests to pass
- Gets merged to `main` by maintainers
- **Feature branches** - Your work
- Created from `development`
- Named descriptively (e.g., `feature/123-add-github-scraping`)
- Merged back to `development` via PR
### Workflow Example
```bash
# 1. Fork and clone
git clone https://github.com/YOUR_USERNAME/Skill_Seekers.git
cd Skill_Seekers
# 2. Add upstream
git remote add upstream https://github.com/yusufkaraaslan/Skill_Seekers.git
# 3. Create feature branch from development
git checkout development
git pull upstream development
git checkout -b my-feature
# 4. Make changes, commit, push
git add .
git commit -m "Add my feature"
git push origin my-feature
# 5. Create PR targeting 'development' branch
```
---
## Related Repositories
Skill Seekers spans multiple repositories. Make sure you're contributing to the right one:
| What you want to work on | Repository |
|--------------------------|-----------|
| Core CLI, scrapers, MCP tools, adaptors | [Skill_Seekers](https://github.com/yusufkaraaslan/Skill_Seekers) (this repo) |
| Website, docs, UI/UX | [skillseekersweb](https://github.com/yusufkaraaslan/skillseekersweb) |
| Preset configs, community configs | [skill-seekers-configs](https://github.com/yusufkaraaslan/skill-seekers-configs) |
| GitHub Action integration | [skill-seekers-action](https://github.com/yusufkaraaslan/skill-seekers-action) |
| Claude Code plugin | [skill-seekers-plugin](https://github.com/yusufkaraaslan/skill-seekers-plugin) |
| Homebrew formula | [homebrew-skill-seekers](https://github.com/yusufkaraaslan/homebrew-skill-seekers) |
---
## Code of Conduct
This project and everyone participating in it is governed by our commitment to fostering an open and welcoming environment. Please be respectful and constructive in all interactions.
---
## How Can I Contribute?
### Reporting Bugs
Before creating bug reports, please check the [existing issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues) to avoid duplicates.
When creating a bug report, include:
- **Clear title and description**
- **Steps to reproduce** the issue
- **Expected behavior** vs actual behavior
- **Screenshots** if applicable
- **Environment details** (OS, Python version, etc.)
- **Error messages** and stack traces
**Example:**
```markdown
**Bug:** MCP tool fails when config has no categories
**Steps to Reproduce:**
1. Create config with empty categories: `"categories": {}`
2. Run `skill-seekers create --config configs/test.json`
3. See error
**Expected:** Should use auto-inferred categories
**Actual:** Crashes with KeyError
**Environment:**
- OS: Ubuntu 22.04
- Python: 3.10.5
- Version: 1.0.0
```
### Suggesting Enhancements
Enhancement suggestions are tracked as [GitHub issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues).
Include:
- **Clear title** describing the enhancement
- **Detailed description** of the proposed functionality
- **Use cases** that would benefit from this enhancement
- **Examples** of how it would work
- **Alternatives considered**
### Adding New Framework Configs
We welcome new framework configurations! To add one:
1. Create a config file in `configs/`
2. Test it thoroughly with different page counts
3. Submit a PR with:
- The config file
- Brief description of the framework
- Test results (number of pages scraped, categories found)
**Example PR:**
```markdown
**Add Svelte Documentation Config**
Adds configuration for Svelte documentation (https://svelte.dev/docs).
- Config: `configs/svelte.json`
- Tested with max_pages: 100
- Successfully categorized: getting_started, components, api, advanced
- Total pages available: ~150
```
### Pull Requests
We actively welcome your pull requests!
**⚠️ IMPORTANT:** All PRs must target the `development` branch, not `main`.
1. Fork the repo and create your branch from `development`
2. If you've added code, add tests
3. If you've changed APIs, update the documentation
4. Ensure the test suite passes
5. Make sure your code follows our coding standards
6. Issue that pull request to `development` branch!
---
## Development Setup
### Prerequisites
- Python 3.10 or higher (required for MCP integration)
- Git
### Setup Steps
1. **Fork and clone the repository**
```bash
git clone https://github.com/YOUR_USERNAME/Skill_Seekers.git
cd Skill_Seekers
```
2. **Install dependencies**
```bash
pip install -e .
pip install -e ".[dev]"
pip install -e ".[all]"
```
3. **Create a feature branch from development**
```bash
git checkout development
git pull upstream development
git checkout -b feature/my-awesome-feature
```
4. **Make your changes**
```bash
# Edit files...
```
5. **Run tests**
```bash
python -m pytest tests/ -v
```
6. **Commit your changes**
```bash
git add .
git commit -m "Add awesome feature"
```
7. **Push to your fork**
```bash
git push origin feature/my-awesome-feature
```
8. **Create a Pull Request**
---
## Pull Request Process
### Before Submitting
- [ ] Tests pass locally (`python -m pytest tests/ -v`)
- [ ] Code follows PEP 8 style guidelines
- [ ] Documentation is updated if needed
- [ ] CHANGELOG.md is updated (if applicable)
- [ ] Commit messages are clear and descriptive
### PR Template
```markdown
## Description
Brief description of what this PR does.
## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation update
## How Has This Been Tested?
Describe the tests you ran to verify your changes.
## Checklist
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
```
### Review Process
1. A maintainer will review your PR within 3-5 business days
2. Address any feedback or requested changes
3. Once approved, a maintainer will merge your PR
4. Your contribution will be included in the next release!
### Code Review Guidelines
**Fix both, don't follow precedent.** When a reviewer flags an anti-pattern in
your PR, do not defend it by pointing at another file that does the same thing.
"`X.py` already does this" is not a justification — it's evidence that two
places need fixing, not that the smell is sanctioned.
The correct response is one of:
- "Good catch — I'll fix both `new_file.py` and `existing_file.py` in this PR."
- "Out of scope here, but I'll file a follow-up to fix `existing_file.py`."
The wrong response is:
- "But `existing_file.py` does the same thing, so this matches the convention."
This rule cuts both ways: maintainers calling out an anti-pattern should be
willing to either accept the broader fix or open the follow-up issue
themselves. Bad-precedent-as-convention is how codebases ossify.
---
## Coding Standards
### Python Style Guide
We follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) with some modifications:
- **Line length:** 100 characters (not 79)
- **Indentation:** 4 spaces
- **Quotes:** Double quotes for strings
- **Naming:**
- Functions/variables: `snake_case`
- Classes: `PascalCase`
- Constants: `UPPER_SNAKE_CASE`
### Code Organization
```python
# 1. Standard library imports
import os
import sys
from pathlib import Path
# 2. Third-party imports
import requests
from bs4 import BeautifulSoup
# 3. Local application imports
from cli.utils import open_folder
# 4. Constants
MAX_PAGES = 1000
DEFAULT_RATE_LIMIT = 0.5
# 5. Functions and classes
def my_function():
"""Docstring describing what this function does."""
pass
```
### Documentation
- All functions should have docstrings
- Use type hints where appropriate
- Add comments for complex logic
```python
def scrape_page(url: str, selectors: dict) -> dict:
"""
Scrape a single page and extract content.
Args:
url: The URL to scrape
selectors: Dictionary of CSS selectors
Returns:
Dictionary containing extracted content
Raises:
RequestException: If page cannot be fetched
"""
pass
```
### Code Quality Tools
We use **Ruff** for linting and code formatting. Ruff is a fast Python linter that combines multiple tools (Flake8, isort, Black, etc.) into one.
**Running Ruff:**
```bash
# Check for linting errors
uvx ruff check src/ tests/
# Auto-fix issues
uvx ruff check --fix src/ tests/
# Format code
uvx ruff format src/ tests/
```
**Common Ruff Rules:**
- **SIM102** - Simplify nested if statements (use `and` instead)
- **SIM117** - Combine multiple `with` statements
- **B904** - Use `from e` for proper exception chaining
- **SIM113** - Use enumerate instead of manual counters
- **B007** - Use `_` for unused loop variables
- **ARG002** - Remove unused function arguments
**CI/CD Integration:**
All pull requests automatically run:
1. `ruff check` - Linting validation
2. `ruff format --check` - Format validation
3. `pytest` - Test suite
Make sure all checks pass before submitting your PR:
```bash
# Run the same checks as CI
uvx ruff check src/ tests/
uvx ruff format --check src/ tests/
pytest tests/ -v
```
---
## Testing
### Running Tests
```bash
# Run all tests
python -m pytest tests/ -v
# Run specific test file
python -m pytest tests/test_mcp_server.py -v
# Run with coverage
python -m pytest tests/ --cov=src/skill_seekers --cov-report=term
```
### Writing Tests
- Tests go in the `tests/` directory
- Test files should start with `test_`
- Use descriptive test names
```python
def test_config_validation_with_missing_fields():
"""Test that config validation fails when required fields are missing."""
config = {"name": "test"} # Missing base_url
result = validate_config(config)
assert result is False
```
### Test Coverage
- Aim for >80% code coverage
- Critical paths should have 100% coverage
- Add tests for bug fixes to prevent regressions
---
## Documentation
### Where to Document
- **README.md** - Overview, quick start, basic usage
- **docs/** - Detailed guides and tutorials
- **CHANGELOG.md** - All notable changes
- **Code comments** - Complex logic and non-obvious decisions
### Documentation Style
- Use clear, simple language
- Include code examples
- Add screenshots for UI-related features
- Keep it up to date with code changes
---
## Project Structure
```
Skill_Seekers/
├── src/skill_seekers/ # Main package (src/ layout)
│ ├── cli/ # CLI commands and entry points
│ │ ├── main.py # Unified CLI entry (COMMAND_MODULES dict)
│ │ ├── source_detector.py # Auto-detects source type
│ │ ├── create_command.py # Unified `create` command routing
│ │ ├── config_validator.py # VALID_SOURCE_TYPES set
│ │ ├── unified_scraper.py # Multi-source orchestrator
│ │ ├── unified_skill_builder.py # Pairwise synthesis + generic merge
│ │ ├── doc_scraper.py # Documentation (web)
│ │ ├── github_scraper.py # GitHub repos
│ │ ├── pdf_scraper.py # PDF files
│ │ ├── word_scraper.py # Word (.docx)
│ │ ├── epub_scraper.py # EPUB books
│ │ ├── video_scraper.py # Video (YouTube, Vimeo, local)
│ │ ├── codebase_scraper.py # Local codebases
│ │ ├── jupyter_scraper.py # Jupyter Notebooks
│ │ ├── html_scraper.py # Local HTML files
│ │ ├── openapi_scraper.py # OpenAPI/Swagger specs
│ │ ├── asciidoc_scraper.py # AsciiDoc files
│ │ ├── pptx_scraper.py # PowerPoint files
│ │ ├── rss_scraper.py # RSS/Atom feeds
│ │ ├── manpage_scraper.py # Man pages
│ │ ├── confluence_scraper.py # Confluence wikis
│ │ ├── notion_scraper.py # Notion pages
│ │ ├── chat_scraper.py # Slack/Discord exports
│ │ ├── adaptors/ # Platform adaptors (Strategy pattern)
│ │ ├── arguments/ # CLI argument definitions (one per source)
│ │ ├── parsers/ # Subcommand parsers (one per source)
│ │ └── storage/ # Cloud storage adaptors
│ ├── services/ # Shared domain logic (marketplace, config publishing, git sources) — used by CLI and MCP
│ ├── mcp/ # MCP server + tools (in-process; thin layer over cli/ + services/)
│ └── sync/ # Sync monitoring
├── configs/ # Preset JSON scraping configs
├── docs/ # Documentation
├── tests/ # 115+ test files (pytest)
└── .github/ # GitHub config
└── workflows/ # CI/CD workflows
```
**Scraper pattern (18 source types):** Each source type is a `SkillConverter` subclass in `cli/<type>_scraper.py` (document-shaped sources subclass `DocumentSkillBuilder`, which provides the whole build side), reached via `skill-seekers create` auto-detection — there is no per-type `main()`. Register new types in: `CONVERTER_REGISTRY` in `skill_converter.py` (this also enables them in unified multi-source configs), `create_command.py:_build_config()`, `source_detector.py`, and `config_validator.py` VALID_SOURCE_TYPES. CLI flags are defined once in the central `parsers/*.py` classes (a drift-guard test enforces this).
### UML Architecture
Full UML class diagrams are maintained in StarUML and synced from source code:
- **[docs/UML_ARCHITECTURE.md](docs/UML_ARCHITECTURE.md)** - Overview with embedded PNG diagrams
- **[docs/UML/skill_seekers.mdj](docs/UML/skill_seekers.mdj)** - StarUML project (open with [StarUML](https://staruml.io/))
- **[docs/UML/exports/](docs/UML/exports/)** - 14 PNG exports (package overview + 13 class diagrams)
- **[docs/UML/html/](docs/UML/html/index.html/index.html)** - HTML API reference
**Key design patterns documented in UML:**
- Strategy + Factory in Adaptors (SkillAdaptor ABC + 20+ implementations)
- Strategy + Factory in Storage (BaseStorageAdaptor + S3/GCS/Azure)
- Template Method in Parsers (SubcommandParser + 28 subclasses)
- Template Method in Analysis (BasePatternDetector + 10 GoF detectors)
- Command pattern in CLI (CLIDispatcher + COMMAND_MODULES lazy dispatch)
When adding new classes or modules, please update the corresponding UML diagram to keep architecture docs in sync.
---
## Release Process
Releases are managed by maintainers:
1. Update version in relevant files
2. Update CHANGELOG.md
3. Create and push version tag
4. GitHub Actions will create the release
5. Announce on relevant channels
---
## Questions?
- 💬 [Open a discussion](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)
- 🐛 [Report a bug](https://github.com/yusufkaraaslan/Skill_Seekers/issues)
- 📧 Contact: yusufkaraaslan.yk@pm.me
---
## Recognition
Contributors will be recognized in:
- README.md contributors section
- CHANGELOG.md for each release
- GitHub contributors page
Thank you for contributing to Skill Seeker! 🎉
+531
View File
@@ -0,0 +1,531 @@
# Skill Seekers Documentation Audit Report
**Date:** 2026-05-30
**Project Version:** 3.6.0 (from `pyproject.toml`)
**Auditor:** Agent Swarm Analysis
**Scope:** All non-generated `.md` files (excludes `output/`, `.skillseeker-cache/`, virtualenvs)
---
## Executive Summary
The Skill Seekers project has **~163 non-generated markdown files** across the repository. While the codebase is actively maintained at **v3.6.0**, the documentation ecosystem suffers from **severe version drift**, **widespread stale CLI references**, **broken internal links**, **incomplete translations**, **near-duplicate files**, and **significant gaps between claimed and actual features**.
### Severity Distribution
| Severity | Count | Description |
|----------|-------|-------------|
| **CRITICAL** | 45+ | Wrong commands, broken links, missing sections, incorrect versions |
| **WARNING** | 60+ | Stale counts, truncated content, minor inaccuracies, deprecated paths |
| **INFO** | 35+ | Missing features from docs, structural suggestions, cosmetic issues |
### Key Finding
> **The English README.md itself is outdated (claims v3.5.0, shows 14 removed CLI subcommands, has 11 broken internal links), and ALL 11 translations lag even further behind (claiming v3.2.0). The documentation as a whole gives new users a systematically incorrect understanding of how to use the tool.**
---
## 1. Version Consistency Crisis
Version numbers are **completely inconsistent** across the documentation ecosystem:
| Location | Claims | Actual | Status |
|----------|--------|--------|--------|
| `pyproject.toml` | — | **3.6.0** | ✅ Source of truth |
| `AGENTS.md` | 3.6.0 | 3.6.0 | ✅ Correct |
| `README.md` (English) | 3.5.0 | 3.6.0 | ❌ Stale |
| **All 11 README translations** | 3.2.0 | 3.6.0 | ❌ **Very stale** |
| `docs/README.md` | 3.2.0 | 3.6.0 | ❌ Stale |
| `docs/getting-started/*.md` | 3.1.03.2.0 | 3.6.0 | ❌ Stale |
| `docs/user-guide/*.md` | 3.1.03.2.0 | 3.6.0 | ❌ Stale |
| `docs/guides/MCP_SETUP.md` | 2.4.0 | 3.6.0 | ❌ Very stale |
| `docs/reference/MCP_REFERENCE.md` | 3.5.0 | 3.6.0 | ❌ Slightly stale |
| `docs/advanced/mcp-server.md` | 3.2.0 | 3.6.0 | ❌ Stale |
| `ROADMAP.md` | 3.2.0 | 3.6.0 | ❌ Stale |
| `QWEN.md` | 3.3.0 | 3.6.0 | ❌ Stale |
| `integrations/INTEGRATIONS.md` | 2.10.0+ | 3.6.0 | ❌ Very stale |
| `api/configs_repo/STATUS.md` | 2.11.0+ | 3.6.0 | ❌ Very stale |
**Impact:** New users cannot determine which version of the tool they are reading documentation for. Features documented at v3.1.0 may have changed significantly by v3.6.0.
---
## 2. README.md (English) — Deep Analysis
### 2.1 CRITICAL Issues
#### A. Broken / Outdated Internal Links (11 links)
| Link in README | Actual Location | Status |
|---|---|---|
| `ASYNC_SUPPORT.md` | **Does not exist anywhere** | ❌ Dead |
| `docs/ENHANCEMENT_MODES.md` | `docs/features/ENHANCEMENT_MODES.md` | ❌ Wrong path |
| `docs/FEATURE_MATRIX.md` | `docs/reference/FEATURE_MATRIX.md` | ❌ Wrong path |
| `docs/GIT_CONFIG_SOURCES.md` | `docs/reference/GIT_CONFIG_SOURCES.md` | ❌ Wrong path |
| `docs/HOW_TO_GUIDES.md#ai-enhancement-new` | `docs/features/HOW_TO_GUIDES.md` | ❌ Wrong path |
| `docs/IMPLEMENTATION_SUMMARY_THREE_STREAM.md` | `docs/archive/historical/IMPLEMENTATION_SUMMARY_THREE_STREAM.md` | ❌ Wrong path |
| `docs/LARGE_DOCUMENTATION.md` | `docs/reference/LARGE_DOCUMENTATION.md` | ❌ Wrong path |
| `docs/MCP_SETUP.md` | `docs/guides/MCP_SETUP.md` | ❌ Wrong path |
| `docs/QUICK_REFERENCE.md` | `docs/archive/legacy/QUICK_REFERENCE.md` | ❌ Wrong path |
| `docs/UNIFIED_SCRAPING.md` | `docs/features/UNIFIED_SCRAPING.md` | ❌ Wrong path |
| `QUICKSTART.md` | `docs/archive/legacy/QUICKSTART.md` | ❌ Wrong path |
#### B. Non-Existent CLI Subcommands (14 removed commands still shown)
The README displays **legacy subcommands that were removed** in the unified CLI redesign:
| Invalid Command Shown | Correct Modern Command |
|---|---|
| `skill-seekers scrape --config ...` | `skill-seekers create --config ...` |
| `skill-seekers scrape --url ...` | `skill-seekers create https://...` |
| `skill-seekers video --url ...` | `skill-seekers create ... --video-url ...` |
| `skill-seekers video --setup` | `skill-seekers create --setup` |
| `skill-seekers pdf --pdf ...` | `skill-seekers create ... --pdf ...` |
| `skill-seekers github --repo ...` | `skill-seekers create ...` (auto-detect) |
| `skill-seekers unified --config ...` | `skill-seekers create configs/..._unified.json` |
| `skill-seekers confluence --space ...` | `skill-seekers create ... --space-key ...` |
| `skill-seekers notion --database-id ...` | `skill-seekers create ... --database-id ...` |
| `skill-seekers chat --export-dir ...` | `skill-seekers create ... --chat-export-path ...` |
| `skill-seekers list-configs` | **Does not exist** |
| `skill-seekers analyze --directory ...` | **Does not exist** |
#### C. Wrong CLI Flags in Examples
- `skill-seekers video --url ...` → should use `--video-url`
- `skill-seekers video --playlist ...` → should use `--video-playlist`
- `skill-seekers confluence --space TEAM` → should use `--space-key`
- `skill-seekers chat --export-dir ./slack-export` → should use `--chat-export-path`
- `skill-seekers package ... --format chroma/faiss/qdrant` → should use `--target`
#### D. Inconsistent Test Counts
- Badge: `3194+ Passing`
- "Quality Assurance" prose: `2,540+ tests`
- Actual codebase: ~3,462 test functions
- **Three different numbers cited in different places**
#### E. Missing Major Features
The README **omits** these significant v3.x features:
- `--preset` (`quick` / `standard` / `comprehensive`)
- `--dry-run`, `--fresh`, `--resume`, `--skip-scrape`
- `--chunk-for-rag`, `--chunk-tokens`, `--chunk-overlap-tokens`
- `--streaming`, `--marketplace`, `--marketplace-category`
- `skill-seekers doctor` (health check)
- `skill-seekers sync-config` (config drift detection)
- `skill-seekers stream` (streaming ingestion)
- `skill-seekers update` (incremental updater)
- `skill-seekers multilang` (multi-language support)
- `skill-seekers quality` (quality scoring)
- `skill-seekers extract-test-examples`
### 2.2 WARNING Issues
- MCP tool count: README says 26, AGENTS.md says 40, actual is 40
- LLM platform comparison table shows only 5 of ~21 supported platforms
- Install-agent names: README says `Kimi` but CLI uses `kimi-code`
- `--target claude` description misleadingly claims auto-copy for Cursor/Windsurf
---
## 3. Translation Analysis (All 11 READMEs)
### 3.1 Summary Table
| Translation | Headings | Version | Test Badge | Architecture Section | Scan Section | IBM Bob | MiniMax | Status |
|-------------|----------|---------|------------|----------------------|--------------|---------|---------|--------|
| English | 210 | 3.5.0 | 3194+ | ✅ | ✅ | ✅ | ✅ | ⚠️ Stale |
| **zh-CN** | 165 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **ja** | 165 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **ko** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **de** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **es** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **fr** | 196 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **pt-BR** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **ru** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **tr** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **ar** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
| **hi** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
### 3.2 Universal Translation Deficiencies
**Every single translation** is missing these major sections that exist in English:
- `## 📚 Documentation` (early quick-link table)
- `## Architecture` (module overview + UML links)
- `### 🛰️ AI-driven project scan (new)`
- `### 🤖 Agent-Agnostic Skill Generation`
- `### 📦 Marketplace Pipeline`
- `IBM Bob` support (agent count says 18 instead of 19)
- `MiniMax AI` platform comparison row and examples
- `pepy.tech` and `Trendshift` badges
### 3.3 Per-Translation Nuances
- **zh-CN, ja, ko** (CJK): Most severely truncated. Installation table has 13 rows vs English 15. Performance table missing Video rows. Smart Rate Limit Management heavily truncated.
- **ko** is the *best* CJK translation: includes Security section at bottom, has correct 15-row install table, includes Video performance rows.
- **fr, tr** (European): Include the early Documentation table (most others don't). Otherwise similar gaps.
- **hi** (Devanagari): Most complete non-European translation. Has all 5 print statements in Three-Stream example. Bootstrap "What you get" bullets present. Good technical transliteration.
- **ar** (Arabic): Most truncated. Three-Stream example has only 2 print statements. Missing Rate Limit Strategies Explained. Some bidi rendering issues in code comments.
- **ru** (Cyrillic): Has some untranslated English fragments ("OpenAPI/Swagger", "README"). Uses Latin "API"/"LOCAL" inconsistently.
---
## 4. Core Project Documentation
### 4.1 AGENTS.md — ✅ MOSTLY ACCURATE
- Version 3.6.0 ✅
- Source types count (17 + config) ✅
- 40 MCP tools ✅
- Minor: Claims "~14 legacy commands" but actual is 16
### 4.2 CLAUDE.md — ✅ ACCURATE
- Architecture descriptions match codebase
- File paths correct
- 24 adaptor files match actual
- Minor: Uses `uvx ruff` while AGENTS.md uses `ruff`
### 4.3 CHANGELOG.md — ✅ ACCURATE
- Latest version [3.6.0] correctly listed
- `[Unreleased]` section holds post-3.6.0 changes appropriately
- Format consistent
### 4.4 ROADMAP.md — 🔴 SEVERELY OUTDATED
- Claims `v3.2.0` is current (actual: 3.6.0)
- Lists `v2.7.0`, `v2.8.0`, `v2.9.0` as future releases
- **Completed tasks NOT checked off**:
- A1.3 `submit_config` MCP tool — EXISTS
- C3.3 "Build how-to guides" — `how_to_guide_builder.py` EXISTS
- C3.4 "Extract configuration patterns" — `config_extractor.py` EXISTS
- C3.5 "Create architectural overview" — `generate_router.py` EXISTS
- E1.1 `fetch_config` MCP tool — EXISTS
- Wrong metrics: "1,880+ tests" (actual: ~3,445), "24 preset configs" (actual: 12), "5 bundled workflows" (actual: 68 YAML files)
### 4.5 TROUBLESHOOTING.md — 🔴 BROKEN PATHS
- Uses **pre-`src/` layout paths** that no longer exist:
- `cli/doc_scraper.py` (×3)
- `python3 mcp/server.py` (×2)
- `pip3 install -r mcp/requirements.txt` — file does not exist
- **Missing critical step**: Never mentions `pip install -e .` which `conftest.py` hard-requires
### 4.6 CONTRIBUTING.md — 🔴 WRONG SETUP
- Tells contributors: `pip install requests beautifulsoup4` and `pip install -r mcp/requirements.txt`
- **Should say**: `pip install -e .` / `pip install -e ".[dev]"` (as AGENTS.md states)
- Coverage paths wrong: `--cov=cli --cov=mcp` should be `--cov=src/skill_seekers`
- **Contradicts AGENTS.md**: Describes pre-commit hook setup, but AGENTS.md says "**No pre-commit hooks, no Makefile**"
- Branch naming: uses `feature/my-awesome-feature` without task ID; AGENTS.md specifies `feature/{task-id}-{description}`
### 4.7 QWEN.md — 🔴 REDUNDANT & OUTDATED
- Version says `v3.3.0` (actual: 3.6.0)
- Claims 26+ MCP tools (actual: 40)
- Shows `skill-seekers package --target langchain``langchain` is a `--format` value, not `--target`
- Shows `--target cursor` — not a valid standalone `--target`
- **Omits `scan` and `doctor` entirely**
- Stale numbers: 67 YAML presets (actual: 68), 123 test files (actual: 143)
- **Recommendation**: Update to 3.6.0 or remove (largely duplicates CLAUDE.md/AGENTS.md)
---
## 5. docs/ Directory Analysis
### 5.1 Structural Issues
#### A. docs/README.md Omits ~40% of Actual Content
The docs hub README lists only a subset of directories. It **omits**:
- `guides/` (7 files)
- `integrations/` (15 files)
- `features/` (5 files)
- `archive/`, `blog/`, `case-studies/`, `plans/`, `roadmap/`, `strategy/`, `agents/`, `superpowers/`
- `zh-CN/` (entire translation tree)
- `UML/`
#### B. Phantom Link
`docs/README.md` links to `advanced/mcp-tools.md`**does not exist**. Only `advanced/mcp-server.md` exists.
### 5.2 Near-Duplicate File Pairs
**Pair 1:** `DOCKER_GUIDE.md` (575 lines) ↔ `DOCKER_DEPLOYMENT.md` (762 lines)
- Both cover Docker deployment, Compose, volumes, networking, monitoring, troubleshooting
- `DOCKER_DEPLOYMENT.md` is more detailed (adds embedding server, sync monitor, Prometheus/Grafana/Loki)
- `DOCKER_GUIDE.md` has broken link: `[Vector Database Integration](docs/strategy/WEEK2_COMPLETE.md)``WEEK2_COMPLETE.md` does not exist
- **Recommendation**: Merge into single file
**Pair 2:** `KUBERNETES_GUIDE.md` (957 lines) ↔ `KUBERNETES_DEPLOYMENT.md` (933 lines)
- Both cover Helm charts, manual deployment, scaling, HA, monitoring, security
- `KUBERNETES_DEPLOYMENT.md` adds Velero backup/restore, cost optimization
- `KUBERNETES_GUIDE.md` uses `skillseekers` namespace; `KUBERNETES_DEPLOYMENT.md` uses `skill-seekers`
- **Recommendation**: Merge into single file
### 5.3 Partial Duplication: Troubleshooting
- `user-guide/06-troubleshooting.md` (~200 lines, v3.1.0)
- Top-level `TROUBLESHOOTING.md` (1094 lines, no version stamp)
- **Significant overlap** in installation, configuration, scraping, enhancement sections
- **Recommendation**: Consolidate or clearly differentiate scope
### 5.4 Stale MCP Documentation
Three different MCP docs claim three different tool counts:
| File | Claims | Actual |
|------|--------|--------|
| `guides/MCP_SETUP.md` | 26 tools / v2.4.0 | 40 tools / v3.6.0 |
| `advanced/mcp-server.md` | 27 tools / v3.2.0 | 40 tools / v3.6.0 |
| `reference/MCP_REFERENCE.md` | 40 tools / v3.5.0 | 40 tools / v3.6.0 |
`guides/MCP_SETUP.md` also claims MCP SDK v1.25.0 and miscounts tool categories (says 27 but lists 28).
### 5.5 Stale Guide Content
- `guides/MIGRATION_GUIDE.md` header says `v3.1.0-dev` but discusses migrating **to v2.7.0** as "latest"
- `integrations/INTEGRATIONS.md` says "Skill Seekers Version: v2.10.0+"
- `integrations/LANGCHAIN.md` says "Skill Seekers Version: v2.9.0+"
- `features/BOOTSTRAP_SKILL.md` says version 2.7.0 in frontmatter example
### 5.6 FEATURE_MATRIX Undercount
Lists only 12 platforms but CLI supports 21+ packaging targets. Missing: DeepSeek, Qwen, OpenRouter, Together, Fireworks, IBM Bob, OpenCode, Kimi, and others.
---
## 6. docs/zh-CN/ Translation Quality — 🔴 CRITICAL FINDING
### 6.1 The Shocking Truth
**The `docs/zh-CN/` directory does NOT contain Chinese translations.**
Out of **21 files audited**:
- **0 files (0%)** are fully translated to Chinese
- **1 file (~15%)** has partial Chinese (`advanced/mcp-server.md` — just section headers and some tool descriptions)
- **20 files (95%)** are **English copies** with minor divergence
### 6.2 Content Drift Pattern
The zh-CN files fall into three categories:
| Category | Files | Description |
|----------|-------|-------------|
| **zh-CN LAGS** | README, CLI_REFERENCE, API_REFERENCE, packaging | English docs are newer (v3.5.0 vs v3.2.0) |
| **zh-CN LEADS** | ARCHITECTURE, core-concepts, scraping, CONFIG_FORMAT, workflows, MCP_REFERENCE, multi-source | zh-CN documents features not yet in English docs (17 sources, unified config, new platforms) |
| **Identical** | 03-your-first-skill, 06-troubleshooting, SKILL_ARCHITECTURE, custom-workflows | Perfect copies |
### 6.3 Broken Link
- `zh-CN/getting-started/04-next-steps.md` links to `05-scan-a-project.md`**does not exist in zh-CN/** (only in `docs/getting-started/`)
### 6.4 Recommendation
If the goal is Chinese-language documentation: **entire directory needs professional translation**.
If the goal is parallel English documentation: **rename directory** (e.g., `docs/extended/`) to avoid misleading users.
---
## 7. Archive, Plans, Strategy & Roadmap Relevance
### 7.1 docs/archive/ — Mostly Appropriate
**Historical (8 files):** All are completed verification reports. Correctly archived. Could compress further.
**Legacy (4 files + index):** Self-aware deprecated docs with redirects. Correctly archived.
**Plans (2 files):** Active skills design from Oct 2025. Status unclear — **verify if implemented**.
**Research (4 files):** PDF research from Oct 2025. Informed current implementation. Keep as historical record.
### 7.2 docs/plans/video/ — 🔴 MISLEADING
All 8 files still say "Status: Planning" but **video support IS implemented** (`video_scraper.py`, `video_models.py`, `video_setup.py`, 60 tests).
**Action:** Update headers to "Implemented" and verify code alignment. Consider moving to `docs/features/`.
### 7.3 docs/strategy/ — Mixed
| File | Verdict |
|------|---------|
| `README.md` | Update — integration guides WERE created (18 files) but doc still says "to be created" |
| `ACTION_PLAN.md` | Archive — historical 4-week plan, partially executed |
| `ARBITRARY_LIMITS_AND_DEAD_CODE_PLAN.md` | Update — Stage 1 completed, Stages 2-3 status unknown |
| `INTEGRATION_STRATEGY.md` | Update — much executed, should reflect success |
| `KIMI_ANALYSIS_COMPARISON.md` | Archive — analysis complete, strategy adopted |
| `STAGE_1_*.md` (3 files) | Archive deeper — historical implementation records |
| `DEEPWIKI_ANALYSIS.md`, `INTEGRATION_TEMPLATES.md` | Keep — still valuable |
### 7.4 docs/roadmap/ — ✅ APPROPRIATE (Future Design)
Intelligence System roadmap (4 files, Jan 2026). No `skill_seekers/intelligence/` package exists yet. These are **future design docs** — correctly kept as research/planning.
### 7.5 docs/agents/ — Archive
EPUB implementation plans (Mar 2026). EPUB scraper EXISTS and is tested (107 tests). These are completed agent-driven development records — **archive deeper**.
### 7.6 docs/superpowers/ — Verify
Scrape-count and SPA detection plan (Mar 2026). References `doc_scraper.py` line numbers. **Verify if implemented** and update status.
---
## 8. Distribution & Examples
### 8.1 distribution/ — 🔴 BROKEN EXAMPLES
#### github-action/README.md
- Lists CLI commands: `scrape`, `github`, `pdf`, `video`, `analyze`, `unified`
- **None exist in v3.6.0**
- Users copying these workflow examples will get `error: argument command: invalid choice`
#### claude-plugin/commands/install-skill.md
- Lists targets: `cursor`, `windsurf`, `continue`, `cline`
- **Not registered adaptors** — will fail
#### claude-plugin/README.md & skills/skill-builder/SKILL.md
- Claims "35 MCP tools" — actual is **40**
### 8.2 examples/ — 🔴 ALL EXAMPLES USE REMOVED COMMANDS
**Every example README** (chroma, cline, continue-dev, cursor, haystack, langchain, llama-index, pinecone, weaviate, windsurf) references:
- `skill-seekers scrape --config ...` → should be `skill-seekers create --config ...`
- `skill-seekers github --repo ...` → should be `skill-seekers create ...`
#### windsurf-fastapi-context/
- References `--split-rules` and `--max-chars` flags on `skill-seekers package`
- **These flags do not exist**
#### Other Example Issues
- Many reference `skill-seekers v2.10.0` — current is 3.6.0
- `cursor-react-skill/README.md`: Uses `--target claude` for Cursor. Cursor uses `.cursorrules` (plain markdown); `--target markdown` would be more appropriate
- `continue-dev-universal/README.md`: References `~/.continue/config.json` for HTTP context providers. Continue.dev has shifted toward YAML config and MCP servers
### 8.3 api/configs_repo/ — 🔴 SEVERELY STALE
#### STATUS.md
- "Last Updated: 2024-12-21"
- "Total Configs: 90 unified format"
- "Format Version: Unified (v2.11.0+)"
- Actual: v3.6.0, submodule shows 22+ official categories
#### Divergence Between Copies
- `api/configs_repo/README.md`: Claims **178 configs / 21 categories / v3.1.0+**
- `skill-seekers-configs/README.md` (root copy): Claims **24 configs / 7 categories**
- **These wildly diverge**
#### CONTRIBUTING.md (configs_repo)
- References `skill-seekers scrape --config` — command removed
### 8.4 skill-seekers-configs/ — 🔴 COUNTS DON'T ADD UP
All TODO files claim `✅ COMPLETE` but counts are contradictory:
| File | Claims | Actually Lists |
|------|--------|----------------|
| `TODO-ai-ml.md` | 40 configs | 34 items |
| `TODO-build-tools.md` | 9 configs | ~15 items |
| `TODO-cloud.md` | 10 configs | ~15 items |
| `TODO-databases.md` | 9 configs | ~20 items |
| `TODO-development-tools.md` | 6 configs | ~12 items |
| `TODO-devops.md` | 7 configs | ~12 items |
| `TODO-game-engines.md` | 7 configs | ~20 items |
| `TODO-testing.md` | 4 configs | ~10 items |
| `TODO-web-frameworks.md` | 12 configs | ~18 items |
---
## 9. Tests, Scripts, Skills, src docs
### 9.1 tests/mcp_integration_test.md — 🔴 BROKEN PATHS
- `python3 mcp/server.py` → should be `python -m skill_seekers.mcp.server_fastmcp`
- `pip3 install -r mcp/requirements.txt` → file doesn't exist
- `python3 cli/doc_scraper.py`, `python3 cli/package_skill.py` → pre-`src/` layout
### 9.2 scripts/skill_header.md — 🔴 DEPRECATED FLAGS
- References `--depth surface/deep/full`**deprecated** (use `--preset`)
- References `--ai-mode none/api/local`**does not exist** in current CLI
### 9.3 skills/skill-seekers/SKILL.md — ⚠️ STALE COUNT
- Claims **35 MCP tools** — actual is **40**
- Otherwise well-structured
### 9.4 src/skill_seekers/mcp/README.md — 🔴 INCONSISTENT & BROKEN LINKS
- Header says "34 tools", later says "40 tools" — inconsistent
- References `docs/MCP_SETUP.md`, `docs/USAGE.md`, `docs/TESTING.md`**none exist**
- References `cli/doc_scraper.py`, `cli/estimate_pages.py`, `cli/package_skill.py` — pre-`src/` layout
- Claims "34 tests | Pass rate: 100%" and "25 tests" in different sections
---
## 10. Missing Documentation
These features exist in the codebase but are **under-documented or undocumented** in user-facing docs:
| Feature | Code Evidence | Doc Status |
|---------|---------------|------------|
| `doctor` command | `doctor_command.py` | Not mentioned in README, getting-started, or user-guide |
| `--preset` flag | Major UX feature in `create` | Mentioned in CLI_REFERENCE but not in README or getting-started |
| `--dry-run`, `--fresh`, `--resume` | Lifecycle management | Undocumented in user guides |
| `--chunk-for-rag`, `--chunk-tokens` | RAG chunking on `package` | Undocumented in packaging guides |
| `--streaming` | Memory-efficient packaging | Undocumented |
| `--marketplace`, `--marketplace-category` | Marketplace publishing | Undocumented |
| `scan` command deep guide | `scan_command.py` | Brief mention only; no deep guide |
| `video` source type deep guide | 7 `video_*.py` files | Mentioned but not deeply documented |
| `epub` source type | `epub_scraper.py` (107 tests) | Unclear coverage in user guides |
| `browser` extra dependency | Playwright for SPA sites | Not in installation table |
| `embedding` extra | Embedding server support | Not in installation table |
| Cloud extras (`s3`, `gcs`, `azure`) | Cloud storage upload | Not in installation table |
| All 68 YAML workflow presets | `workflows/` directory | Only mentioned as "67" or "68" |
---
## 11. Unneeded / Redundant Documentation
| File(s) | Issue | Recommendation |
|---------|-------|----------------|
| `DOCKER_GUIDE.md` + `DOCKER_DEPLOYMENT.md` | Near-duplicates | Merge into one |
| `KUBERNETES_GUIDE.md` + `KUBERNETES_DEPLOYMENT.md` | Near-duplicates | Merge into one |
| `user-guide/06-troubleshooting.md` + `TROUBLESHOOTING.md` | Heavy overlap | Consolidate or differentiate scope |
| `QWEN.md` | Duplicates CLAUDE.md/AGENTS.md, less accurate | Update or remove |
| `docs/archive/historical/*` (8 files) | Historical artifacts | Compress or move to external wiki |
| `docs/strategy/STAGE_1_*.md` (3 files) | Implementation records | Merge into one summary |
| `docs/agents/*` (2 files) | Completed EPUB plans | Archive deeper |
| `skill-seekers-configs/README.md` (root copy) | Wildly diverges from submodule | Remove or sync with `api/configs_repo/` |
---
## 12. Recommendations (Prioritized)
### P0 — Critical (Fix Immediately)
1. **Update English README.md** to v3.6.0 and remove all 14 legacy CLI subcommand examples
2. **Fix all 11 broken internal links** in README.md
3. **Update or remove ALL 11 README translations** — they are 4 minor versions behind and missing major sections
4. **Update ALL example READMEs** to use `skill-seekers create` instead of removed `scrape`/`github`/`video`/`pdf`/`unified` commands
5. **Fix `distribution/github-action/README.md`** to use valid CLI commands
6. **Update `ROADMAP.md`** version header, check off completed tasks, remove obsolete v2.x release planning
7. **Rewrite `TROUBLESHOOTING.md`** for `src/` layout and modern CLI paths
8. **Fix `CONTRIBUTING.md`** setup instructions to match AGENTS.md (`pip install -e .`)
### P1 — High (Fix Soon)
9. **Unify version stamps** to 3.6.0 across all docs (or automate via CI build step)
10. **Consolidate Docker and K8s duplicate files**
11. **Update MCP tool counts** to consistently say **40** across all docs
12. **Update `guides/MCP_SETUP.md`** to match current v3.6.0 reality
13. **Update `integrations/INTEGRATIONS.md`** from v2.10.0+ to v3.6.0
14. **Fix `tests/mcp_integration_test.md`** paths to `server_fastmcp.py`
15. **Fix `src/skill_seekers/mcp/README.md`** broken links and inconsistent counts
16. **Update `FEATURE_MATRIX.md`** to include all 21+ packaging targets
17. **Update `scripts/skill_header.md`** to remove deprecated `--depth` and non-existent `--ai-mode`
18. **Sync or remove `skill-seekers-configs/README.md`** (root copy)
### P2 — Medium (Quality Improvements)
19. **Update `docs/README.md`** to reflect actual directory structure (add missing directories)
20. **Update `docs/plans/video/*.md`** from "Planning" to "Implemented"
21. **Update `docs/strategy/README.md`** and `INTEGRATION_STRATEGY.md` to show executed deliverables
22. **Add missing features to README**: `--preset`, `--dry-run`, `doctor`, `sync-config`, `stream`, `update`, `multilang`, `quality`
23. **Add missing optional deps** to installation table: `browser`, `embedding`, `s3`, `gcs`, `azure`, `rag-upload`
24. **Add `scan` and `doctor` deep guides** to getting-started or user-guide
25. **Update `guides/MIGRATION_GUIDE.md`** to discuss actual v3.5→v3.6 migration
### P3 — Low (Nice to Have)
26. **Automate version stamping** in CI to prevent future drift
27. **Archive completed agent plans** (EPUB, scrape-count) deeper or compress
28. **Add documentation update checklist** to `.github/PULL_REQUEST_TEMPLATE.md`
29. **Regenerate UML diagrams** to reflect scan feature and other recent changes
30. **Consider professional translation** of `docs/zh-CN/` or rename to `docs/extended/` if not actually Chinese
---
## Appendix: Files Audited by Area
| Area | Files Count | Key Finding |
|------|-------------|-------------|
| Root READMEs | 12 | All translations 4 versions behind |
| Core docs | 7 | ROADMAP, TROUBLESHOOTING, CONTRIBUTING severely outdated |
| docs/ main | ~86 | Version inconsistency, duplicate files, missing structure |
| docs/zh-CN/ | 21 | **0% actually translated to Chinese** |
| docs/archive/ | 18 | Appropriately archived, some could compress deeper |
| docs/plans/video/ | 8 | Misleadingly labeled "Planning" — already implemented |
| docs/strategy/ | 11 | Partially executed, needs update |
| docs/roadmap/ | 4 | Future design docs — appropriately kept |
| docs/integrations/ | 18 | Successfully created strategy deliverables |
| distribution/ | 6 | GitHub Action uses removed commands |
| examples/ | 13 | ALL use removed CLI commands |
| api/configs_repo/ | 6 | STATUS.md severely stale |
| skill-seekers-configs/ | 13 | Counts don't add up |
| .github/ | 5 | Templates functional but could reference `doctor` |
| tests/ | 1 | Broken paths to old MCP server |
| scripts/ | 1 | Deprecated flags |
| skills/ | 1 | Stale MCP tool count |
| src/ | 1 | Broken links, inconsistent counts |
---
*End of Report*
+75
View File
@@ -0,0 +1,75 @@
# Skill Seekers - Multi-stage Docker Build
# Optimized for production deployment with minimal image size
# Stage 1: Builder - Install dependencies and build
FROM python:3.12-slim as builder
WORKDIR /build
# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
g++ \
git \
&& rm -rf /var/lib/apt/lists/*
# Copy dependency files
COPY pyproject.toml README.md ./
COPY src/ src/
# Install dependencies and build package
RUN pip install --no-cache-dir --upgrade pip uv && \
uv pip install --system --no-cache -e . && \
uv pip install --system --no-cache ".[all-llms]"
# Stage 2: Runtime - Minimal production image
FROM python:3.12-slim
LABEL maintainer="Skill Seekers <noreply@skillseekers.dev>"
LABEL description="Skill Seekers - Convert documentation to AI skills"
LABEL version="2.9.0"
# Install runtime dependencies only
RUN apt-get update && apt-get install -y --no-install-recommends \
git \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user
RUN useradd -m -u 1000 -s /bin/bash skillseeker && \
mkdir -p /app /data /configs /output && \
chown -R skillseeker:skillseeker /app /data /configs /output
WORKDIR /app
# Copy Python packages from builder
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY --from=builder /usr/local/bin/skill-seekers* /usr/local/bin/
# Copy application code
COPY --chown=skillseeker:skillseeker src/ src/
COPY --chown=skillseeker:skillseeker configs/ configs/
COPY --chown=skillseeker:skillseeker pyproject.toml README.md ./
# Switch to non-root user
USER skillseeker
# Set environment variables
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PATH="/home/skillseeker/.local/bin:$PATH" \
SKILL_SEEKERS_HOME=/data \
SKILL_SEEKERS_OUTPUT=/output
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD skill-seekers --version || exit 1
# Default volumes
VOLUME ["/data", "/configs", "/output"]
# Expose MCP server port (HTTP mode)
EXPOSE 8765
# Default command - show help
CMD ["skill-seekers", "--help"]
+57
View File
@@ -0,0 +1,57 @@
# Skill Seekers MCP Server - Docker Image
# Optimized for MCP server deployment (stdio + HTTP modes)
FROM python:3.12-slim
LABEL maintainer="Skill Seekers <noreply@skillseekers.dev>"
LABEL description="Skill Seekers MCP Server - 35 tools for AI skills generation"
LABEL version="3.3.0"
WORKDIR /app
# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
git \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user
RUN useradd -m -u 1000 -s /bin/bash mcp && \
mkdir -p /app /data /configs /output && \
chown -R mcp:mcp /app /data /configs /output
# Copy application files
COPY --chown=mcp:mcp src/ src/
COPY --chown=mcp:mcp configs/ configs/
COPY --chown=mcp:mcp pyproject.toml README.md ./
# Install dependencies
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e ".[all-llms]" && \
pip install --no-cache-dir mcp
# Switch to non-root user
USER mcp
# Environment variables
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
MCP_TRANSPORT=http \
MCP_PORT=8765 \
SKILL_SEEKERS_HOME=/data \
SKILL_SEEKERS_OUTPUT=/output
# Health check for HTTP mode
HEALTHCHECK --interval=30s --timeout=10s --start-period=10s --retries=3 \
CMD curl -f http://localhost:${MCP_PORT}/health || exit 1
# Volumes
VOLUME ["/data", "/configs", "/output"]
# Expose MCP server port (default 8765, overridden by $PORT on cloud platforms)
EXPOSE ${MCP_PORT:-8765}
# Start MCP server in HTTP mode by default
# Uses shell form so $PORT/$MCP_PORT env vars are expanded at runtime
# Cloud platforms (Render, Railway, etc.) set $PORT automatically
CMD python -m skill_seekers.mcp.server_fastmcp --http --host 0.0.0.0 --port ${PORT:-${MCP_PORT:-8765}}
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Yusuf Karaaslan
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+719
View File
@@ -0,0 +1,719 @@
# QWEN.md - Skill Seekers
Comprehensive context file for AI coding agents working with the Skill Seekers project.
---
## Project Overview
**Skill Seekers** (v3.6.0) is a Python CLI tool and MCP server that converts documentation sites, GitHub repositories, PDFs, videos, notebooks, wikis, and 17+ source types into structured AI-ready skills for 21+ LLM platforms and RAG pipelines.
**Tagline:** "The data layer for AI systems" — sits between raw documentation and every AI system that consumes it (Claude, Gemini, LangChain, LlamaIndex, Cursor, etc.).
### Key Capabilities
- **18 source types:** documentation (web), GitHub, PDF, Word (.docx), EPUB, video, local codebase, Jupyter, HTML, OpenAPI, AsciiDoc, PowerPoint, RSS/Atom, man pages, Confluence, Notion, Slack/Discord chat
- **21+ export targets:** Claude, Gemini, OpenAI, DeepSeek, Qwen, Fireworks, Together, OpenRouter, IBM BoB, Kimi, MiniMax, OpenCode, LangChain, LlamaIndex, Haystack, Pinecone, Chroma, Weaviate, Qdrant, FAISS, Markdown, and more
- **Unified pipeline:** One scraping command → export to any platform without re-scraping
- **MCP server:** 40 tools for AI assistants to scrape, package, and manage skills
- **AI enhancement:** Optional Claude-powered enhancement for better skill quality
### Project Links
| Resource | Link |
|----------|------|
| Website | https://skillseekersweb.com/ |
| PyPI | https://pypi.org/project/skill-seekers/ |
| GitHub | https://github.com/yusufkaraaslan/Skill_Seekers |
| Configs | https://github.com/yusufkaraaslan/skill-seekers-configs |
| MCP | https://modelcontextprotocol.io |
---
## Project Structure
```
Skill_Seekers/
├── src/skill_seekers/ # Main package (src/ layout)
│ ├── cli/ # CLI commands (97 files)
│ │ ├── adaptors/ # Platform adaptors (Strategy pattern)
│ │ ├── arguments/ # CLI argument definitions
│ │ ├── parsers/ # Subcommand parsers
│ │ ├── storage/ # Cloud storage adaptors
│ │ ├── main.py # Unified CLI entry point
│ │ ├── source_detector.py # Auto-detects source type
│ │ ├── create_command.py # Unified `create` command
│ │ ├── config_validator.py # Config validation
│ │ ├── unified_scraper.py # Multi-source orchestrator
│ │ └── unified_skill_builder.py # Skill merging
│ ├── mcp/ # MCP server
│ │ ├── server.py # Main MCP server
│ │ ├── server_fastmcp.py # FastMCP implementation
│ │ └── tools/ # MCP tools (10 files)
│ ├── sync/ # Sync monitoring (Pydantic)
│ ├── benchmark/ # Benchmarking framework
│ ├── embedding/ # FastAPI embedding server
│ └── workflows/ # 68 YAML workflow presets
├── tests/ # ~143 test files (pytest)
├── configs/ # Preset JSON scraping configs
├── docs/ # Documentation
├── templates/ # GitHub Actions, etc.
├── scripts/ # Utility scripts
└── pyproject.toml # Project configuration
```
---
## Setup & Installation
### Required: Install in Editable Mode
```bash
# ALWAYS run this first — tests hard-exit if package not installed
pip install -e .
# With dev tools (pytest, ruff, mypy, coverage)
pip install -e ".[dev]"
# With all optional dependencies
pip install -e ".[all]"
```
**Note:** `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not.
### Environment Variables
Copy `.env.example` to `.env` and configure:
```bash
# Required for AI enhancement
ANTHROPIC_API_KEY=sk-ant-...
# Optional: LLM platforms
GOOGLE_API_KEY=... # Gemini
OPENAI_API_KEY=... # OpenAI/ChatGPT
# Optional: GitHub (increases rate limits)
GITHUB_TOKEN=...
# MCP Server config
MCP_TRANSPORT=http
MCP_PORT=8765
```
---
## Build / Test / Lint Commands
### Testing
```bash
# Run ALL tests (required before commits)
pytest tests/ -v
# Run single test file
pytest tests/test_scraper_features.py -v
# Run single test function
pytest tests/test_scraper_features.py::test_detect_language -v
# Run single test class method
pytest tests/test_adaptors/test_claude_adaptor.py::TestClaudeAdaptor::test_package -v
# Skip slow/integration tests
pytest tests/ -v -m "not slow and not integration"
# With coverage
pytest tests/ --cov=src/skill_seekers --cov-report=term
```
### Linting & Formatting
```bash
# Lint (ruff)
ruff check src/ tests/
ruff check src/ tests/ --fix
# Format (ruff)
ruff format --check src/ tests/
ruff format src/ tests/
# Type check (mypy)
mypy src/skill_seekers --show-error-codes --pretty
```
### Pytest Configuration
From `pyproject.toml`:
- `addopts = "-v --tb=short --strict-markers"`
- `asyncio_mode = "auto"`
- `asyncio_default_fixture_loop_scope = "function"`
**Test markers:** `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`
**Test count:** 123 test files (107 in `tests/`, 16 in `tests/test_adaptors/`)
---
## CLI Usage
### Core Commands
```bash
# Unified create command (auto-detects source type)
skill-seekers create https://docs.react.dev/
skill-seekers create facebook/react
skill-seekers create manual.pdf
skill-seekers create notebook.ipynb
# Package for specific platform
skill-seekers package output/react --target claude # Claude AI (ZIP)
skill-seekers package output/react --target gemini # Gemini (tar.gz)
skill-seekers package output/react --target openai # OpenAI
skill-seekers package output/react --target cursor # .cursorrules
# Multi-source unified scraping
skill-seekers create configs/react_unified.json
```
### All 20+ Commands
| Command | Description |
|---------|-------------|
| `create` | Unified create (auto-detects source) |
| `scan` | AI-detect project tech stack and emit configs |
| `doctor` | Health check for dependencies and configuration |
| `scrape` | Scrape documentation website |
| `github` | Scrape GitHub repository |
| `pdf` | Extract from PDF |
| `word` | Extract from Word (.docx) |
| `epub` | Extract from EPUB |
| `video` | Extract from video |
| `jupyter` | Extract from Jupyter notebook |
| `html` | Extract from local HTML |
| `openapi` | Extract from OpenAPI spec |
| `asciidoc` | Extract from AsciiDoc |
| `pptx` | Extract from PowerPoint |
| `rss` | Extract from RSS/Atom feed |
| `manpage` | Extract from man pages |
| `confluence` | Extract from Confluence |
| `notion` | Extract from Notion |
| `chat` | Extract from Slack/Discord |
| `unified` | Multi-source scraping |
| `analyze` | Analyze local codebase |
| `enhance` | AI enhancement |
| `package` | Package skill |
| `upload` | Upload to platform |
| `install-agent` | Install to AI agent |
---
## Code Style & Conventions
### Formatting Rules (ruff)
- **Line length:** 100 characters
- **Target Python:** 3.10+
- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM
- **Ignored rules:** E501, F541, ARG002, B007, I001, SIM114
### Naming Conventions
| Type | Convention | Example |
|------|------------|---------|
| Files | `snake_case.py` | `source_detector.py` |
| Classes | `PascalCase` | `SkillAdaptor`, `ClaudeAdaptor` |
| Functions | `snake_case` | `get_adaptor()`, `detect_language()` |
| Constants | `UPPER_CASE` | `ADAPTORS`, `DEFAULT_CHUNK_TOKENS` |
| Private | `_prefix` | `_read_existing_content()` |
### Type Hints
- Gradual typing with modern syntax
- Use `str | None` not `Optional[str]`
- Use `list[str]` not `List[str]`
- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`
### Docstrings
- Module-level docstring on every file
- Google-style for public functions/classes
- Include `Args:`, `Returns:`, `Raises:` sections
### Error Handling
```python
# Use specific exceptions
raise ValueError("Invalid config: missing 'sources'")
raise RuntimeError("Scraping failed after 3 retries")
# Chain exceptions
try:
...
except Exception as e:
raise RuntimeError(f"Failed to process {source}") from e
# Guard optional imports
try:
from .claude import ClaudeAdaptor
except ImportError:
ClaudeAdaptor = None
```
### Import Patterns
```python
# Standard library → third-party → first-party
import os
import sys
from pathlib import Path
import requests
from beautifulsoup4 import BeautifulSoup
from skill_seekers.cli.adaptors import ClaudeAdaptor
from skill_seekers.cli.source_detector import SourceDetector
# Guard optional imports
try:
from .gemini import GeminiAdaptor
except ImportError:
GeminiAdaptor = None
# Re-exports (use noqa)
from .base import SkillAdaptor, SkillMetadata # noqa: F401
```
---
## Key Architectural Patterns
### 1. Adaptor (Strategy) Pattern
All platform logic in `cli/adaptors/`. Each adaptor inherits `SkillAdaptor`:
```python
from skill_seekers.cli.adaptors.base import SkillAdaptor, SkillMetadata
class ClaudeAdaptor(SkillAdaptor):
PLATFORM = "claude"
PLATFORM_NAME = "Claude AI (Anthropic)"
def format_skill_md(self, skill_dir: Path, metadata: SkillMetadata) -> str:
"""Format SKILL.md with YAML frontmatter"""
...
def package(self, skill_dir: Path, output_path: Path, ...) -> Path:
"""Package as ZIP with SKILL.md, references/, scripts/"""
...
def upload(self, package_path: Path, api_key: str) -> str:
"""Upload to Claude API"""
...
```
**Registered in:** `cli/adaptors/__init__.py``ADAPTORS` dict
### 2. Scraper Pattern
Each source type has 3 files:
```
cli/<type>_scraper.py # Main scraper class + main()
arguments/<type>.py # CLI argument definitions
parsers/<type>_parser.py # ArgumentParser setup
```
**Example:** `pdf_scraper.py``PdfToSkillConverter` class
**Registered in:**
- `parsers/__init__.py``PARSERS` list
- `main.py``COMMAND_MODULES` dict
- `config_validator.py``VALID_SOURCE_TYPES` set
### 3. Unified Pipeline
**`unified_scraper.py`** orchestrates multi-source scraping:
```python
class UnifiedScraper:
def __init__(self, config_path: str, merge_mode: str = "rule-based"):
self.config = load_config(config_path)
self.scraped_data = {
"documentation": [],
"github": [],
"pdf": [],
# ... 18 source types
}
def run(self) -> Path:
# 1. Scrape all sources
for source in self.config["sources"]:
self._scrape_source(source)
# 2. Merge (pairwise synthesis or generic)
merged = self._merge_sources()
# 3. Build unified skill
return self._build_skill(merged)
```
**`unified_skill_builder.py`** uses:
- **Pairwise synthesis** for docs+github+pdf combos
- **`_generic_merge()`** for other combinations
### 4. Source Detection
**`source_detector.py`** auto-detects from user input:
```python
class SourceDetector:
@classmethod
def detect(cls, source: str) -> SourceInfo:
# Check file extensions
if source.endswith(".pdf"):
return cls._detect_pdf(source)
if source.endswith(".ipynb"):
return cls._detect_jupyter(source)
# Check GitHub patterns
if cls.GITHUB_REPO_PATTERN.match(source):
return cls._detect_github(source)
# Check URLs
parsed = urlparse(source)
if parsed.scheme in ("http", "https"):
return cls._detect_web(source)
# Check local directories
if os.path.isdir(source):
return cls._detect_local(source)
```
### 5. MCP Tools
**`mcp/tools/`** grouped by category:
- `scrape_tools.py` — Scraping tools
- `package_tools.py` — Packaging tools
- `enhance_tools.py` — Enhancement tools
- `install_tools.py` — Installation tools
- `vector_db_tools.py` — Vector DB tools
- `workflow_tools.py` — Workflow tools
**`scrape_generic_tool`** handles all new source types dynamically.
---
## Configuration
### Unified Config Format
```json
{
"name": "react-skill",
"description": "React documentation skill",
"sources": [
{
"type": "documentation",
"url": "https://react.dev/",
"config": {
"max_pages": 100,
"include_patterns": ["**/*.md"],
"exclude_patterns": ["**/blog/**"]
}
},
{
"type": "github",
"repo": "facebook/react",
"config": {
"include": ["src/", "packages/"],
"exclude": ["**/*.test.tsx"]
}
}
],
"merge_mode": "rule-based",
"output": "output/react"
}
```
### Valid Source Types
```python
VALID_SOURCE_TYPES = {
"documentation", "github", "pdf", "local", "word",
"video", "epub", "jupyter", "html", "openapi",
"asciidoc", "pptx", "confluence", "notion", "rss",
"manpage", "chat"
}
```
### Merge Modes
- **`rule-based`** — Deterministic merging with conflict resolution rules
- **`claude-enhanced`** — AI-powered merging (requires `ANTHROPIC_API_KEY`)
---
## Git Workflow
### Branch Structure
```
main (production, protected)
│ (maintainer merges only)
development (integration, default PR target)
│ (all contributor PRs)
feature branches
```
### PR Process
1. Fork and clone
2. Create feature branch from `development`
3. Make changes, commit, push
4. Create PR targeting **`development`** (NOT `main`)
5. Wait for tests + review
```bash
git checkout development
git pull upstream development
git checkout -b my-feature
# ... make changes
git commit -m "Add feature X"
git push origin my-feature
# Create PR → development
```
---
## Testing Practices
### Test Organization
```
tests/
├── conftest.py # Fixtures, setup
├── test_adaptors/ # Adaptor tests (16 files)
├── test_scraper_features.py # Core scraper tests
├── test_source_detector.py # Source detection tests
├── test_config_validation.py # Config validation
├── test_mcp_*.py # MCP tests
├── test_*_e2e.py # End-to-end tests
└── fixtures/ # Test fixtures
```
### Test Patterns
```python
import pytest
from skill_seekers.cli.source_detector import SourceDetector
class TestSourceDetector:
"""Test source detection"""
def test_detect_pdf(self, tmp_path):
"""Test PDF detection"""
pdf_file = tmp_path / "test.pdf"
pdf_file.touch()
result = SourceDetector.detect(str(pdf_file))
assert result.type == "pdf"
@pytest.mark.asyncio
async def test_async_scraping(self):
"""Test async scraping"""
# asyncio_mode = "auto" — decorator often implicit
...
@pytest.mark.slow
def test_slow_operation(self):
"""Mark slow tests for optional skipping"""
...
@pytest.mark.integration
def test_external_api(self):
"""Mark integration tests requiring external services"""
...
```
### Fixtures
```python
# tests/conftest.py
import pytest
from pathlib import Path
@pytest.fixture
def sample_config(tmp_path):
"""Create sample config file"""
config = {
"name": "test-skill",
"sources": [{"type": "documentation", "url": "https://example.com"}]
}
config_file = tmp_path / "config.json"
config_file.write_text(json.dumps(config))
return str(config_file)
@pytest.fixture
def mock_response():
"""Mock HTTP response"""
class MockResponse:
status_code = 200
text = "<html><body>Test</body></html>"
return MockResponse()
```
---
## Development Guidelines
### Before Commits
```bash
# 1. Lint
ruff check src/ tests/
ruff format --check src/ tests/
# 2. Type check
mypy src/skill_seekers
# 3. Test (ALL must pass)
pytest tests/ -v
```
### Adding New Source Types
1. Create scraper: `cli/<type>_scraper.py` with `<Type>ToSkillConverter` class
2. Create arguments: `arguments/<type>.py`
3. Create parser: `parsers/<type>_parser.py`
4. Register in `parsers/__init__.py``PARSERS`
5. Register in `main.py``COMMAND_MODULES`
6. Add to `config_validator.py``VALID_SOURCE_TYPES`
7. Add detection to `source_detector.py`
8. Add to `unified_scraper.py``scraped_data` dict
9. Write tests in `tests/test_<type>_scraper.py`
### Adding New Platform Adaptors
1. Create adaptor: `cli/adaptors/<platform>.py` inheriting `SkillAdaptor`
2. Implement: `format_skill_md()`, `package()`, `upload()`
3. Register in `cli/adaptors/__init__.py``ADAPTORS` dict
4. Add to `package_skill.py` → target mapping
5. Write tests in `tests/test_adaptors/test_<platform>.py`
### Adding New MCP Tools
1. Create tool in `mcp/tools/<category>_tools.py`
2. Use `@mcp.tool()` decorator
3. Register in `mcp/server.py` or `mcp/server_fastmcp.py`
4. Write tests in `tests/test_mcp_*.py`
---
## Common Issues & Solutions
### Tests Fail with Import Error
**Problem:** `ModuleNotFoundError: No module named 'skill_seekers'`
**Solution:** Install in editable mode first:
```bash
pip install -e .
```
### Optional Dependency Missing
**Problem:** `ImportError: No module named 'mammoth'`
**Solution:** Install optional dependency:
```bash
pip install "skill-seekers[docx]"
# or
pip install "skill-seekers[all]"
```
### Rate Limiting
**Problem:** GitHub API rate limited (60/hour anonymous)
**Solution:** Set `GITHUB_TOKEN` in `.env`:
```bash
GITHUB_TOKEN=ghp_...
```
### Async Scraping Issues
**Problem:** Event loop errors in async tests
**Solution:** Use `@pytest.mark.asyncio` decorator (auto mode enabled)
---
## Key Files Reference
| File | Purpose |
|------|---------|
| `pyproject.toml` | Project config, dependencies, tool settings |
| `src/skill_seekers/cli/main.py` | Unified CLI entry point |
| `src/skill_seekers/cli/source_detector.py` | Auto-detect source types |
| `src/skill_seekers/cli/config_validator.py` | Config validation |
| `src/skill_seekers/cli/unified_scraper.py` | Multi-source orchestrator |
| `src/skill_seekers/cli/adaptors/base.py` | Adaptor interface |
| `src/skill_seekers/cli/adaptors/__init__.py` | Adaptor registry |
| `src/skill_seekers/mcp/server.py` | MCP server |
| `tests/conftest.py` | Test fixtures |
| `AGENTS.md` | Quick reference for AI agents |
---
## Version & Release
- **Current version:** 3.3.0 (from `pyproject.toml`)
- **Version source:** `src/skill_seekers/_version.py` reads from `pyproject.toml`
- **Release process:** Tag → GitHub Actions → PyPI publish
- **Changelog:** `CHANGELOG.md` (Keep a Changelog format)
---
## Related Repositories
| Repository | Purpose |
|------------|---------|
| [Skill_Seekers](https://github.com/yusufkaraaslan/Skill_Seekers) | Core CLI & MCP (this repo) |
| [skillseekersweb](https://github.com/yusufkaraaslan/skillseekersweb) | Website & docs |
| [skill-seekers-configs](https://github.com/yusufkaraaslan/skill-seekers-configs) | Community configs |
| [skill-seekers-action](https://github.com/yusufkaraaslan/skill-seekers-action) | GitHub Action |
| [skill-seekers-plugin](https://github.com/yusufkaraaslan/skill-seekers-plugin) | Claude Code plugin |
| [homebrew-skill-seekers](https://github.com/yusufkaraaslan/homebrew-skill-seekers) | Homebrew tap |
---
## Quick Commands Cheat Sheet
```bash
# Setup
pip install -e ".[dev]"
cp .env.example .env
# Edit .env with API keys
# Development
ruff check src/ tests/ --fix
ruff format src/ tests/
mypy src/skill_seekers
pytest tests/ -v
# Test subsets
pytest tests/test_adaptors/ -v
pytest tests/ -m "not slow and not integration"
pytest tests/ --cov=src/skill_seekers
# Usage
skill-seekers create https://docs.python.org/
skill-seekers package output/python --target claude
skill-seekers create configs/react_unified.json
```
+1374
View File
File diff suppressed because it is too large Load Diff
+1341
View File
File diff suppressed because it is too large Load Diff
+1380
View File
File diff suppressed because it is too large Load Diff
+1380
View File
File diff suppressed because it is too large Load Diff
+1377
View File
File diff suppressed because it is too large Load Diff
+1376
View File
File diff suppressed because it is too large Load Diff
+1376
View File
File diff suppressed because it is too large Load Diff
+1375
View File
File diff suppressed because it is too large Load Diff
+1380
View File
File diff suppressed because it is too large Load Diff
+1358
View File
File diff suppressed because it is too large Load Diff
+1381
View File
File diff suppressed because it is too large Load Diff
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目:`yusufkaraaslan/Skill_Seekers`
- 原始仓库:https://github.com/yusufkaraaslan/Skill_Seekers
- 导入方式:上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源,不代表 WeHub 是原项目作者
+1376
View File
File diff suppressed because it is too large Load Diff
+433
View File
@@ -0,0 +1,433 @@
# Skill Seekers Roadmap
Transform Skill Seekers into the easiest way to create Claude AI skills from **any knowledge source** - documentation websites, PDFs, codebases, GitHub repos, Office docs, and more - with both CLI and MCP interfaces.
---
## 🎯 Current Status: v3.6.0 ✅
**Latest Release:** v3.6.0 (May 2026)
**What Works:**
-**18 source types** — documentation, GitHub, PDF, video, Word, EPUB, Jupyter, local HTML, OpenAPI, AsciiDoc, PowerPoint, RSS/Atom, man pages, Confluence, Notion, Slack/Discord, local codebase
- ✅ Unified multi-source scraping with generic merge for any source combination
- ✅ 40 MCP tools fully functional
- ✅ Multi-platform support (21 platforms: Claude, Gemini, OpenAI, DeepSeek, Qwen, Fireworks, Together, OpenRouter, IBM BoB, Kimi, MiniMax, OpenCode, LangChain, LlamaIndex, Haystack, Pinecone, ChromaDB, FAISS, Weaviate, Qdrant, Markdown)
- ✅ Auto-upload to all platforms
- ✅ 12 preset configs (including unified configs)
- ✅ Large docs support (40K+ pages with router skills)
- ✅ C3.x codebase analysis suite (C3.1-C3.10)
- ✅ Bootstrap skill feature - self-hosting capability
- ✅ 3,445+ tests passing
- ✅ Unified `create` command with auto-detection for all 18 source types
- ✅ 68 YAML workflow presets
- ✅ Cloud storage integration (S3, GCS, Azure)
- ✅ Source auto-detection via `source_detector.py`
**Recent Improvements (v3.2.0):**
-**10 new source types**: Word, EPUB, video, Jupyter, local HTML, OpenAPI, AsciiDoc, PowerPoint, RSS/Atom, man pages, Confluence, Notion, Slack/Discord
-**Generic merge system**: `_generic_merge()` in `unified_skill_builder.py` handles arbitrary source combinations
-**Unified CLI**: `create` command auto-detects all 18 source types
-**Workflow Presets**: YAML-based enhancement presets with CLI management
-**Progressive Disclosure**: Default help shows 13 universal flags, detailed help per source
-**Bug Fixes**: Markdown parser h1 filtering, paragraph length filtering
-**Docs Cleanup**: Removed 47 stale planning/QA/release markdown files
---
## 🧭 Development Philosophy
**Small tasks → Pick one → Complete → Move on**
Instead of rigid milestones, we use a **flexible task-based approach**:
- 136 small, independent tasks across 10 categories
- Pick any task, any order
- Start small, ship often
- No deadlines, just continuous progress
**Philosophy:** Small steps → Consistent progress → Compound results
---
## 📋 Task-Based Roadmap (136 Tasks, 10 Categories)
### 🌐 **Category A: Community & Sharing**
Small tasks that build community features incrementally
#### A1: Config Sharing (Website Feature)
- [x] **Task A1.1:** Create simple JSON API endpoint to list configs ✅ **COMPLETE**
- **Status:** Live at https://api.skillseekersweb.com
- **Features:** 6 REST endpoints, auto-categorization, auto-tags, filtering, SSL enabled
- [x] **Task A1.2:** Add MCP tool `fetch_config` to download from website ✅ **COMPLETE**
- **Features:** List 24 configs, filter by category, download by name
- [x] **Task A1.3:** Add MCP tool `submit_config` to submit custom configs ✅ **COMPLETE**
- **Purpose:** Allow users to submit custom configs via MCP (creates GitHub issue)
- **Completed:** May 2026
- [ ] **Task A1.4:** Create static config catalog website (GitHub Pages)
- **Purpose:** Read-only catalog to browse/search configs
- **Time:** 2-3 hours
- [ ] **Task A1.5:** Add config rating/voting system
- **Purpose:** Community feedback on config quality
- **Time:** 3-4 hours
- [ ] **Task A1.6:** Admin review queue for submitted configs
- **Approach:** Use GitHub Issues with labels
- **Time:** 1-2 hours
- [x] **Task A1.7:** Add MCP tool `install_skill` for one-command workflow ✅ **COMPLETE**
- **Features:** fetch → scrape → enhance → package → upload
- **Completed:** December 21, 2025
- [ ] **Task A1.8:** Add smart skill detection and auto-install
- **Purpose:** Auto-detect missing skills from user queries
- **Time:** 4-6 hours
**Start Next:** Pick A1.4 (static config catalog website)
#### A2: Knowledge Sharing (Website Feature)
- [ ] **Task A2.1:** Design knowledge database schema
- [ ] **Task A2.2:** Create API endpoint to upload knowledge (.zip files)
- [ ] **Task A2.3:** Add MCP tool `fetch_knowledge` to download from site
- [ ] **Task A2.4:** Add knowledge preview/description
- [ ] **Task A2.5:** Add knowledge categorization (by framework/topic)
- [ ] **Task A2.6:** Add knowledge search functionality
**Start Small:** Pick A2.1 first (schema design, no coding)
#### A3: Simple Website Foundation
- [ ] **Task A3.1:** Create single-page static site (GitHub Pages)
- [ ] **Task A3.2:** Add config gallery view
- [ ] **Task A3.3:** Add "Submit Config" link
- [ ] **Task A3.4:** Add basic stats
- [ ] **Task A3.5:** Add simple blog using GitHub Issues
- [ ] **Task A3.6:** Add RSS feed for updates
**Start Small:** Pick A3.1 first (single HTML page)
---
### 🛠️ **Category B: New Input Formats**
Add support for non-HTML documentation sources
#### B1: PDF Documentation Support ✅ **COMPLETE (v3.0.0)**
- [x] **Task B1.1:** Research PDF parsing libraries ✅
- [x] **Task B1.2:** Create simple PDF text extractor (POC) ✅
- [x] **Task B1.3:** Add PDF page detection and chunking ✅
- [x] **Task B1.4:** Extract code blocks from PDFs ✅
- [x] **Task B1.5:** Add PDF image extraction ✅
- [x] **Task B1.6:** Create `pdf_scraper.py` CLI tool ✅
- [x] **Task B1.7:** Add MCP tool `scrape_pdf`
- [x] **Task B1.8:** Create PDF config format ✅
#### B2: Microsoft Word (.docx) Support ✅ **COMPLETE (v3.2.0)**
- [x] **Task B2.1-B2.7:** Word document parsing and scraping ✅
#### B3: Excel/Spreadsheet (.xlsx) Support
- [ ] **Task B3.1-B3.6:** Spreadsheet parsing and API extraction
#### B4: Markdown Files Support ✅ **COMPLETE (v3.1.0)**
- [x] **Task B4.1-B4.6:** Local markdown directory scraping ✅
#### B5: Additional Source Types ✅ **COMPLETE (v3.2.0)**
- [x] **EPUB** - `epub_scraper.py`
- [x] **Video** - `video_scraper.py` (YouTube, Vimeo, local files) ✅
- [x] **Jupyter Notebook** - `jupyter_scraper.py`
- [x] **Local HTML** - `html_scraper.py`
- [x] **OpenAPI/Swagger** - `openapi_scraper.py`
- [x] **AsciiDoc** - `asciidoc_scraper.py`
- [x] **PowerPoint** - `pptx_scraper.py`
- [x] **RSS/Atom** - `rss_scraper.py`
- [x] **Man pages** - `manpage_scraper.py`
- [x] **Confluence** - `confluence_scraper.py`
- [x] **Notion** - `notion_scraper.py`
- [x] **Slack/Discord** - `chat_scraper.py`
---
### 💻 **Category C: Codebase Knowledge**
Generate skills from actual code repositories
#### C1: GitHub Repository Scraping
- [ ] **Task C1.1-C1.12:** GitHub API integration and code analysis
#### C2: Local Codebase Scraping
- [ ] **Task C2.1-C2.8:** Local directory analysis and API extraction
#### C3: Code Pattern Recognition
- [x] **Task C3.1:** Detect common patterns (singleton, factory, etc.) ✅ **v2.6.0**
- 10 GoF patterns, 9 languages, 87% precision
- [x] **Task C3.2:** Extract usage examples from test files ✅ **v2.6.0**
- 5 categories, 9 languages, 80%+ high-confidence examples
- [x] **Task C3.3:** Build "how to" guides from code ✅ **COMPLETE**
- [x] **Task C3.4:** Extract configuration patterns ✅ **COMPLETE**
- [x] **Task C3.5:** Create architectural overview ✅ **COMPLETE**
- [x] **Task C3.6:** AI Enhancement for Pattern Detection ✅ **v2.6.0**
- Claude API integration for enhanced insights
- [x] **Task C3.7:** Architectural Pattern Detection ✅ **v2.6.0**
- Detects 8 architectural patterns, framework-aware
**Start Next:** Pick C1.1 (GitHub API client)
---
### 🔌 **Category D: Context7 Integration**
- [ ] **Task D1.1-D1.4:** Research and planning
- [ ] **Task D2.1-D2.5:** Basic integration
---
### 🚀 **Category E: MCP Enhancements**
Small improvements to existing MCP tools
#### E1: New MCP Tools
- [x] **Task E1.3:** Add `scrape_pdf` MCP tool ✅
- [x] **Task E1.1:** Add `fetch_config` MCP tool ✅ **COMPLETE**
- [ ] **Task E1.2:** Add `fetch_knowledge` MCP tool
- [ ] **Task E1.4-E1.9:** Additional format scrapers
#### E2: MCP Quality Improvements
- [ ] **Task E2.1:** Add error handling to all tools
- [ ] **Task E2.2:** Add structured logging
- [ ] **Task E2.3:** Add progress indicators
- [ ] **Task E2.4:** Add validation for all inputs
- [ ] **Task E2.5:** Add helpful error messages
- [x] **Task E2.6:** Add retry logic for network failures ✅ **Utilities ready**
---
### ⚡ **Category F: Performance & Reliability**
Technical improvements to existing features
#### F1: Core Scraper Improvements
- [ ] **Task F1.1:** Add URL normalization
- [ ] **Task F1.2:** Add duplicate page detection
- [ ] **Task F1.3:** Add memory-efficient streaming
- [ ] **Task F1.4:** Add HTML parser fallback
- [x] **Task F1.5:** Add network retry with exponential backoff ✅
- [ ] **Task F1.6:** Fix package path output bug
#### F2: Incremental Updates
- [ ] **Task F2.1-F2.5:** Track modifications, update only changed content
---
### 🎨 **Category G: Tools & Utilities**
Small standalone tools that add value
#### G1: Config Tools
- [ ] **Task G1.1:** Create `validate_config.py`
- [ ] **Task G1.2:** Create `test_selectors.py`
- [ ] **Task G1.3:** Create `auto_detect_selectors.py` (AI-powered)
- [ ] **Task G1.4:** Create `compare_configs.py`
- [ ] **Task G1.5:** Create `optimize_config.py`
#### G2: Skill Quality Tools
- [ ] **Task G2.1-G2.5:** Quality analysis and reporting
---
### 📚 **Category H: Community Response**
- [ ] **Task H1.1-H1.5:** Address open GitHub issues
---
### 🎓 **Category I: Content & Documentation**
- [ ] **Task I1.1-I1.6:** Video tutorials
- [ ] **Task I2.1-I2.5:** Written guides
---
### 🧪 **Category J: Testing & Quality**
- [ ] **Task J1.1-J1.6:** Test expansion and coverage
---
## 🎯 Recommended Starting Tasks
### Quick Wins (1-2 hours each):
1. **H1.1** - Respond to Issue #8
2. **J1.1** - Install MCP package
3. **A3.1** - Create GitHub Pages site
4. **B1.1** - Research PDF parsing
5. **F1.1** - Add URL normalization
### Medium Tasks (3-5 hours each):
6.**A1.1** - JSON API for configs (COMPLETE)
7. **G1.1** - Config validator script
8. **C1.1** - GitHub API client
9. **I1.1** - Video script writing
10. **E2.1** - Error handling for MCP tools
---
## 📊 Release History
### ✅ v2.6.0 - C3.x Codebase Analysis Suite (January 14, 2026)
**Focus:** Complete codebase analysis with multi-platform support
**Completed Features:**
- C3.x suite (C3.1-C3.8): Pattern detection, test extraction, architecture analysis
- Multi-platform support: Claude, Gemini, OpenAI, Markdown
- Platform adaptor architecture
- 18 MCP tools (up from 9)
- 700+ tests passing
- Unified multi-source scraping maturity
### ✅ v2.1.0 - Test Coverage & Quality (November 29, 2025)
**Focus:** Test coverage and unified scraping
**Completed Features:**
- Fixed 12 unified scraping tests
- GitHub repository scraping with unlimited local analysis
- PDF extraction and conversion
- 427 tests passing
### ✅ v1.0.0 - Production Release (October 19, 2025)
**First stable release**
**Core Features:**
- Documentation scraping with BFS
- Smart categorization
- Language detection
- Pattern extraction
- 12 preset configurations
- MCP server with 9 tools
- Large documentation support (40K+ pages)
- Auto-upload functionality
---
## 📅 Release Planning
### Release: v3.7.0 (Estimated: Q3 2026)
**Focus:** Developer Experience & Integrations
**Planned Features:**
- CI/CD integration examples
- Docker containerization
- Enhanced scraping formats (Sphinx, Docusaurus detection)
- Performance optimizations
- Real-time documentation monitoring
---
## 🔮 Long-term Vision (v3.0+)
### Major Features Under Consideration
#### Advanced Scraping
- Real-time documentation monitoring
- Automatic skill updates
- Change notifications
- Multi-language documentation support
#### Collaboration
- Collaborative skill curation
- Shared skill repositories
- Community ratings and reviews
- Skill marketplace
#### AI & Intelligence
- Enhanced AI analysis
- Better conflict detection algorithms
- Automatic documentation quality scoring
- Semantic understanding and natural language queries
#### Ecosystem
- VS Code extension
- IntelliJ/PyCharm plugin
- Interactive TUI mode
- Skill diff and merge tools
---
## 📈 Metrics & Goals
### Current State (v3.6.0) ✅
- ✅ 18 source types supported (17 + config)
- ✅ 12 preset configs
- ✅ 3,445+ tests (excellent coverage)
- ✅ 40 MCP tools
- ✅ 21 platform adaptors
- ✅ C3.x codebase analysis suite complete
- ✅ Multi-source synthesis with generic merge for any combination
### Goals for v3.7+
- 🎯 Professional website live
- 🎯 50+ preset configs
- 🎯 Video tutorial series (5+ videos)
- 🎯 100+ GitHub stars
- 🎯 Community contributions flowing
### Goals for v3.0+
- 🎯 Auto-detection for 80%+ of sites
- 🎯 <1 minute skill generation
- 🎯 Active community marketplace
- 🎯 Quality scoring system
- 🎯 Real-time monitoring
---
## 🤝 How to Influence the Roadmap
### Priority System
Features are prioritized based on:
1. **User impact** - How many users will benefit?
2. **Technical feasibility** - How complex is the implementation?
3. **Community interest** - How many upvotes/requests?
4. **Strategic alignment** - Does it fit our vision?
### Ways to Contribute
1. **Vote on Features** - ⭐ Star feature request issues
2. **Contribute Code** - Pick any task from the 136 available
3. **Share Feedback** - Open issues, share success stories
4. **Help with Documentation** - Write tutorials, improve docs
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
---
## 🎨 Flexibility Rules
1. **Pick any task, any order** - No rigid dependencies
2. **Start small** - Research tasks before implementation
3. **One task at a time** - Focus, complete, move on
4. **Switch anytime** - Not enjoying it? Pick another!
5. **Document as you go** - Each task should update docs
6. **Test incrementally** - Each task should have a quick test
7. **Ship early** - Don't wait for "complete" features
---
## 📊 Progress Tracking
**Completed Tasks:** 15+ (C3.1, C3.2, C3.3, C3.4, C3.5, C3.6, C3.7, A1.1, A1.2, A1.3, A1.7, E1.1, E1.3, E2.6, F1.5)
**In Progress:** v3.7.0 planning
**Total Available Tasks:** 136
**No pressure, no deadlines, just progress!**
---
## 🔗 Related Projects
- [Model Context Protocol](https://modelcontextprotocol.io/)
- [Claude Code](https://claude.ai/code)
- [Anthropic Claude](https://claude.ai)
- Documentation frameworks we support: Docusaurus, GitBook, VuePress, Sphinx, MkDocs
---
## 📚 Learn More
- **Project Board**: https://github.com/users/yusufkaraaslan/projects/2
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)
- **Contributing**: [CONTRIBUTING.md](CONTRIBUTING.md)
- **Discussions**: https://github.com/yusufkaraaslan/Skill_Seekers/discussions
- **Issues**: https://github.com/yusufkaraaslan/Skill_Seekers/issues
---
**Last Updated:** May 30, 2026
**Philosophy:** Small steps → Consistent progress → Compound results
**Together, we're building the future of documentation-to-AI skill conversion!** 🚀
+485
View File
@@ -0,0 +1,485 @@
# Troubleshooting Guide
Common issues and solutions when using Skill Seeker.
---
## Installation Issues
### Python Not Found
**Error:**
```
python3: command not found
```
**Solutions:**
1. **Check if Python is installed:**
```bash
which python3
python --version # Try without the 3
```
2. **Install Python:**
- **macOS:** `brew install python3`
- **Linux:** `sudo apt install python3 python3-pip`
- **Windows:** Download from python.org, check "Add to PATH"
3. **Use python instead of python3:**
```bash
python -m skill_seekers --help
```
### Module Not Found
**Error:**
```
ModuleNotFoundError: No module named 'requests'
ModuleNotFoundError: No module named 'bs4'
ModuleNotFoundError: No module named 'mcp'
```
**Solutions:**
1. **Install the package in editable mode (critical first step):**
```bash
pip install -e .
```
2. **Install MCP extras if needed:**
```bash
pip install -e ".[mcp]"
```
3. **Use --user flag if permission denied:**
```bash
pip install --user -e .
```
4. **Check pip is working:**
```bash
pip3 --version
```
### Permission Denied
**Error:**
```
Permission denied: '/usr/local/lib/python3.x/...'
```
**Solutions:**
1. **Use --user flag:**
```bash
pip3 install --user -e .
```
2. **Use sudo (not recommended):**
```bash
sudo pip install -e .
```
3. **Use virtual environment (best practice):**
```bash
python3 -m venv venv
source venv/bin/activate
pip install -e .
```
---
## Runtime Issues
### File Not Found
**Error:**
```
FileNotFoundError: [Errno 2] No such file or directory: 'src/skill_seekers/cli/main.py'
```
**Solutions:**
1. **Check you're in the Skill_Seekers directory:**
```bash
pwd
# Should show: .../Skill_Seekers
ls
# Should show: README.md, src/, configs/, tests/
```
2. **Change to the correct directory:**
```bash
cd ~/Projects/Skill_Seekers # Adjust path
```
3. **Ensure the package is installed:**
```bash
pip install -e .
```
### Config File Not Found
**Error:**
```
❌ Error: Config file not found: configs/myconfig.json
```
**Understanding Config Locations:**
The tool searches for configs in this order:
1. Exact path as provided
2. `./configs/` (current directory)
3. `~/.config/skill-seekers/configs/` (user config directory)
4. SkillSeekersWeb.com API (preset configs)
**Solutions:**
1. **Place config in user directory (recommended for custom configs):**
```bash
mkdir -p ~/.config/skill-seekers/configs
cp myconfig.json ~/.config/skill-seekers/configs/
# Now you can use it from anywhere
skill-seekers create --config myconfig.json
```
2. **Place config in current directory (project-specific):**
```bash
mkdir -p configs
cp myconfig.json configs/
skill-seekers create --config configs/myconfig.json
```
3. **Use absolute path:**
```bash
skill-seekers create --config /full/path/to/myconfig.json
```
4. **Check if it's a preset config (auto-downloads):**
```bash
# List all available presets
skill-seekers estimate --all
# Use preset (auto-fetched from API)
skill-seekers create --config react.json
```
5. **Create new config interactively:**
```bash
skill-seekers create --interactive
```
---
## MCP Setup Issues
### MCP Server Not Loading
**Symptoms:**
- Tools don't appear in Claude Code
- "List all available configs" doesn't work
**Solutions:**
1. **Check configuration file:**
```bash
cat ~/.config/claude-code/mcp.json
```
2. **Verify paths are ABSOLUTE (not placeholders):**
```json
{
"mcpServers": {
"skill-seeker": {
"command": "python",
"args": [
"-m",
"skill_seekers.mcp.server_fastmcp"
]
}
}
}
```
❌ **Bad:** `$REPO_PATH` or `/path/to/Skill_Seekers`
✅ **Good:** `/Users/john/Projects/Skill_Seekers`
3. **Test server manually:**
```bash
cd ~/Projects/Skill_Seekers
python -m skill_seekers.mcp.server_fastmcp
# Should start without errors (Ctrl+C to stop)
```
4. **Re-run setup script:**
```bash
./setup_mcp.sh
# Select "y" for auto-configure
```
5. **RESTART Claude Code completely:**
- Quit (don't just close window)
- Reopen
### Placeholder Paths in Config
**Problem:** Config has `$REPO_PATH` or `/Users/username/` instead of real paths
**Solution:**
```bash
# Get your actual path
cd ~/Projects/Skill_Seekers
pwd
# Copy this path
# Edit config
nano ~/.config/claude-code/mcp.json
# Replace ALL instances of placeholders with your actual path
# Save (Ctrl+O, Enter, Ctrl+X)
# Restart Claude Code
```
### Tools Appear But Don't Work
**Symptoms:**
- Tools listed but commands fail
- "Error executing tool" messages
**Solutions:**
1. **Check working directory:**
```json
{
"cwd": "/FULL/PATH/TO/Skill_Seekers"
}
```
2. **Verify package is installed:**
```bash
pip list | grep skill-seekers
python -c "import skill_seekers; print(skill_seekers.__version__)"
```
3. **Test CLI tools directly:**
```bash
skill-seekers create --help
```
---
## Scraping Issues
### Slow or Hanging
**Solutions:**
1. **Check network connection:**
```bash
ping google.com
curl -I https://docs.yoursite.com
```
2. **Use smaller max_pages for testing:**
```bash
skill-seekers create --config configs/test.json --max-pages 5
```
3. **Increase rate_limit in config:**
```json
{
"rate_limit": 1.0 // Increase from 0.5
}
```
### No Content Extracted
**Problem:** Pages scraped but content is empty
**Solutions:**
1. **Check selector in config:**
```bash
# Test with browser dev tools
# Look for: article, main, div[role="main"], div.content
```
2. **Verify website is accessible:**
```bash
curl https://docs.example.com
```
3. **Try different selectors:**
```json
{
"selectors": {
"main_content": "article" // Try: main, div.content, etc.
}
}
```
### Rate Limiting / 429 Errors
**Error:**
```
HTTP Error 429: Too Many Requests
```
**Solutions:**
1. **Increase rate_limit:**
```json
{
"rate_limit": 2.0 // Wait 2 seconds between requests
}
```
2. **Reduce max_pages:**
```json
{
"max_pages": 50 // Scrape fewer pages
}
```
3. **Try again later:**
```bash
# Wait an hour and retry
```
---
## Platform-Specific Issues
### macOS
**Issue:** Can't run `./setup_mcp.sh`
**Solution:**
```bash
chmod +x setup_mcp.sh
./setup_mcp.sh
```
**Issue:** Homebrew not installed
**Solution:**
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
### Linux
**Issue:** pip3 not found
**Solution:**
```bash
sudo apt update
sudo apt install python3-pip
```
**Issue:** Permission errors
**Solution:**
```bash
# Use --user flag
pip3 install --user -e .
```
### Windows (WSL)
**Issue:** Python not in PATH
**Solution:**
1. Reinstall Python
2. Check "Add Python to PATH"
3. Or add manually to PATH
**Issue:** Line ending errors
**Solution:**
```bash
dos2unix setup_mcp.sh
./setup_mcp.sh
```
---
## Verification Commands
Use these to check your setup:
```bash
# 1. Check Python
python3 --version # Should be 3.10+
# 2. Check package is installed
pip list | grep skill-seekers
python -c "import skill_seekers; print(skill_seekers.__version__)"
# 3. Check source layout
ls src/skill_seekers/cli/
ls src/skill_seekers/mcp/
ls configs/
# 4. Check MCP config
cat ~/.config/claude-code/mcp.json
# 5. Test scraper
skill-seekers create --help
# 6. Test MCP server
timeout 3 python -m skill_seekers.mcp.server_fastmcp || echo "Server OK"
# 7. Check git repo
git status
git log --oneline -5
```
---
## Getting Help
If none of these solutions work:
1. **Check existing issues:**
https://github.com/yusufkaraaslan/Skill_Seekers/issues
2. **Open a new issue with:**
- Your OS (macOS 13, Ubuntu 22.04, etc.)
- Python version (`python3 --version`)
- Full error message
- What command you ran
- Output of verification commands above
3. **Include this debug info:**
```bash
# System info
uname -a
python3 --version
pip3 --version
# Skill Seeker info
cd ~/Projects/Skill_Seekers # Your path
pwd
git log --oneline -1
ls -la cli/ mcp/ configs/
# MCP config (if using MCP)
cat ~/.config/claude-code/mcp.json
```
---
## Quick Fixes Checklist
- [ ] In the Skill_Seekers directory? (`pwd`)
- [ ] Python 3.10+ installed? (`python3 --version`)
- [ ] Package installed? (`pip list | grep skill-seekers`)
- [ ] Config file exists? (`ls configs/yourconfig.json`)
- [ ] Internet connection working? (`ping google.com`)
- [ ] For MCP: Config uses absolute paths? (not `$REPO_PATH`)
- [ ] For MCP: Claude Code restarted? (quit and reopen)
---
**Still stuck?** Open an issue: https://github.com/yusufkaraaslan/Skill_Seekers/issues/new
+2
View File
@@ -0,0 +1,2 @@
# configs_repo is now a git submodule, tracked in .gitmodules
# configs_repo/
+267
View File
@@ -0,0 +1,267 @@
# Skill Seekers Config API
FastAPI backend for discovering and downloading Skill Seekers configuration files.
## 🚀 Endpoints
### Base URL
- **Production**: `https://skillseekersweb.com`
- **Local**: `http://localhost:8000`
### Available Endpoints
#### 1. **GET /** - API Information
Returns API metadata and available endpoints.
```bash
curl https://skillseekersweb.com/
```
**Response:**
```json
{
"name": "Skill Seekers Config API",
"version": "1.0.0",
"endpoints": {
"/api/configs": "List all available configs",
"/api/configs/{name}": "Get specific config details",
"/api/categories": "List all categories",
"/docs": "API documentation"
},
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
"website": "https://skillseekersweb.com"
}
```
---
#### 2. **GET /api/configs** - List All Configs
Returns list of all available configs with metadata.
**Query Parameters:**
- `category` (optional) - Filter by category (e.g., `web-frameworks`)
- `tag` (optional) - Filter by tag (e.g., `javascript`)
- `type` (optional) - Filter by type (`single-source` or `unified`)
```bash
# Get all configs
curl https://skillseekersweb.com/api/configs
# Filter by category
curl https://skillseekersweb.com/api/configs?category=web-frameworks
# Filter by tag
curl https://skillseekersweb.com/api/configs?tag=javascript
# Filter by type
curl https://skillseekersweb.com/api/configs?type=unified
```
**Response:**
```json
{
"version": "1.0.0",
"total": 24,
"filters": null,
"configs": [
{
"name": "react",
"description": "React framework for building user interfaces...",
"type": "single-source",
"category": "web-frameworks",
"tags": ["javascript", "frontend", "documentation"],
"primary_source": "https://react.dev/",
"max_pages": 300,
"file_size": 1055,
"last_updated": "2025-11-30T09:26:07+00:00",
"download_url": "https://skillseekersweb.com/api/download/react.json",
"config_file": "react.json"
}
]
}
```
---
#### 3. **GET /api/configs/{name}** - Get Specific Config
Returns detailed information about a specific config.
```bash
curl https://skillseekersweb.com/api/configs/react
```
**Response:**
```json
{
"name": "react",
"description": "React framework for building user interfaces...",
"type": "single-source",
"category": "web-frameworks",
"tags": ["javascript", "frontend", "documentation"],
"primary_source": "https://react.dev/",
"max_pages": 300,
"file_size": 1055,
"last_updated": "2025-11-30T09:26:07+00:00",
"download_url": "https://skillseekersweb.com/api/download/react.json",
"config_file": "react.json"
}
```
---
#### 4. **GET /api/categories** - List Categories
Returns all available categories with config counts.
```bash
curl https://skillseekersweb.com/api/categories
```
**Response:**
```json
{
"total_categories": 5,
"categories": {
"web-frameworks": 7,
"game-engines": 2,
"devops": 2,
"css-frameworks": 1,
"uncategorized": 12
}
}
```
---
#### 5. **GET /api/download/{config_name}** - Download Config File
Downloads the actual config JSON file.
```bash
# Download react config
curl -O https://skillseekersweb.com/api/download/react.json
# Download with just name (auto-adds .json)
curl -O https://skillseekersweb.com/api/download/react
```
---
#### 6. **GET /health** - Health Check
Health check endpoint for monitoring.
```bash
curl https://skillseekersweb.com/health
```
**Response:**
```json
{
"status": "healthy",
"service": "skill-seekers-api"
}
```
---
#### 7. **GET /docs** - API Documentation
Interactive OpenAPI documentation (Swagger UI).
Visit: `https://skillseekersweb.com/docs`
---
## 📦 Metadata Fields
Each config includes the following metadata:
| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Config identifier (e.g., "react") |
| `description` | string | What the config is used for |
| `type` | string | "single-source" or "unified" |
| `category` | string | Auto-categorized (e.g., "web-frameworks") |
| `tags` | array | Relevant tags (e.g., ["javascript", "frontend"]) |
| `primary_source` | string | Main documentation URL or repo |
| `max_pages` | int | Estimated page count for scraping |
| `file_size` | int | Config file size in bytes |
| `last_updated` | string | ISO 8601 date of last update |
| `download_url` | string | Direct download link |
| `config_file` | string | Filename (e.g., "react.json") |
---
## 🏗️ Categories
Configs are auto-categorized into:
- **web-frameworks** - Web development frameworks (React, Django, FastAPI, etc.)
- **game-engines** - Game development engines (Godot, Unity, etc.)
- **devops** - DevOps tools (Kubernetes, Ansible, etc.)
- **css-frameworks** - CSS frameworks (Tailwind, etc.)
- **development-tools** - Dev tools (Claude Code, etc.)
- **gaming** - Gaming platforms (Steam, etc.)
- **uncategorized** - Other configs
---
## 🏷️ Tags
Common tags include:
- **Language**: `javascript`, `python`, `php`
- **Domain**: `frontend`, `backend`, `devops`, `game-development`
- **Type**: `documentation`, `github`, `pdf`, `multi-source`
- **Tech**: `css`, `testing`, `api`
---
## 🚀 Local Development
### Setup
```bash
# Install dependencies
cd api
pip install -r requirements.txt
# Run server
python main.py
```
API will be available at `http://localhost:8000`
### Testing
```bash
# Test health check
curl http://localhost:8000/health
# List all configs
curl http://localhost:8000/api/configs
# Get specific config
curl http://localhost:8000/api/configs/react
# Download config
curl -O http://localhost:8000/api/download/react.json
```
---
## 📝 Deployment
### Render
This API is configured for Render deployment via `render.yaml`.
1. Push to GitHub
2. Connect repository to Render
3. Render auto-deploys from `render.yaml`
4. Configure custom domain: `skillseekersweb.com`
---
## 🔗 Links
- **API Documentation**: https://skillseekersweb.com/docs
- **GitHub Repository**: https://github.com/yusufkaraaslan/Skill_Seekers
- **Main Project**: https://github.com/yusufkaraaslan/Skill_Seekers#readme
+6
View File
@@ -0,0 +1,6 @@
"""
Skill Seekers Config API
FastAPI backend for discovering and downloading config files
"""
__version__ = "1.0.0"
+366
View File
@@ -0,0 +1,366 @@
#!/usr/bin/env python3
"""
Config Analyzer - Extract metadata from Skill Seekers config files
"""
import json
import subprocess
from datetime import datetime
from pathlib import Path
from typing import Any
class ConfigAnalyzer:
"""Analyzes Skill Seekers config files and extracts metadata"""
# Category mapping based on config content
CATEGORY_MAPPING = {
"web-frameworks": ["react", "vue", "django", "fastapi", "laravel", "astro", "hono"],
"game-engines": ["godot", "unity", "unreal"],
"devops": ["kubernetes", "ansible", "docker", "terraform"],
"css-frameworks": ["tailwind", "bootstrap", "bulma"],
"development-tools": ["claude-code", "vscode", "git"],
"gaming": ["steam"],
"testing": ["pytest", "jest", "test"],
}
# Tag extraction keywords
TAG_KEYWORDS = {
"javascript": ["react", "vue", "astro", "hono", "javascript", "js", "node"],
"python": ["django", "fastapi", "ansible", "python", "flask"],
"php": ["laravel", "php"],
"frontend": ["react", "vue", "astro", "tailwind", "frontend", "ui"],
"backend": ["django", "fastapi", "laravel", "backend", "server", "api"],
"css": ["tailwind", "css", "styling"],
"game-development": ["godot", "unity", "unreal", "game"],
"devops": ["kubernetes", "ansible", "docker", "k8s", "devops"],
"documentation": ["docs", "documentation"],
"testing": ["test", "testing", "pytest", "jest"],
}
def __init__(self, config_dir: Path, base_url: str = "https://api.skillseekersweb.com"):
"""
Initialize config analyzer
Args:
config_dir: Path to configs directory
base_url: Base URL for download links
"""
self.config_dir = Path(config_dir)
self.base_url = base_url
if not self.config_dir.exists():
raise ValueError(f"Config directory not found: {self.config_dir}")
def analyze_all_configs(self) -> list[dict[str, Any]]:
"""
Analyze all config files and extract metadata
Returns:
List of config metadata dicts
"""
configs = []
# Find all JSON files recursively in configs directory and subdirectories
for config_file in sorted(self.config_dir.rglob("*.json")):
# Skip test/example configs in test-examples directory
if "test-examples" in config_file.parts:
continue
try:
metadata = self.analyze_config(config_file)
if metadata: # Skip invalid configs
configs.append(metadata)
except Exception as e:
print(f"Warning: Failed to analyze {config_file.name}: {e}")
continue
return configs
def analyze_config(self, config_path: Path) -> dict[str, Any] | None:
"""
Analyze a single config file and extract metadata
Args:
config_path: Path to config JSON file
Returns:
Config metadata dict or None if invalid
"""
try:
# Read config file
with open(config_path) as f:
config_data = json.load(f)
# Skip if no name field
if "name" not in config_data:
return None
name = config_data["name"]
description = config_data.get("description", "")
# Determine config type
config_type = self._determine_type(config_data)
# Get primary source (base_url or repo)
primary_source = self._get_primary_source(config_data, config_type)
# Use directory name as category (official/{category}/{name}.json)
# Fall back to keyword-based categorization if not in a named subdirectory
category = self._categorize_config(name, description, config_data, config_path)
# Extract tags
tags = self._extract_tags(name, description, config_data)
# Get file metadata
file_size = config_path.stat().st_size
last_updated = self._get_last_updated(config_path)
# Generate download URL
download_url = f"{self.base_url}/api/download/{config_path.name}"
# Get max_pages (for estimation)
max_pages = self._get_max_pages(config_data)
return {
"name": name,
"description": description,
"type": config_type,
"category": category,
"tags": tags,
"primary_source": primary_source,
"max_pages": max_pages,
"file_size": file_size,
"last_updated": last_updated,
"download_url": download_url,
"config_file": config_path.name,
}
except json.JSONDecodeError as e:
print(f"Invalid JSON in {config_path.name}: {e}")
return None
except Exception as e:
print(f"Error analyzing {config_path.name}: {e}")
return None
def get_config_by_name(self, name: str) -> dict[str, Any] | None:
"""
Get config metadata by name
Args:
name: Config name (e.g., "react", "django")
Returns:
Config metadata or None if not found
"""
configs = self.analyze_all_configs()
for config in configs:
if config["name"] == name:
return config
return None
def _determine_type(self, config_data: dict[str, Any]) -> str:
"""
Determine if config is single-source or unified
Args:
config_data: Config JSON data
Returns:
"single-source" or "unified"
"""
# Unified configs have "sources" array
if "sources" in config_data:
return "unified"
# Check for merge_mode (another indicator of unified configs)
if "merge_mode" in config_data:
return "unified"
return "single-source"
def _get_primary_source(self, config_data: dict[str, Any], config_type: str) -> str:
"""
Get primary source URL/repo
Args:
config_data: Config JSON data
config_type: "single-source" or "unified"
Returns:
Primary source URL or repo name
"""
if config_type == "unified":
# Get first source
sources = config_data.get("sources", [])
if sources:
first_source = sources[0]
if first_source.get("type") == "documentation":
return first_source.get("base_url", "")
elif first_source.get("type") == "github":
return f"github.com/{first_source.get('repo', '')}"
elif first_source.get("type") == "pdf":
return first_source.get("pdf_url", "PDF file")
return "Multiple sources"
# Single-source configs
if "base_url" in config_data:
return config_data["base_url"]
elif "repo" in config_data:
return f"github.com/{config_data['repo']}"
elif "pdf_url" in config_data or "pdf" in config_data:
return "PDF file"
return "Unknown"
def _categorize_config(
self,
name: str,
description: str,
config_data: dict[str, Any],
config_path: Path | None = None,
) -> str:
"""
Categorize config using directory structure first, then keyword fallback.
The configs_repo organizes files as official/{category}/{name}.json so the
parent directory name is the authoritative category.
Args:
name: Config name
description: Config description
config_data: Full config data
config_path: Path to config file (used to read directory-based category)
Returns:
Category name
"""
# Primary: use directory structure (official/{category}/{name}.json)
if config_path is not None:
parent = config_path.parent.name
# Exclude generic/root directories from being used as categories
if parent not in ("official", "community", "configs", "configs_repo", "."):
return parent
# Fallback: keyword matching against config name
name_lower = name.lower()
for category, keywords in self.CATEGORY_MAPPING.items():
if any(keyword in name_lower for keyword in keywords):
return category
# Fallback: description hints
desc_lower = description.lower()
if "framework" in desc_lower or "library" in desc_lower:
if any(word in desc_lower for word in ["web", "frontend", "backend", "api"]):
return "web-frameworks"
if "game" in desc_lower or "engine" in desc_lower:
return "game-engines"
if "devops" in desc_lower or "deployment" in desc_lower or "infrastructure" in desc_lower:
return "devops"
return "uncategorized"
def _extract_tags(self, name: str, description: str, config_data: dict[str, Any]) -> list[str]:
"""
Extract relevant tags from config
Args:
name: Config name
description: Config description
config_data: Full config data
Returns:
List of tags
"""
tags = set()
name_lower = name.lower()
desc_lower = description.lower()
# Check against tag keywords
for tag, keywords in self.TAG_KEYWORDS.items():
if any(keyword in name_lower or keyword in desc_lower for keyword in keywords):
tags.add(tag)
# Add config type as tag
config_type = self._determine_type(config_data)
if config_type == "unified":
tags.add("multi-source")
# Add source type tags
if "base_url" in config_data or (
config_type == "unified"
and any(s.get("type") == "documentation" for s in config_data.get("sources", []))
):
tags.add("documentation")
if "repo" in config_data or (
config_type == "unified"
and any(s.get("type") == "github" for s in config_data.get("sources", []))
):
tags.add("github")
if (
"pdf" in config_data
or "pdf_url" in config_data
or (
config_type == "unified"
and any(s.get("type") == "pdf" for s in config_data.get("sources", []))
)
):
tags.add("pdf")
return sorted(list(tags))
def _get_max_pages(self, config_data: dict[str, Any]) -> int | None:
"""
Get max_pages value from config
Args:
config_data: Config JSON data
Returns:
max_pages value or None
"""
# Single-source configs
if "max_pages" in config_data:
return config_data["max_pages"]
# Unified configs - get from first documentation source
if "sources" in config_data:
for source in config_data["sources"]:
if source.get("type") == "documentation" and "max_pages" in source:
return source["max_pages"]
return None
def _get_last_updated(self, config_path: Path) -> str:
"""
Get last updated date from git history
Args:
config_path: Path to config file
Returns:
ISO format date string
"""
try:
# Try to get last commit date for this file
result = subprocess.run(
["git", "log", "-1", "--format=%cI", str(config_path)],
cwd=config_path.parent.parent,
capture_output=True,
text=True,
timeout=5,
)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
except Exception:
pass
# Fallback to file modification time
mtime = config_path.stat().st_mtime
return datetime.fromtimestamp(mtime).isoformat()
+204
View File
@@ -0,0 +1,204 @@
#!/usr/bin/env python3
"""
Skill Seekers Config API
FastAPI backend for listing available skill configs
"""
from pathlib import Path
from typing import Any
from config_analyzer import ConfigAnalyzer
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
app = FastAPI(
title="Skill Seekers Config API",
description="API for discovering and downloading Skill Seekers configuration files",
version="1.0.0",
docs_url="/docs",
redoc_url="/redoc",
)
# CORS middleware - allow all origins for public API
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Initialize config analyzer
# Try configs_repo first (production), fallback to configs (local development)
CONFIG_DIR = Path(__file__).parent / "configs_repo" / "official"
if not CONFIG_DIR.exists():
CONFIG_DIR = Path(__file__).parent.parent / "configs"
analyzer = ConfigAnalyzer(CONFIG_DIR)
@app.get("/")
async def root():
"""Root endpoint - API information"""
return {
"name": "Skill Seekers Config API",
"version": "1.0.0",
"endpoints": {
"/api/configs": "List all available configs",
"/api/configs/{name}": "Get specific config details",
"/api/categories": "List all categories",
"/api/download/{name}": "Download config file",
"/docs": "API documentation",
},
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
"configs_repository": "https://github.com/yusufkaraaslan/skill-seekers-configs",
"website": "https://api.skillseekersweb.com",
}
@app.get("/api/configs")
async def list_configs(
category: str | None = None, tag: str | None = None, type: str | None = None
) -> dict[str, Any]:
"""
List all available configs with metadata
Query Parameters:
- category: Filter by category (e.g., "web-frameworks")
- tag: Filter by tag (e.g., "javascript")
- type: Filter by type ("single-source" or "unified")
Returns:
- version: API version
- total: Total number of configs
- filters: Applied filters
- configs: List of config metadata
"""
try:
# Get all configs
all_configs = analyzer.analyze_all_configs()
# Apply filters
configs = all_configs
filters_applied = {}
if category:
configs = [c for c in configs if c.get("category") == category]
filters_applied["category"] = category
if tag:
configs = [c for c in configs if tag in c.get("tags", [])]
filters_applied["tag"] = tag
if type:
configs = [c for c in configs if c.get("type") == type]
filters_applied["type"] = type
return {
"version": "1.0.0",
"total": len(configs),
"filters": filters_applied if filters_applied else None,
"configs": configs,
}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Error analyzing configs: {str(e)}")
@app.get("/api/configs/{name}")
async def get_config(name: str) -> dict[str, Any]:
"""
Get detailed information about a specific config
Path Parameters:
- name: Config name (e.g., "react", "django")
Returns:
- Full config metadata including all fields
"""
try:
config = analyzer.get_config_by_name(name)
if not config:
raise HTTPException(status_code=404, detail=f"Config '{name}' not found")
return config
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Error loading config: {str(e)}")
@app.get("/api/categories")
async def list_categories() -> dict[str, Any]:
"""
List all available categories with config counts
Returns:
- categories: Dict of category names to config counts
- total_categories: Total number of categories
"""
try:
configs = analyzer.analyze_all_configs()
# Count configs per category
category_counts = {}
for config in configs:
cat = config.get("category", "uncategorized")
category_counts[cat] = category_counts.get(cat, 0) + 1
return {"total_categories": len(category_counts), "categories": category_counts}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Error analyzing categories: {str(e)}")
@app.get("/api/download/{config_name}")
async def download_config(config_name: str):
"""
Download a specific config file
Path Parameters:
- config_name: Config filename (e.g., "react.json", "django.json")
Returns:
- JSON file for download
"""
try:
# Validate filename (prevent directory traversal)
if ".." in config_name or "/" in config_name or "\\" in config_name:
raise HTTPException(status_code=400, detail="Invalid config name")
# Ensure .json extension
if not config_name.endswith(".json"):
config_name = f"{config_name}.json"
# Search recursively in all subdirectories
config_path = None
for found_path in CONFIG_DIR.rglob(config_name):
config_path = found_path
break
if not config_path or not config_path.exists():
raise HTTPException(status_code=404, detail=f"Config file '{config_name}' not found")
return FileResponse(path=config_path, media_type="application/json", filename=config_name)
except HTTPException:
raise
except Exception as e:
raise HTTPException(status_code=500, detail=f"Error downloading config: {str(e)}")
@app.get("/health")
async def health_check():
"""Health check endpoint for monitoring"""
return {"status": "healthy", "service": "skill-seekers-api"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
+3
View File
@@ -0,0 +1,3 @@
fastapi==0.115.0
uvicorn[standard]==0.32.0
python-multipart==0.0.12
+25
View File
@@ -0,0 +1,25 @@
coverage:
status:
project:
default:
target: auto
threshold: 1%
informational: false
patch:
default:
# Patch coverage is informational only — doesn't fail CI.
# Rationale: version bumps, docs, config, and test-only commits have
# naturally low or zero patch coverage and shouldn't block releases.
informational: true
comment:
layout: "reach, diff, flags, files"
behavior: default
require_changes: false
# Ignore files that don't need coverage
ignore:
- "tests/**/*"
- "docs/**/*"
- "**/__init__.py"
- "src/skill_seekers/_version.py"
+32
View File
@@ -0,0 +1,32 @@
{
"name": "astrovalley",
"description": "Space farming/automation game with combat and exploration - GitHub repo with deep codebase analysis",
"version": "1.0.0",
"target": "claude",
"sources": [
{
"type": "github",
"repo": "yusufkaraaslan/AstroValley",
"clone_path": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/AstroValley",
"enable_codebase_analysis": true,
"analysis_depth": "deep",
"include_issues": true,
"include_pull_requests": true,
"include_discussions": false,
"max_issues": 100,
"codebase_options": {
"build_api_reference": true,
"build_dependency_graph": true,
"detect_patterns": true,
"extract_test_examples": true,
"extract_config_patterns": true,
"detect_architecture": true
}
}
],
"synthesis_strategy": "comprehensive",
"ai_enhancement": {
"mode": "auto",
"enable": true
}
}
+276
View File
@@ -0,0 +1,276 @@
{
"name": "blender",
"description": "Complete Blender 3D creation suite knowledge base combining official documentation and source code analysis. Use for comprehensive understanding of 3D modeling, animation, rendering, compositing, video editing, game development, Python scripting, and Blender's internal architecture.",
"merge_mode": "claude-enhanced",
"sources": [
{
"type": "documentation",
"base_url": "https://docs.blender.org/manual/en/latest/",
"extract_api": true,
"selectors": {
"main_content": "article[role='main']",
"title": "h1",
"code_blocks": "pre code, div.highlight pre"
},
"url_patterns": {
"include": [
"/getting_started/",
"/interface/",
"/editors/",
"/modeling/",
"/sculpt_paint/",
"/grease_pencil/",
"/animation/",
"/physics/",
"/render/",
"/scene_layout/",
"/compositing/",
"/video_editing/",
"/files/",
"/addons/",
"/advanced/",
"/troubleshooting/"
],
"exclude": [
"/_static/",
"/_images/",
"/search.html",
"/genindex.html",
"/glossary.html",
"/index.html$"
]
},
"categories": {
"getting_started": [
"getting_started",
"installing",
"configuration",
"introduction",
"quickstart",
"about"
],
"interface": [
"interface",
"window_system",
"keymap",
"controls",
"operators",
"tools",
"ui",
"navigation"
],
"modeling": [
"modeling",
"mesh",
"curve",
"surface",
"metaball",
"text",
"volume",
"geometry_nodes",
"modifiers",
"mesh_tools",
"edit_mode"
],
"sculpting": [
"sculpt",
"sculpting",
"brush",
"texture_paint",
"vertex_paint",
"weight_paint",
"dynamic_paint"
],
"grease_pencil": [
"grease_pencil",
"2d_animation",
"drawing",
"stroke"
],
"animation": [
"animation",
"keyframe",
"rigging",
"armature",
"constraints",
"drivers",
"shape_keys",
"motion_paths",
"timeline",
"dope_sheet",
"graph_editor",
"nla"
],
"physics": [
"physics",
"simulation",
"particles",
"hair",
"fluid",
"cloth",
"soft_body",
"rigid_body",
"dynamic_paint",
"force_fields"
],
"shading": [
"shading",
"shader",
"material",
"texture",
"nodes",
"shader_nodes",
"lighting",
"world"
],
"rendering": [
"render",
"eevee",
"cycles",
"workbench",
"freestyle",
"camera",
"output",
"color_management",
"optimization"
],
"compositing": [
"compositing",
"compositor",
"nodes",
"color_correction",
"filters",
"matte"
],
"video_editing": [
"video_editing",
"vse",
"sequencer",
"strips",
"effects",
"preview"
],
"scene_layout": [
"scene",
"object",
"collection",
"properties",
"outliner",
"view_layers"
],
"files_assets": [
"files",
"import",
"export",
"asset",
"library",
"data_blocks",
"linking",
"append"
],
"addons": [
"addon",
"plugin",
"extension",
"import_export"
],
"scripting": [
"scripting",
"python",
"api",
"bpy",
"operator",
"custom",
"automation"
],
"advanced": [
"advanced",
"command_line",
"app_templates",
"extensions",
"limits"
],
"troubleshooting": [
"troubleshooting",
"crash",
"recover",
"gpu",
"graphics"
]
},
"rate_limit": 0.5,
"max_pages": 1500
},
{
"type": "github",
"repo": "blender/blender",
"github_token": null,
"code_analysis_depth": "deep",
"include_code": true,
"include_issues": true,
"max_issues": 200,
"include_changelog": true,
"include_releases": true,
"include_wiki": true,
"file_patterns": [
"source/blender/blenkernel/**/*.h",
"source/blender/blenkernel/**/*.c",
"source/blender/blenkernel/**/*.cc",
"source/blender/blenlib/**/*.h",
"source/blender/blenlib/**/*.c",
"source/blender/blenlib/**/*.cc",
"source/blender/editors/**/*.h",
"source/blender/editors/**/*.c",
"source/blender/editors/**/*.cc",
"source/blender/makesdna/**/*.h",
"source/blender/makesrna/**/*.c",
"source/blender/makesrna/**/*.cc",
"source/blender/render/**/*.h",
"source/blender/render/**/*.c",
"source/blender/render/**/*.cc",
"source/blender/python/**/*.h",
"source/blender/python/**/*.c",
"source/blender/python/**/*.cc",
"source/blender/python/**/*.py",
"source/blender/depsgraph/**/*.h",
"source/blender/depsgraph/**/*.cc",
"source/blender/draw/**/*.h",
"source/blender/draw/**/*.c",
"source/blender/draw/**/*.cc",
"source/blender/gpu/**/*.h",
"source/blender/gpu/**/*.c",
"source/blender/gpu/**/*.cc",
"source/blender/nodes/**/*.h",
"source/blender/nodes/**/*.c",
"source/blender/nodes/**/*.cc",
"source/blender/windowmanager/**/*.h",
"source/blender/windowmanager/**/*.c",
"source/blender/windowmanager/**/*.cc",
"intern/cycles/**/*.h",
"intern/cycles/**/*.cpp",
"scripts/startup/bl_ui/**/*.py",
"scripts/modules/**/*.py",
"release/scripts/startup/**/*.py",
"README.md",
"CONTRIBUTING.md",
"BUILD.md",
"CODE_OF_CONDUCT.md"
],
"exclude_patterns": [
"**/tests/**",
"**/__pycache__/**",
"build_files/**",
"doc/**"
],
"analysis_features": {
"detect_patterns": true,
"extract_tests": true,
"build_guides": true,
"extract_config": true,
"build_api_reference": true,
"analyze_dependencies": true,
"detect_architecture": true
}
}
]
}
+261
View File
@@ -0,0 +1,261 @@
{
"name": "claude-code-unified",
"description": "Claude Code CLI - unified knowledge base combining official documentation and leaked source code analysis. Use for understanding Claude Code internals, architecture, tools, commands, IDE integrations, MCP, plugins, skills, and development workflows.",
"merge_mode": "claude-enhanced",
"workflows": [
"complex-merge",
"cli-tooling",
"architecture-comprehensive",
"api-documentation",
"security-focus",
"ai-assistant-internals"
],
"workflow_vars": {
"depth": "comprehensive",
"merge_strategy": "priority",
"source_priority_order": "official_docs,code,community"
},
"sources": [
{
"type": "documentation",
"base_url": "https://code.claude.com/docs/en/",
"start_urls": [
"https://code.claude.com/docs/en/overview",
"https://code.claude.com/docs/en/quickstart",
"https://code.claude.com/docs/en/common-workflows",
"https://code.claude.com/docs/en/claude-code-on-the-web",
"https://code.claude.com/docs/en/desktop",
"https://code.claude.com/docs/en/chrome",
"https://code.claude.com/docs/en/vs-code",
"https://code.claude.com/docs/en/jetbrains",
"https://code.claude.com/docs/en/github-actions",
"https://code.claude.com/docs/en/gitlab-ci-cd",
"https://code.claude.com/docs/en/slack",
"https://code.claude.com/docs/en/sub-agents",
"https://code.claude.com/docs/en/plugins",
"https://code.claude.com/docs/en/discover-plugins",
"https://code.claude.com/docs/en/skills",
"https://code.claude.com/docs/en/output-styles",
"https://code.claude.com/docs/en/hooks-guide",
"https://code.claude.com/docs/en/headless",
"https://code.claude.com/docs/en/mcp",
"https://code.claude.com/docs/en/third-party-integrations",
"https://code.claude.com/docs/en/amazon-bedrock",
"https://code.claude.com/docs/en/google-vertex-ai",
"https://code.claude.com/docs/en/microsoft-foundry",
"https://code.claude.com/docs/en/network-config",
"https://code.claude.com/docs/en/llm-gateway",
"https://code.claude.com/docs/en/devcontainer",
"https://code.claude.com/docs/en/sandboxing",
"https://code.claude.com/docs/en/setup",
"https://code.claude.com/docs/en/iam",
"https://code.claude.com/docs/en/security",
"https://code.claude.com/docs/en/data-usage",
"https://code.claude.com/docs/en/monitoring-usage",
"https://code.claude.com/docs/en/costs",
"https://code.claude.com/docs/en/analytics",
"https://code.claude.com/docs/en/plugin-marketplaces",
"https://code.claude.com/docs/en/settings",
"https://code.claude.com/docs/en/terminal-config",
"https://code.claude.com/docs/en/model-config",
"https://code.claude.com/docs/en/memory",
"https://code.claude.com/docs/en/statusline",
"https://code.claude.com/docs/en/cli-reference",
"https://code.claude.com/docs/en/interactive-mode",
"https://code.claude.com/docs/en/slash-commands",
"https://code.claude.com/docs/en/checkpointing",
"https://code.claude.com/docs/en/hooks",
"https://code.claude.com/docs/en/plugins-reference",
"https://code.claude.com/docs/en/troubleshooting",
"https://code.claude.com/docs/en/legal-and-compliance"
],
"selectors": {
"main_content": "#content-area, #content-container, article, main",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": [
"/docs/en/"
],
"exclude": [
"/docs/fr/",
"/docs/de/",
"/docs/it/",
"/docs/ja/",
"/docs/es/",
"/docs/ko/",
"/docs/zh-CN/",
"/docs/zh-TW/",
"/docs/ru/",
"/docs/id/",
"/docs/pt/",
"/changelog",
"github.com"
]
},
"categories": {
"getting_started": [
"overview",
"quickstart",
"common-workflows"
],
"ide_integrations": [
"vs-code",
"jetbrains",
"desktop",
"chrome",
"claude-code-on-the-web",
"slack"
],
"ci_cd": [
"github-actions",
"gitlab-ci-cd"
],
"building": [
"sub-agents",
"subagent",
"plugins",
"discover-plugins",
"skills",
"output-styles",
"hooks-guide",
"headless",
"programmatic"
],
"mcp": [
"mcp",
"model-context-protocol"
],
"deployment": [
"third-party-integrations",
"amazon-bedrock",
"google-vertex-ai",
"microsoft-foundry",
"network-config",
"llm-gateway",
"devcontainer",
"sandboxing"
],
"administration": [
"setup",
"iam",
"security",
"data-usage",
"monitoring-usage",
"costs",
"analytics",
"plugin-marketplaces"
],
"configuration": [
"settings",
"terminal-config",
"model-config",
"memory",
"statusline"
],
"reference": [
"cli-reference",
"interactive-mode",
"slash-commands",
"checkpointing",
"hooks",
"plugins-reference"
],
"troubleshooting": [
"troubleshooting"
],
"legal": [
"legal-and-compliance"
]
},
"rate_limit": 0.5,
"max_pages": 250
},
{
"type": "local",
"path": "/Users/yusufkaraaslan/Github/cli-template",
"name": "claude-code-source",
"description": "Leaked Claude Code CLI TypeScript source code (main.tsx, tools, commands, components, Ink renderer, bridge, coordinator, plugins, skills, MCP, vim, voice, remote, server, memdir, tasks, state, schemas, entrypoints)",
"weight": 0.6,
"enhance_level": 3,
"include_code": true,
"file_patterns": [
"*.ts",
"*.tsx",
"*.md",
"*.json"
],
"skip_patterns": [
".git/",
"node_modules/",
"__pycache__/",
"*.map",
"*.js",
"dist/",
"build/"
],
"categories": {
"core": [
"main.tsx",
"commands.ts",
"tools.ts",
"Tool.ts",
"QueryEngine.ts",
"context.ts",
"cost-tracker.ts"
],
"commands": [
"commands/"
],
"tools": [
"tools/"
],
"ui": [
"components/",
"hooks/",
"screens/",
"ink/"
],
"integrations": [
"bridge/",
"coordinator/",
"remote/",
"server/",
"services/"
],
"features": [
"plugins/",
"skills/",
"vim/",
"voice/",
"tasks/",
"memdir/",
"state/",
"buddy/"
],
"infrastructure": [
"entrypoints/",
"migrations/",
"schemas/",
"query/",
"upstreamproxy/",
"native-ts/",
"outputStyles/",
"utils/",
"types/",
"constants/"
]
},
"analysis_features": {
"detect_patterns": true,
"extract_tests": true,
"build_guides": true,
"extract_config": true,
"build_api_reference": true,
"analyze_dependencies": true,
"detect_architecture": true
}
}
]
}
+164
View File
@@ -0,0 +1,164 @@
{
"name": "claude-code",
"description": "Claude Code CLI and development environment. Use for Claude Code features, tools, workflows, MCP integration, plugins, hooks, configuration, deployment, and AI-assisted development.",
"merge_mode": "rule-based",
"sources": [
{
"type": "documentation",
"_migration_note": "TODO: Migrate to external skill-seekers-configs repo. Kept temporarily to preserve PR #244 work.",
"base_url": "https://code.claude.com/docs/en/",
"start_urls": [
"https://code.claude.com/docs/en/overview",
"https://code.claude.com/docs/en/quickstart",
"https://code.claude.com/docs/en/common-workflows",
"https://code.claude.com/docs/en/claude-code-on-the-web",
"https://code.claude.com/docs/en/desktop",
"https://code.claude.com/docs/en/chrome",
"https://code.claude.com/docs/en/vs-code",
"https://code.claude.com/docs/en/jetbrains",
"https://code.claude.com/docs/en/github-actions",
"https://code.claude.com/docs/en/gitlab-ci-cd",
"https://code.claude.com/docs/en/slack",
"https://code.claude.com/docs/en/sub-agents",
"https://code.claude.com/docs/en/plugins",
"https://code.claude.com/docs/en/discover-plugins",
"https://code.claude.com/docs/en/skills",
"https://code.claude.com/docs/en/output-styles",
"https://code.claude.com/docs/en/hooks-guide",
"https://code.claude.com/docs/en/headless",
"https://code.claude.com/docs/en/mcp",
"https://code.claude.com/docs/en/third-party-integrations",
"https://code.claude.com/docs/en/amazon-bedrock",
"https://code.claude.com/docs/en/google-vertex-ai",
"https://code.claude.com/docs/en/microsoft-foundry",
"https://code.claude.com/docs/en/network-config",
"https://code.claude.com/docs/en/llm-gateway",
"https://code.claude.com/docs/en/devcontainer",
"https://code.claude.com/docs/en/sandboxing",
"https://code.claude.com/docs/en/setup",
"https://code.claude.com/docs/en/iam",
"https://code.claude.com/docs/en/security",
"https://code.claude.com/docs/en/data-usage",
"https://code.claude.com/docs/en/monitoring-usage",
"https://code.claude.com/docs/en/costs",
"https://code.claude.com/docs/en/analytics",
"https://code.claude.com/docs/en/plugin-marketplaces",
"https://code.claude.com/docs/en/settings",
"https://code.claude.com/docs/en/terminal-config",
"https://code.claude.com/docs/en/model-config",
"https://code.claude.com/docs/en/memory",
"https://code.claude.com/docs/en/statusline",
"https://code.claude.com/docs/en/cli-reference",
"https://code.claude.com/docs/en/interactive-mode",
"https://code.claude.com/docs/en/slash-commands",
"https://code.claude.com/docs/en/checkpointing",
"https://code.claude.com/docs/en/hooks",
"https://code.claude.com/docs/en/plugins-reference",
"https://code.claude.com/docs/en/troubleshooting",
"https://code.claude.com/docs/en/legal-and-compliance"
],
"selectors": {
"main_content": "#content-area, #content-container, article, main",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": [
"/docs/en/"
],
"exclude": [
"/docs/fr/",
"/docs/de/",
"/docs/it/",
"/docs/ja/",
"/docs/es/",
"/docs/ko/",
"/docs/zh-CN/",
"/docs/zh-TW/",
"/docs/ru/",
"/docs/id/",
"/docs/pt/",
"/changelog",
"github.com"
]
},
"categories": {
"getting_started": [
"overview",
"quickstart",
"common-workflows"
],
"ide_integrations": [
"vs-code",
"jetbrains",
"desktop",
"chrome",
"claude-code-on-the-web",
"slack"
],
"ci_cd": [
"github-actions",
"gitlab-ci-cd"
],
"building": [
"sub-agents",
"subagent",
"plugins",
"discover-plugins",
"skills",
"output-styles",
"hooks-guide",
"headless",
"programmatic"
],
"mcp": [
"mcp",
"model-context-protocol"
],
"deployment": [
"third-party-integrations",
"amazon-bedrock",
"google-vertex-ai",
"microsoft-foundry",
"network-config",
"llm-gateway",
"devcontainer",
"sandboxing"
],
"administration": [
"setup",
"iam",
"security",
"data-usage",
"monitoring-usage",
"costs",
"analytics",
"plugin-marketplaces"
],
"configuration": [
"settings",
"terminal-config",
"model-config",
"memory",
"statusline"
],
"reference": [
"cli-reference",
"interactive-mode",
"slash-commands",
"checkpointing",
"hooks",
"plugins-reference"
],
"troubleshooting": [
"troubleshooting"
],
"legal": [
"legal-and-compliance"
]
},
"rate_limit": 0.5,
"max_pages": 250
}
]
}
+49
View File
@@ -0,0 +1,49 @@
{
"name": "godot",
"description": "Complete Godot Engine knowledge base combining official documentation and source code analysis",
"merge_mode": "claude-enhanced",
"sources": [
{
"type": "documentation",
"base_url": "https://docs.godotengine.org/en/stable/",
"extract_api": true,
"selectors": {
"main_content": "div[role='main']",
"title": "title",
"code_blocks": "pre"
},
"url_patterns": {
"include": [],
"exclude": ["/search.html", "/_static/", "/_images/"]
},
"categories": {
"getting_started": ["introduction", "getting_started", "step_by_step"],
"scripting": ["scripting", "gdscript", "c_sharp"],
"2d": ["2d", "canvas", "sprite", "animation"],
"3d": ["3d", "spatial", "mesh", "shader"],
"physics": ["physics", "collision", "rigidbody"],
"api": ["api", "class", "reference", "method"]
},
"rate_limit": 0.5,
"max_pages": 500
},
{
"type": "github",
"repo": "godotengine/godot",
"enable_codebase_analysis": true,
"code_analysis_depth": "deep",
"fetch_issues": true,
"max_issues": 100,
"fetch_changelog": true,
"fetch_releases": true,
"file_patterns": [
"core/**/*.h",
"core/**/*.cpp",
"scene/**/*.h",
"scene/**/*.cpp",
"servers/**/*.h",
"servers/**/*.cpp"
]
}
]
}
+108
View File
@@ -0,0 +1,108 @@
{
"name": "godot",
"description": "Godot Engine 4.x - Complete open source game engine (documentation + source code + signal flow analysis)",
"output_dir": "output/godot-unified/",
"sources": [
{
"type": "local",
"path": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/godot-docs",
"name": "documentation",
"description": "Official Godot 4.x documentation (RST + Markdown)",
"weight": 0.4,
"enhance_level": 3,
"file_patterns": ["*.rst", "*.md"],
"skip_patterns": [
"build/",
"_build/",
".git/",
"node_modules/",
"__pycache__/"
],
"categories": {
"getting_started": ["getting_started", "introduction", "tutorial"],
"core_concepts": ["classes", "nodes", "scenes", "signals"],
"api": ["api", "reference", "class_reference"],
"tutorials": ["tutorials", "how_to", "examples"],
"advanced": ["advanced", "performance", "optimization"]
}
},
{
"type": "local",
"path": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/godot",
"name": "source_code",
"description": "Godot Engine C++ source code + GDScript core",
"weight": 0.6,
"enhance_level": 3,
"languages": ["C++", "GDScript", "Python", "GodotShader"],
"skip_patterns": [
".git/",
"thirdparty/",
"tests/",
"doc/",
"misc/",
"drivers/",
"platform/android/",
"platform/ios/",
"platform/web/",
"*.obj",
"*.o",
"*.a",
"*.so"
],
"focus_dirs": [
"core/",
"scene/",
"servers/",
"modules/gdscript/",
"editor/"
],
"analysis_depth": "full",
"extract_patterns": true,
"extract_tests": true,
"extract_signals": true,
"extract_config": true
}
],
"merge_strategy": "unified",
"conflict_resolution": "code_first",
"detect_conflicts": true,
"analysis_features": {
"pattern_detection": true,
"test_extraction": true,
"how_to_guides": true,
"config_extraction": true,
"architecture_overview": true,
"signal_flow_analysis": true,
"api_reference": true,
"dependency_graph": true
},
"enhancement": {
"enabled": true,
"level": 3,
"mode": "LOCAL"
},
"chunking": {
"enabled": true,
"chunk_size": 1000,
"chunk_overlap": 200
},
"output_formats": [
"claude",
"markdown"
],
"metadata": {
"version": "4.x",
"framework": "godot",
"language": "cpp+gdscript",
"tags": ["game-engine", "godot", "cpp", "gdscript", "signals", "nodes"],
"documentation_url": "https://docs.godotengine.org/",
"repository_url": "https://github.com/godotengine/godot"
}
}
+114
View File
@@ -0,0 +1,114 @@
{
"name": "httpx",
"description": "Use this skill when working with HTTPX, a fully featured HTTP client for Python 3 with sync and async APIs. HTTPX provides a familiar requests-like interface with support for HTTP/2, connection pooling, and comprehensive middleware capabilities.",
"version": "1.0.0",
"base_url": "https://www.python-httpx.org/",
"sources": [
{
"type": "documentation",
"base_url": "https://www.python-httpx.org/",
"selectors": {
"main_content": "article.md-content__inner",
"title": "h1",
"code_blocks": "pre code"
}
},
{
"type": "github",
"repo": "encode/httpx",
"code_analysis_depth": "deep",
"enable_codebase_analysis": true,
"fetch_issues": true,
"fetch_changelog": true,
"fetch_releases": true,
"max_issues": 50
}
],
"selectors": {
"main_content": "article.md-content__inner",
"title": "h1",
"code_blocks": "pre code",
"navigation": "nav.md-tabs",
"sidebar": "nav.md-nav--primary"
},
"url_patterns": {
"include": [
"/quickstart/",
"/advanced/",
"/api/",
"/async/",
"/http2/",
"/compatibility/"
],
"exclude": [
"/changelog/",
"/contributing/",
"/exceptions/"
]
},
"categories": {
"getting_started": [
"quickstart",
"install",
"introduction",
"overview"
],
"core_concepts": [
"client",
"request",
"response",
"timeout",
"pool"
],
"async": [
"async",
"asyncio",
"trio",
"concurrent"
],
"http2": [
"http2",
"http/2",
"multiplexing"
],
"advanced": [
"authentication",
"middleware",
"transport",
"proxy",
"ssl",
"streaming"
],
"api_reference": [
"api",
"reference",
"client",
"request",
"response"
],
"compatibility": [
"requests",
"migration",
"compatibility"
]
},
"rate_limit": 0.5,
"max_pages": 100,
"metadata": {
"author": "Encode",
"language": "Python",
"framework_type": "HTTP Client",
"use_cases": [
"Making HTTP requests",
"REST API clients",
"Async HTTP operations",
"HTTP/2 support",
"Connection pooling"
],
"related_skills": [
"requests",
"aiohttp",
"urllib3"
]
}
}
+71
View File
@@ -0,0 +1,71 @@
{
"name": "medusa-mercurjs",
"description": "Complete Medusa v2 + MercurJS multi-vendor e-commerce framework knowledge. Use when building headless commerce applications, implementing multi-vendor marketplaces, or understanding Medusa modules/workflows.",
"merge_mode": "rule-based",
"sources": [
{
"type": "documentation",
"base_url": "https://docs.medusajs.com",
"llms_txt_url": "https://docs.medusajs.com/llms-full.txt",
"extract_api": true,
"selectors": {
"main_content": "main, article, .content",
"title": "h1",
"code_blocks": "pre"
},
"url_patterns": {
"include": [
"/learn",
"/resources"
],
"exclude": []
},
"categories": {
"installation": ["installation", "install", "docker", "update"],
"fundamentals": ["fundamentals", "api-routes", "data-models", "modules", "module-links", "workflows", "events-and-subscribers", "scheduled-jobs", "custom-cli-scripts", "admin", "environment-variables"],
"customization": ["customization", "custom-features", "extend-features", "integrate-systems", "customize-admin"],
"debugging_testing": ["debugging-and-testing", "logging", "testing", "test-tools", "instrumentation", "feature-flags", "debug-workflows"],
"deployment": ["deployment", "production", "deploy", "general"],
"commerce_modules": ["commerce-modules", "product", "cart", "order", "payment", "pricing", "tax", "inventory", "fulfillment", "customer", "promotion", "auth", "region", "currency", "sales-channel", "stock-location", "api-key", "user"],
"infrastructure_modules": ["infrastructure-modules", "caching", "event", "file", "locking", "notification", "workflow-engine", "analytics"],
"storefront": ["storefront-development", "publishable-api-keys", "checkout", "products", "customers", "regions"],
"integrations": ["integrations", "sanity", "contentful", "stripe", "paypal", "shipstation", "sentry"],
"cli_tools": ["medusa-cli", "commands", "build", "develop", "plugin", "db"],
"references": ["references", "medusa-workflows", "helper-steps", "service-factory-reference", "data-model-repository-reference", "test-tools-reference", "fulfillment", "auth", "notification-provider", "file-provider", "locking-service", "caching-service"],
"recipes": ["recipes", "erp", "marketplace", "b2b", "subscriptions", "digital-products", "bundled-products"],
"admin_components": ["admin-components", "widgets", "ui-routes"],
"examples": ["examples", "guides", "how-to-tutorials", "tutorials"]
},
"rate_limit": 0.3,
"max_pages": 500
},
{
"type": "documentation",
"base_url": "https://docs.mercurjs.com/",
"llms_txt_url": "https://docs.mercurjs.com/llms-full.txt",
"extract_api": true,
"selectors": {
"main_content": "main, article",
"title": "h1",
"code_blocks": "pre"
},
"url_patterns": {
"include": ["/"],
"exclude": []
},
"categories": {
"quick_start": ["introduction", "get-started"],
"components": ["components", "backend", "admin-panel", "vendor-panel", "storefront"],
"core_concepts": ["core-concepts", "seller", "commission", "payouts", "order-splitting", "reviews", "requests", "notifications", "marketplace-settings"],
"product": ["product", "core-commerce-modules", "core-infrastructure-modules", "framework"],
"integrations": ["integrations", "algolia", "resend", "stripe"],
"api_admin": ["api-reference/admin", "admin-algolia", "admin-api-keys", "admin-attributes", "admin-auth", "admin-campaigns", "admin-claims", "admin-collections", "admin-commission", "admin-currencies", "admin-customers", "admin-draft-orders", "admin-exchanges", "admin-fulfillment", "admin-inventory", "admin-invites", "admin-notifications", "admin-orders", "admin-payments", "admin-price-lists", "admin-products", "admin-promotions", "admin-regions", "admin-reservations", "admin-returns", "admin-sales-channels", "admin-sellers", "admin-shipping", "admin-stock-locations", "admin-stores", "admin-tax", "admin-uploads", "admin-users"],
"api_store": ["api-reference/store", "store-auth", "store-carts", "store-collections", "store-currencies", "store-customers", "store-fulfillment", "store-orders", "store-payment", "store-products", "store-regions", "store-returns"],
"api_vendor": ["api-reference/vendor", "vendor-auth", "vendor-fulfillment", "vendor-inventory", "vendor-orders", "vendor-payouts", "vendor-products", "vendor-returns", "vendor-sellers", "vendor-shipping", "vendor-stock-locations", "vendor-uploads"],
"help": ["help", "llm", "mcp", "support"]
},
"rate_limit": 0.3,
"max_pages": 300
}
]
}
+69
View File
@@ -0,0 +1,69 @@
{
"name": "react",
"description": "Complete React knowledge base combining official documentation and React codebase insights. Use when working with React, understanding API changes, or debugging React internals.",
"version": "1.1.0",
"merge_mode": "rule-based",
"sources": [
{
"type": "documentation",
"base_url": "https://react.dev/",
"extract_api": true,
"selectors": {
"main_content": "article",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": [],
"exclude": [
"/blog/",
"/community/"
]
},
"categories": {
"getting_started": [
"learn",
"installation",
"quick-start"
],
"components": [
"components",
"props",
"state"
],
"hooks": [
"hooks",
"usestate",
"useeffect",
"usecontext"
],
"api": [
"api",
"reference"
],
"advanced": [
"context",
"refs",
"portals",
"suspense"
]
},
"rate_limit": 0.5
},
{
"type": "github",
"repo": "facebook/react",
"enable_codebase_analysis": true,
"code_analysis_depth": "deep",
"fetch_issues": true,
"max_issues": 100,
"fetch_changelog": true,
"fetch_releases": true,
"file_patterns": [
"packages/react/src/**/*.js",
"packages/react-dom/src/**/*.js"
]
}
],
"base_url": "https://react.dev/"
}
+125
View File
@@ -0,0 +1,125 @@
{
"name": "unity-addressables",
"description": "Unity Addressables asset management system - Use when implementing asset loading, remote content delivery, asset bundles, memory management, or dynamic asset loading in Unity projects.",
"version": "1.0.0",
"merge_mode": "claude-enhanced",
"base_url": "https://docs.unity3d.com/Packages/com.unity.addressables@2.3/manual/index.html",
"sources": [
{
"type": "documentation",
"base_url": "https://docs.unity3d.com/Packages/com.unity.addressables@2.3/manual/index.html",
"browser": true,
"extract_api": true,
"selectors": {
"main_content": ".content-wrap, .section, article, main",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": [
"com.unity.addressables"
],
"exclude": [
"/changelog/",
"/license/"
]
},
"categories": {
"getting_started": [
"index",
"getting-started",
"installation",
"quickstart",
"AddressableAssetsGettingStarted"
],
"concepts": [
"AddressableAssetsDevelopmentCycle",
"AddressableAssetsOverview",
"asset-references",
"labels",
"groups",
"profiles"
],
"loading": [
"load-assets",
"LoadingAddressableAssets",
"MemoryManagement",
"AsyncOperationHandle",
"UnloadingAddressableAssets",
"synchronous-addressables"
],
"api": [
"api",
"reference",
"Addressables",
"AddressablesAPI"
],
"building": [
"Builds",
"BuildLayoutReport",
"ContentUpdateWorkflow",
"AddressableAssetSettings",
"build-scripting"
],
"advanced": [
"remote-content-distribution",
"content-catalogs",
"diagnostic-tools",
"CustomOperations",
"TransformInternalId",
"ccd"
]
},
"rate_limit": 0.5
},
{
"type": "github",
"repo": "Unity-Technologies/Addressables-Sample",
"include_code": true,
"language": "C#",
"enable_codebase_analysis": true,
"code_analysis_depth": "deep",
"fetch_issues": false,
"fetch_changelog": false,
"fetch_releases": false,
"file_patterns": [
"**/*.cs"
]
}
],
"analysis_features": {
"pattern_detection": true,
"test_extraction": true,
"how_to_guides": true,
"config_extraction": true,
"architecture_overview": true,
"api_reference": true,
"dependency_graph": true
},
"enhancement": {
"enabled": true,
"level": 3,
"mode": "LOCAL"
},
"chunking": {
"enabled": true,
"chunk_size": 1000,
"chunk_overlap": 200
},
"output_formats": [
"claude",
"markdown"
],
"metadata": {
"version": "2.3",
"framework": "unity",
"language": "csharp",
"tags": ["unity", "addressables", "asset-management", "asset-bundles", "content-delivery", "csharp"],
"documentation_url": "https://docs.unity3d.com/Packages/com.unity.addressables@latest/",
"repository_url": "https://github.com/Unity-Technologies/Addressables-Sample"
}
}
+128
View File
@@ -0,0 +1,128 @@
{
"name": "dotween",
"description": "DOTween (HOTween v2) animation engine for Unity - Use when implementing tweening animations, sequences, easing, UI animations, or any programmatic animation in Unity C# projects.",
"version": "1.0.0",
"merge_mode": "claude-enhanced",
"base_url": "https://dotween.demigiant.com/documentation.php",
"sources": [
{
"type": "documentation",
"base_url": "https://dotween.demigiant.com/documentation.php",
"extract_api": true,
"selectors": {
"main_content": "#content, .documentation, article, main",
"title": "h1, h2",
"code_blocks": "pre code, .code"
},
"url_patterns": {
"include": [
"/documentation",
"/getstarted",
"/pro",
"/support"
],
"exclude": [
"/download",
"/credits"
]
},
"categories": {
"getting_started": [
"getstarted",
"setup",
"installation",
"initialization"
],
"core_api": [
"DOTween",
"Tweener",
"Sequence",
"Tween",
"TweenParams",
"DOVirtual"
],
"shortcuts": [
"shortcuts",
"transform",
"material",
"rigidbody",
"camera",
"audio",
"light",
"spriterenderer"
],
"easing": [
"ease",
"easing",
"animationcurve",
"custom-ease"
],
"sequences": [
"sequence",
"append",
"insert",
"join",
"prepend",
"callbacks"
],
"advanced": [
"paths",
"blendable",
"DOTweenAnimation",
"DOTweenVisualManager",
"pro"
]
},
"rate_limit": 1.0
},
{
"type": "github",
"repo": "Demigiant/dotween",
"include_code": true,
"language": "C#",
"enable_codebase_analysis": true,
"code_analysis_depth": "deep",
"fetch_issues": true,
"max_issues": 50,
"fetch_changelog": false,
"fetch_releases": true
}
],
"analysis_features": {
"pattern_detection": true,
"test_extraction": true,
"how_to_guides": true,
"config_extraction": true,
"architecture_overview": true,
"api_reference": true,
"dependency_graph": true
},
"enhancement": {
"enabled": true,
"level": 3,
"mode": "LOCAL",
"agent": "kimi",
"timeout": "unlimited"
},
"chunking": {
"enabled": true,
"chunk_size": 1000,
"chunk_overlap": 200
},
"output_formats": [
"claude",
"markdown"
],
"metadata": {
"version": "1.2",
"framework": "unity",
"language": "csharp",
"tags": ["unity", "dotween", "tweening", "animation", "easing", "csharp"],
"documentation_url": "https://dotween.demigiant.com/documentation.php",
"repository_url": "https://github.com/Demigiant/dotween"
}
}
+136
View File
@@ -0,0 +1,136 @@
{
"name": "spine-unity",
"description": "Spine 2D skeletal animation runtime for Unity - Use when implementing Spine animations, SkeletonAnimation components, skin management, or spine-unity integration in Unity projects.",
"version": "1.0.0",
"merge_mode": "claude-enhanced",
"base_url": "http://en.esotericsoftware.com/spine-unity",
"sources": [
{
"type": "documentation",
"base_url": "http://en.esotericsoftware.com/spine-unity",
"extract_api": true,
"selectors": {
"main_content": "#wiki_page_content, .wiki-content, article, main",
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": [
"/spine-unity",
"/spine-runtime",
"/spine-api",
"/spine-applying-animations",
"/spine-attachments",
"/spine-events",
"/spine-skeleton",
"/spine-skins",
"/spine-slots"
],
"exclude": [
"/spine-corona",
"/spine-cocos2d",
"/spine-godot",
"/spine-unreal",
"/spine-flutter",
"/spine-sfml",
"/spine-monogame",
"/spine-love",
"/spine-haxe",
"/spine-ts",
"/spine-phaser",
"/spine-pixi",
"/spine-lwjgl",
"/spine-libgdx",
"/spine-changelog"
]
},
"categories": {
"getting_started": [
"Installation",
"Download",
"Assets",
"Runtime Documentation",
"FAQ"
],
"components": [
"Main Components",
"Utility Components",
"SkeletonAnimation",
"SkeletonGraphic",
"SkeletonMecanim",
"BoneFollower"
],
"animation": [
"Events",
"AnimationState",
"callback",
"Timeline",
"Examples"
],
"rendering": [
"Rendering",
"shader",
"material"
],
"advanced": [
"On-Demand Loading",
"UPM",
"Packages"
]
},
"rate_limit": 1.0
},
{
"type": "github",
"repo": "EsotericSoftware/spine-runtimes",
"include_code": true,
"language": "C#",
"enable_codebase_analysis": true,
"code_analysis_depth": "deep",
"fetch_issues": true,
"max_issues": 50,
"fetch_changelog": true,
"fetch_releases": true,
"file_patterns": [
"spine-unity/Assets/Spine/**/*.cs",
"spine-unity/Assets/Spine Examples/**/*.cs",
"spine-csharp/src/**/*.cs"
]
}
],
"analysis_features": {
"pattern_detection": true,
"test_extraction": true,
"how_to_guides": true,
"config_extraction": true,
"architecture_overview": true,
"api_reference": true,
"dependency_graph": true
},
"enhancement": {
"enabled": true,
"level": 3,
"mode": "LOCAL"
},
"chunking": {
"enabled": true,
"chunk_size": 1000,
"chunk_overlap": 200
},
"output_formats": [
"claude",
"markdown"
],
"metadata": {
"version": "4.2",
"framework": "unity",
"language": "csharp",
"tags": ["unity", "spine", "2d-animation", "skeletal-animation", "csharp"],
"documentation_url": "http://en.esotericsoftware.com/spine-unity",
"repository_url": "https://github.com/EsotericSoftware/spine-runtimes"
}
}
@@ -0,0 +1,11 @@
{
"name": "skill-seekers",
"description": "Transform 17 source types (docs, GitHub, PDFs, videos, Jupyter, Confluence, Notion, Slack, and more) into AI-ready skills and RAG knowledge for 16+ LLM platforms.",
"version": "3.3.0",
"author": {
"name": "Yusuf Karaaslan"
},
"homepage": "https://github.com/yusufkaraaslan/Skill_Seekers",
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
"license": "MIT"
}
+6
View File
@@ -0,0 +1,6 @@
{
"skill-seekers": {
"command": "python",
"args": ["-m", "skill_seekers.mcp.server_fastmcp"]
}
}
+93
View File
@@ -0,0 +1,93 @@
# Skill Seekers — Claude Code Plugin
Transform 18 source types into AI-ready skills and RAG knowledge, directly from Claude Code.
## Installation
### From the Official Plugin Directory
```
/plugin install skill-seekers@claude-plugin-directory
```
Or browse for it in `/plugin > Discover`.
### Local Installation (for development)
```bash
claude --plugin-dir ./path/to/skill-seekers-plugin
```
### Prerequisites
The plugin requires `skill-seekers` to be installed:
```bash
pip install skill-seekers[mcp]
```
## What's Included
### MCP Server (40 tools)
The plugin bundles the Skill Seekers MCP server providing tools for:
- Scraping documentation, GitHub repos, PDFs, videos, and 13 other source types
- Packaging skills for 21+ LLM platforms
- Exporting to vector databases (Weaviate, Chroma, FAISS, Qdrant)
- Managing configs, workflows, and sources
### Slash Commands
| Command | Description |
|---------|-------------|
| `/skill-seekers:create-skill <source>` | Create a skill from any source (auto-detects type) |
| `/skill-seekers:sync-config <config>` | Sync config URLs against live docs |
| `/skill-seekers:install-skill <source>` | End-to-end: fetch, scrape, enhance, package, install |
### Agent Skill
The **skill-builder** skill is automatically available to Claude. It detects source types and uses the appropriate MCP tools to build skills autonomously.
## Usage Examples
```
# Create a skill from a documentation site
/skill-seekers:create-skill https://react.dev
# Create from a GitHub repo, targeting LangChain
/skill-seekers:create-skill pallets/flask --target langchain
# Full install workflow with AI enhancement
/skill-seekers:install-skill https://fastapi.tiangolo.com --enhance
# Sync an existing config
/skill-seekers:sync-config react
```
Or just ask Claude naturally:
> "Create an AI skill from the React documentation"
> "Scrape the Flask GitHub repo and package it for OpenAI"
> "Export my skill to a Chroma vector database"
The skill-builder agent skill will automatically detect the intent and use the right tools.
## Remote MCP Alternative
By default, the plugin runs the MCP server locally via `python -m skill_seekers.mcp.server_fastmcp`. To use a remote server instead, edit `.mcp.json`:
```json
{
"skill-seekers": {
"type": "http",
"url": "https://your-hosted-server.com/mcp"
}
}
```
## Supported Source Types
Documentation (web), GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML files, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence, Notion, Slack/Discord exports.
## License
MIT — https://github.com/yusufkaraaslan/Skill_Seekers
@@ -0,0 +1,62 @@
---
description: Create an AI skill from any source (URL, repo, PDF, video, notebook, etc.)
---
# Create Skill
Create an AI-ready skill from a source. The source type is auto-detected.
## Usage
```
/skill-seekers:create-skill <source> [--preset <level>] [--output <dir>]
```
## Instructions
When the user provides a source via `$ARGUMENTS`, run the `skill-seekers create` command to generate a skill.
1. Parse the arguments: extract the source (first argument) and any flags.
2. If no `--preset` is specified, default to `quick` for fast results.
3. If no `--output` is specified, default to `./output`.
4. Run the create command:
```bash
skill-seekers create "$SOURCE" --preset quick --output "$OUTPUT"
```
5. After completion, read the generated `SKILL.md` and summarize what was created.
6. If the user wants to target a specific platform (e.g., Claude, OpenAI, LangChain), run the package command after:
```bash
skill-seekers package "$SKILL_DIR" --target "$PLATFORM"
```
## Presets
- `-p quick` — 1-2 minutes, basic skill
- `-p standard` — 5-10 minutes, good coverage
- `-p comprehensive` — 20-60 minutes, full analysis
## Source Types (auto-detected)
- **URL** (https://...) — Documentation scraping
- **owner/repo** or github.com URL — GitHub repo analysis
- **file.pdf** — PDF extraction
- **file.ipynb** — Jupyter notebook
- **file.docx** — Word document
- **file.epub** — EPUB book
- **YouTube/Vimeo URL** — Video transcript
- **./directory** — Local codebase analysis
- **file.yaml** with OpenAPI — API spec
- **file.pptx** — PowerPoint
- **file.adoc** — AsciiDoc
- **file.html** — HTML page
- **file.rss** — RSS/Atom feed
- **cmd.1** — Man page
## Examples
```
/skill-seekers:create-skill https://react.dev
/skill-seekers:create-skill pallets/flask -p standard
/skill-seekers:create-skill ./docs/api.pdf
/skill-seekers:create-skill https://youtube.com/watch?v=abc123
```
@@ -0,0 +1,43 @@
---
description: One-command skill creation and packaging for a target platform
---
# Install Skill
End-to-end workflow: create a skill from any source, then package it for a target LLM platform.
## Usage
```
/skill-seekers:install-skill <source> [--target <platform>] [--preset <level>]
```
## Instructions
When the user provides a source via `$ARGUMENTS`:
1. Parse the arguments: extract source, `--target` (default: claude), `--preset` (default: quick).
2. Run the create command:
```bash
skill-seekers create "$SOURCE" --preset "$PRESET" --output ./output
```
3. Find the generated skill directory (look for the directory containing SKILL.md in ./output/).
4. Run the package command for the target platform:
```bash
skill-seekers package "$SKILL_DIR" --target "$TARGET"
```
5. Report what was created and where to find the packaged output.
## Target Platforms
`claude` (default), `openai`, `gemini`, `langchain`, `llama-index`, `haystack`, `markdown`, `chroma`, `weaviate`, `faiss`, `qdrant`, `pinecone`, `deepseek`, `kimi`, `qwen`, `fireworks`, `openrouter`, `together`, `minimax`, `opencode`, `ibm-bob`
> **IDE-specific installs** (`cursor`, `windsurf`, `continue`, `cline`) are not standalone `--target` values. Use `skill-seekers install-agent` or manually copy the output from `--target markdown` or `--target claude`.
## Examples
```
/skill-seekers:install-skill https://react.dev --target claude
/skill-seekers:install-skill pallets/flask --target langchain -p standard
/skill-seekers:install-skill ./docs/api.pdf --target openai
```
@@ -0,0 +1,32 @@
---
description: Sync a scraping config's URLs against the live documentation site
---
# Sync Config
Synchronize a Skill Seekers config file with the current state of a documentation site. Detects new pages, removed pages, and URL changes.
## Usage
```
/skill-seekers:sync-config <config-path-or-name>
```
## Instructions
When the user provides a config path or preset name via `$ARGUMENTS`:
1. If it's a preset name (e.g., `react`, `godot`), look for it in the `configs/` directory or fetch from the API.
2. Run the sync command:
```bash
skill-seekers sync-config "$CONFIG"
```
3. Report what changed: new URLs found, removed URLs, and any conflicts.
4. Ask the user if they want to update the config and re-scrape.
## Examples
```
/skill-seekers:sync-config configs/react.json
/skill-seekers:sync-config react
```
@@ -0,0 +1,69 @@
---
name: skill-builder
description: Automatically detect source types and build AI skills using Skill Seekers. Use when the user wants to create skills from documentation, repos, PDFs, videos, or other knowledge sources.
---
# Skill Builder
You have access to the Skill Seekers MCP server which provides 40 tools for converting knowledge sources into AI-ready skills.
## When to Use This Skill
Use this skill when the user:
- Wants to create an AI skill from a documentation site, GitHub repo, PDF, video, or other source
- Needs to convert documentation into a format suitable for LLM consumption
- Wants to update or sync existing skills with their source documentation
- Needs to export skills to vector databases (Weaviate, Chroma, FAISS, Qdrant)
- Asks about scraping, converting, or packaging documentation for AI
## Source Type Detection
Automatically detect the source type from user input:
| Input Pattern | Source Type | Tool to Use |
|---------------|-------------|-------------|
| `https://...` (not GitHub/YouTube) | Documentation | `scrape_docs` |
| `owner/repo` or `github.com/...` | GitHub | `scrape_github` |
| `*.pdf` | PDF | `scrape_pdf` |
| YouTube/Vimeo URL or video file | Video | `scrape_video` |
| Local directory path | Codebase | `scrape_codebase` |
| `*.ipynb`, `*.html`, `*.yaml` (OpenAPI), `*.adoc`, `*.pptx`, `*.rss`, `*.1`-`.8` | Various | `scrape_generic` |
| JSON config file | Unified | Use config with `scrape_docs` |
## Recommended Workflow
1. **Detect source type** from the user's input
2. **Generate or fetch config** using `generate_config` or `fetch_config` if needed
3. **Estimate scope** with `estimate_pages` for documentation sites
4. **Scrape the source** using the appropriate scraping tool
5. **Enhance** with `enhance_skill` if the user wants AI-powered improvements
6. **Package** with `package_skill` for the target platform
7. **Export to vector DB** if requested using `export_to_*` tools
## Available MCP Tools
### Config Management
- `generate_config` — Generate a scraping config from a URL
- `list_configs` — List available preset configs
- `validate_config` — Validate a config file
### Scraping (use based on source type)
- `scrape_docs` — Documentation sites
- `scrape_github` — GitHub repositories
- `scrape_pdf` — PDF files
- `scrape_video` — Video transcripts
- `scrape_codebase` — Local code analysis
- `scrape_generic` — Jupyter, HTML, OpenAPI, AsciiDoc, PPTX, RSS, manpage, Confluence, Notion, chat
### Post-processing
- `enhance_skill` — AI-powered skill enhancement
- `package_skill` — Package for target platform
- `upload_skill` — Upload to platform API
- `install_skill` — End-to-end install workflow
### Advanced
- `detect_patterns` — Design pattern detection in code
- `extract_test_examples` — Extract usage examples from tests
- `build_how_to_guides` — Generate how-to guides from tests
- `split_config` — Split large configs into focused skills
- `export_to_weaviate`, `export_to_chroma`, `export_to_faiss`, `export_to_qdrant` — Vector DB export
+147
View File
@@ -0,0 +1,147 @@
# Skill Seekers GitHub Action
Transform documentation, GitHub repos, PDFs, videos, and 13 other source types into AI-ready skills and RAG knowledge — directly in your CI/CD pipeline.
## Quick Start
```yaml
- uses: yusufkaraaslan/skill-seekers-action@v3
with:
source: 'https://react.dev'
```
## Inputs
| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| `source` | Yes | — | Source URL, file path, or `owner/repo` |
| `command` | No | `create` | Command: `create`, `scan`, `doctor`, `enhance`, `enhance-status`, `package`, `upload`, `install`, `install-agent`, `estimate`, `extract-test-examples`, `resume`, `quality`, `config`, `workflows`, `sync-config`, `stream`, `update`, `multilang` |
| `target` | No | `claude` | Target platform: `claude`, `openai`, `gemini`, `langchain`, `llama-index`, `markdown` |
| `config` | No | — | Path to JSON config file |
| `output-dir` | No | `output` | Output directory |
| `extra-args` | No | — | Additional CLI arguments |
## Outputs
| Output | Description |
|--------|-------------|
| `skill-dir` | Path to the generated skill directory |
| `skill-name` | Name of the generated skill |
## Examples
### Auto-update documentation skill weekly
```yaml
name: Update AI Skills
on:
schedule:
- cron: '0 6 * * 1' # Every Monday 6am UTC
workflow_dispatch:
jobs:
update-skills:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: yusufkaraaslan/skill-seekers-action@v3
with:
source: 'https://react.dev'
target: 'langchain'
- uses: actions/upload-artifact@v4
with:
name: react-skill
path: output/
```
### Generate skill from GitHub repo
```yaml
- uses: yusufkaraaslan/skill-seekers-action@v3
with:
source: 'pallets/flask'
command: 'create'
target: 'claude'
```
### Process PDF documentation
```yaml
- uses: actions/checkout@v4
- uses: yusufkaraaslan/skill-seekers-action@v3
with:
source: 'docs/api-reference.pdf'
command: 'create'
```
### Unified multi-source build with config
```yaml
- uses: actions/checkout@v4
- uses: yusufkaraaslan/skill-seekers-action@v3
with:
config: 'configs/my-project.json'
command: 'create'
target: 'openai'
```
### Commit generated skill back to repo
```yaml
- uses: actions/checkout@v4
- uses: yusufkaraaslan/skill-seekers-action@v3
id: generate
with:
source: 'https://fastapi.tiangolo.com'
- name: Commit skill
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add output/
git diff --staged --quiet || git commit -m "Update AI skill: ${{ steps.generate.outputs.skill-name }}"
git push
```
## Environment Variables
Pass API keys as environment variables for AI-enhanced skills:
```yaml
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
## Supported Source Types
| Type | Example Source |
|------|---------------|
| Documentation (web) | `https://react.dev` |
| GitHub repo | `pallets/flask` or `https://github.com/pallets/flask` |
| PDF | `docs/manual.pdf` |
| Video | `https://youtube.com/watch?v=...` |
| Local codebase | `./src` |
| Jupyter Notebook | `analysis.ipynb` |
| OpenAPI/Swagger | `openapi.yaml` |
| Word (.docx) | `docs/guide.docx` |
| EPUB | `book.epub` |
| PowerPoint | `slides.pptx` |
| AsciiDoc | `docs/guide.adoc` |
| HTML | `page.html` |
| RSS/Atom | `feed.rss` |
| Man pages | `tool.1` |
| Confluence | Via config file |
| Notion | Via config file |
| Chat (Slack/Discord) | Via config file |
## License
MIT
+92
View File
@@ -0,0 +1,92 @@
name: 'Skill Seekers - AI Knowledge Builder'
description: 'Transform documentation, repos, PDFs, videos, and 13 other source types into AI skills and RAG knowledge'
author: 'Yusuf Karaaslan'
branding:
icon: 'book-open'
color: 'blue'
inputs:
source:
description: 'Source URL, file path, or owner/repo for GitHub repos'
required: true
command:
description: 'Command to run: create (auto-detect), scrape, github, pdf, video, analyze, unified'
required: false
default: 'create'
target:
description: 'Output target platform: claude, openai, gemini, langchain, llama-index, markdown, cursor, windsurf'
required: false
default: 'claude'
config:
description: 'Path to JSON config file (for unified/advanced scraping)'
required: false
output-dir:
description: 'Output directory for generated skills'
required: false
default: 'output'
extra-args:
description: 'Additional CLI arguments to pass to skill-seekers'
required: false
default: ''
outputs:
skill-dir:
description: 'Path to the generated skill directory'
value: ${{ steps.run.outputs.skill-dir }}
skill-name:
description: 'Name of the generated skill'
value: ${{ steps.run.outputs.skill-name }}
runs:
using: 'composite'
steps:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install Skill Seekers
shell: bash
run: pip install skill-seekers
- name: Run Skill Seekers
id: run
shell: bash
env:
ANTHROPIC_API_KEY: ${{ env.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ env.OPENAI_API_KEY }}
GOOGLE_API_KEY: ${{ env.GOOGLE_API_KEY }}
GITHUB_TOKEN: ${{ env.GITHUB_TOKEN }}
run: |
set -euo pipefail
OUTPUT_DIR="${{ inputs.output-dir }}"
mkdir -p "$OUTPUT_DIR"
CMD="${{ inputs.command }}"
SOURCE="${{ inputs.source }}"
TARGET="${{ inputs.target }}"
CONFIG="${{ inputs.config }}"
EXTRA="${{ inputs.extra-args }}"
# Build the command
if [ "$CMD" = "create" ]; then
skill-seekers create "$SOURCE" --target "$TARGET" --output "$OUTPUT_DIR" $EXTRA
elif [ -n "$CONFIG" ]; then
skill-seekers "$CMD" --config "$CONFIG" --target "$TARGET" --output "$OUTPUT_DIR" $EXTRA
else
skill-seekers "$CMD" "$SOURCE" --target "$TARGET" --output "$OUTPUT_DIR" $EXTRA
fi
# Find the generated skill directory
SKILL_DIR=$(find "$OUTPUT_DIR" -name "SKILL.md" -exec dirname {} \; | head -1)
SKILL_NAME=$(basename "$SKILL_DIR" 2>/dev/null || echo "unknown")
echo "skill-dir=$SKILL_DIR" >> "$GITHUB_OUTPUT"
echo "skill-name=$SKILL_NAME" >> "$GITHUB_OUTPUT"
echo "### Skill Generated" >> "$GITHUB_STEP_SUMMARY"
echo "- **Name:** $SKILL_NAME" >> "$GITHUB_STEP_SUMMARY"
echo "- **Directory:** $SKILL_DIR" >> "$GITHUB_STEP_SUMMARY"
echo "- **Target:** $TARGET" >> "$GITHUB_STEP_SUMMARY"
+109
View File
@@ -0,0 +1,109 @@
# Skill Seekers — Smithery MCP Registry
Publishing guide for the Skill Seekers MCP server on [Smithery](https://smithery.ai).
## Status
- **Namespace created:** `yusufkaraaslan`
- **Server created:** `yusufkaraaslan/skill-seekers`
- **Server page:** https://smithery.ai/servers/yusufkaraaslan/skill-seekers
- **Release status:** Needs re-publish (initial release failed — Smithery couldn't scan GitHub URL as MCP endpoint)
## Publishing
Smithery requires a live, scannable MCP HTTP endpoint for URL-based publishing. Two options:
### Option A: Publish via Web UI (Recommended)
1. Go to https://smithery.ai/servers/yusufkaraaslan/skill-seekers/releases
2. The server already exists — create a new release
3. For the "Local" tab: follow the prompts to publish as a stdio server
4. For the "URL" tab: provide a hosted HTTP endpoint URL
### Option B: Deploy HTTP endpoint first, then publish via CLI
1. Deploy the MCP server on Render/Railway/Fly.io:
```bash
# Using existing Dockerfile.mcp
docker build -f Dockerfile.mcp -t skill-seekers-mcp .
# Deploy to your hosting provider
```
2. Publish the live URL:
```bash
npx @smithery/cli@latest auth login
npx @smithery/cli@latest mcp publish "https://your-deployed-url/mcp" \
-n yusufkaraaslan/skill-seekers
```
### CLI Authentication (already done)
```bash
# Install via npx (no global install needed)
npx @smithery/cli@latest auth login
npx @smithery/cli@latest namespace show # Should show: yusufkaraaslan
```
### After Publishing
Update the server page with metadata:
**Display name:** Skill Seekers — AI Skill & RAG Toolkit
**Description:**
> Transform 18 source types into AI-ready skills and RAG knowledge. Ingest documentation sites, GitHub repos, PDFs, Jupyter notebooks, videos, Confluence, Notion, Slack/Discord exports, and more. Package for 21+ LLM platforms including Claude, GPT, Gemini, LangChain, LlamaIndex, and vector databases.
**Tags:** `ai`, `rag`, `documentation`, `skills`, `preprocessing`, `mcp`, `knowledge-base`, `vector-database`
## User Installation
Once published, users can add the server to their MCP client:
```bash
# Via Smithery CLI (adds to Claude Desktop, Cursor, etc.)
smithery mcp add yusufkaraaslan/skill-seekers --client claude
# Or configure manually — users need skill-seekers installed:
pip install skill-seekers[mcp]
```
### Manual MCP Configuration
For clients that use JSON config (Claude Desktop, Claude Code, Cursor):
```json
{
"mcpServers": {
"skill-seekers": {
"command": "python",
"args": ["-m", "skill_seekers.mcp.server_fastmcp"]
}
}
}
```
## Available Tools (40)
| Category | Tools | Description |
|----------|-------|-------------|
| Config | 3 | Generate, list, validate scraping configs |
| Sync | 1 | Sync config URLs against live docs |
| Scraping | 11 | Scrape docs, GitHub, PDF, video, codebase, generic (10 types) |
| Packaging | 4 | Package, upload, enhance, install skills |
| Splitting | 2 | Split large configs, generate routers |
| Sources | 5 | Fetch, submit, manage config sources |
| Vector DB | 4 | Export to Weaviate, Chroma, FAISS, Qdrant |
| Workflows | 5 | List, get, create, update, delete workflows |
| Quality | 3 | Quality checks, estimate scope, resume operations |
| Agent Install | 2 | Install agent, extract test examples |
## Maintenance
- Update description/tags on major releases
- No code changes needed — users always get the latest via `pip install`
## Notes
- Smithery CLI v4.7.0 removed the `--transport stdio` flag from the docs
- The CLI `publish` command only supports URL-based (external) publishing
- For local/stdio servers, use the web UI at smithery.ai/servers/new
- The namespace and server entity are already created; only the release needs to succeed
+111
View File
@@ -0,0 +1,111 @@
# Skill Seekers Docker Compose
# Complete deployment with MCP server and vector databases
version: '3.8'
services:
# Main Skill Seekers CLI application
skill-seekers:
build:
context: .
dockerfile: Dockerfile
image: skill-seekers:latest
container_name: skill-seekers
environment:
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- GOOGLE_API_KEY=${GOOGLE_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
- GITHUB_TOKEN=${GITHUB_TOKEN}
volumes:
- ./data:/data
- ./configs:/configs:ro
- ./output:/output
networks:
- skill-seekers-net
command: ["skill-seekers", "--help"]
# MCP Server (HTTP mode)
mcp-server:
build:
context: .
dockerfile: Dockerfile.mcp
image: skill-seekers-mcp:latest
container_name: skill-seekers-mcp
ports:
- "8765:8765"
environment:
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- GOOGLE_API_KEY=${GOOGLE_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
- GITHUB_TOKEN=${GITHUB_TOKEN}
- MCP_TRANSPORT=http
- MCP_PORT=8765
volumes:
- ./data:/data
- ./configs:/configs:ro
- ./output:/output
networks:
- skill-seekers-net
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 10s
# Weaviate Vector Database
weaviate:
image: semitechnologies/weaviate:latest
container_name: weaviate
ports:
- "8080:8080"
environment:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'none'
ENABLE_MODULES: ''
CLUSTER_HOSTNAME: 'node1'
volumes:
- weaviate-data:/var/lib/weaviate
networks:
- skill-seekers-net
restart: unless-stopped
# Qdrant Vector Database
qdrant:
image: qdrant/qdrant:latest
container_name: qdrant
ports:
- "6333:6333"
- "6334:6334"
volumes:
- qdrant-data:/qdrant/storage
networks:
- skill-seekers-net
restart: unless-stopped
# Chroma Vector Database
chroma:
image: ghcr.io/chroma-core/chroma:latest
container_name: chroma
ports:
- "8000:8000"
environment:
IS_PERSISTENT: 'TRUE'
PERSIST_DIRECTORY: '/chroma/data'
volumes:
- chroma-data:/chroma/data
networks:
- skill-seekers-net
restart: unless-stopped
networks:
skill-seekers-net:
driver: bridge
volumes:
weaviate-data:
qdrant-data:
chroma-data:
+263
View File
@@ -0,0 +1,263 @@
# Documentation Architecture
> **How Skill Seekers documentation is organized (v3.6.0 - 18 source types)**
---
## Philosophy
Our documentation follows these principles:
1. **Progressive Disclosure** - Start simple, add complexity as needed
2. **Task-Oriented** - Organized by what users want to do
3. **Single Source of Truth** - One authoritative reference per topic
4. **Version Current** - Always reflect the latest release
---
## Directory Structure
```
docs/
├── README.md # Entry point - navigation hub
├── ARCHITECTURE.md # This file
├── getting-started/ # New users (lowest cognitive load)
│ ├── 01-installation.md
│ ├── 02-quick-start.md
│ ├── 03-your-first-skill.md
│ └── 04-next-steps.md
├── user-guide/ # Common tasks (practical focus)
│ ├── 01-core-concepts.md
│ ├── 02-scraping.md
│ ├── 03-enhancement.md
│ ├── 04-packaging.md
│ ├── 05-workflows.md
│ └── 06-troubleshooting.md
├── reference/ # Technical details (comprehensive)
│ ├── CLI_REFERENCE.md
│ ├── MCP_REFERENCE.md
│ ├── CONFIG_FORMAT.md
│ └── ENVIRONMENT_VARIABLES.md
└── advanced/ # Power users (specialized)
├── mcp-server.md
├── mcp-tools.md
├── custom-workflows.md
└── multi-source.md
```
---
## Category Guidelines
### Getting Started
**Purpose:** Get new users to their first success quickly
**Characteristics:**
- Minimal prerequisites
- Step-by-step instructions
- Copy-paste ready commands
- Screenshots/output examples
**Files:**
- `01-installation.md` - Install the tool
- `02-quick-start.md` - 3 commands to first skill
- `03-your-first-skill.md` - Complete walkthrough
- `04-next-steps.md` - Where to go after first success
---
### User Guide
**Purpose:** Teach common tasks and concepts
**Characteristics:**
- Task-oriented
- Practical examples
- Best practices
- Common patterns
**Files:**
- `01-core-concepts.md` - How it works
- `02-scraping.md` - All 18 source types (docs, GitHub, PDF, video, Word, EPUB, Jupyter, HTML, OpenAPI, AsciiDoc, PPTX, RSS, man pages, Confluence, Notion, Slack/Discord, local codebase)
- `03-enhancement.md` - AI enhancement
- `04-packaging.md` - Platform export
- `05-workflows.md` - Workflow presets
- `06-troubleshooting.md` - Problem solving
---
### Reference
**Purpose:** Authoritative technical information
**Characteristics:**
- Comprehensive
- Precise
- Organized for lookup
- Always accurate
**Files:**
- `CLI_REFERENCE.md` - All CLI commands (including 17 source-type subcommands)
- `MCP_REFERENCE.md` - 40 MCP tools
- `CONFIG_FORMAT.md` - JSON schema (covers all 18 source types)
- `ENVIRONMENT_VARIABLES.md` - All env vars (including Confluence, Notion, Slack tokens)
---
### Advanced
**Purpose:** Specialized topics for power users
**Characteristics:**
- Assumes basic knowledge
- Deep dives
- Complex scenarios
- Integration topics
**Files:**
- `mcp-server.md` - MCP server setup
- `mcp-tools.md` - Advanced MCP usage
- `custom-workflows.md` - Creating workflows
- `multi-source.md` - Unified scraping
---
## Naming Conventions
### Files
- **getting-started:** `01-topic.md` (numbered for order)
- **user-guide:** `01-topic.md` (numbered for order)
- **reference:** `TOPIC_REFERENCE.md` (uppercase, descriptive)
- **advanced:** `topic.md` (lowercase, specific)
### Headers
- H1: Title with version
- H2: Major sections
- H3: Subsections
- H4: Details
Example:
```markdown
# Topic Guide
> **Skill Seekers v3.6.0**
## Major Section
### Subsection
#### Detail
```
---
## Cross-References
Link to related docs using relative paths:
```markdown
<!-- Within same directory -->
See [Troubleshooting](06-troubleshooting.md)
<!-- Up one directory, then into reference -->
See [CLI Reference](../reference/CLI_REFERENCE.md)
<!-- Up two directories (to root) -->
See [Contributing](../../CONTRIBUTING.md)
```
---
## Maintenance
### Keeping Docs Current
1. **Update with code changes** - Docs must match implementation
2. **Version in header** - Keep version current
3. **Last updated date** - Track freshness
4. **Deprecate old files** - Don't delete, redirect
### Review Checklist
Before committing docs:
- [ ] Commands actually work (tested)
- [ ] No phantom commands documented
- [ ] Links work
- [ ] Version number correct
- [ ] Date updated
---
## Adding New Documentation
### New User Guide
1. Add to `user-guide/` with next number
2. Update `docs/README.md` navigation
3. Add to table of contents
4. Link from related guides
### New Reference
1. Add to `reference/` with `_REFERENCE` suffix
2. Update `docs/README.md` navigation
3. Link from user guides
4. Add to troubleshooting if relevant
### New Advanced Topic
1. Add to `advanced/` with descriptive name
2. Update `docs/README.md` navigation
3. Link from appropriate user guide
---
## Deprecation Strategy
When content becomes outdated:
1. **Don't delete immediately** - Breaks external links
2. **Add deprecation notice**:
```markdown
> ⚠️ **DEPRECATED**: This document is outdated.
> See [New Guide](path/to/new.md) for current information.
```
3. **Move to archive** after 6 months:
```
docs/archive/legacy/
```
4. **Update navigation** to remove deprecated links
---
## Contributing
### Doc Changes
1. Edit relevant file
2. Test all commands
3. Update version/date
4. Submit PR
### New Doc
1. Choose appropriate category
2. Follow naming conventions
3. Add to README.md
4. Cross-link related docs
---
## See Also
- [Docs README](README.md) - Navigation hub
- [Contributing Guide](../CONTRIBUTING.md) - How to contribute
- [Repository README](../README.md) - Project overview
+496
View File
@@ -0,0 +1,496 @@
# Best Practices for High-Quality Skills
**Target Audience:** Anyone creating Claude skills | Already scraped documentation? Make it better!
**Time:** 5-10 minutes to review | Apply as you build
**Result:** Skills that Claude understands better and activates more reliably
---
## Quick Checklist
Before uploading a skill, check:
- [ ] SKILL.md has clear "When to Use" triggers
- [ ] At least 5 code examples included
- [ ] Prerequisites documented (if any)
- [ ] Troubleshooting section present
- [ ] Quality score 90+ (Grade A)
```bash
# Check your skill quality
skill-seekers quality output/myskill/
```
---
## 1. Structure Your SKILL.md Clearly
### Use Consistent Sections
Claude looks for specific sections to understand your skill:
```markdown
# Skill Name
## Description
Brief explanation of what this skill enables.
## When to Use This Skill
- User asks about [specific topic]
- User needs help with [specific task]
- User mentions [keywords]
## Prerequisites
What needs to be true before using this skill.
## Quick Reference
Most common commands or patterns.
## Detailed Guide
Step-by-step instructions with examples.
## Troubleshooting
Common issues and solutions.
```
### Why This Matters
Claude uses the "When to Use" section to decide if your skill matches the user's question. Vague triggers = skill doesn't activate.
**Bad Example:**
```markdown
## When to Use This Skill
Use this skill for API-related questions.
```
**Good Example:**
```markdown
## When to Use This Skill
- User asks about Steam Inventory API methods
- User needs to implement item drops in a Steam game
- User wants to grant promotional items to players
- User mentions: SteamInventory, GetAllItems, AddPromoItems
```
---
## 2. Include Real Code Examples
Skills with 5+ code examples work significantly better. Claude learns patterns from examples.
### What Works
**Include a variety:**
- Basic usage (getting started)
- Common patterns (day-to-day use)
- Advanced usage (edge cases)
- Error handling (when things go wrong)
**Example from a good SKILL.md:**
```markdown
## Quick Reference
### Get All Items
```cpp
SteamInventoryResult_t resultHandle;
bool success = SteamInventory()->GetAllItems(&resultHandle);
```
### Grant Promotional Items
```cpp
void CInventory::GrantPromoItems()
{
SteamItemDef_t newItems[2];
newItems[0] = 110;
newItems[1] = 111;
SteamInventory()->AddPromoItems(&s_GenerateRequestResult, newItems, 2);
}
```
### Handle Async Results
```cpp
void OnSteamInventoryResult(SteamInventoryResultReady_t *pResult)
{
if (pResult->m_result == k_EResultOK) {
// Process items
}
}
```
```
### What to Avoid
**Generic placeholder text:**
```markdown
## Quick Reference
*Quick reference patterns will be added as you use the skill.*
```
**Code without context:**
```markdown
`GetAllItems()`
```
---
## 3. Document Prerequisites
Claude can check conditions before proceeding. This prevents errors mid-execution.
### Good Pattern
```markdown
## Before You Start
Make sure you have:
- [ ] Python 3.10+ installed
- [ ] API key set in environment (`export API_KEY=...`)
- [ ] Network access to api.example.com
### Quick Check
```bash
python3 --version # Should show 3.10+
echo $API_KEY # Should not be empty
curl api.example.com/health # Should return 200
```
```
### Why It Matters
Without prerequisites, Claude might:
1. Start a complex workflow
2. Fail halfway through
3. Leave the user with a broken state
With prerequisites, Claude can:
1. Check conditions first
2. Report what's missing
3. Guide the user to fix issues before starting
---
## 4. Add Troubleshooting Sections
Real-world usage hits errors. Document the common ones.
### Template
```markdown
## Troubleshooting
### "Connection refused" error
**Cause:** Service not running or firewall blocking
**Solution:**
1. Check if the service is running: `systemctl status myservice`
2. Verify firewall settings: `sudo ufw status`
3. Test connectivity: `curl -v https://api.example.com`
### "Permission denied" error
**Cause:** Insufficient file permissions
**Solution:**
1. Check file permissions: `ls -la /path/to/file`
2. Fix permissions: `chmod 644 /path/to/file`
3. Verify ownership: `chown user:group /path/to/file`
### "Rate limited" error
**Cause:** Too many API requests
**Solution:**
1. Wait 60 seconds before retrying
2. Implement exponential backoff in your code
3. Consider caching responses
```
### What to Include
- Error message (exact text users see)
- Cause (why it happens)
- Solution (step-by-step fix)
- Prevention (how to avoid it)
---
## 5. Organize Reference Files
The `references/` directory should be easy to navigate.
### Good Structure
```
output/myskill/
├── SKILL.md # Main entry point
└── references/
├── index.md # Category overview
├── getting_started.md # Installation, setup
├── api_reference.md # API methods, classes
├── guides.md # How-to tutorials
└── advanced.md # Complex scenarios
```
### Category Guidelines
| Category | Contains | Keywords |
|----------|----------|----------|
| getting_started | Installation, setup, quickstart | intro, install, setup, quickstart |
| api_reference | Methods, classes, parameters | api, method, function, class, reference |
| guides | Step-by-step tutorials | guide, tutorial, how-to, example |
| concepts | Architecture, design patterns | concept, overview, architecture |
| advanced | Complex scenarios, internals | advanced, internal, extend |
### Navigation Table in SKILL.md
```markdown
## Navigation
| Topic | File | Description |
|-------|------|-------------|
| Getting Started | references/getting_started.md | Installation and setup |
| API Reference | references/api_reference.md | Complete API documentation |
| Guides | references/guides.md | Step-by-step tutorials |
| Advanced | references/advanced.md | Complex scenarios |
```
---
## 6. Run Quality Checks
Always check quality before uploading:
```bash
# Check quality score
skill-seekers quality output/myskill/
# Expected output:
# ✅ Grade: A (Score: 95)
# ✅ No errors
# ⚠️ 1 warning: Consider adding more code examples
```
### Quality Targets
| Grade | Score | Status |
|-------|-------|--------|
| A | 90-100 | Ready to upload |
| B | 80-89 | Good, minor improvements possible |
| C | 70-79 | Review warnings before uploading |
| D | 60-69 | Needs work |
| F | < 60 | Significant issues |
### Common Issues
**"Missing SKILL.md"**
- Run the scraper first
- Or create manually
**"No code examples found"**
- Add code blocks to SKILL.md
- Run enhancement: `skill-seekers enhance output/myskill/`
**"Generic description"**
- Rewrite "When to Use" section
- Add specific keywords and use cases
---
## 7. Test Your Skill
Before uploading, test with Claude:
### Manual Testing
1. Upload the skill to Claude
2. Ask a question your skill should answer
3. Check if Claude activates the skill
4. Verify the response uses skill content
### Test Questions
For a Steam Inventory skill:
```
"How do I get all items in a player's Steam inventory?"
"What's the API call for granting promotional items?"
"Show me how to handle async inventory results"
```
### What to Look For
**Good activation:**
- Claude references your skill
- Response includes examples from your SKILL.md
- Specific, accurate information
**Poor activation:**
- Claude gives generic answer
- No skill reference
- Information doesn't match your docs
---
## Real-World Example
### Before Improvement
```markdown
# React Skill
## Description
React documentation.
## When to Use
For React questions.
## Quick Reference
See references.
```
**Quality Score:** 45 (Grade F)
### After Improvement
```markdown
# React Skill
## Description
Complete React 18+ documentation including hooks, components, and best practices.
## When to Use This Skill
- User asks about React hooks (useState, useEffect, useContext)
- User needs help with React component lifecycle
- User wants to implement React patterns (render props, HOCs, custom hooks)
- User mentions: React, JSX, virtual DOM, fiber, concurrent mode
## Prerequisites
- Node.js 16+ for development
- Basic JavaScript/ES6 knowledge
## Quick Reference
### useState Hook
```jsx
const [count, setCount] = useState(0);
```
### useEffect Hook
```jsx
useEffect(() => {
document.title = `Count: ${count}`;
}, [count]);
```
### Custom Hook
```jsx
function useWindowSize() {
const [size, setSize] = useState({ width: 0, height: 0 });
useEffect(() => {
const handleResize = () => {
setSize({ width: window.innerWidth, height: window.innerHeight });
};
window.addEventListener('resize', handleResize);
handleResize();
return () => window.removeEventListener('resize', handleResize);
}, []);
return size;
}
```
## Troubleshooting
### "Invalid hook call"
**Cause:** Hook called outside component or conditionally
**Solution:**
1. Only call hooks at top level of function components
2. Don't call hooks inside loops or conditions
3. Check for multiple React copies: `npm ls react`
```
**Quality Score:** 94 (Grade A)
---
## Summary
| Practice | Why It Matters | Quick Check |
|----------|---------------|-------------|
| Clear structure | Claude knows where to look | Has all standard sections? |
| Code examples (5+) | Claude learns patterns | Count code blocks |
| Prerequisites | Prevents mid-task failures | Prerequisites section exists? |
| Troubleshooting | Handles real-world errors | Common errors documented? |
| Organized references | Easy navigation | Categories make sense? |
| Quality check | Catches issues early | Score 90+? |
| Testing | Confirms it works | Claude activates skill? |
**Final command before upload:**
```bash
skill-seekers quality output/myskill/
```
That's it! Follow these practices and your skills will work better with Claude.
---
## 8. Tips for Specific Source Types
Skill Seekers supports **18 source types**. Here are tips for getting the best results from each category:
### Documentation (Web)
- Always test CSS selectors before large scrapes: `skill-seekers create --max-pages 3 --verbose`
- Use `--async` for large sites (2-3x faster)
### GitHub Repos
- Use `--analysis-depth c3x` for deep analysis (patterns, tests, architecture)
- Set `GITHUB_TOKEN` to avoid rate limits
### PDFs & Office Documents (PDF, Word, EPUB, PPTX)
- Use `--enable-ocr` for scanned PDFs
- For Word/PPTX, embedded images are extracted automatically; add `--extract-images` for PDFs
- EPUB works best with DRM-free files
### Video
- Run `skill-seekers create --setup` first to install GPU-optimized dependencies
- YouTube and Vimeo URLs are auto-detected; local video files also work
### Jupyter Notebooks
- Ensure notebooks are saved (unsaved cell outputs won't be captured)
- Both code cells and markdown cells are extracted
### OpenAPI/Swagger Specs
- Both YAML and JSON specs are supported (OpenAPI 3.x and Swagger 2.0)
- Endpoints, schemas, and examples are parsed into structured API reference
### AsciiDoc & Man Pages
- AsciiDoc requires `asciidoctor` (install via your package manager or gem)
- Man pages in sections `.1` through `.8` are supported
### RSS/Atom Feeds
- Useful for converting blog posts and changelogs into skills
- Set `--max-items` to limit how many entries are extracted
### Confluence & Notion
- API mode requires authentication tokens (see FAQ for setup)
- Export directory mode works offline with HTML/Markdown exports
### Slack & Discord
- Use official export tools (Slack Workspace Export, DiscordChatExporter)
- Specify `--platform slack` or `--platform discord` explicitly
---
## See Also
- [Enhancement Guide](features/ENHANCEMENT.md) - AI-powered SKILL.md improvement
- [Upload Guide](guides/UPLOAD_GUIDE.md) - How to upload skills to Claude
- [CLI Reference](reference/CLI_REFERENCE.md) - Complete command reference
---
## Contributing
This guide was contributed by the [AI Writing Guide](https://github.com/jmagly/ai-writing-guide) project, which uses Skill Seekers for documentation-to-skill conversion. Best practices here are informed by research on production-grade agentic workflows.
Found an issue or want to improve this guide? PRs welcome!
+704
View File
@@ -0,0 +1,704 @@
# Skill Seekers — Bug Audit
> **Addendum (2026-06-11):** Follow-up review during the Grand Unification work
> (see [UNIFICATION_PLAN.md](UNIFICATION_PLAN.md)) found that two fixes claimed
> below were no-ops as shipped and have since been fixed properly:
> 1. **MCP-03 (`dry_run` for unified configs)** — the injected flag never
> reached `UnifiedScraper`; it is now genuinely honored (constructor param,
> preview-and-return, no directory creation) for both CLI and MCP paths.
> 2. **ENH timeout fallback** — the raw-config fallbacks for enhancement
> settings (incl. timeout/"unlimited") were dead code because
> `ExecutionContext.get()` never raises; the gate now uses
> `ExecutionContext.is_initialized()`.
>
> The findings below are preserved unchanged as a historical record.
**Date:** 2026-06-10
**Scope:** Full audit of `src/skill_seekers/` (~80K LOC): scan/config pipeline, CLI core & dispatch, platform adaptors, MCP server & tools, codebase-analysis engines (AST/pattern/dependency/guide/router), document & media & remote scrapers, enhancement + unified builder, argument/parser system, and infra (embedding, sync, storage, benchmark, presets).
**Method:** Two-pass. Pass 1 — 10 parallel review agents, one per subsystem, surfaced candidate defects. Pass 2 — a second set of agents (plus manual spot-checks) re-read the actual code behind **every** finding, checked reachability/trigger, marked each `Confirmed` / `Corrected` / `False-Positive`, deepened it, and hunted for additional related bugs. Only `Confirmed` and `Corrected` findings appear in the body; dismissed items are recorded in Appendix A for transparency.
**Severity rubric**
- **Critical** — silent data loss/corruption of the primary output, or a core feature completely (and silently) non-functional.
- **High** — wrong result / broken feature under common conditions, or a crash on realistic input.
- **Medium** — wrong behaviour under specific conditions, misleading output, or latent correctness hazard with a plausible trigger.
- **Low** — cosmetic, rare trigger, dead code, type-annotation, or defensive gap.
**No security/RCE findings.** Subprocess calls use argv arrays (no `shell=True`); path-traversal guards in `config_publisher`, `marketplace_publisher`, and `workflow_tools` were verified present and correct.
---
## Summary counts
| Severity | Count |
|----------|-------|
| Critical | 5 |
| High | 18 |
| Medium | 45 |
| Low | 67 |
| **Total confirmed/corrected** | **135** |
| of which newly found in the verification pass (★) | 33 |
| Dismissed (false-positive / no-impact) | 7 (Appendix A) |
IDs are subsystem-prefixed: `SCAN` (scan+config), `CLI` (core/dispatch), `ADP` (adaptors), `MCP`, `CBA` (codebase-analysis: AST/pattern/dependency), `CBB` (codebase-analysis: example/guide/router), `DOC` (document scrapers), `MED` (media/structured/remote scrapers), `ENH` (enhancement + unified builder), `INF` (args/parsers + infra). `★` marks bugs newly discovered during the verification pass.
---
## Resolution status (this PR)
Worked through the audit Critical → Low. Every Critical and High finding is fixed, and the entire Medium tier is fixed except one deferred item. Each fix carries a regression test where behavioral; the rest are covered by existing suites. Lint + format are clean throughout.
| Tier | Total | Fixed | Remaining |
|------|-------|-------|-----------|
| Critical | 5 | 5 | 0 |
| High | 18 | 18 | 0 |
| Medium | 45* | 44 | 1 (DOC-04) |
| Low | 67* | 0 | 67 (not in scope for this PR) |
\* Medium/Low counts include sub-findings; the body tables enumerate each fixed ID.
**Delivered as 8 commits:**
1. Critical + High behavioral bugs (each with a regression test).
2. `--output` honored for every source type + config-file precedence (CLI-01/02, CFG-01/02).
3. `ExecutionContext` made the real single source of truth (RT-01..11): ~12 inert flags wired, 4 bypasses fixed, dead `rag` section removed, `--languages`/`--dry-run` wired.
4. Medium batch 1 — codebase-analysis, enhancement, infra, scan/config, adaptors.
5. Medium batch 2 — guides + document/remote scrapers.
6. DOC-07/MED-04 — SKILL.md nav links fixed across all 7 scrapers.
7. ADP-01 — `--streaming` wired into the 8 RAG/vector adaptors (was dead for every target).
8. CLI-04 — all 15 commands migrated off `_reconstruct_argv` to namespace dispatch (also fixed a pre-existing live bug: `skill-seekers workflows …` was broken).
**Remaining / out of scope:**
- **DOC-04** (Medium, deferred): `DEFAULT_MAX_PAGES = -1` means a config without `max_pages` crawls unbounded. The fix (a finite default) is a **user-visible behavior change** for existing configs, so it's left for a maintainer product decision rather than changed unilaterally.
- **Low tier (67):** cosmetic / rare-trigger / dead-code / type-annotation items — catalogued in the body, not addressed in this PR.
- **`tests/test_bootstrap_skill*.py`** were hardened (out of band) to guard a fork-bomb: the bootstrap/`create` subprocess tests spawn real `skill-seekers create` runs (and LLM enhance agents). Because of this, the *full* `pytest tests/` is unsafe to run repeatedly; verification used targeted per-module suites + `-m "not slow and not integration"`.
---
## Independent re-review of this PR — 2026-06-10 (second pass)
After the fixes above, every commit in this PR was re-reviewed independently (one reviewer per commit) to verify the fixes are correct, complete, and don't introduce regressions — i.e. not taking the audit's own self-grade at face value. The reviewers confirmed the large majority of fixes are correct, and surfaced the following issues, **all now resolved in this PR** (each with a regression test unless noted):
### Regressions this PR had introduced (now fixed)
| ID | Sev | Fix |
|----|-----|-----|
| **ENH-03-bis** | Blocker | The ENH-03 retry path was mis-indented: when the exit-75 retry failed, `new_mtime` was unbound → `UnboundLocalError`. Re-nested under the returncode/exists guards. Tests: `test_enhance_skill_local.py::TestHeadlessSuccessGate::{test_failed_retry_returns_false_without_crashing,test_successful_retry_counts_as_success}`. |
| **R1 (CLI-04-bis)** | High | `skill-seekers estimate <cfg>` crawled **unlimited** instead of capping at 1000 — the unified `estimate_parser`'s `--max-discovery` had no default (`None`), which `estimate_pages()` treats as unlimited. Added `default=DEFAULT_MAX_DISCOVERY`. Test: `test_estimate_pages.py::TestEstimateParserDefault`. |
| **R2 (CBA-14-bis)** | Medium | The JS method regex required `)` immediately before `{`, silently dropping **TypeScript** class methods with return-type annotations (`greet(): string {`). Made the return type optional in the pattern. Test: `test_code_analyzer.py::...::test_typescript_class_methods_with_return_types_not_dropped`. |
| **R3 (CLI-02-bis)** | Medium | `github_scraper._save_data()` did `os.makedirs("output")` but wrote to a `--output`-derived `data_file`, raising `FileNotFoundError` for `create owner/repo --output <nested/path>`. Now creates `dirname(data_file)`. Test: `test_github_scraper.py::...::test_save_data_creates_nested_output_parent`. |
### Audit fixes that were incomplete / over-claimed (now corrected)
| ID | Sev | Resolution |
|----|-----|-----------|
| **I1 (MCP-03)** | High | `skip_scrape` was a no-op (set an attribute no converter read). Now honored at the `SkillConverter.run()` chokepoint for single-source paths and by `UnifiedScraper.run()` for multi-source configs, which reloads each configured source from `.skillseeker-cache`/cached extraction files before conflict detection and build. Tests: `test_skill_converter.py::TestSkipScrape`, `test_unified_scraper_orchestration.py::TestUnifiedSkipScrape`, `test_unified_mcp_integration.py::test_mcp_scrape_docs_unified_skip_scrape_sets_attribute_without_warning`. |
| **I2 (RT-06)** | Medium | Config-file **inline** `stages` never ran — `_build_inline_engine` re-read `args.enhance_stage` (None for config-only) instead of the resolved list. Now takes the resolved `inline_stages`. Test: `test_workflow_runner.py::...::test_inline_stages_from_config_execute`. |
| **I3 (ENH-06)** | Low | The OpenAI branch of `_call_api` still hardcoded `timeout=120`; now forwards the caller's `request_timeout`. Test: `test_agent_client.py::...::test_openai_forwards_caller_timeout`. |
| **I4 (RT-04)** | Low | RT-04 claimed `--timeout` is wired on `create`, but the flag isn't registered there, so the mapping is inert on that path (it only fires for config dicts carrying `timeout`). **Claim corrected here** — no CLI flag added (avoids new surface area for an over-claim). |
### Low-tier / pre-existing issues fixed opportunistically
| ID | Area | Fix |
|----|------|-----|
| **CBA-09** | Kotlin | `is_suspend` used a substring test, mis-flagging a function *named* `suspendCoroutine`. Now word-boundary (`\bsuspend\b`). Test in `test_kotlin_support.py`. |
| **P1** | chat | Discord 429 retry was unbounded (hang risk) — the Slack path in the same commit was bounded; Discord now caps consecutive 429s at 3 and stops with partial data. |
| **P2** | doc | `_normalize_url` stripped `?ref=`, which is content-bearing on some sites (GitHub `?ref=branch`, SPA routers) → could collapse distinct pages. `ref` removed from the strip set (`ref_src` kept). Tests in `test_scraper_features.py::TestNormalizeUrl`. |
| **P3** | doc | `--dry-run` supplied via config still created empty output dirs (the ctor used the param, not `self.dry_run`). Now uses `self.dry_run`/`self.resume`. |
| **P5** | adaptors | `claude.py` enhancement did rename-then-write, leaving only `SKILL.md.backup` if the write failed. Now writes to a temp file, copies the backup, then `os.replace()` (atomic). |
| **P6** | mcp | `submit_config` dedup used GitHub's fuzzy title search (`[CONFIG] react` matched `react-native`). Added an exact-title guard. |
| **P8** | scrapers | asciidoc/jupyter `_generate_index` still had inline filename logic (drift risk vs the nav helper). Both index generators now route through `_ref_filename`, so index links and reference files share one source of truth. |
| **P9** | config | `config status` omitted the Moonshot key row; added it. |
### Deferred (genuine larger work — not safe to do inline; tracked here)
- **Streaming adaptor format (ADP-01 follow-up):** all 8 RAG/vector adaptors emit a *generic* streamed format rather than each platform's native shape (only the unregistered example classes override the converter). `--streaming` works; per-platform fidelity is a follow-up.
- **Unified multi-source `skip_scrape` (I1 remainder):** fixed in #405 by rebuilding `scraped_data` from cached per-source files before the unified build, and by forwarding `--skip-scrape` through the config-source `create` path.
- **Kotlin brace-depth / `is_suspend` corpus robustness (CBA-08, CBA-11):** the brace-depth nesting check is fooled by braces inside strings/comments; a robust fix needs a string/comment-aware scan (shared with the still-open CBA-11).
### Low-tier disposition (2026-06-11)
After fixing the ~12 impactful Low items above, the remaining Low tier was triaged
and batch-fixed. Issues #405#408 track the 4 larger deferred items.
**Fixed in this PR (data-loss/crash + minor correctness):**
DOC-11, DOC-13, DOC-14, DOC-15, DOC-17, MED-06, MED-07, MED-08, MED-11, MED-12,
MED-14, MED-17, MED-18, ENH-13, ENH-14, ENH-15, ENH-16, ENH-17, ADP-03, ADP-04,
ADP-05, ADP-06, MCP-11, MCP-15, MCP-16, MCP-17, SCAN-05, SCAN-06, SCAN-07, CLI-08,
CLI-09, INF-06, CBA-12, CBA-16, CBB-09, CBB-10, CBB-11, CBB-14, CBB-15, CBB-16
(plus the re-review R/I/P items). MED-16 was already fixed in batch 2.
**Intentionally NOT changed (with reason):**
- **Product decisions (left as-is, like DOC-04):** MCP-10 (sync caps at 500 — a
sensible safety bound), CLI-07 (a failed *optional* enhancement returning a
non-zero exit from `create` is a behavior change — needs a maintainer call).
- **Cosmetic / doc-only (dropped):** ADP-07 (docstring), CLI-05 (a misleading
success-log path), MED-13 (type-annotation lie), MCP-08 (param typing), MED-09,
MED-10 (already corrected/dead per Appendix).
- **Still deferred (genuinely larger work):** MED-15 (OpenAPI external/multi-file
`$ref` bundling — needs a pre-bundle pass; the MED-07 *cycle* guard is fixed),
MCP-09 (a truly structured MCP error channel — the text already carries ✅/❌
markers).
- The original **DOC-04** decision stands (issue #408): keep `-1` (unbounded).
---
## Fix log — 2026-06-10
All 5 Criticals + all 18 High findings fixed with regression tests (lint/format clean; affected suites green). A few mechanical/wide fixes (CLI-01, CLI-02, CBB-02/03/12, MCP-02..05) are covered by the broad suite rather than dedicated new tests.
### Critical
| ID | Status | Fix summary |
|----|--------|-------------|
| ENH-01 | ✅ Fixed | `enhance_skill.py`, `agent_client.py`, `adaptors/claude.py` refuse to save/return a `max_tokens`-truncated (or empty) response; unified-scraper path covered via `agent_client`. |
| SCAN-01 | ✅ Fixed | Schema hint `code_analysis_depth` `"standard"``"deep"`; pinned to the validator by a regression test. |
| CBA-13 | ✅ Fixed | GDScript class dict key `"bases"``"base_classes"` (+ `docstring`, `_offset_to_line`). |
| CBB-01 | ✅ Fixed | `guide_enhancer._enhance_via_local` now calls the real `_call_ai`. |
| MCP-01 | ✅ Fixed | `config_publisher` clone/pull/push authenticate via an explicit token URL (origin stays tokenless). |
| ★MCP-12 | ✅ Fixed | `get_source` `KeyError` translated to a helpful `ValueError`. |
| ★MCP-13 | ✅ Fixed | cached-repo re-pull uses the token URL (was tokenless origin). |
| ★MCP-14 | ✅ Fixed | feature branch restored in a `finally` so the cache can't be stranded. |
| MCP-07 | ✅ Fixed | commit `action` uses pre-copy existence (`add` vs `update`). |
| ADP-06 | ✅ Fixed | `adaptors/claude.py` validates non-empty before renaming the original (folded into the ENH-01 fix). |
### High
| ID | Status | Fix summary |
|----|--------|-------------|
| SCAN-02 | ✅ Fixed | Guard the non-numeric `confidence` parse so one bad AI entry can't crash the scan. |
| CLI-01 | ✅ Fixed | `_build_config` reads the negative dests (`not no_issues`/`no_changelog`/`no_releases`); `--no-*` opt-outs now work. |
| CLI-02 | ✅ Fixed | `--output` now honored across the whole single-source create path: base `SkillConverter` + all 16 overriding converters read `config["output_dir"]` (incl. 5 type-annotated `skill_dir: str = …` that a first pass missed), and `data_file`/`data_dir`/`checkpoint` are derived from `skill_dir` so intermediate artifacts follow `--output` too (behavior-preserving fallback to `output/<name>`). `_build_config` passes `output_dir` for every branch. The unified/`config` multi-source path also honors `--output`: `unified_scraper`/`unified_skill_builder` derive the final skill from `config["output_dir"]`, and the docs/github sub-converters now write **straight into the `.skillseeker-cache/` cache** (their sub-config gets `output_dir`), removing the `output/{subname}` hardcodes and the move-to-cache dance. (`pdf_extractor_poc.py` keeps an `output/` *default* but already accepts `image_dir` as a constructor arg — standalone PoC, not the config flow.) |
| CBA-01 | ✅ Fixed | Singleton precedence: `(has_instance_method or has_init_control) and confidence >= 0.5`. |
| CBA-02 | ✅ Fixed | `total_emissions` iterates `.values()` (was `.items()` → always 2×#signals). |
| CBA-03 | ✅ Fixed | JS/TS class body delimited by the matching brace (brace-count), not the first `}`. |
| CBA-04 | ✅ Fixed | New `_base_root`/`_matches_base` normalize generic/qualified bases so subclass detection matches. |
| CBB-02 | ✅ Fixed | Removed the positional verification overwrite that misattributed checks to steps. |
| CBB-03 | ✅ Fixed | `_generate_examples_from_github` works on a copy + index set instead of mutating shared insights. |
| CBB-12 | ✅ Fixed | `dependencies=list(imports)` — per-example copy (no shared mutable list). |
| ENH-02 | ✅ Fixed | `SKILL_SEEKER_PROVIDER` override so a Moonshot/Kimi `sk-` key isn't misrouted to OpenAI. |
| ENH-03 | ✅ Fixed | Headless success gates on mtime advance, not file growth (matches background/daemon). |
| ENH-04 | ✅ Fixed | Kimi parser uses `re.DOTALL` + record-boundary anchor (multi-line / apostrophe safe). |
| ENH-05 | ✅ Fixed | Only read the response file when `output_file` was requested; dropped the stray-`.json` glob. |
| MCP-02 | ✅ Fixed | Package-path regex matches `Output:` / `Package created:` (the strings actually printed). |
| MCP-03 | ✅ Fixed | `scrape_docs` honors `skip_scrape` and applies `dry_run` for unified configs too. |
| MCP-04 | ✅ Fixed | `submit_config` searches for an existing open issue before creating (idempotent). |
| MCP-05 | ✅ Fixed | `install_skill` detects failure by the specific `_run_converter` markers, not any `❌`. |
Regression tests added — Critical: `test_agent_client.py::TestCallApiTruncation`, `test_scan_command.py::TestGenerateSchemaHintDepth`, `test_code_analyzer.py::TestGDScriptParsing`, `test_guide_enhancer.py::TestEnhanceViaLocalRegression`, `test_config_publisher.py` (re-pull + KeyError + add/update). High: `test_agent_client.py` (`TestProviderOverride`, `TestKimiOutputParsing`, `TestCallLocalStrayJson`), `test_scan_command.py` (non-numeric confidence), `test_enhance_skill_local.py::TestHeadlessSuccessGate`, `test_pattern_recognizer.py::TestBaseClassMatching`, `test_code_analyzer.py` (JS multi-method), `test_signal_flow_analyzer.py`. (CBB-02/03/12, CLI-01, MCP-02..05 are covered by the broad suite rather than dedicated new tests.)
### Config-override follow-up (found while auditing "what else is overridden by hardcoded values")
| ID | Status | Fix summary |
|----|--------|-------------|
| CFG-01 | ✅ Fixed | **Web `--config` `selectors`/`url_patterns` silently ignored.** `_merge_json_config` only fills missing keys, but `_build_config` pre-set the hardcoded `selectors`/`url_patterns` before merging → the user's config-file values were shadowed. Now applied via `setdefault` **after** the merge, so the file wins. (Verified `doc_scraper` actually reads `config["selectors"]`/`url_patterns`.) |
| CFG-02 | ✅ Fixed | **Web config-file `workers`/`async_mode`/`browser_wait_until`/`browser_extra_wait` dropped.** `ExecutionContext._load_config_file` copied only `max_pages`/`rate_limit`/`browser`; now copies all scraping-tuning keys. Test: `test_execution_context.py::...test_simple_web_config_format`. |
| CFG-03 | ️ Noted | `--config` is only merged for `web`/`github` source types; for `pdf`/`video`/etc. the documented path is the unified `sources` config (which reads per-source keys correctly). Low impact; left as a design note. |
### Runtime-config (ExecutionContext) audit — is every field wired through args→context→consumer, with no bypasses?
The singleton claims "all components read from this context instead of parsing their own argv." Audit found it was **not** the single source of truth: ~12 registered flags never reached the context, several consumers bypassed it, and two sections were dead. All fixed (regression tests in `test_execution_context.py`).
| ID | Status | Fix summary |
|----|--------|-------------|
| RT-01 | ✅ Fixed | **5 analysis `--skip-*` flags inert.** `--skip-config-patterns`/`--skip-api-reference`/`--skip-dependency-graph`/`--skip-docs`/`--no-comments` were never mapped in `_args_to_data`, yet `create_command` reads those exact `ctx.analysis.*` fields → the skipped steps ran anyway. Now mapped. |
| RT-02 | ✅ Fixed | **`--skip-config` orphan dest** now aliased to `skip_config_patterns`. |
| RT-03 | ✅ Fixed | **`--dry-run` dead on `create`.** `ctx.output.dry_run` was set but never put in the converter config; `_build_config` now passes `"dry_run"`, and `doc_scraper` reads `config["dry_run"]` (was only honoring the unused ctor param). |
| RT-04 | ✅ Fixed | **`--timeout` not wired on `create`** → mapped to `ctx.enhancement.timeout` in `_args_to_data`. |
| RT-05 | ✅ Fixed | **`get_agent_client()` dropped `api_key`** → a CLI `--api-key` was lost and AgentClient env-detected only. Now forwarded. |
| RT-06 | ✅ Fixed | **Config-file `workflows`/`stages`/`workflow_vars` never ran**`run_workflows`/`collect_workflow_vars` read only argv. Now fall back to `ctx.enhancement.*` so config-declared workflows execute. |
| RT-07 | ✅ Fixed | **Confluence/Notion `max_pages` bypass** — read `getattr(self.args, …)` (dropping config-file `max_pages`); now read `ctx.scraping.max_pages`. |
| RT-08 | ✅ Fixed | **`unified_scraper` dropped `ctx.enhancement.timeout` and `api_key`** (re-read raw config / built `AgentClient(mode="api")`); now use the context. |
| RT-09 | ✅ Fixed | **`scraping.languages` was a dead field with a name collision** (default `["en"]`, read nowhere; `--languages` is a *code*-filter the local converter read from argv, which would have broken if fed `["en"]`). Repurposed the field as the code-language filter (`default=None`), wired `--languages` into it, and `create_command` now reads `ctx.scraping.languages`. |
| RT-10 | ✅ Fixed (removed) | **`rag.*` section was write-only/dead.** Chunking lives in the separate `package` command, which parses its own (richer) args and runs as its own process — the create/scrape context never chunks and `ctx.rag` was read nowhere. Removed `RAGSettings` from the context. As cleanup, `common.py` now sources `DEFAULT_CHUNK_TOKENS`/`_OVERLAP` from `defaults.json` (was hardcoded `512`/`50`), so `defaults.json`'s `rag` block is the single source of truth for package chunking. *(Deviation: the earlier plan said "wire rag via package"; on inspection that's unsound — `create` and `package` are separate processes, so a create-time context can't reach package. Removal is the honest fix.)* |
| RT-11 | ✅ Kept (not a bug) | **`source.*`** is read nowhere in production (consumers use `create_command.self.source_info`), but it is **tested public API** and legitimately belongs in an execution context. Retained — removing tested API to chase a cosmetic dual-representation would be over-reach. |
### Medium tier — batch 1 (codebase-analysis + enhancement + infra + scan/config + adaptors)
All ✅ Fixed; verified by existing suites + new regression tests where behavioral.
| ID | Fix summary |
|----|-------------|
| CBA-05 / CBA-06 | Parenthesized the `A and B or C` precedence so the `"protocol"`/`"event"` clause no longer boosts *every* pattern type. |
| CBA-07 | `_resolve_import` converts dotted module names → slash paths + suffix-matches `file_nodes`, so the dependency graph has edges (cycle detection was empty for all dotted-import languages). |
| CBA-08 | Kotlin top-level-fn detection uses brace-depth, not the brittle `indent > 4`. |
| CBA-09 | Kotlin `is_suspend` reads `match.group(0)` (the matched modifiers), not a fixed look-behind window. |
| CBA-10 | `gd_resource` regex ends with `\s*[\]\s]` so `script_class` is captured on compact headers. |
| CBA-14 | JS method extractor requires a trailing `{` (declarations only) + a fuller keyword blocklist, so call-sites like `setTimeout(...)` aren't counted as methods (unmasked by the CBA-03 brace fix). |
| CBA-15 | GDScript signal doc comment = nearest preceding non-blank line (was always `None` for col-0/indented signals). |
| ENH-06 | `_call_api` threads the caller's `timeout` (was hardcoded 120s, killing large prompts). |
| ENH-07 | `scrape_all_sources` reports the count of items scraped (was the bucket count, always ~17) and returns it. |
| ENH-08 | `run()` aborts with non-zero when every source failed (was building an empty skill + exit 0). |
| ENH-09 | "Official Documentation" URL filters `type == "documentation"` (was `sources[0]`). |
| ENH-10 | `DEBUG:` synthesis logs downgraded to `logger.debug`. |
| ENH-11 | Gemini API honors `max_output_tokens`/`timeout` and rejects a truncated reply. |
| ENH-12 | Moonshot/Kimi added to the auto-detect priority in `enhance_command._pick_mode` + `enhance_skill_local._detect_api_target` (Moonshot-only users were dropped to LOCAL). |
| SCAN-03 | `get_api_key`/`set_api_key` include `moonshot``MOONSHOT_API_KEY`. |
| SCAN-04 | `scan --dry-run` resolves with `auto_fetch=False` (no network, no stray `./configs/` write). |
| MCP-06 | `list_configs` falls back to `sources[].base_url` for unified configs. |
| ADP-02 | `chroma`/`pinecone` split `except ImportError` from `except Exception` (don't misreport a broken-but-installed package as "not installed"). |
| CLI-03 | `CreateCommand.execute` calls `ExecutionContext.reset()` first so a 2nd in-process create rebuilds the context. |
| INF-01 | Embedding cache key includes `normalize` (`{model}:{int(normalize)}:{text}`); threaded through `server.py`. |
| INF-02 | `cleanup_old` strips the full `_YYYYMMDD_HHMMSS` timestamp via regex (retention was per-name-per-day). |
| INF-03 | `memory()` samples RSS on a background thread to capture the true peak. |
| INF-04 | `compare()` guards division by zero on instant ops. |
| INF-05 | `check_header_changes` returns `True` when the server provides no `Last-Modified`/`ETag` validators. |
### Medium tier — batch 2 (guides + document scrapers + remote scrapers)
All ✅ Fixed; verified by the guide/scraper suites + new regression tests where behavioral.
| ID | Fix summary |
|----|-------------|
| CBB-04 | `_extract_steps_python` descends one level into the test-function wrapper (else module body) instead of `ast.walk`, which flattened nested control flow out of context. |
| CBB-05 | `ai_enhancer` batch analysis maps back by an echoed `index` (was positional → a dropped/reordered entry shifted every later example's analysis). |
| CBB-06 | `_is_test_class` matches a base that *is* a `TestCase` (bare/`*TestCase`), not the `"Test" in base.id` substring that matched `LatestConfig`/`TestableMixin`. |
| CBB-07 | Index TOC sorts by a difficulty rank map (was alphabetical → advanced/beginner/intermediate). |
| CBB-08 | c3x failure returns `analysis_type="c3x_failed"` (distinct from empty) and the temp dir is removed in `finally` (was leaked every run). |
| CBB-13 | `guide_enhancer` step-enhancements map by the model's explicit `step_index`; positional fallback only when none are indexed (extracted to `_parse_step_enhancements`). |
| DOC-02 | `smart_categorize` assigns to the highest-scoring category (was the first over threshold → config-order dependent). |
| DOC-03 | Link extraction strips tracking params (`utm_*`/`fbclid`/…) + sorts the query before dedup, so tracking variants don't re-crawl the same page (preserves `?lang=`). |
| DOC-05 | PDF cross-page code merge requires a real continuation token / unbalanced bracket / block-opener colon (was `any([...])` ≈ always true) and keeps both pages' `code_blocks_count` consistent. |
| DOC-06 | Async `--dry-run` caps the preview at 20 even for unlimited configs (matches sync; was previewing the whole site). |
| MED-01 | `github_fetcher` follows `Link: rel="next"` pagination (was a single ≤100 page). Regression test added. |
| MED-02 | Slack: per-channel try/except so one channel can't abort all; `conversations_history` retries on 429 with `Retry-After`. |
| MED-03 | Discord: `ClientTimeout(30)`, 429 `Retry-After` retry, and a guarded `before` cursor (`.get("id")`). |
### Medium tier — batch 3 (systemic nav-link fix)
| ID | Status | Fix summary |
|----|--------|-------------|
| DOC-07 + MED-04 | ✅ Fixed | **Broken SKILL.md nav links across all 7 scrapers** (pdf/html/word/epub/jupyter/asciidoc/pptx). Each scraper now routes its SKILL.md nav, `index.md`, and reference-file writer through ONE filename helper (`_reference_filename`, or the pre-existing `_ref_filename` for asciidoc/jupyter), so nav links match the actual range/basename filenames instead of pointing at nonexistent `sanitize(title).md`. Regression test: `test_pdf_scraper.py::...test_reference_filename_matches_nav_and_index`. |
| ADP-01 | ✅ Fixed | **`--streaming` was dead for every target.** `StreamingAdaptorMixin` was never inherited, so `hasattr(adaptor, "package_streaming")` was always False. Fixed the mixin's fragile `sys.path` import (now `from skill_seekers.cli.streaming_ingest import …`) and made all 8 RAG/vector adaptors (langchain, llama-index, chroma, haystack, weaviate, qdrant, faiss, pinecone) inherit it. `--streaming` now produces a real streamed package; non-streaming targets (claude/markdown) still fall back with the announced message. Regression + end-to-end test in `test_adaptors/test_langchain_adaptor.py`. |
| CLI-04 | ✅ Fixed | **`_reconstruct_argv` argv round-trip removed; all 15 legacy commands migrated to namespace dispatch.** Each command's `main(args=None)` now accepts the central parsed namespace (re-parses only when invoked standalone via the `skill-seekers-<cmd>` entry points). The dispatcher calls `module.main(args=args)` directly and `_reconstruct_argv` is deleted. Fixed the drift the round-trip had been masking: aligned 4 central positional dests to the modules (`skill_directory``skill_dir`, `input_file``input` via `metavar`), added `--welcome` to the config parser, and backfill module-parser defaults for options the central parser doesn't expose. **This also fixed a pre-existing live bug: `skill-seekers workflows …` was broken** (`unrecognized arguments: --workflows-action`) and now works. Verified: 345 command-suite tests + all 16 commands dispatch cleanly. |
Deferred (Medium):
- **DOC-04** — `DEFAULT_MAX_PAGES = -1` (unbounded default crawl). Changing the default to finite is a user-visible behavior change; left for an explicit decision (not selected for fixing).
> **Test-harness note:** the *full* `pytest tests/` run is unsafe to execute repeatedly — `@pytest.mark.slow` tests like `test_bootstrap_skill::test_bootstrap_script_runs` and the `test_create_integration_basic` subprocess tests invoke the real `create` pipeline, which spawns live LLM agents (a fork-bomb). Verify with targeted per-module test files and `-m "not slow and not integration"`.
---
## Triage index — Critical & High
| ID | Sev | Title | File |
|----|-----|-------|------|
| ENH-01 | Critical | Truncated SKILL.md silently overwrites the original | `enhance_skill.py` / `agent_client.py` / `adaptors/claude.py` |
| SCAN-01 | Critical | Scan AI github-config generation always rejected by validator | `scan_command.py:426` |
| CBB-01 | Critical | LOCAL-mode guide enhancement calls a non-existent method (silent no-op) | `guide_enhancer.py:307` |
| MCP-01 | Critical | `push_config` strips its own token → broken for private repos | `mcp/config_publisher.py:170,210` |
| CBA-13 ★ | Critical | GDScript class dicts use `"bases"`; consumer reads `"base_classes"` | `code_analyzer.py:1986` |
| SCAN-02 | High | Non-numeric AI `confidence` crashes the whole scan | `scan_command.py:394` |
| CLI-01 | High | `--no-issues/--no-changelog/--no-releases` are dead on `create` | `create_command.py:270` |
| CLI-02 | High | `--output` ignored for all non-local sources; enhancement targets wrong dir | `create_command.py:215` |
| CBA-01 | High | Singleton precedence bug bypasses the confidence gate | `pattern_recognizer.py:453` |
| CBA-02 | High | `total_emissions` iterates `.items()` → always 2×#signals | `signal_flow_analyzer.py:172` |
| CBA-03 | High | JS/TS class body truncated at first `}` → later methods dropped | `code_analyzer.py:301` |
| CBA-04 | High | Subclass detection misses generic/qualified bases | `pattern_recognizer.py:819` |
| CBB-02 | High | Verification points overwritten/misattributed across steps | `how_to_guide_builder.py:185` |
| MCP-02 | High | `install_skill` package-path regex never matches real output | `tools/packaging_tools.py:602` |
| MCP-03 | High | `scrape_docs` drops `skip_scrape` and `dry_run` (unified) | `tools/scraping_tools.py:144` |
| MCP-04 | High | `submit_config` has no idempotency guard → duplicate issues | `tools/source_tools.py:570` |
| MCP-05 | High | `install_skill` infers failure from `"❌"` substring | `tools/packaging_tools.py:523` |
| ENH-02 | High | Direct Moonshot key misrouted to OpenAI | `agent_client.py:202` |
| ENH-03 | High | Headless "success" requires the file to *grow* | `enhance_skill_local.py:900` |
| ENH-04 | High | Kimi stdout parser drops all multi-line responses | `agent_client.py:467` |
| ENH-05 | High | Stray `.json` in agent cwd returned as the LLM response | `agent_client.py:419` |
| CBB-03 | High | `common_problems.remove()` mutates shared insights | `generate_router.py:421` |
| CBB-12 ★ | High | `dependencies=imports` shares one mutable list across examples | `test_example_extractor.py:394` |
---
# Critical
## ENH-01 — Truncated SKILL.md silently overwrites the original `Confirmed · High`
**Location:** `cli/enhance_skill.py:75-94`; `cli/agent_client.py:286-301`; reachable save path `cli/adaptors/claude.py:384-405`; unified path `cli/unified_scraper.py:2079`.
**Trigger:** Enhancement output exceeds `max_tokens` (4096 in `enhance_skill.py`/`agent_client.py`; 8192 in the unified path), or the model otherwise stops with `stop_reason="max_tokens"`.
**Mechanism:** All call sites read `message.content[0].text` / iterate for `block.text` with **no** check of `message.stop_reason` (Anthropic) or `choices[0].finish_reason` (OpenAI). The reachable save path renames the original to `SKILL.md.backup` **first** (claude.py:400), then writes the new content (claude.py:404). So a truncated body overwrites the live file; the only intact copy is the backup.
**Impact:** Any non-trivial skill gets silently chopped to ~4k output tokens (broken frontmatter/fences) and saved as "complete." Silent data loss of the primary deliverable; the run reports success.
```python
message = client.messages.create(model=..., max_tokens=4096, ...)
enhanced_content = message.content[0].text # no stop_reason check
if skill_md_path.exists():
skill_md_path.rename(skill_md_path.with_suffix(".md.backup")) # original moved away first
skill_md_path.write_text(enhanced_content, ...) # truncated body wins
```
**Fix:** Check the stop reason before touching the original; abort on truncation. Also write to a temp file and `os.replace` only after a successful, complete write (so failure can never leave the dir without a `SKILL.md`).
```python
if getattr(message, "stop_reason", None) == "max_tokens":
print("❌ Response truncated (max_tokens); leaving original SKILL.md intact.")
return False
```
## SCAN-01 — Scan AI github-config generation always rejected by the validator `Confirmed · High`
**Location:** `cli/scan_command.py:426` (`_GENERATE_SCHEMA_HINT`); validated at `scan_command.py:555` via `UniSkillConfigValidator(data).validate()`; rejection at `cli/config_validator.py:266-272` against `VALID_DEPTH_LEVELS` (`config_validator.py:71`).
**Trigger:** Any unmapped detection that falls through to `generate_config_with_ai`. The hint sits inside the `"type": "github"` source block, and `"standard"` is a plain literal, so a prompt-compliant model copies it verbatim.
**Mechanism:** The schema hint tells the AI to emit `"code_analysis_depth": "standard"`, but the validator's github-source branch only accepts `{surface, deep, full}`. `generate_config_with_ai` validates AI output and rejects on raise; the retry reuses the same hint, so attempt 2 reproduces the same invalid value. After `max_attempts` the detection lands in `result.failed`.
**Impact:** The headline `scan` feature — "generate a config for an unmapped framework" — systematically produces nothing for any library with a github source. Two wasted AI calls per detection, zero output.
```python
"code_analysis_depth": "standard", # scan_command.py:426 — not a valid level
# config_validator.py:71
VALID_DEPTH_LEVELS = {"surface", "deep", "full"}
```
**Fix:** `"code_analysis_depth": "deep",` (or `"surface"`).
## CBB-01 — LOCAL-mode guide enhancement calls a non-existent method `Confirmed · High`
**Location:** `cli/guide_enhancer.py:307`.
**Trigger:** `build-how-to-guides` runs with AI enhancement when no API key is set (`auto``AgentClient.mode == "local"`) or `ai_mode="local"`.
**Mechanism:** `_enhance_via_local` calls `self._call_claude_local(prompt)`, which is defined nowhere (only `_call_ai` at :263 exists; every other call site uses it). The `AttributeError` is swallowed by the broad `except Exception` in `enhance_guide` (:106-109), which logs and returns the original `guide_data`.
**Impact:** Every LOCAL-mode (no-API-key) guide build silently returns **unenhanced** guides while logging a generic "AI enhancement failed." Local-mode guide enhancement has never worked.
```python
def _enhance_via_local(self, guide_data: dict) -> dict:
prompt = self._create_enhancement_prompt(guide_data)
response = self._call_claude_local(prompt) # NameError: no such method
```
**Fix:** `response = self._call_ai(prompt)`.
## MCP-01 — `push_config` strips its own auth token → broken for private repos `Confirmed · High`
**Location:** `mcp/config_publisher.py:170,210` (compounded by ★MCP-12 / ★MCP-13).
**Trigger:** Any `push_config` to a private/authenticated source repo.
**Mechanism:** The clone uses a token-injected URL, but line 170 immediately runs `origin.set_url(git_url)` (tokenless) "to scrub the token," and the push at line 210 goes through that tokenless `origin`. `marketplace_publisher.py:160` does it correctly — it pushes to a freshly token-injected URL.
**Impact:** The push fails with an auth error *after* the commit, leaving the cache repo with an orphan commit. `push_config` to the primary (private) config source is effectively non-functional. ★MCP-12 and ★MCP-13 (below) make the second run fail at pull and leave the cache on a stranded branch — together they fully break the feature and corrupt the cache.
```python
repo_obj.remotes.origin.set_url(git_url) # :170 scrubs token from origin
...
repo.remotes.origin.push(target_branch) # :210 pushes via tokenless origin
```
**Fix:** `repo.git.push(self.git_repo.inject_token(git_url, token), target_branch)`.
## CBA-13 ★ — GDScript class dicts use key `"bases"`; consumer reads `"base_classes"` `Confirmed · High`
**Location:** `cli/code_analyzer.py:1986` vs `cli/pattern_recognizer.py:367`.
**Trigger:** Any GDScript file with `class_name Foo extends Bar` run through `PatternRecognizer`.
**Mechanism:** `_analyze_gdscript` emits class dicts with the key `"bases"` — the only one of 11 analyzers to do so (the other 10 use `"base_classes"`). `_convert_to_signatures` reads `cls.get("base_classes", [])`, so for GDScript the base list is always `[]`.
**Impact:** All inheritance-based pattern detection (Strategy/Template-Method `subclasses`, Observer/Chain families — every `name in cls.base_classes` check) is silently dead for GDScript, a first-class target (C3.10 signal-flow pipeline).
```python
classes.append({
"name": class_name,
"bases": [extends] if extends else [], # wrong key
...
})
```
**Fix:** rename to `"base_classes"`.
---
# High
## SCAN-02 — Non-numeric AI `confidence` crashes the whole scan `Confirmed · High`
**Location:** `cli/scan_command.py:394`; propagates through `run_scan``_amain` (only catches `KeyboardInterrupt`/`RuntimeError`).
**Trigger:** AI returns `"confidence": "high"` / a list / explicit `null` (LLMs do this despite the "0-1" instruction). Missing key is safe (defaults to `0.0`).
**Mechanism:** `float(entry.get("confidence", 0.0))` is inside the per-entry loop, which is **not** wrapped in try/except (only `client.call` is). `float("high")` raises `ValueError`; it unwinds out of `asyncio.run`.
**Impact:** One malformed entry aborts the entire command with a traceback, discarding all valid detections.
```python
confidence = float(entry.get("confidence", 0.0)) # unguarded
```
**Fix:** wrap in `try/except (TypeError, ValueError): continue`.
## CLI-01 — `--no-issues/--no-changelog/--no-releases` are dead on `create` `Confirmed · High`
**Location:** `cli/create_command.py:270-283`; dests in `arguments/create.py:298-318` & `arguments/github.py:49-69`; consumer `github_scraper.py:257-266`.
**Trigger:** `skill-seekers create owner/repo --no-issues` (or `--no-changelog`/`--no-releases`).
**Mechanism:** The parser defines the **negative** dests `no_issues/no_changelog/no_releases`, but `_build_config` reads the never-defined positive keys `include_issues/include_changelog/include_releases` via `getattr(..., True)`, so they're always `True`. No `no_*→include_*` translation exists anywhere. `include_code` has no flag at all.
**Impact:** `--no-issues` still fetches every issue (potential rate-limit exhaustion); the three opt-out flags are no-ops on the only entry point that exists.
```python
"include_issues": getattr(self.args, "include_issues", True), # attr never exists
```
**Fix:** `"include_issues": not getattr(self.args, "no_issues", False)` (and the others).
## CLI-02 — `--output` ignored for all non-local sources; enhancement targets the wrong dir `Confirmed · High`
**Location:** `cli/create_command.py:215-430` (`_build_config`) & `:450-457` (`_run_enhancement`); scrapers `doc_scraper.py:209`, `notion_scraper.py:99`, `pdf_scraper.py:79`, `unified_scraper.py:120`.
**Trigger:** `create <web/github/pdf/... source> --output /tmp/x` (anything but `output/{name}`), especially with `--enhance-level > 0`.
**Mechanism:** `_build_config` only forwards `output_dir` for the `local` branch; the other scrapers hardcode `self.skill_dir = f"output/{name}"`. But `_run_enhancement` computes `skill_dir = ctx.output.output_dir`, which *does* honor `--output`.
**Impact:** Output always lands in `output/{name}` (flag silently ignored); enhancement then runs against the empty `--output` dir and fails — and since `_run_enhancement` swallows exceptions (★CLI-06), the run still exits 0 with an un-enhanced skill in the wrong place.
**Fix:** have every converter read `config.get("output_dir") or f"output/{self.name}"`, and set `config["output_dir"]` in every `_build_config` branch.
## CBA-01 — Singleton precedence bug bypasses the confidence gate `Confirmed · High`
**Location:** `cli/pattern_recognizer.py:453`.
**Mechanism:** `if has_instance_method or has_init_control and confidence >= 0.5:` parses as `has_instance_method or (...)`. Any class with a method named `instance`/`getInstance` is flagged Singleton at confidence 0.4, bypassing the 0.5 gate.
**Impact:** False-positive Singletons for any class exposing an `instance`-style accessor (e.g. ORM models).
**Fix:** `if (has_instance_method or has_init_control) and confidence >= 0.5:`.
## CBA-02 — `total_emissions` iterates `.items()` → always 2× signal count `Confirmed · High`
**Location:** `cli/signal_flow_analyzer.py:172` (line 171 correctly uses `.values()`).
**Mechanism:** `sum(len(emits) for emits in self.signal_emissions.items())` — each `emits` is a `(key, list)` 2-tuple, so `len` is always 2. Result = `2 × (#emitting signals)`, decoupled from real emission count.
**Impact:** "Total Emissions" in every Godot signal-flow report (`signal_flow.json`, `signal_reference.md`) is meaningless.
**Fix:** `.values()`.
## CBA-03 — JS/TS class body truncated at first `}` `Confirmed · High`
**Location:** `cli/code_analyzer.py:301`.
**Mechanism:** `class_block_end = content.find("}", class_block_start)` finds the **first** brace, not the matching one (C#/Java/Kotlin brace-count; JS doesn't). For any real JS class, the body ends at the first method's closing brace.
**Impact:** All methods after the first are dropped from JS/TS classes, breaking method-count pattern heuristics (Builder/Observer/Strategy). (Fixing this surfaces ★CBA-14.)
**Fix:** use the brace-counting loop already used in `_analyze_csharp`/`_analyze_java`.
## CBA-04 — Subclass detection misses generic/qualified bases `Confirmed · High`
**Location:** `cli/pattern_recognizer.py:819` (also 805/926/1297/1324/1476).
**Mechanism:** `class_sig.name in cls.base_classes` is exact-match, but C# bases are stored verbatim (`BaseStrategy<Foo>`), so `"BaseStrategy" in ["BaseStrategy<Foo>"]` is False. (Java strips generics, so Java works; C#/qualified `Namespace.Base` fail.)
**Impact:** Strategy/Template-Method built on C# generic bases are missed entirely; sibling/family evidence lost.
**Fix:** normalize bases before compare: `b.split("<",1)[0].split(".")[-1].strip()`.
## CBB-02 — Verification points overwritten / misattributed across steps `Confirmed · High`
**Location:** `cli/how_to_guide_builder.py:185-187`.
**Mechanism:** `_extract_steps_python` (:222-226) already pairs each step with its following assertion. Then `analyze_workflow` does `step.verification = verifications[i]` positionally, but `steps` (non-assert statements) and `verifications` (assert lines) are different filtered lists with divergent lengths/order.
**Impact:** Guides show verification code under the wrong step (e.g. step 1 shows a check that validates step 4's result), and the correct pairing is thrown away.
**Fix:** remove the positional overwrite loop; rely on the per-step pairing already done.
## CBB-03 — `common_problems.remove()` mutates shared insights `Confirmed · High`
**Location:** `cli/generate_router.py:421,436` (readers at :865, :1022).
**Mechanism:** `common_problems = self.github_issues.get("common_problems", [])` is the shared list (no copy); `.remove(issue)` deletes from it in place for up to 3 skills, **before** the "Common Issues" section and the issues-reference file read it.
**Impact:** Up to 3 of the most label-relevant issues vanish from the summary and `github_issues.md`; output is order-dependent and non-idempotent.
**Fix:** iterate `list(...)` copy and track used issues by id; never `.remove()` the shared list.
## CBB-12 ★ — `dependencies=imports` shares one mutable list across all examples `Corrected · High→Medium (latent)`
**Location:** `cli/test_example_extractor.py:394,446,490,532` (list built once at :191).
**Mechanism:** The same `imports` list object is assigned as `dependencies` to every `TestExample` from a file, and `how_to_guide_builder._detect_prerequisites` aliases it again into `metadata["required_imports"]`. Aliasing is real, **but** all current consumers only read the list — none mutate it — so there is no observable corruption today.
**Impact:** Latent: any future `.append`/`.sort` on one example's `dependencies` would silently affect all siblings (and the guide prerequisites). Reclassified **Medium** (latent) after verification.
**Fix:** `dependencies=list(imports)` at each call site.
## MCP-02 — `install_skill` package-path regex never matches real output `Confirmed · High`
**Location:** `mcp/tools/packaging_tools.py:602` vs `cli/package_skill.py:177`.
**Mechanism:** The regex matches `"saved to:"`, but `package_skill.py` prints `"Output: <path>"` and `"✅ Package created: <path>"`. The match always fails, so `zip_path` falls back to a constructed name (`{name}.zip`) that differs from real adaptor names (`{dir}-gemini.tar.gz`, etc.).
**Impact:** The upload phase receives a guessed/non-existent path while the workflow reports success.
**Fix:** match what is actually printed: `r"(?im)^\s*(?:Output|✅ Package created):\s*(.+\.(?:zip|tar\.gz))\s*$"` (better: return the path structurally).
## MCP-03 — `scrape_docs` drops `skip_scrape` entirely and `dry_run` for unified configs `Confirmed · High`
**Location:** `mcp/tools/scraping_tools.py:144-216` (wrapper `server_fastmcp.py:353-381`).
**Mechanism:** The FastMCP wrapper declares `skip_scrape`, `dry_run`, `enhance_local` and forwards them, but the impl never reads `skip_scrape`/`enhance_local`, and only sets `converter.dry_run` in the non-unified branch. `server_legacy.py` honored all three.
**Impact:** `skip_scrape=True` re-scrapes from the network (20-45 min wasted, ignores cache); `dry_run=True` performs a full scrape for unified configs. Real server_fastmcp ↔ server_legacy drift.
**Fix:** read `skip_scrape`, set `converter.dry_run = dry_run` and `converter.skip_scrape = skip_scrape` in both branches.
## MCP-04 — `submit_config` has no idempotency guard `Confirmed · High`
**Location:** `mcp/tools/source_tools.py:529-574`.
**Mechanism:** Calls `repo.create_issue(title=f"[CONFIG] {name}", ...)` unconditionally; unlike scan's `maybe_publish` (`_find_existing_issue`), it never searches for an existing open issue.
**Impact:** Re-runs create duplicate `[CONFIG] {name}` issues, polluting the review queue.
**Fix:** query `is:issue is:open in:title "{name}"` before creating; return the existing URL if found.
## MCP-05 — `install_skill` infers scrape failure from `"❌"` substring `Confirmed · Medium-High`
**Location:** `mcp/tools/packaging_tools.py:523`.
**Mechanism:** `if "❌" in scrape_output:` discards the structured returncode that `_run_converter` already provides.
**Impact:** A single skipped page logging `❌` aborts the workflow; a real failure without that emoji proceeds to package an empty skill and reports success. (★MCP-15 is the identical issue for the packaging phase, which has no check at all.)
**Fix:** return a status flag from `scrape_docs_tool`/`_run_converter` and branch on it.
## ENH-02 — Direct Moonshot/Kimi key misrouted to OpenAI `Confirmed · High`
**Location:** `cli/agent_client.py:202-217`.
**Mechanism:** `_detect_provider_from_key` returns `"moonshot"` for an `sk-` key only if `MOONSHOT_API_KEY` env equals it; otherwise any `sk-` key → `"openai"`. Moonshot keys are `sk-`-prefixed, so a key passed via `--api-key` without the env var builds an `OpenAI` client with `gpt-4o`.
**Impact:** Direct Moonshot-key usage fails auth (hits `api.openai.com`) or routes to the wrong provider, surfacing as a misleading OpenAI error.
**Fix:** accept an explicit provider hint (e.g. `SKILL_SEEKER_PROVIDER`) before defaulting to OpenAI.
## ENH-03 — Headless "success" requires the file to *grow* `Confirmed · High`
**Location:** `cli/enhance_skill_local.py:900` (headless), `:1087` (background), `:1213` (daemon).
**Mechanism:** Headless gates success on `new_mtime > initial_mtime and new_size > initial_size`; background/daemon gate only on `returncode == 0`. The three modes disagree.
**Impact:** A legitimate condensing/same-size rewrite is reported as failure in the default headless mode ("SKILL.md was not updated"), so CI/automation treats valid runs as errors.
**Fix:** gate on `new_mtime > initial_mtime` (file changed), not strictly larger size; share one predicate across modes.
## ENH-04 — Kimi stdout parser drops all multi-line responses `Confirmed · High`
**Location:** `cli/agent_client.py:467`.
**Mechanism:** `re.findall(r"TextPart\(type='text', text='(.+?)'\)", raw_output)` has no `re.DOTALL`, so `.` doesn't match newlines; every multi-line SKILL.md matches nothing and the function returns the raw `TurnBegin(...)/StepBegin(...)` debug dump. The non-greedy `'` also splits on embedded apostrophes.
**Impact:** With `--agent kimi`, SKILL.md is replaced by Kimi's raw debug log (or fragmented at apostrophes).
**Fix:** add `re.DOTALL` and anchor the terminator to the next record boundary.
## ENH-05 — Stray `.json` in agent cwd returned as the LLM response `Confirmed · High (conditional)`
**Location:** `cli/agent_client.py:385-393, 419-435` *(downgraded from Critical — conditional trigger)*.
**Mechanism:** When `output_file` is None, the prompt is `prompt.md` and no "write response" instruction is added, yet the code returns `temp/response.json` if present, else **any** `*.json` in the temp dir. The guard `!= "prompt.json"` is dead (the file is `prompt.md`). The subprocess cwd is `temp_path`, so any incidental `.json` the agent writes becomes the "response," shadowing stdout.
**Impact:** Non-deterministic: enhancement can return an unrelated JSON file's contents instead of the model's actual output. Conditional on the agent writing a stray `.json`, hence High not Critical.
**Fix:** only consult the response file when `output_file` was explicitly requested (and match exactly that file); otherwise return stdout.
---
# Medium
## Scan + config
- **SCAN-03 — `get_api_key("moonshot")` env fallback dead.** `config_manager.py:314``env_map` omits `moonshot`, so a user who exports `MOONSHOT_API_KEY` (no config-file key) gets `None`, while `config_command.py:329` shows "(from environment)". `set_api_key`'s error text also omits moonshot. *Fix:* add `"moonshot": "MOONSHOT_API_KEY"`. `Confirmed`
- **★SCAN-04 — `--dry-run` writes `./configs/` and hits the network.** `scan_command.py:1162` calls `resolve_config_path(lookup, auto_fetch=allow_network)` (default True) in the dry-run branch; on an API hit `fetch_config_from_api` performs HTTP GETs and writes `./configs/<name>.json`, contradicting the "DRY RUN — no files written" promise (`:1287`). *Fix:* pass `auto_fetch=False` in dry-run. `Confirmed`
## CLI core
- **CLI-03 — `ExecutionContext.reset()` never called in production.** `execution_context.py:207-228` — the singleton guard returns the existing instance; `reset()` exists but only tests call it. A 2nd in-process `create` reuses the 1st's name/output/settings. Latent today (MCP runs scrapes in subprocesses; CLI is one-shot). *Fix:* call `reset()` at the top of `CreateCommand.execute()`. `Confirmed`
- **★CLI-04 — Legacy `_reconstruct_argv` round-trips the full namespace.** `main.py:179-186` re-serializes every namespace attribute to argv for ~14 legacy commands; attributes unknown to a sub-parser are emitted as `--unknown-flag value`, which the sub-parser can reject. Brittle; the documented migration to `COMMAND_CLASSES` is the fix. `Confirmed`
## Adaptors
- **ADP-01 — `--streaming` is dead for every adaptor.** `streaming_adaptor.py:21` + `package_skill.py:149``StreamingAdaptorMixin` is never imported or inherited by any registered adaptor, so `hasattr(adaptor,"package_streaming")` is always False. *Corrected:* the fallback **announces** "Streaming not supported… using standard packaging" (not silent), but the feature is fully dead. *Fix:* have RAG adaptors inherit the mixin or register the streaming variants. `Corrected`
- **ADP-02 — `except (ImportError, Exception)` masks all import errors.** `chroma.py:236`, `pinecone_adaptor.py:328``Exception` subsumes `ImportError`, so a broken-but-installed package (e.g. chromadb/py3.14 pydantic-v1) is misreported as "not installed." *Fix:* split the handlers. `Confirmed`
## MCP
- **MCP-06 — `list_configs` reads top-level `base_url`; unified configs store it under `sources[0]`.** `tools/config_tools.py:136` — every config `generate_config` produces lists a blank URL. *Fix:* fall back to `sources[].base_url`. `Confirmed`
- **★MCP-12 — `config_publisher` `get_source` raises `KeyError`; the `if not source:` branch is dead.** `config_publisher.py:140-143` — an unknown source name raises a raw `KeyError` (quoted message) instead of the curated "Available sources" error. *Fix:* wrap in `try/except KeyError`. `Confirmed`
- **★MCP-13 — `config_publisher` tokenless re-pull on the 2nd run.** `config_publisher.py:161-172` — on a cached repo it pulls via `origin` (scrubbed tokenless by the prior run's :170) before re-injecting the token, so a 2nd push to a private repo fails at pull. Compounds MCP-01. *Fix:* `origin.set_url(clone_url)` before pulling. `Confirmed`
## Codebase analysis (AST / pattern / dependency)
- **CBA-05 — "protocol" boosts ANY pattern.** `pattern_recognizer.py:1538-1542``type=="Strategy" and "duck typing" in s or "protocol" in s` precedence → the `"protocol"` clause fires for any Python pattern. *Fix:* parenthesize the OR group. `Confirmed`
- **CBA-06 — "event" boosts ANY pattern (+ false "EventEmitter detected").** `pattern_recognizer.py:1559-1563` — same precedence flaw for JS/TS. `Confirmed`
- **CBA-07 — `_resolve_import` never maps dotted module → file path.** `dependency_analyzer.py:784-814``file_nodes` keys are file paths (`src/pkg/mod.py`) while `imported_module` is `pkg.mod`; no dot→slash conversion. The dependency graph is essentially edgeless and cycle detection returns empty for **all dotted-import languages** (Python/Java/C#/Kotlin/Go/Rust/Ruby/PHP). *Corrected:* Godot resource graphs (slash paths) **do** resolve. *Fix:* convert dots to slashes + suffix-match against `file_nodes`. `Confirmed`
- **CBA-08 — Kotlin top-level-fn `indent > 4` heuristic.** `code_analyzer.py:1379-1383` — tab/2-space-indented methods are misclassified as top-level, and indented top-level fns are dropped. *Fix:* track brace depth instead of indentation. `Confirmed`
- **CBA-09 — Kotlin `is_suspend` look-behind reads before the modifier.** `code_analyzer.py:1385` — the 50-char window precedes the match (which already includes `suspend`), so real `suspend fun` reads False and a neighboring `suspend` reads True. *Fix:* `"suspend" in match.group(0)`. `Confirmed`
- **CBA-10 — `gd_resource` regex drops `script_class` on compact headers.** `code_analyzer.py:1875` — trailing `\s+` can't match `]`, so `[gd_resource type="X" script_class="Y"]` yields `script_class=None`. *Fix:* end with `\s*[\]\s]`. `Confirmed`
- **★CBA-14 — JS method extractor matches call-sites and keywords.** `code_analyzer.py:391-419``(\w+)\s*\(...\)` matches any `ident(args)`; only 4 keywords are blocklisted, so `this.helper(b)`, `setTimeout(...)` become "methods." Masked today by CBA-03 (truncation); surfaces once CBA-03 is fixed. *Fix:* anchor to `\s*\{` declarations and exclude the full JS keyword set. `Confirmed`
- **★CBA-15 — GDScript signal doc comment read from the wrong line.** `code_analyzer.py:2043-2047` — for a signal at column 0, `content[:start].split("\n")[-1]` is the empty pre-`signal` text; the doc comment is `lines[-2]`. So `## doc \n signal x` always yields `documentation=None`. *Fix:* use `lines[-2]`. `Confirmed`
## Codebase analysis (example / guide / router)
- **CBB-04 — `ast.walk` flattens nested statements.** `how_to_guide_builder.py:204-206` — statements inside `if`/`for`/`with`/`try` are hoisted into the flat step list, losing control-flow context. *Fix:* iterate top-level body only. `Confirmed`
- **CBB-05 — AI `ai_analysis` aligned by position, not `example_id`.** Real site `ai_enhancer.py:360-362` (the extractor re-merge at `test_example_extractor.py:1088` is actually order-safe). If the model drops/reorders an entry, every later example gets another's analysis (and `tutorial_group`, which drives guide grouping). *Fix:* key by `example_id`. `Confirmed`
- **CBB-06 — `_is_test_class` `"Test" in base.id` over-matches.** `test_example_extractor.py:218-225` — any base whose name *contains* "Test" (`LatestConfig`, `TestableMixin`) flags the class as a test. *Fix:* `endswith("TestCase")` / explicit bases. `Confirmed`
- **CBB-07 — Index TOC sorts complexity alphabetically.** `how_to_guide_builder.py:789``sorted(..., key=g.complexity_level)``advanced, beginner, intermediate`. *Fix:* explicit rank map. `Confirmed`
- **CBB-08 — c3x failures look like empty results; temp dir leaks.** `unified_codebase_analyzer.py:288-305` — broad `except` returns `analysis_type:"c3x"` with empty arrays + an easily-ignored `error` key; `tempfile.mkdtemp` (:255) is never cleaned. *Fix:* mark a distinct failure type and `shutil.rmtree` in `finally`. `Confirmed`
- **★CBB-13 — Guide step-description index trust.** `guide_enhancer.py:552-560` + `how_to_guide_builder.py:1050-1055` — step explanations routed by AI-supplied `step_index` with an enumerate fallback; a dropped entry shifts every subsequent explanation onto the wrong step. *Fix:* require an explicit in-range `step_index`; skip otherwise. `Confirmed`
## Document scrapers
- **DOC-02 — `smart_categorize` assigns to first category over threshold, not max.** `doc_scraper.py:1837-1840``break`s on the first category reaching score 2 (config-order dependent); siblings use `max(scores,...)`. *Corrected severity High→Medium.* *Fix:* collect all scores, assign to `max`. `Confirmed`
- **DOC-03 — Link extraction keeps query strings.** `doc_scraper.py:432-437` — only `#fragment` is stripped, so `?lang=`/`?utm=` variants are distinct pages (crawler-trap risk in unlimited mode). *Fix:* drop tracking params / normalize query before dedup. `Confirmed`
- **DOC-04 — `DEFAULT_MAX_PAGES = -1` ⇒ unbounded default crawl.** `doc_scraper.py:1443-1450`; `defaults.json:7` — a config without `max_pages` enters unlimited mode. *Fix:* finite default (e.g. 1000); require explicit `-1` for unlimited. `Confirmed (by-design footgun)`
- **DOC-05 — PDF cross-page code merge heuristic is ~always true; count drift.** `pdf_extractor_poc.py:606-623``any([not endswith("}"), not endswith(";"), ...])` merges unrelated adjacent blocks; `code_blocks_count` decremented on next page but not incremented on current. *Fix:* require a real continuation token; recompute counts from `len(code_samples)`. `Confirmed`
- **DOC-06 — Async unlimited dry-run doesn't cap at 20.** `doc_scraper.py:1613-1620` — sets `preview_limit=inf` before checking `dry_run` (sync caps at 20 unconditionally), so async `--dry-run` on a default config previews the entire site. *Fix:* set `preview_limit=20` when `dry_run` first. `Confirmed`
- **DOC-07 — Broken SKILL.md nav links (pdf/html/word/epub).** `pdf_scraper.py:530`, `html_scraper.py:1562`, `word_scraper.py:631`, `epub_scraper.py:800` — nav links to `references/{sanitize(title)}.md` but files are written with range/basename names (`_pX-pY`, `_sX-sY`, raw basename). Dead links for any multi-section/multi-category source. (Same root cause as MED-04 — see systemic pattern S2.) *Fix:* derive nav names from the same filename helper that writes the files. `Confirmed`
## Media / structured / remote scrapers
- **MED-01 — GitHub issues capped at 100 (no Link pagination).** `github_fetcher.py:299-345` — single page, `per_page=min(max,100)`, no `rel="next"`; PR filtering reduces further. *Corrected severity High→Medium.* *Fix:* loop on the `next` link. `Confirmed`
- **MED-02 — Slack 429 aborts all channels.** `chat_scraper.py:648-722``conversations_history` has no 429/retry, and the per-channel loop is in one try whose `except SlackApiError` raises a fatal `RuntimeError`, losing already-fetched channels. *Fix:* per-channel try/except + `Retry-After` backoff. `Confirmed`
- **MED-03 — Discord session has no timeout; 429 silently breaks.** `chat_scraper.py:827-878` — no `ClientTimeout`; a 429 is treated as a generic non-200 → silent `break`. *Fix:* add timeout + honor `Retry-After`. `Confirmed`
- **MED-04 — Broken SKILL.md nav links (jupyter/asciidoc/pptx).** `jupyter_scraper.py:966`, `asciidoc_scraper.py:841`, `pptx_scraper.py:1467` — same nav vs file naming mismatch as DOC-07. *Verified:* Confluence/RSS are consistent (not affected). *Fix:* shared filename helper. `Confirmed`
## Enhancement + unified builder
- **ENH-06 — `_call_api` ignores the caller's `timeout`.** `agent_client.py:295-309``call()` computes a timeout (default 45 min) but `_call_api` hardcodes `timeout=120`. Large prompts fail at 2 min with a misleading "connection error." *Fix:* thread `timeout` through. `Confirmed`
- **ENH-07 — `scrape_all_sources` reports the bucket count, not sources scraped.** `unified_scraper.py:256``len(self.scraped_data)` is always ~17. *Fix:* `sum(len(v) for v in ...)`. `Confirmed`
- **ENH-08 — All-sources-fail still builds and returns 0.** `unified_scraper.py:252-254` — per-source exceptions swallowed; `run()` proceeds to `build_skill` on empty data and exits 0. *Fix:* count scraped sources; return non-zero on zero. `Confirmed`
- **ENH-09 — "Official Documentation" reads `sources[0]` regardless of type.** `unified_skill_builder.py:280` — prints `N/A` (or a non-docs URL) if github is listed first. *Fix:* filter `type == "documentation"`. `Confirmed`
- **ENH-10 — `DEBUG:` log lines left in production synthesis.** `unified_skill_builder.py:327-332``logger.info("DEBUG: ...")` and `logger.warning("DEBUG: ... NOT FOUND!")` fire on every docs+github synthesis. *Fix:* `logger.debug`, drop the alarm wording. `Confirmed`
- **★ENH-11 — Gemini API mode ignores `timeout` and `max_tokens`.** `agent_client.py:312-315``generate_content(prompt)` with no `generation_config`/timeout; output capped at the model default, request unbounded. With ENH-01 the truncation is doubly invisible. *Fix:* pass `max_output_tokens`/`request_options` and check `finish_reason`. `Confirmed`
- **★ENH-12 — Provider-priority lists are inconsistent; Moonshot dropped in `_pick_mode`.** `agent_client.py:84-90` honors `MOONSHOT_API_KEY`, but `enhance_command._pick_mode` and `enhance_skill_local._detect_api_target` only check claude/gemini/openai. A Moonshot-only user is silently dropped to LOCAL mode by the `enhance` command. *Fix:* add a moonshot/kimi branch; share one priority order. `Confirmed`
## Args / infra
- **INF-01 — Embedding cache key omits `normalize`.** `embedding/generator.py:414-418`; `server.py:138,182,209` — cache keyed `(model,text)` only; a `normalize=False` request after a cached `normalize=True` returns the wrong (normalized) vector. *Fix:* include `int(normalize)` in the key. `Confirmed`
- **INF-02 — `cleanup_old` folds the date into the group key.** `benchmark/runner.py:293-295` — filename is `{name}_{%Y%m%d_%H%M%S}`; `"_".join(parts[:-1])` leaves `{name}_{date}`, so retention is per-name-per-day → old files never pruned. *Fix:* `parts[:-2]`. `Confirmed`
- **INF-03 — `peak_mb` never samples during the op.** `benchmark/framework.py:184-192` — only `max(before, after)`; transient spikes invisible, so leak/memory recommendations under-report. *Fix:* sample RSS on a background thread during `yield`. `Confirmed`
- **INF-04 — `compare()` divides by zero on instant ops.** `benchmark/runner.py:160,195` — no guard on `current.duration == 0`. *Fix:* guard with `float("inf")`. `Confirmed`
- **INF-05 — Header change-detection misses validator-less servers.** `sync/detector.py:270-277` — returns `False` (unchanged) when neither `Last-Modified` nor `ETag` is present, so `batch_check_headers` silently drops genuinely-changed pages (the `except` branch returns `True`, inconsistently). *Fix:* return `True` when no validators are available. `Confirmed`
---
# Low
## Scan + config
- **SCAN-05 — API-fetched configs written to `./configs/` regardless of `--out`.** `config_fetcher.py:196` — hardcoded `destination="configs"` pollutes CWD (scan re-writes to out_dir anyway). *Fix:* thread the out dir / temp dir through. `Confirmed`
- **SCAN-06 — Module-level `logging.basicConfig` in a library.** `config_validator.py:33` — can win over scan's `--verbose` depending on import order. *Fix:* remove from the library module. `Confirmed`
- **SCAN-07 — Duplicate-slug detections double-count.** `scan_command.py:1149-1199` — the diff dedups by slug, but the main loop double-writes/double-counts duplicate-slug detections. *Fix:* dedup detections by `_config_filename_for`. `Confirmed`
## CLI core
- **CLI-05 — `run()` success log prints `output/{name}` for local builds.** `skill_converter.py:46` + `codebase_scraper.py:2154` — artifacts go to `output_dir` (honors `--output`) but the base log shows `skill_dir`. *Fix:* set `self.skill_dir = str(self.output_dir)`. `Confirmed`
- **CLI-06 — confluence/notion `max_pages` dead `DEFAULTS` fallback.** `create_command.py:404,415``getattr(..., DEFAULTS[...])` fallback is unreachable (`max_pages` always present as `None`); harmless because `None` is treated as unlimited like `-1`. *Corrected (no behavioral impact).* `Corrected`
- **★CLI-07 — `_run_enhancement`/`_run_workflows` swallow all exceptions; exit 0.** `create_command.py:504-516` — enhancement/workflow failure never affects the exit code (CI can't detect it). Compounds CLI-02. *Fix:* track success, return non-zero or emit a clear failure summary. `Confirmed`
- **★CLI-08 — `merge_sources(mode="ai-enhanced")` not recognized.** `merge_sources.py:766-769` — only `"claude-enhanced"` is special-cased, so the canonical `"ai-enhanced"` falls through to rule-based merging for direct callers. *Fix:* accept both spellings. `Confirmed`
- **★CLI-09 — Module-level `logging.basicConfig` in `merge_sources.py:36`.** Same library-logging hazard as SCAN-06 (imported transitively via `generate_router`). *Fix:* remove. `Confirmed`
## Adaptors
- **ADP-03 — Weaviate adaptor uses v3-only API; pin allows v4.** `weaviate.py:341-379`; pin `weaviate-client>=3.25.0``weaviate.Client`, `schema.create_class`, `client.batch` all raise `AttributeError` on v4. *Fix:* cap `<4` or port to the v4 API. `Confirmed`
- **ADP-04 — OpenAI upload uses `client.beta.vector_stores`.** `openai.py:260-280`; pin `openai>=1.0.0` — promoted to `client.vector_stores` in newer SDKs; deprecated path may raise. *Fix:* prefer top-level with fallback; pin a known-good range. `Confirmed (Conf: Low)`
- **ADP-05 — Unquoted YAML name/description in Claude frontmatter.** `claude.py:76-80` — colon-space / leading-quote in `description` produces invalid YAML (OpenCode/IBM-Bob quote; Claude doesn't). *Fix:* quote/escape the values. `Confirmed`
- **ADP-06 — Enhance renames SKILL.md to `.backup` before writing; empty response loses it.** `openai_compatible.py:349-357` (+ openai/gemini/claude) — a successful-but-empty/None response leaves the dir with no `SKILL.md`. *Fix:* validate non-empty before renaming; write-temp-then-swap. `Confirmed`
- **ADP-07 — LlamaIndex uses `format="hex"` while base docstring claims uuid5.** `llama_index.py:43` vs `base.py:481` — doc/behavior mismatch only (hex ids are valid). *Fix:* correct the docstring. `Confirmed`
- *Verified correct:* the thin OpenAI-compatible subclasses (kimi/deepseek/qwen/openrouter/together/fireworks/minimax) have correct, distinct model ids / endpoints / env vars / registry keys; vector-DB dimensions/metrics are consistent.
## MCP
- **MCP-07 — `config_publisher` commit `action` term is meaningless.** `config_publisher.py:204``target_file.exists()` is always True post-copy. *Fix:* capture existence before copy, or use `"update" if force else "add"`. `Confirmed`
- **MCP-08 — Three tool params typed `str` but default `None`.** `server_fastmcp.py:1342,1403,1404` — should be `str | None`; schema advertises non-nullable. `Confirmed`
- **MCP-09 — `enhance_skill` returns identical shape on success/failure.** `tools/packaging_tools.py:329-344` — callers can't distinguish without substring scanning. *Fix:* structured error signal. `Confirmed`
- **MCP-10 — `sync_config` wrapper hardcodes `max_pages=500`.** `server_fastmcp.py:279` vs `tools/sync_config_tools.py:40` — the impl's `DEFAULTS` (-1/unlimited) default is dead; MCP path silently caps at 500. *Fix:* default to `None`, forward only if set. `Confirmed`
- **MCP-11 — Unlimited temp path is non-unique and uses `str.replace`.** `tools/scraping_tools.py:169``config_path.replace(".json", "_unlimited_temp.json")` collides on concurrency and rewrites any `.json` substring. *Fix:* `tempfile.mkstemp`. `Confirmed`
- **★MCP-14 — `config_publisher` left on the feature branch after a mid-flow exception.** `config_publisher.py:198-219` — no try/finally to restore `branch`; a push failure strands the cache on `config/<name>`. *Fix:* restore branch in `finally`. `Confirmed`
- **★MCP-15 — `package_skill_tool` returncode swallowed in `install_skill`.** `tools/packaging_tools.py:85-143` — no success check after packaging; a failed package still yields a fabricated `zip_path`. *Fix:* structured success signal; abort on failure. `Confirmed`
- **★MCP-16 — `generate_config` silently overwrites an existing config.** `tools/config_tools.py:80-84` — no `exists()`/`force` guard; clobbers hand-edited configs. *Fix:* add a `force` arg and guard. `Confirmed`
- **★MCP-17 — `_run_converter` attaches a handler to the shared `skill_seekers` logger.** `tools/scraping_tools.py:41-70` — concurrent converter tools cross-contaminate captured logs. *Fix:* per-call scoped capture. `Confirmed`
## Codebase analysis
- **CBA-11 — Comment extractors match `//` / `/* */` inside string literals.** `code_analyzer.py:581,588` — URLs in strings yield phantom comments. *Fix:* strip string literals first. `Confirmed`
- **CBA-12 — Single-letter CSS classes `c`/`r` matched as C/R.** `language_detector.py:688,708` — bare-name branch returns confidence 1.0 for ambiguous tokens. *Fix:* require a prefix for `c`/`r`. `Confirmed`
- **★CBA-16 — Go grouped-import line numbers off by one.** `dependency_analyzer.py:495` — uses `match.start()` (at `import`) instead of `match.start(1)` (after `import (\n`). *Fix:* `block_start = match.start(1)`. `Confirmed`
## Codebase analysis (guides)
- **CBB-09 — Duplicate `LANGUAGE_ALIASES`; `c++→cpp` but no `cpp` in `PATTERNS`.** `test_example_extractor.py:731-743` — C++ test files silently yield nothing. *Fix:* remove the dup; add `cpp` patterns or drop the alias. `Confirmed`
- **CBB-10 — Space-form `ENV KEY VALUE` dropped; `_infer_purpose` substring-matches.** `config_extractor.py:601-611,350` — legacy ENV skipped; `"db" in "dbeaver"`. *Fix:* handle space form; token-boundary match. `Confirmed`
- **CBB-11 — Step code fences hardcoded ```python```.** `how_to_guide_builder.py:650,663,679` — wrong highlighting for non-Python guides despite `guide.language`. *Fix:* `f"```{guide.language}"`. `Confirmed`
- **★CBB-14 — Basic-mode maps best_practices onto step explanations by index.** `how_to_guide_builder.py:1098-1101` — unrelated best-practice strings attached to steps positionally. *Fix:* render as a separate list. `Confirmed`
- **★CBB-15 — `ast.unparse(node.body)` on a statement list for `setUp`.** `test_example_extractor.py:266-271` — relies on undocumented unparse-of-list behavior. *Fix:* `"\n".join(ast.unparse(s) for s in node.body)`. `Confirmed`
- **★CBB-16 — Regex `group(1)` assumption → `Test: None` for Kotlin/GDScript/C#.** `test_example_extractor.py:762-769` — alternation patterns put the name in group 2/3; `group(1)` is None, and the body-slice re-search can land inside the current body. *Fix:* `next((g for g in match.groups() if g), match.group(0))`. `Confirmed`
- **★CBB-17 — Redundant triple-fetch of `ai_analysis` with divergent null handling.** `how_to_guide_builder.py:957-961, 995, 1004-1012` — maintenance hazard. *Fix:* fetch once, reuse. `Confirmed`
## Document scrapers
- **DOC-08 — llms.txt sections with identical titles overwrite each other.** `doc_scraper.py:867-875` — same title → same URL → same `{safe_title}_{url_hash}` filename. *Corrected:* narrow (identical-title sections only), not the general case. *Fix:* add a section index/counter. `Corrected`
- **DOC-09 — Parallel/async submit guard off-by-one (`<=` vs `<`).** `doc_scraper.py:1523,1647` vs `1457` — parallel mode over-scrapes `max_pages` by ≥1. *Fix:* use `<`. `Confirmed`
- **DOC-10 — `is_valid_url` rejects the exact base page.** `doc_scraper.py:301-303` — trailing-slash `base_dir` drops the slashless base URL. *Corrected:* negligible — the start page is enqueued directly via `start_urls`. *Fix:* also accept `url == base_url`. `Corrected (negligible)`
- **DOC-11 — EPUB section images written as 0-byte files with broken links.** `epub_scraper.py:984-994` + `598-613``data=b""` passes the `isinstance(bytes)` check → 0-byte PNG + `![Image]` link. *Fix:* `if isinstance(...) and len(data) > 0`. `Confirmed`
- **DOC-12 — `_extract_html_as_markdown` returns `links=[]` → BFS dead-ends.** `doc_scraper.py:509-510,667-711` — a `.md` URL serving HTML contributes no further links. *Fix:* extract same-prefix links. `Confirmed`
- **DOC-13 — `_extract_tables` drops body rows equal to the header.** `html_scraper.py:868-872` (+ shared helper, confluence) — `cells != headers` drops legitimate data rows. *Fix:* skip the first row only when it came from `<thead>`. `Confirmed`
- **DOC-14 — Non-atomic checkpoint/page writes corrupt on interrupt.** `doc_scraper.py:334, 874` (the earlier-cited :771 is unrelated) — direct `open(...,'w')`; a second Ctrl-C truncates the checkpoint, and `load_checkpoint` silently starts fresh, losing progress. *Fix:* temp-file + `os.replace`. `Confirmed`
- **DOC-15 — `int(img.get("width") or 0)` crashes on `"100%"`/`"50px"`.** `html_scraper.py:911-912` (crash), `word_scraper.py:826` (silently drops image), `epub_scraper.py:991` (crash). *Fix:* parse leading digits defensively. `Confirmed`
- **★DOC-16 — Async dry-run discovered links bypass the cap.** `doc_scraper.py:1647-1664` — combined with DOC-06, async `--dry-run` issues real GETs across the whole site. *Fix:* the DOC-06 fix bounds this. `Confirmed`
- **★DOC-17 — pdf legacy image branch writes raw `img["data"]` with no guard.** `pdf_scraper.py:341-356``b""` → 0-byte file; missing/non-bytes → `KeyError`/`TypeError` aborts the reference-file write. *Fix:* `isinstance`/non-empty guard + `.get`. `Confirmed`
- **★DOC-18 — `smart_categorize` output is config-order-dependent.** `doc_scraper.py:1809-1846` — corollary of DOC-02; identical configs differing only in key order produce different groupings. *Fix:* same as DOC-02 (pick max, stable tiebreak). `Confirmed`
## Media / structured / remote scrapers
- **MED-05 — `VideoScraperResult.to_dict()` drops `config`.** `video_models.py:827-848``--from-json` reload loses clip range/languages/whisper model. *Corrected:* latent — no current reader of `result.config`. *Fix:* serialize/restore `config`. `Corrected (latent)`
- **MED-06 — `.man` files dropped when `--sections` is set.** `man_scraper.py:317-353` — section-less `.man` pages have `section_num=None`, never in the list. *Fix:* only filter when `section_num is not None`. `Confirmed`
- **MED-07 — OpenAPI `$ref` has no cycle set.** `openapi_scraper.py:775-855` — self-referential schemas expand to the depth-10 cap (bounded — no hang). *Fix:* track visited ref names; emit a stub on cycle. `Confirmed`
- **MED-08 — RSS link-following has no global time budget.** `rss_scraper.py:191-206,484-489` — serial, 15s/req + 1s sleep; ~13 min on a 50-entry feed with slow hosts. *Fix:* time budget or concurrency. `Confirmed`
- **MED-09 — Confluence pagination compares against constant `limit`.** `confluence_scraper.py:455-528``if len(batch) < limit` uses 50, not the requested `min(...)`. *Corrected:* sloppy but not a reachable premature-stop (outer `while` already bounds it). *Fix:* compare against the requested size. `Corrected`
- **MED-10 — OCR keyframe temp JPEGs never deleted.** `video_visual.py:702-715``delete=False`, never unlinked. *Corrected:* `extract_keyframes` is dead code (no production caller); the active pipeline writes intended output frames and cleans stale ones. *Fix:* delete the dead function or have it clean up. `Corrected (dead code)`
- **MED-11 — Confluence `created` only set for version-1 pages.** `confluence_scraper.py:494-498` — empty for all edited pages. *Fix:* use `history.createdDate`. `Confirmed`
- **MED-12 — `get_releases()` / file-tree walk uncapped.** `github_scraper.py:686-708, 939-968` — unlike issues' `islice`, can exhaust the rate-limit quota on large repos. *Fix:* `islice` / depth cap. `Confirmed`
- **MED-13 — `max_pages: int` assigned `float("inf")`.** `notion_scraper.py:96-98`, `confluence_scraper.py:241-243` — type-annotation lie; works only because comparisons accept inf. *Fix:* `sys.maxsize` or annotate `float`. `Confirmed`
- **MED-14 — Video AI ref cleaning accepts any response >50% of input.** `video_scraper.py:299-312` — no `stop_reason` check; a truncated (>50%) response overwrites the reference file. *Fix:* require `stop_reason == "end_turn"`. `Confirmed`
- **★MED-15 — OpenAPI external/relative `$ref` left as an unresolved stub.** `openapi_scraper.py:916-943` — only `#/`-refs resolve; multi-file bundles emit stubs with a debug-only log. *Fix:* pre-bundle, or warn + mark `_ref_unresolved`. `Confirmed`
- **★MED-16 — Discord `before = batch[-1]["id"]` `KeyError` on malformed message.** `chat_scraper.py:875` — aborts the whole fetch. *Fix:* `.get("id")` + break. `Confirmed`
- **★MED-17 — Slack `conversations_list` is single-page.** `chat_scraper.py:654-662` — no `next_cursor`; >200 channels truncated. *Fix:* paginate via `response_metadata.next_cursor`. `Confirmed`
- **★MED-18 — GitHub "all" mode `//2` quota split under-fetches.** `github_fetcher.py:289-297` — rigid open/closed split never reallocates unused quota; magnifies MED-01. *Fix:* fetch open then request `max - len(open)` closed. `Confirmed`
## Enhancement + builder
- **ENH-13 — Unknown provider → `_init_api_client` returns None, mode stays "api".** `agent_client.py:219-254``_call_api` then silently returns None (no fallback to LOCAL). *Fix:* `else:` raise or set `self.mode = "local"`. `Confirmed`
- **ENH-14 — `quality_metrics --threshold` parsed but never used.** `quality_metrics.py:542``main()` always returns 0, so quality gating is a no-op (also a 0-10 vs percentage scale mismatch). *Fix:* compare the score and return non-zero. `Confirmed`
- **ENH-15 — `prompt_file.write_text` missing `encoding="utf-8"`.** `agent_client.py:393``UnicodeEncodeError` on non-UTF-8 locales for emoji/CJK prompts (every read uses utf-8). *Fix:* add `encoding="utf-8"`. `Confirmed`
- **★ENH-16 — `_call_api` error classification by name substring.** `agent_client.py:317-353``"auth"/"rate" in type-name` misfires; a 429 whose class lacks "rate" gets a generic message. *Fix:* branch on HTTP status / typed SDK exceptions. `Confirmed (diagnostic)`
- **★ENH-17 — PDF content dropped when base SKILL.md lacks a Reference heading.** `unified_skill_builder.py:619-623, 659-663``insertion_index == -1` → assembled PDF lines silently discarded (no append-at-end fallback, unlike the docs+github+pdf path). *Fix:* append at end when no heading found. `Confirmed`
## Args / infra
- **INF-06 — `--skip-config` is an orphan flag.** `arguments/create.py:427-433` — dest `skip_config` read nowhere (the real flag is `--skip-config-patterns`). *Fix:* remove or alias. `Confirmed`
- **INF-07 — Azure server-side copy uses an unauthenticated source URL.** `storage/azure_storage.py:223``start_copy_from_url(source_blob.url)` (no SAS) → 403 on private containers (S3/GCS are fine). *Fix:* append a short-lived read SAS. `Confirmed`
- **INF-08 — `presets/` package is dead code.** `presets/manager.py`, `presets/analyze_presets.py` — imported by no command (analyze parser not registered). Latent: `resolve_enhance_level` assumes `enhance_level` defaults to `None`, but the analyze parser forces `0`, so `--enhance` would be dead if revived. *Fix:* delete, or set the default back to `None`. `Confirmed`
- **★INF-09 — `create --no-preserve-code-blocks` / `--no-preserve-paragraphs` inert.** `arguments/create.py:810-822` — registered for `create` but read nowhere in `_args_to_data`/`_build_config` (only the `package` command consumes them). *Fix:* wire into the RAG config or drop from `create`. `Confirmed`
- **★INF-10 — `create --preset/-p` consumed nowhere.** `arguments/create.py:75-82` — the only reader lives in the dead `presets` package; `--preset comprehensive` is parsed and discarded. *Fix:* wire to analysis depth/features or remove. `Confirmed`
---
# Appendix A — Investigated and dismissed / corrected severity
These were checked against the source and are **not** defects (or were materially downgraded). Recorded for coverage.
- **C6 → DOC-01 (pdf `next(iter(...))` StopIteration).** *Downgraded Critical → Low (not in body).* The `elif self.categories:` guard means a plain dict is non-empty there, so `next(iter(values()))` cannot raise; it is only a defensive inconsistency vs the siblings (which pass `, None`). Low-priority robustness fix: add the `, None` default at `pdf_scraper.py:192`.
- **CLI `_validate_arguments` false "not applicable" warnings — FALSE POSITIVE.** `create_command.py:111-136`. Verified every argument dict key equals its argparse dest, so the compatible-set membership check aligns; warnings only fire for genuinely-incompatible args.
- **CBA GDScript `class_name` line via `.count("\n")` — FALSE POSITIVE (correctness).** `code_analyzer.py:1988`. Produces the identical line number as `_offset_to_line`; only a style/perf nit.
- **CBB JS/TS config extraction duplicate settings — FALSE POSITIVE.** `config_extractor.py:584-586`. The three regexes are mutually exclusive per declaration; tested, no duplicate emissions. (A separate `version = 1.5``1` truncation defect exists but was not the claim.)
- **DOC PDF chapter-boundary chunk inverts start/end — FALSE POSITIVE.** `pdf_extractor_poc.py:660-669`. The non-empty-`current_chunk` guard guarantees `end_page ≥ start_page`.
- **ENH `enhance --model` dropped — CORRECTED (dead path).** `enhance_skill_local.py:1312-1340` does drop `--model`, but it's an unreachable legacy `main()`; the shipped `enhance` command routes through `enhance_command.py`, which forwards `--model` correctly to `enhance_skill.py` (which accepts it).
- **Scan archive/diff ordering & `set_api_key` display name — FALSE POSITIVE.** `scan_command.py:1137-1143` ordering is correct by design; `config_manager.py` `capitalize()` is display-only (the real defect is the env_map omission, SCAN-03).
---
# Appendix B — Systemic patterns
Cross-cutting root causes worth fixing once rather than per-site:
- **S1 — Truncated/empty LLM output accepted as complete.** No `stop_reason`/`finish_reason` check: ENH-01 (`enhance_skill`/`agent_client`), ENH-11 (Gemini), MED-14 (video), ADP-06 (rename-before-write). Add a single "complete and non-empty?" gate before any overwrite, and always write-temp-then-swap.
- **S2 — SKILL.md nav links derived differently from reference filenames.** DOC-07 (pdf/html/word/epub) + MED-04 (jupyter/asciidoc/pptx): 7 scrapers link to `sanitize(title).md` while files use range/basename names. Extract one `_ref_filename()` helper per scraper and use it for both writing and nav. (★MED-N5: PPTX duplicates the naming logic in 3 places.)
- **S3 — Success/failure inferred from emoji/text substrings.** MCP-05 (`"❌"`), MCP-09/★MCP-15 (enhance/package), ENH-04 (Kimi regex). Return structured status/returncodes; never scan stdout for `❌`.
- **S4 — `A or B and C` precedence bugs.** CBA-01, CBA-05, CBA-06, CBB-06. Audit every multi-clause boolean for missing parentheses.
- **S5 — Shared-mutable-list aliasing.** CBB-03 (`common_problems.remove`), CBB-12 (`dependencies=imports`). Copy lists at construction (`list(...)`).
- **S6 — Pagination / `max_pages` inconsistency.** MED-01/★MED-18 (github), MED-02/★MED-17 (slack), MED-03/★MED-16 (discord), MED-09 (confluence), DOC-04/DOC-09 (doc_scraper), MED-13 (`inf`-as-`int`). Standardize a cursor-following helper with explicit caps.
- **S7 — Negative→positive flag translation gaps & inert flags.** CLI-01 (`--no-*`), INF-06/INF-09/INF-10 (`--skip-config`, `--no-preserve-*`, `--preset`). Add a registration test asserting every declared arg's dest is read somewhere.
- **S8 — Library modules calling `logging.basicConfig`.** SCAN-06, ★CLI-09. Only the entry point should configure logging.
- **S9 — Provider/model/timeout plumbing not threaded to the SDK call.** ENH-02, ENH-06, ENH-11, ENH-12. Centralize provider detection and request-option passing in one place.
---
# Suggested fix order
1. **S1 + ENH-01** (silent data loss of the primary artifact) — highest blast radius, low risk.
2. **SCAN-01, CBB-01, MCP-01(+★MCP-12/13/14), CBA-13** (core features silently broken) — small, surgical fixes.
3. **CLI-01, CLI-02(+★CLI-07), SCAN-02** (user-visible flag/crash bugs on the main `create`/`scan` paths).
4. **S3 + S6** (MCP install/upload reliability; scraper pagination correctness).
5. **CBA-01..04 + ★CBA-13..16** (pattern/dependency-analysis correctness — many share the precedence/regex root causes in S4).
6. Medium/Low by subsystem as code is touched.
+762
View File
@@ -0,0 +1,762 @@
# Docker Deployment Guide
Complete guide for deploying Skill Seekers using Docker.
## Table of Contents
- [Quick Start](#quick-start)
- [Building Images](#building-images)
- [Running Containers](#running-containers)
- [Docker Compose](#docker-compose)
- [Configuration](#configuration)
- [Data Persistence](#data-persistence)
- [Networking](#networking)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)
## Quick Start
### Single Container Deployment
```bash
# Pull pre-built image (when available)
docker pull skillseekers/skillseekers:latest
# Or build locally
docker build -t skillseekers:latest .
# Run MCP server
docker run -d \
--name skillseekers-mcp \
-p 8765:8765 \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-e GITHUB_TOKEN=$GITHUB_TOKEN \
-v skillseekers-data:/app/data \
--restart unless-stopped \
skillseekers:latest
```
### Multi-Service Deployment
```bash
# Start all services
docker-compose up -d
# Check status
docker-compose ps
# View logs
docker-compose logs -f
```
## Building Images
### 1. Production Image
The Dockerfile uses multi-stage builds for optimization:
```dockerfile
# Build stage
FROM python:3.12-slim as builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt
# Runtime stage
FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "-m", "skill_seekers.mcp.server_fastmcp"]
```
**Build the image:**
```bash
# Standard build
docker build -t skillseekers:latest .
# Build with specific features
docker build \
--build-arg INSTALL_EXTRAS="all-llms,embedding" \
-t skillseekers:full \
.
# Build with cache
docker build \
--cache-from skillseekers:latest \
-t skillseekers:v3.6.0 \
.
```
### 2. Development Image
```dockerfile
# Dockerfile.dev
FROM python:3.12
WORKDIR /app
RUN pip install -e ".[dev]"
COPY . .
CMD ["python", "-m", "skill_seekers.mcp.server_fastmcp", "--reload"]
```
**Build and run:**
```bash
docker build -f Dockerfile.dev -t skillseekers:dev .
docker run -it \
--name skillseekers-dev \
-p 8765:8765 \
-v $(pwd):/app \
skillseekers:dev
```
### 3. Image Optimization
**Reduce image size:**
```bash
# Multi-stage build
FROM python:3.12-slim as builder
...
FROM python:3.12-alpine # Smaller base
# Remove build dependencies
RUN pip install --no-cache-dir ... && \
rm -rf /root/.cache
# Use .dockerignore
echo ".git" >> .dockerignore
echo "tests/" >> .dockerignore
echo "*.pyc" >> .dockerignore
```
**Layer caching:**
```dockerfile
# Copy requirements first (changes less frequently)
COPY requirements.txt .
RUN pip install -r requirements.txt
# Copy code later (changes more frequently)
COPY . .
```
## Running Containers
### 1. MCP Server
```bash
# HTTP transport (recommended for production)
docker run -d \
--name skillseekers-mcp \
-p 8765:8765 \
-e MCP_TRANSPORT=http \
-e MCP_PORT=8765 \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v skillseekers-data:/app/data \
--restart unless-stopped \
skillseekers:latest
# stdio transport (for local tools)
docker run -it \
--name skillseekers-stdio \
-e MCP_TRANSPORT=stdio \
skillseekers:latest
```
### 2. Embedding Server
```bash
docker run -d \
--name skillseekers-embed \
-p 8000:8000 \
-e OPENAI_API_KEY=$OPENAI_API_KEY \
-e VOYAGE_API_KEY=$VOYAGE_API_KEY \
-v skillseekers-cache:/app/cache \
--restart unless-stopped \
skillseekers:latest \
python -m skill_seekers.embedding.server --host 0.0.0.0 --port 8000
```
### 3. Sync Monitor
```bash
docker run -d \
--name skillseekers-sync \
-e SYNC_WEBHOOK_URL=$SYNC_WEBHOOK_URL \
-v skillseekers-configs:/app/configs \
--restart unless-stopped \
skillseekers:latest \
skill-seekers-sync start --config configs/react.json
```
### 4. Interactive Commands
```bash
# Run scraping
docker run --rm \
-e GITHUB_TOKEN=$GITHUB_TOKEN \
-v $(pwd)/output:/app/output \
skillseekers:latest \
skill-seekers create --config configs/react.json
# Generate skill
docker run --rm \
-v $(pwd)/output:/app/output \
skillseekers:latest \
skill-seekers package output/react/
# Interactive shell
docker run --rm -it \
skillseekers:latest \
/bin/bash
```
## Docker Compose
### 1. Basic Setup
**docker-compose.yml:**
```yaml
version: '3.8'
services:
mcp-server:
image: skillseekers:latest
container_name: skillseekers-mcp
ports:
- "8765:8765"
environment:
- MCP_TRANSPORT=http
- MCP_PORT=8765
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- GITHUB_TOKEN=${GITHUB_TOKEN}
- LOG_LEVEL=INFO
volumes:
- skillseekers-data:/app/data
- skillseekers-logs:/app/logs
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
embedding-server:
image: skillseekers:latest
container_name: skillseekers-embed
ports:
- "8000:8000"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- VOYAGE_API_KEY=${VOYAGE_API_KEY}
volumes:
- skillseekers-cache:/app/cache
command: ["python", "-m", "skill_seekers.embedding.server", "--host", "0.0.0.0"]
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
nginx:
image: nginx:alpine
container_name: skillseekers-nginx
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./certs:/etc/nginx/certs:ro
depends_on:
- mcp-server
- embedding-server
restart: unless-stopped
volumes:
skillseekers-data:
skillseekers-logs:
skillseekers-cache:
```
### 2. With Monitoring Stack
**docker-compose.monitoring.yml:**
```yaml
version: '3.8'
services:
# ... (previous services)
prometheus:
image: prom/prometheus:latest
container_name: skillseekers-prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
restart: unless-stopped
grafana:
image: grafana/grafana:latest
container_name: skillseekers-grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
volumes:
- grafana-data:/var/lib/grafana
- ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
restart: unless-stopped
loki:
image: grafana/loki:latest
container_name: skillseekers-loki
ports:
- "3100:3100"
volumes:
- loki-data:/loki
restart: unless-stopped
volumes:
prometheus-data:
grafana-data:
loki-data:
```
### 3. Commands
```bash
# Start services
docker-compose up -d
# Start with monitoring
docker-compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
# Check status
docker-compose ps
# View logs
docker-compose logs -f mcp-server
# Scale services
docker-compose up -d --scale mcp-server=3
# Stop services
docker-compose down
# Stop and remove volumes
docker-compose down -v
```
## Configuration
### 1. Environment Variables
**Using .env file:**
```bash
# .env
ANTHROPIC_API_KEY=sk-ant-...
GITHUB_TOKEN=ghp_...
OPENAI_API_KEY=sk-...
VOYAGE_API_KEY=...
LOG_LEVEL=INFO
MCP_PORT=8765
```
**Load in docker-compose:**
```yaml
services:
mcp-server:
env_file:
- .env
```
### 2. Config Files
**Mount configuration:**
```bash
docker run -d \
-v $(pwd)/configs:/app/configs:ro \
skillseekers:latest
```
**docker-compose.yml:**
```yaml
services:
mcp-server:
volumes:
- ./configs:/app/configs:ro
```
### 3. Secrets Management
**Docker Secrets (Swarm mode):**
```bash
# Create secrets
echo $ANTHROPIC_API_KEY | docker secret create anthropic_key -
echo $GITHUB_TOKEN | docker secret create github_token -
# Use in service
docker service create \
--name skillseekers-mcp \
--secret anthropic_key \
--secret github_token \
skillseekers:latest
```
**docker-compose.yml (Swarm):**
```yaml
version: '3.8'
secrets:
anthropic_key:
external: true
github_token:
external: true
services:
mcp-server:
secrets:
- anthropic_key
- github_token
environment:
- ANTHROPIC_API_KEY_FILE=/run/secrets/anthropic_key
```
## Data Persistence
### 1. Named Volumes
```bash
# Create volume
docker volume create skillseekers-data
# Use in container
docker run -v skillseekers-data:/app/data skillseekers:latest
# Backup volume
docker run --rm \
-v skillseekers-data:/data \
-v $(pwd):/backup \
alpine \
tar czf /backup/backup.tar.gz /data
# Restore volume
docker run --rm \
-v skillseekers-data:/data \
-v $(pwd):/backup \
alpine \
sh -c "cd /data && tar xzf /backup/backup.tar.gz --strip 1"
```
### 2. Bind Mounts
```bash
# Mount host directory
docker run -v /opt/skillseekers/output:/app/output skillseekers:latest
# Read-only mount
docker run -v $(pwd)/configs:/app/configs:ro skillseekers:latest
```
### 3. Data Migration
```bash
# Export from container
docker cp skillseekers-mcp:/app/data ./data-backup
# Import to new container
docker cp ./data-backup new-container:/app/data
```
## Networking
### 1. Bridge Network (Default)
```bash
# Containers can communicate by name
docker network create skillseekers-net
docker run --network skillseekers-net skillseekers:latest
```
### 2. Host Network
```bash
# Use host network stack
docker run --network host skillseekers:latest
```
### 3. Custom Network
**docker-compose.yml:**
```yaml
networks:
frontend:
driver: bridge
backend:
driver: bridge
internal: true # No external access
services:
nginx:
networks:
- frontend
mcp-server:
networks:
- frontend
- backend
database:
networks:
- backend
```
## Monitoring
### 1. Health Checks
```yaml
services:
mcp-server:
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
```
### 2. Resource Limits
```yaml
services:
mcp-server:
deploy:
resources:
limits:
cpus: '2.0'
memory: 4G
reservations:
cpus: '1.0'
memory: 2G
```
### 3. Logging
```yaml
services:
mcp-server:
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
labels: "service=mcp"
# Or use syslog
logging:
driver: "syslog"
options:
syslog-address: "udp://192.168.1.100:514"
```
### 4. Metrics
```bash
# Docker stats
docker stats skillseekers-mcp
# cAdvisor for metrics
docker run -d \
--name cadvisor \
-p 8080:8080 \
-v /:/rootfs:ro \
-v /var/run:/var/run:ro \
-v /sys:/sys:ro \
-v /var/lib/docker:/var/lib/docker:ro \
gcr.io/cadvisor/cadvisor:latest
```
## Troubleshooting
### Common Issues
#### 1. Container Won't Start
```bash
# Check logs
docker logs skillseekers-mcp
# Inspect container
docker inspect skillseekers-mcp
# Run with interactive shell
docker run -it --entrypoint /bin/bash skillseekers:latest
```
#### 2. Port Already in Use
```bash
# Find process using port
sudo lsof -i :8765
# Kill process
kill -9 <PID>
# Or use different port
docker run -p 8766:8765 skillseekers:latest
```
#### 3. Volume Permission Issues
```bash
# Run as specific user
docker run --user $(id -u):$(id -g) skillseekers:latest
# Fix permissions
docker run --rm \
-v skillseekers-data:/data \
alpine chown -R 1000:1000 /data
```
#### 4. Network Connectivity
```bash
# Test connectivity
docker exec skillseekers-mcp ping google.com
# Check DNS
docker exec skillseekers-mcp cat /etc/resolv.conf
# Use custom DNS
docker run --dns 8.8.8.8 skillseekers:latest
```
#### 5. High Memory Usage
```bash
# Set memory limit
docker run --memory=4g skillseekers:latest
# Check memory usage
docker stats skillseekers-mcp
# Enable memory swappiness
docker run --memory=4g --memory-swap=8g skillseekers:latest
```
### Debug Commands
```bash
# Enter running container
docker exec -it skillseekers-mcp /bin/bash
# View environment variables
docker exec skillseekers-mcp env
# Check processes
docker exec skillseekers-mcp ps aux
# View logs in real-time
docker logs -f --tail 100 skillseekers-mcp
# Inspect container details
docker inspect skillseekers-mcp | jq '.[]'
# Export container filesystem
docker export skillseekers-mcp > container.tar
```
## Production Best Practices
### 1. Image Management
```bash
# Tag images with versions
docker build -t skillseekers:2.9.0 .
docker tag skillseekers:2.9.0 skillseekers:latest
# Use private registry
docker tag skillseekers:latest registry.example.com/skillseekers:latest
docker push registry.example.com/skillseekers:latest
# Scan for vulnerabilities
docker scan skillseekers:latest
```
### 2. Security
```bash
# Run as non-root user
RUN useradd -m -s /bin/bash skillseekers
USER skillseekers
# Read-only root filesystem
docker run --read-only --tmpfs /tmp skillseekers:latest
# Drop capabilities
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE skillseekers:latest
# Use security scanning
trivy image skillseekers:latest
```
### 3. Resource Management
```yaml
services:
mcp-server:
# CPU limits
cpus: 2.0
cpu_shares: 1024
# Memory limits
mem_limit: 4g
memswap_limit: 8g
mem_reservation: 2g
# Process limits
pids_limit: 200
```
### 4. Backup & Recovery
```bash
# Backup script
#!/bin/bash
docker-compose down
tar czf backup-$(date +%Y%m%d).tar.gz volumes/
docker-compose up -d
# Automated backups
0 2 * * * /opt/skillseekers/backup.sh
```
## Next Steps
- See [KUBERNETES_DEPLOYMENT.md](./KUBERNETES_DEPLOYMENT.md) for Kubernetes deployment
- Review [PRODUCTION_DEPLOYMENT.md](./PRODUCTION_DEPLOYMENT.md) for general production guidelines
- Check [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common issues
---
**Need help?** Open an issue on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers/issues).
+863
View File
@@ -0,0 +1,863 @@
# Frequently Asked Questions (FAQ)
**Version:** 3.6.0
**Last Updated:** 2026-03-15
---
## General Questions
### What is Skill Seekers?
Skill Seekers is a Python tool that converts 18 source types — documentation websites, GitHub repos, PDFs, videos, Word docs, EPUB books, Jupyter notebooks, local HTML files, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence wikis, Notion pages, Slack/Discord exports, and local codebases — into AI-ready formats for 21+ platforms: LLM platforms (Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown), RAG frameworks (LangChain, LlamaIndex, Haystack), vector databases (ChromaDB, FAISS, Weaviate, Qdrant, Pinecone), and AI coding assistants (Cursor, Windsurf, Cline, Continue.dev, Roo, Aider, Bolt, Kilo, Kimi Code).
**Use Cases:**
- Create custom documentation skills for your favorite frameworks
- Analyze GitHub repositories and extract code patterns
- Convert PDF manuals into searchable AI skills
- Import knowledge from Confluence, Notion, or Slack/Discord
- Extract content from videos (YouTube, Vimeo, local files)
- Convert Jupyter notebooks, EPUB books, or PowerPoint slides into skills
- Parse OpenAPI/Swagger specs into API reference skills
- Combine multiple sources (docs + code + PDFs + more) into unified skills
### Which platforms are supported?
**Supported Platforms (30+):**
*LLM Platforms (12):*
1. **Claude AI** - ZIP format with YAML frontmatter
2. **Google Gemini** - tar.gz format for Grounded Generation
3. **OpenAI ChatGPT** - ZIP format for Vector Stores
4. **MiniMax** - ZIP format
5. **OpenCode** - ZIP format
6. **Kimi** - ZIP format
7. **DeepSeek** - ZIP format
8. **Qwen** - ZIP format
9. **OpenRouter** - ZIP format for multi-model routing
10. **Together AI** - ZIP format for open-source models
11. **Fireworks AI** - ZIP format for fast inference
12. **Generic Markdown** - ZIP format with markdown files
*RAG Frameworks:*
13. **LangChain** - Document objects for QA chains and agents
14. **LlamaIndex** - TextNodes for query engines
15. **Haystack** - Document objects for enterprise RAG
*Vector Databases:*
16. **ChromaDB** - Direct collection upload
17. **FAISS** - Index files for local similarity search
18. **Weaviate** - Vector objects with schema creation
19. **Qdrant** - Points with payload indexing
20. **Pinecone** - Ready-to-upsert format
*AI Coding Assistants (9):*
21. **Cursor** - .cursorrules persistent context
22. **Windsurf** - .windsurfrules AI coding rules
23. **Cline** - .clinerules + MCP integration
24. **Continue.dev** - HTTP context server (all IDEs)
25. **Roo** - .roorules AI coding rules
26. **Aider** - Terminal AI coding assistant
27. **Bolt** - Web IDE AI context
28. **Kilo** - IDE AI context
29. **Kimi Code** - IDE AI context
Each platform has a dedicated adaptor for optimal formatting and upload.
### Is it free to use?
**Tool:** Yes, Skill Seekers is 100% free and open-source (MIT license).
**API Costs:**
- **Scraping:** Free (just bandwidth)
- **AI Enhancement (API mode):** ~$0.15-0.30 per skill (Claude API)
- **AI Enhancement (LOCAL mode):** Free! (uses your Claude Code Max plan)
- **Upload:** Free (platform storage limits apply)
**Recommendation:** Use LOCAL mode for free AI enhancement or skip enhancement entirely.
### How do I set up video extraction?
**Quick setup:**
```bash
# 1. Install video support
pip install skill-seekers[video-full]
# 2. Auto-detect GPU and install visual deps
skill-seekers create --setup
```
The `--setup` command auto-detects your GPU vendor (NVIDIA CUDA, AMD ROCm, or CPU-only) and installs the correct PyTorch variant along with easyocr and other visual extraction dependencies. This avoids the ~2GB NVIDIA CUDA download that would happen if easyocr were installed via pip on non-NVIDIA systems.
**What it detects:**
- **NVIDIA:** Uses `nvidia-smi` to find CUDA version → installs matching `cu124`/`cu121`/`cu118` PyTorch
- **AMD:** Uses `rocminfo` to find ROCm version → installs matching ROCm PyTorch
- **CPU-only:** Installs lightweight CPU-only PyTorch
### What source types are supported?
Skill Seekers supports **18 source types**:
| # | Source Type | CLI Command | Auto-Detection |
|---|------------|-------------|----------------|
| 1 | Documentation (web) | `scrape` / `create <url>` | HTTP/HTTPS URLs |
| 2 | GitHub repo | `github` / `create owner/repo` | `owner/repo` or github.com URLs |
| 3 | PDF | `pdf` / `create file.pdf` | `.pdf` extension |
| 4 | Word (.docx) | `word` / `create file.docx` | `.docx` extension |
| 5 | EPUB | `epub` / `create file.epub` | `.epub` extension |
| 6 | Video | `video` / `create <url/file>` | YouTube/Vimeo URLs, video extensions |
| 7 | Local codebase | `analyze` / `create ./path` | Directory paths |
| 8 | Jupyter Notebook | `jupyter` / `create file.ipynb` | `.ipynb` extension |
| 9 | Local HTML | `html` / `create file.html` | `.html`/`.htm` extensions |
| 10 | OpenAPI/Swagger | `openapi` / `create spec.yaml` | `.yaml`/`.yml` with OpenAPI content |
| 11 | AsciiDoc | `asciidoc` / `create file.adoc` | `.adoc`/`.asciidoc` extensions |
| 12 | PowerPoint | `pptx` / `create file.pptx` | `.pptx` extension |
| 13 | RSS/Atom | `rss` / `create feed.rss` | `.rss`/`.atom` extensions |
| 14 | Man pages | `manpage` / `create cmd.1` | `.1`-`.8`/`.man` extensions |
| 15 | Confluence | `confluence` | API or export directory |
| 16 | Notion | `notion` | API or export directory |
| 17 | Slack/Discord | `chat` | Export directory or API |
The `create` command auto-detects the source type from your input, so you often don't need to specify a subcommand.
### How long does it take to create a skill?
**Typical Times:**
- Documentation scraping: 5-45 minutes (depends on size)
- GitHub analysis: 1-5 minutes (basic) or 20-60 minutes (C3.x deep analysis)
- PDF extraction: 30 seconds - 5 minutes
- Video extraction: 2-10 minutes (depends on length and visual analysis)
- Word/EPUB/PPTX: 10-60 seconds
- Jupyter notebook: 10-30 seconds
- OpenAPI spec: 5-15 seconds
- Confluence/Notion import: 1-5 minutes (depends on space size)
- AI enhancement: 30-60 seconds (LOCAL or API mode)
- Total workflow: 10-60 minutes
**Speed Tips:**
- Use `--async` for 2-3x faster scraping
- Use `--skip-scrape` to rebuild without re-scraping
- Skip AI enhancement for faster workflow
---
## Installation & Setup
### How do I install Skill Seekers?
```bash
# Basic installation
pip install skill-seekers
# With all platform support
pip install skill-seekers[all-llms]
# Development installation
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git
cd Skill_Seekers
pip install -e ".[all-llms,dev]"
```
### What Python version do I need?
**Required:** Python 3.10 or higher
**Tested on:** Python 3.10, 3.11, 3.12, 3.13
**OS Support:** Linux, macOS, Windows (WSL recommended)
**Check your version:**
```bash
python --version # Should be 3.10+
```
### Why do I get "No module named 'skill_seekers'" error?
**Common Causes:**
1. Package not installed
2. Wrong Python environment
**Solutions:**
```bash
# Install package
pip install skill-seekers
# Or for development
pip install -e .
# Verify installation
skill-seekers --version
```
### How do I set up API keys?
```bash
# Claude AI (for enhancement and upload)
export ANTHROPIC_API_KEY=sk-ant-...
# Google Gemini (for upload)
export GOOGLE_API_KEY=AIza...
# OpenAI ChatGPT (for upload)
export OPENAI_API_KEY=sk-...
# GitHub (for higher rate limits)
export GITHUB_TOKEN=ghp_...
# Make permanent (add to ~/.bashrc or ~/.zshrc)
echo 'export ANTHROPIC_API_KEY=sk-ant-...' >> ~/.bashrc
```
---
## Usage Questions
### How do I scrape documentation?
**Using preset config:**
```bash
skill-seekers create --config react
```
**Using custom URL:**
```bash
skill-seekers create https://docs.example.com --name my-framework
```
**From custom config file:**
```bash
skill-seekers create --config configs/my-framework.json
```
### How do I create skills for an entire project at once?
Use `skill-seekers scan` — an AI agent inspects the project's manifests, README, Dockerfile/CI, and source imports, then emits one config per detected framework plus a `<project>-codebase.json` for your own code:
```bash
# AI-detect frameworks and emit configs
skill-seekers scan ./my-react-app --out ./configs/scanned/
# → react.json, typescript.json, vite.json, my-react-app-codebase.json
# Then build any of them
skill-seekers create ./configs/scanned/react.json
```
Re-running `scan` against the same output directory reports diffs (added / version-bumped / removed) so you can keep skills in sync with your real dependencies. See [CLI Reference: scan](reference/CLI_REFERENCE.md#scan) for all flags.
### Can I analyze GitHub repositories?
Yes! Skill Seekers has powerful GitHub analysis:
```bash
# Basic analysis (fast)
skill-seekers create https://github.com/facebook/react
# Deep C3.x analysis (includes patterns, tests, guides)
skill-seekers create https://github.com/vercel/next.js --depth full
```
**C3.x Features:**
- Design pattern detection (10 GoF patterns)
- Test example extraction
- How-to guide generation
- Configuration pattern extraction
- Architectural overview
- API reference generation
### Can I extract content from PDFs?
Yes! PDF extraction with OCR support:
```bash
# Basic PDF extraction
skill-seekers create --pdf manual.pdf --name product-manual
# With OCR (for scanned PDFs)
skill-seekers create --pdf scanned.pdf --ocr
# Images and tables are extracted automatically
skill-seekers create --pdf document.pdf
```
### How do I scrape a Jupyter Notebook?
```bash
# Extract cells, outputs, and markdown from a notebook
skill-seekers create analysis.ipynb --name data-analysis
# Or use auto-detection
skill-seekers create analysis.ipynb
```
Jupyter extraction preserves code cells, markdown cells, and cell outputs. It works with `.ipynb` files from JupyterLab, Google Colab, and other notebook environments.
### How do I import from Confluence or Notion?
**Confluence:**
```bash
# From Confluence Cloud API
export CONFLUENCE_URL=https://yourorg.atlassian.net
export CONFLUENCE_TOKEN=your-api-token
export CONFLUENCE_EMAIL=your-email@example.com
skill-seekers create --space-key MYSPACE --name my-wiki
# From a Confluence HTML/XML export directory
skill-seekers create --conf-export-path ./confluence-export --name my-wiki
```
**Notion:**
```bash
# From Notion API
export NOTION_TOKEN=secret_...
skill-seekers create --database-id DATABASE_ID --name my-notes
# From a Notion HTML/Markdown export directory
skill-seekers create --notion-export-path ./notion-export --name my-notes
```
### How do I convert Word, EPUB, or PowerPoint files?
```bash
# Word document
skill-seekers create report.docx --name quarterly-report
# EPUB book
skill-seekers create handbook.epub --name dev-handbook
# PowerPoint presentation
skill-seekers create slides.pptx --name training-deck
# Or use auto-detection for any of them
skill-seekers create report.docx
skill-seekers create handbook.epub
skill-seekers create slides.pptx
```
### How do I parse an OpenAPI/Swagger spec?
```bash
# From a local YAML/JSON file
skill-seekers create api-spec.yaml --name my-api
# Auto-detection works too
skill-seekers create api-spec.yaml
```
OpenAPI extraction parses endpoints, schemas, parameters, and examples into a structured API reference skill.
### How do I extract content from RSS feeds or man pages?
```bash
# RSS/Atom feed
skill-seekers create https://blog.example.com/feed.xml --name blog-feed
# Man page
skill-seekers create grep.1 --name grep-manual
```
### How do I import from Slack or Discord?
```bash
# From a Slack export directory
skill-seekers create --chat-export-path ./slack-export --platform slack --name team-knowledge
# From a Discord export directory
skill-seekers create --chat-export-path ./discord-export --platform discord --name server-archive
```
### Can I combine multiple sources?
Yes! Unified multi-source scraping:
**Create unified config** (`configs/unified/my-framework.json`):
```json
{
"name": "my-framework",
"sources": {
"documentation": {
"type": "docs",
"base_url": "https://docs.example.com"
},
"github": {
"type": "github",
"repo_url": "https://github.com/org/repo"
},
"pdf": {
"type": "pdf",
"pdf_path": "manual.pdf"
}
}
}
```
**Run unified scraping:**
```bash
skill-seekers create --config configs/unified/my-framework.json
```
### How do I upload skills to platforms?
```bash
# Upload to Claude AI
export ANTHROPIC_API_KEY=sk-ant-...
skill-seekers upload output/react-claude.zip --target claude
# Upload to Google Gemini
export GOOGLE_API_KEY=AIza...
skill-seekers upload output/react-gemini.tar.gz --target gemini
# Upload to OpenAI ChatGPT
export OPENAI_API_KEY=sk-...
skill-seekers upload output/react-openai.zip --target openai
```
**Or use complete workflow (uploads by default):**
```bash
skill-seekers install --config react --target claude
```
---
## Platform-Specific Questions
### What's the difference between platforms?
| Feature | Claude AI | Google Gemini | OpenAI ChatGPT | Markdown |
|---------|-----------|---------------|----------------|----------|
| Format | ZIP + YAML | tar.gz | ZIP | ZIP |
| Upload API | Projects API | Corpora API | Vector Stores | N/A |
| Model | Sonnet 4.5 | Gemini 2.0 Flash | GPT-4o | N/A |
| Max Size | 32MB | 10MB | 512MB | N/A |
| Use Case | Claude Code | Grounded Gen | ChatGPT Custom | Export |
**Choose based on:**
- Claude AI: Best for Claude Code integration
- Google Gemini: Best for Grounded Generation in Gemini
- OpenAI ChatGPT: Best for ChatGPT Custom GPTs
- MiniMax/Kimi/DeepSeek/Qwen: Best for Chinese LLM ecosystem
- OpenRouter/Together/Fireworks: Best for multi-model routing or open-source model access
- Markdown: Generic export for other tools
### Can I use multiple platforms at once?
Yes! Package and upload to all platforms:
```bash
# Package for all platforms
for platform in claude gemini openai minimax kimi deepseek qwen openrouter together fireworks markdown; do
skill-seekers package output/react/ --target $platform
done
# Install + upload per platform (one --target per run)
for platform in claude gemini openai; do
skill-seekers install --config react --target $platform
done
```
### How do I use skills in Claude Code?
1. **Install skill to Claude Code directory:**
```bash
skill-seekers install-agent output/react/ --agent claude
```
2. **Use in Claude Code:**
```
Use the react skill to explain React hooks
```
3. **Or upload to Claude AI:**
```bash
skill-seekers upload output/react-claude.zip --target claude
```
---
## Features & Capabilities
### What is AI enhancement?
AI enhancement transforms basic skills (2-3/10 quality) into production-ready skills (8-9/10 quality) using LLMs.
**Two Modes (via AgentClient):**
1. **API Mode:** Multi-provider AI API calls -- Anthropic, Moonshot/Kimi, Gemini, OpenAI (fast, costs ~$0.15-0.30)
2. **LOCAL Mode:** Any supported coding agent -- Claude Code, Kimi, Codex, Copilot, OpenCode, or custom (free with agent subscription)
**What it improves:**
- Better organization and structure
- Clearer explanations
- More examples and use cases
- Better cross-references
- Improved searchability
**Usage:**
```bash
# API mode (if ANTHROPIC_API_KEY is set)
skill-seekers enhance output/react/
# LOCAL mode (free! — auto-selected when no API key is set; pick the agent)
skill-seekers enhance output/react/ --agent claude
# Background mode
skill-seekers enhance output/react/ --background
skill-seekers enhance-status output/react/ --watch
```
### What are C3.x features?
C3.x features are advanced codebase analysis capabilities:
- **C3.1:** Design pattern detection (Singleton, Factory, Strategy, etc.)
- **C3.2:** Test example extraction (real usage examples from tests)
- **C3.3:** How-to guide generation (educational guides from test workflows)
- **C3.4:** Configuration pattern extraction (env vars, config files)
- **C3.5:** Architectural overview (system architecture analysis)
- **C3.6:** AI enhancement (Claude API integration for insights)
- **C3.7:** Architectural pattern detection (MVC, MVVM, Repository, etc.)
- **C3.8:** Standalone codebase scraping (300+ line SKILL.md from code alone)
**Enable C3.x:**
```bash
# All C3.x features enabled by default
skill-seekers create --directory /path/to/repo
# Skip specific features
skill-seekers create --directory . --skip-patterns --skip-how-to-guides
```
### What are router skills?
Router skills help Claude navigate large documentation (>500 pages) by providing a table of contents and keyword index.
**When to use:**
- Documentation with 500+ pages
- Complex multi-section docs
- Large API references
**Generate router (from split sub-skill configs):**
```bash
python -m skill_seekers.cli.generate_router configs/godot_*.json --name godot
```
(Also available as the `generate_router` MCP tool.)
### What preset configurations are available?
**24 preset configs:**
- Web: react, vue, angular, svelte, nextjs
- Python: django, flask, fastapi, sqlalchemy, pytest
- Game Dev: godot, pygame, unity
- DevOps: docker, kubernetes, terraform, ansible
- Unified: react-unified, vue-unified, nextjs-unified, etc.
**List all:**
```bash
skill-seekers estimate --all
```
---
## Troubleshooting
### Scraping is very slow, how can I speed it up?
**Solutions:**
1. **Use async mode** (2-3x faster):
```bash
skill-seekers create --config react --async
```
2. **Increase rate limit** (faster requests):
```json
{
"rate_limit": 0.1 // Faster (but may hit rate limits)
}
```
3. **Limit pages**:
```json
{
"max_pages": 100 // Stop after 100 pages
}
```
### Why are some pages missing?
**Common Causes:**
1. **URL patterns exclude them**
2. **Max pages limit reached**
3. **BFS didn't reach them**
**Solutions:**
```bash
# Check URL patterns in config
{
"url_patterns": {
"include": ["/docs/"], // Make sure your pages match
"exclude": [] // Remove overly broad exclusions
}
}
# Increase max pages
{
"max_pages": 1000 // Default is 500
}
# Use verbose mode to see what's being scraped
skill-seekers create --config react --verbose
```
### How do I fix "NetworkError: Connection failed"?
**Solutions:**
1. **Check internet connection**
2. **Verify URL is accessible**:
```bash
curl -I https://docs.example.com
```
3. **Increase timeout**:
```json
{
"timeout": 30 // 30 seconds
}
```
4. **Check rate limiting**:
```json
{
"rate_limit": 1.0 // Slower requests
}
```
### Tests are failing, what should I do?
**Quick fixes:**
```bash
# Ensure package is installed
pip install -e ".[all-llms,dev]"
# Clear caches
rm -rf .pytest_cache/ **/__pycache__/
# Run specific failing test
pytest tests/test_file.py::test_name -vv
# Check for missing dependencies
pip install -e ".[all-llms,dev]"
```
**If still failing:**
1. Check [Troubleshooting Guide](../TROUBLESHOOTING.md)
2. Report issue on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers/issues)
---
## MCP Server Questions
### How do I start the MCP server?
```bash
# stdio mode (Claude Code, VS Code + Cline)
skill-seekers-mcp
# HTTP mode (Cursor, Windsurf, IntelliJ)
skill-seekers-mcp --transport http --port 8765
```
### What MCP tools are available?
**40 MCP tools:**
*Core Tools (9):*
1. `list_configs` - List preset configurations
2. `generate_config` - Generate config from docs URL
3. `validate_config` - Validate config structure
4. `estimate_pages` - Estimate page count
5. `scrape_docs` - Scrape documentation
6. `package_skill` - Package to .zip (supports `--format` and `--target`)
7. `upload_skill` - Upload to platform (supports `--target`)
8. `enhance_skill` - AI enhancement
9. `install_skill` - Complete workflow
*Extended Tools (10):*
10. `scrape_github` - GitHub analysis
11. `scrape_pdf` - PDF extraction
12. `unified_scrape` - Multi-source scraping
13. `merge_sources` - Merge docs + code
14. `detect_conflicts` - Find discrepancies
15. `split_config` - Split large configs
16. `generate_router` - Generate router skills
17. `add_config_source` - Register git repos
18. `fetch_config` - Fetch configs from git
19. `list_config_sources` - List registered sources
20. `remove_config_source` - Remove config source
*Vector DB Tools (4):*
21. `export_to_chroma` - Export to ChromaDB
22. `export_to_weaviate` - Export to Weaviate
23. `export_to_faiss` - Export to FAISS
24. `export_to_qdrant` - Export to Qdrant
*Cloud Tools (3):*
25. `cloud_upload` - Upload to S3/GCS/Azure
26. `cloud_download` - Download from cloud storage
### How do I configure MCP for Claude Code?
**Add to `claude_desktop_config.json`:**
```json
{
"mcpServers": {
"skill-seekers": {
"command": "skill-seekers-mcp"
}
}
}
```
**Restart Claude Code**, then use:
```
Use skill-seekers MCP tools to scrape React documentation
```
---
## Advanced Questions
### Can I use Skill Seekers programmatically?
Yes! Full API for Python integration:
```python
from skill_seekers.cli.doc_scraper import scrape_all, build_skill
from skill_seekers.cli.adaptors import get_adaptor
# Scrape documentation
pages = scrape_all(
base_url='https://docs.example.com',
selectors={'main_content': 'article'},
config={'name': 'example'}
)
# Build skill
skill_path = build_skill(
config_name='example',
output_dir='output/example'
)
# Package for platform
adaptor = get_adaptor('claude')
package_path = adaptor.package(skill_path, 'output/')
```
**See:** [API Reference](reference/API_REFERENCE.md)
### How do I create custom configurations?
**Create config file** (`configs/my-framework.json`):
```json
{
"name": "my-framework",
"description": "My custom framework documentation",
"base_url": "https://docs.example.com/",
"selectors": {
"main_content": "article", // CSS selector
"title": "h1",
"code_blocks": "pre code"
},
"url_patterns": {
"include": ["/docs/", "/api/"],
"exclude": ["/blog/", "/changelog/"]
},
"categories": {
"getting_started": ["intro", "quickstart"],
"api": ["api", "reference"]
},
"rate_limit": 0.5,
"max_pages": 500
}
```
**Use config:**
```bash
skill-seekers create --config configs/my-framework.json
```
### Can I contribute preset configs?
Yes! We welcome config contributions:
1. **Create config** in `configs/` directory
2. **Test it** thoroughly:
```bash
skill-seekers create --config configs/your-framework.json
```
3. **Submit PR** on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers)
**Guidelines:**
- Name: `{framework-name}.json`
- Include all required fields
- Add to appropriate category
- Test with real documentation
### How do I debug scraping issues?
```bash
# Verbose output
skill-seekers create --config react --verbose
# Dry run (no actual scraping)
skill-seekers create --config react --dry-run
# Single page test
skill-seekers create https://docs.example.com/intro --max-pages 1
# Check config (also available as the validate_config MCP tool)
python -m skill_seekers.cli.config_validator configs/react.json
```
---
## Getting More Help
### Where can I find documentation?
**Main Documentation:**
- [README](../README.md) - Project overview
- [Usage Guide](guides/USAGE.md) - Detailed usage
- [API Reference](reference/API_REFERENCE.md) - Programmatic usage
- [Troubleshooting](../TROUBLESHOOTING.md) - Common issues
**Guides:**
- [MCP Setup](guides/MCP_SETUP.md)
- [Testing Guide](guides/TESTING_GUIDE.md)
- [Migration Guide](guides/MIGRATION_GUIDE.md)
- [Quick Reference](QUICK_REFERENCE.md)
### How do I report bugs?
1. **Check existing issues:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
2. **Create new issue** with:
- Skill Seekers version (`skill-seekers --version`)
- Python version (`python --version`)
- Operating system
- Config file (if relevant)
- Error message and stack trace
- Steps to reproduce
### How do I request features?
1. **Check roadmap:** [ROADMAP.md](../ROADMAP.md)
2. **Create feature request:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
3. **Join discussions:** https://github.com/yusufkaraaslan/Skill_Seekers/discussions
### Is there a community?
Yes!
- **GitHub Discussions:** https://github.com/yusufkaraaslan/Skill_Seekers/discussions
- **Issue Tracker:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2
---
**Version:** 3.6.0
**Last Updated:** 2026-03-15
**Questions? Ask on [GitHub Discussions](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)**
+933
View File
@@ -0,0 +1,933 @@
# Kubernetes Deployment Guide
Complete guide for deploying Skill Seekers on Kubernetes.
## Table of Contents
- [Prerequisites](#prerequisites)
- [Quick Start with Helm](#quick-start-with-helm)
- [Manual Deployment](#manual-deployment)
- [Configuration](#configuration)
- [Scaling](#scaling)
- [High Availability](#high-availability)
- [Monitoring](#monitoring)
- [Ingress & Load Balancing](#ingress--load-balancing)
- [Storage](#storage)
- [Security](#security)
- [Troubleshooting](#troubleshooting)
## Prerequisites
### 1. Kubernetes Cluster
**Minimum requirements:**
- Kubernetes v1.21+
- kubectl configured
- 2 nodes (minimum)
- 4 CPU cores total
- 8 GB RAM total
**Cloud providers:**
- **AWS:** EKS (Elastic Kubernetes Service)
- **GCP:** GKE (Google Kubernetes Engine)
- **Azure:** AKS (Azure Kubernetes Service)
- **Local:** Minikube, kind, k3s
### 2. Required Tools
```bash
# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# Verify installations
kubectl version --client
helm version
```
### 3. Cluster Access
```bash
# Verify cluster connection
kubectl cluster-info
kubectl get nodes
# Create namespace
kubectl create namespace skillseekers
kubectl config set-context --current --namespace=skillseekers
```
## Quick Start with Helm
### 1. Install with Default Values
```bash
# Add Helm repository (when available)
helm repo add skillseekers https://charts.skillseekers.io
helm repo update
# Install release
helm install skillseekers skillseekers/skillseekers \
--namespace skillseekers \
--create-namespace
# Or install from local chart
helm install skillseekers ./helm/skillseekers \
--namespace skillseekers \
--create-namespace
```
### 2. Install with Custom Values
```bash
# Create values file
cat > values-prod.yaml <<EOF
replicaCount: 3
secrets:
anthropicApiKey: "sk-ant-..."
githubToken: "ghp_..."
openaiApiKey: "sk-..."
resources:
limits:
cpu: 2000m
memory: 4Gi
requests:
cpu: 1000m
memory: 2Gi
ingress:
enabled: true
className: nginx
hosts:
- host: api.skillseekers.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: skillseekers-tls
hosts:
- api.skillseekers.example.com
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 70
EOF
# Install with custom values
helm install skillseekers ./helm/skillseekers \
--namespace skillseekers \
--create-namespace \
--values values-prod.yaml
```
### 3. Helm Commands
```bash
# List releases
helm list -n skillseekers
# Get status
helm status skillseekers -n skillseekers
# Upgrade release
helm upgrade skillseekers ./helm/skillseekers \
--namespace skillseekers \
--values values-prod.yaml
# Rollback
helm rollback skillseekers 1 -n skillseekers
# Uninstall
helm uninstall skillseekers -n skillseekers
```
## Manual Deployment
### 1. Secrets
Create secrets for API keys:
```yaml
# secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: skillseekers-secrets
namespace: skillseekers
type: Opaque
stringData:
ANTHROPIC_API_KEY: "sk-ant-..."
GITHUB_TOKEN: "ghp_..."
OPENAI_API_KEY: "sk-..."
VOYAGE_API_KEY: "..."
```
```bash
kubectl apply -f secrets.yaml
```
### 2. ConfigMap
```yaml
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: skillseekers-config
namespace: skillseekers
data:
MCP_TRANSPORT: "http"
MCP_PORT: "8765"
LOG_LEVEL: "INFO"
CACHE_TTL: "86400"
```
```bash
kubectl apply -f configmap.yaml
```
### 3. Deployment
```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: skillseekers-mcp
namespace: skillseekers
labels:
app: skillseekers
component: mcp-server
spec:
replicas: 3
selector:
matchLabels:
app: skillseekers
component: mcp-server
template:
metadata:
labels:
app: skillseekers
component: mcp-server
spec:
containers:
- name: mcp-server
image: skillseekers:2.9.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8765
name: http
protocol: TCP
env:
- name: MCP_TRANSPORT
valueFrom:
configMapKeyRef:
name: skillseekers-config
key: MCP_TRANSPORT
- name: MCP_PORT
valueFrom:
configMapKeyRef:
name: skillseekers-config
key: MCP_PORT
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: skillseekers-secrets
key: ANTHROPIC_API_KEY
- name: GITHUB_TOKEN
valueFrom:
secretKeyRef:
name: skillseekers-secrets
key: GITHUB_TOKEN
resources:
requests:
cpu: 1000m
memory: 2Gi
limits:
cpu: 2000m
memory: 4Gi
livenessProbe:
httpGet:
path: /health
port: 8765
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: 8765
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
volumeMounts:
- name: data
mountPath: /app/data
- name: cache
mountPath: /app/cache
volumes:
- name: data
persistentVolumeClaim:
claimName: skillseekers-data
- name: cache
emptyDir: {}
```
```bash
kubectl apply -f deployment.yaml
```
### 4. Service
```yaml
# service.yaml
apiVersion: v1
kind: Service
metadata:
name: skillseekers-mcp
namespace: skillseekers
labels:
app: skillseekers
component: mcp-server
spec:
type: ClusterIP
ports:
- port: 8765
targetPort: 8765
protocol: TCP
name: http
selector:
app: skillseekers
component: mcp-server
```
```bash
kubectl apply -f service.yaml
```
### 5. Verify Deployment
```bash
# Check pods
kubectl get pods -n skillseekers
# Check services
kubectl get svc -n skillseekers
# Check logs
kubectl logs -n skillseekers -l app=skillseekers --tail=100 -f
# Port forward for testing
kubectl port-forward -n skillseekers svc/skillseekers-mcp 8765:8765
# Test endpoint
curl http://localhost:8765/health
```
## Configuration
### 1. Resource Requests & Limits
```yaml
resources:
requests:
cpu: 500m # Guaranteed CPU
memory: 1Gi # Guaranteed memory
limits:
cpu: 2000m # Maximum CPU
memory: 4Gi # Maximum memory
```
### 2. Environment Variables
```yaml
env:
# From ConfigMap
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: skillseekers-config
key: LOG_LEVEL
# From Secret
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: skillseekers-secrets
key: ANTHROPIC_API_KEY
# Direct value
- name: MCP_TRANSPORT
value: "http"
```
### 3. Multi-Environment Setup
```bash
# Development
helm install skillseekers-dev ./helm/skillseekers \
--namespace skillseekers-dev \
--values values-dev.yaml
# Staging
helm install skillseekers-staging ./helm/skillseekers \
--namespace skillseekers-staging \
--values values-staging.yaml
# Production
helm install skillseekers-prod ./helm/skillseekers \
--namespace skillseekers-prod \
--values values-prod.yaml
```
## Scaling
### 1. Manual Scaling
```bash
# Scale deployment
kubectl scale deployment skillseekers-mcp -n skillseekers --replicas=5
# Verify
kubectl get pods -n skillseekers
```
### 2. Horizontal Pod Autoscaler (HPA)
```yaml
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: skillseekers-mcp
namespace: skillseekers
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: skillseekers-mcp
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 60
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
- type: Pods
value: 2
periodSeconds: 15
selectPolicy: Max
```
```bash
kubectl apply -f hpa.yaml
# Monitor autoscaling
kubectl get hpa -n skillseekers --watch
```
### 3. Vertical Pod Autoscaler (VPA)
```yaml
# vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: skillseekers-mcp
namespace: skillseekers
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: skillseekers-mcp
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: mcp-server
minAllowed:
cpu: 500m
memory: 1Gi
maxAllowed:
cpu: 4000m
memory: 8Gi
```
## High Availability
### 1. Pod Disruption Budget
```yaml
# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: skillseekers-mcp
namespace: skillseekers
spec:
minAvailable: 2
selector:
matchLabels:
app: skillseekers
component: mcp-server
```
### 2. Pod Anti-Affinity
```yaml
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- skillseekers
topologyKey: kubernetes.io/hostname
```
### 3. Node Affinity
```yaml
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role
operator: In
values:
- worker
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: node-type
operator: In
values:
- high-cpu
```
### 4. Multi-Zone Deployment
```yaml
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: skillseekers
```
## Monitoring
### 1. Prometheus Metrics
```yaml
# servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: skillseekers-mcp
namespace: skillseekers
spec:
selector:
matchLabels:
app: skillseekers
endpoints:
- port: metrics
interval: 30s
path: /metrics
```
### 2. Grafana Dashboard
```bash
# Import dashboard
kubectl apply -f grafana/dashboard.json
```
### 3. Logging with Fluentd
```yaml
# fluentd-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
data:
fluent.conf: |
<source>
@type tail
path /var/log/containers/skillseekers*.log
pos_file /var/log/fluentd-skillseekers.pos
tag kubernetes.*
format json
</source>
<match **>
@type elasticsearch
host elasticsearch
port 9200
</match>
```
## Ingress & Load Balancing
### 1. Nginx Ingress
```yaml
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: skillseekers
namespace: skillseekers
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rate-limit: "100"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- api.skillseekers.example.com
secretName: skillseekers-tls
rules:
- host: api.skillseekers.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: skillseekers-mcp
port:
number: 8765
```
### 2. TLS with cert-manager
```bash
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
# Create ClusterIssuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
EOF
```
## Storage
### 1. Persistent Volume
```yaml
# pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: skillseekers-data
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
hostPath:
path: /mnt/skillseekers-data
```
### 2. Persistent Volume Claim
```yaml
# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: skillseekers-data
namespace: skillseekers
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: standard
```
### 3. StatefulSet (for stateful workloads)
```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: skillseekers-cache
spec:
serviceName: skillseekers-cache
replicas: 3
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
```
## Security
### 1. Network Policies
```yaml
# networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: skillseekers-mcp
namespace: skillseekers
spec:
podSelector:
matchLabels:
app: skillseekers
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: skillseekers
ports:
- protocol: TCP
port: 8765
egress:
- to:
- namespaceSelector: {}
ports:
- protocol: TCP
port: 443 # HTTPS
- protocol: TCP
port: 80 # HTTP
```
### 2. Pod Security Policy
```yaml
# psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: skillseekers-restricted
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'persistentVolumeClaim'
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
```
### 3. RBAC
```yaml
# rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: skillseekers
namespace: skillseekers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: skillseekers
namespace: skillseekers
rules:
- apiGroups: [""]
resources: ["configmaps", "secrets"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: skillseekers
namespace: skillseekers
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: skillseekers
subjects:
- kind: ServiceAccount
name: skillseekers
namespace: skillseekers
```
## Troubleshooting
### Common Issues
#### 1. Pods Not Starting
```bash
# Check pod status
kubectl get pods -n skillseekers
# Describe pod
kubectl describe pod <pod-name> -n skillseekers
# Check events
kubectl get events -n skillseekers --sort-by='.lastTimestamp'
# Check logs
kubectl logs <pod-name> -n skillseekers
```
#### 2. Image Pull Errors
```bash
# Check image pull secrets
kubectl get secrets -n skillseekers
# Create image pull secret
kubectl create secret docker-registry regcred \
--docker-server=registry.example.com \
--docker-username=user \
--docker-password=password \
-n skillseekers
# Use in pod spec
spec:
imagePullSecrets:
- name: regcred
```
#### 3. Resource Constraints
```bash
# Check node resources
kubectl top nodes
# Check pod resources
kubectl top pods -n skillseekers
# Increase resources
kubectl edit deployment skillseekers-mcp -n skillseekers
```
#### 4. Service Not Accessible
```bash
# Check service
kubectl get svc -n skillseekers
kubectl describe svc skillseekers-mcp -n skillseekers
# Check endpoints
kubectl get endpoints -n skillseekers
# Port forward
kubectl port-forward svc/skillseekers-mcp 8765:8765 -n skillseekers
```
### Debug Commands
```bash
# Execute command in pod
kubectl exec -it <pod-name> -n skillseekers -- /bin/bash
# Copy files from pod
kubectl cp skillseekers/<pod-name>:/app/data ./data
# Check pod networking
kubectl exec <pod-name> -n skillseekers -- nslookup google.com
# View full pod spec
kubectl get pod <pod-name> -n skillseekers -o yaml
# Restart deployment
kubectl rollout restart deployment skillseekers-mcp -n skillseekers
```
## Best Practices
1. **Always set resource requests and limits**
2. **Use namespaces for environment separation**
3. **Enable autoscaling for variable workloads**
4. **Implement health checks (liveness & readiness)**
5. **Use Secrets for sensitive data**
6. **Enable monitoring and logging**
7. **Implement Pod Disruption Budgets for HA**
8. **Use RBAC for access control**
9. **Enable Network Policies**
10. **Regular backup of persistent volumes**
## Next Steps
- Review [PRODUCTION_DEPLOYMENT.md](./PRODUCTION_DEPLOYMENT.md) for general guidelines
- See [DOCKER_DEPLOYMENT.md](./DOCKER_DEPLOYMENT.md) for container-specific details
- Check [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common issues
---
**Need help?** Open an issue on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers/issues).
+827
View File
@@ -0,0 +1,827 @@
# Production Deployment Guide
Complete guide for deploying Skill Seekers in production environments.
## Table of Contents
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Configuration](#configuration)
- [Deployment Options](#deployment-options)
- [Monitoring & Observability](#monitoring--observability)
- [Security](#security)
- [Scaling](#scaling)
- [Backup & Disaster Recovery](#backup--disaster-recovery)
- [Troubleshooting](#troubleshooting)
## Prerequisites
### System Requirements
**Minimum:**
- CPU: 2 cores
- RAM: 4 GB
- Disk: 10 GB
- Python: 3.10+
**Recommended (for production):**
- CPU: 4+ cores
- RAM: 8+ GB
- Disk: 50+ GB SSD
- Python: 3.12+
### Dependencies
**Required:**
```bash
# System packages (Ubuntu/Debian)
sudo apt update
sudo apt install -y python3.12 python3.12-venv python3-pip \
git curl wget build-essential libssl-dev
# System packages (RHEL/CentOS)
sudo yum install -y python312 python312-devel git curl wget \
gcc gcc-c++ openssl-devel
```
**Optional (for specific features):**
```bash
# OCR support (PDF scraping)
sudo apt install -y tesseract-ocr
# Cloud storage
# (Install provider-specific SDKs via pip)
# Embedding generation
# (GPU support requires CUDA)
```
## Installation
### 1. Production Installation
```bash
# Create dedicated user
sudo useradd -m -s /bin/bash skillseekers
sudo su - skillseekers
# Create virtual environment
python3.12 -m venv /opt/skillseekers/venv
source /opt/skillseekers/venv/bin/activate
# Install package
pip install --upgrade pip
pip install skill-seekers[all]
# Verify installation
skill-seekers --version
```
### 2. Configuration Directory
```bash
# Create config directory
mkdir -p ~/.config/skill-seekers/{configs,output,logs,cache}
# Set permissions
chmod 700 ~/.config/skill-seekers
```
### 3. Environment Variables
Create `/opt/skillseekers/.env`:
```bash
# API Keys
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIza...
OPENAI_API_KEY=sk-...
VOYAGE_API_KEY=...
# GitHub Tokens (use skill-seekers config --github for multiple)
GITHUB_TOKEN=ghp_...
# Cloud Storage (optional)
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs-key.json
AZURE_STORAGE_CONNECTION_STRING=...
# MCP Server
MCP_TRANSPORT=http
MCP_PORT=8765
# Sync Monitoring (optional)
SYNC_WEBHOOK_URL=https://...
SLACK_WEBHOOK_URL=https://hooks.slack.com/...
# Logging
LOG_LEVEL=INFO
LOG_FILE=/var/log/skillseekers/app.log
```
**Security Note:** Never commit `.env` files to version control!
```bash
# Secure the env file
chmod 600 /opt/skillseekers/.env
```
## Configuration
### 1. GitHub Configuration
Use the interactive configuration wizard:
```bash
skill-seekers config --github
```
This will:
- Add GitHub personal access tokens
- Configure rate limit strategies
- Test token validity
- Support multiple profiles (work, personal, etc.)
### 2. API Keys Configuration
```bash
skill-seekers config --api-keys
```
Configure:
- Claude API (Anthropic)
- Gemini API (Google)
- OpenAI API
- Voyage AI (embeddings)
### 3. Connection Testing
```bash
skill-seekers config --test
```
Verifies:
- ✅ GitHub token(s) validity and rate limits
- ✅ Claude API connectivity
- ✅ Gemini API connectivity
- ✅ OpenAI API connectivity
- ✅ Cloud storage access (if configured)
## Deployment Options
### Option 1: Systemd Service (Recommended)
Create `/etc/systemd/system/skillseekers-mcp.service`:
```ini
[Unit]
Description=Skill Seekers MCP Server
After=network.target
[Service]
Type=simple
User=skillseekers
Group=skillseekers
WorkingDirectory=/opt/skillseekers
EnvironmentFile=/opt/skillseekers/.env
ExecStart=/opt/skillseekers/venv/bin/python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=skillseekers-mcp
# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/skillseekers /var/log/skillseekers
[Install]
WantedBy=multi-user.target
```
**Enable and start:**
```bash
sudo systemctl daemon-reload
sudo systemctl enable skillseekers-mcp
sudo systemctl start skillseekers-mcp
sudo systemctl status skillseekers-mcp
```
### Option 2: Docker Deployment
See [Docker Deployment Guide](./DOCKER_DEPLOYMENT.md) for detailed instructions.
**Quick Start:**
```bash
# Build image
docker build -t skillseekers:latest .
# Run container
docker run -d \
--name skillseekers-mcp \
-p 8765:8765 \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-e GITHUB_TOKEN=$GITHUB_TOKEN \
-v /opt/skillseekers/data:/app/data \
--restart unless-stopped \
skillseekers:latest
```
### Option 3: Kubernetes Deployment
See [Kubernetes Deployment Guide](./KUBERNETES_DEPLOYMENT.md) for detailed instructions.
**Quick Start:**
```bash
# Install with Helm
helm install skillseekers ./helm/skillseekers \
--namespace skillseekers \
--create-namespace \
--set secrets.anthropicApiKey=$ANTHROPIC_API_KEY \
--set secrets.githubToken=$GITHUB_TOKEN
```
### Option 4: Docker Compose
See [Docker Compose Guide](./DOCKER_COMPOSE.md) for multi-service deployment.
```bash
# Start all services
docker-compose up -d
# Check status
docker-compose ps
# View logs
docker-compose logs -f
```
## Monitoring & Observability
### 1. Health Checks
**MCP Server Health:**
```bash
# HTTP transport
curl http://localhost:8765/health
# Expected response:
{
"status": "healthy",
"version": "2.9.0",
"uptime": 3600,
"tools": 25
}
```
### 2. Logging
**Configure structured logging:**
```python
# config/logging.yaml
version: 1
formatters:
json:
format: '{"time":"%(asctime)s","level":"%(levelname)s","msg":"%(message)s"}'
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: /var/log/skillseekers/app.log
maxBytes: 10485760 # 10MB
backupCount: 5
formatter: json
loggers:
skill_seekers:
level: INFO
handlers: [file]
```
**Log aggregation options:**
- **ELK Stack:** Elasticsearch + Logstash + Kibana
- **Grafana Loki:** Lightweight log aggregation
- **CloudWatch Logs:** For AWS deployments
- **Stackdriver:** For GCP deployments
### 3. Metrics
**Prometheus metrics endpoint:**
```bash
# Add to MCP server
from prometheus_client import start_http_server, Counter, Histogram
# Metrics
scraping_requests = Counter('scraping_requests_total', 'Total scraping requests')
scraping_duration = Histogram('scraping_duration_seconds', 'Scraping duration')
# Start metrics server
start_http_server(9090)
```
**Key metrics to monitor:**
- Request rate
- Response time (p50, p95, p99)
- Error rate
- Memory usage
- CPU usage
- Disk I/O
- GitHub API rate limit remaining
- Claude API token usage
### 4. Alerting
**Example Prometheus alert rules:**
```yaml
groups:
- name: skillseekers
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 5m
annotations:
summary: "High error rate detected"
- alert: HighMemoryUsage
expr: process_resident_memory_bytes > 2e9 # 2GB
for: 10m
annotations:
summary: "Memory usage above 2GB"
- alert: GitHubRateLimitLow
expr: github_rate_limit_remaining < 100
for: 1m
annotations:
summary: "GitHub rate limit low"
```
## Security
### 1. API Key Management
**Best Practices:**
**DO:**
- Store keys in environment variables or secret managers
- Use different keys for dev/staging/prod
- Rotate keys regularly (quarterly minimum)
- Use least-privilege IAM roles for cloud services
- Monitor key usage for anomalies
**DON'T:**
- Commit keys to version control
- Share keys via email/Slack
- Use production keys in development
- Grant overly broad permissions
**Recommended Secret Managers:**
- **Kubernetes Secrets** (for K8s deployments)
- **AWS Secrets Manager** (for AWS)
- **Google Secret Manager** (for GCP)
- **Azure Key Vault** (for Azure)
- **HashiCorp Vault** (cloud-agnostic)
### 2. Network Security
**Firewall Rules:**
```bash
# Allow only necessary ports
sudo ufw enable
sudo ufw allow 22/tcp # SSH
sudo ufw allow 8765/tcp # MCP server (if public)
sudo ufw deny incoming
sudo ufw allow outgoing
```
**Reverse Proxy (Nginx):**
```nginx
# /etc/nginx/sites-available/skillseekers
server {
listen 80;
server_name api.skillseekers.example.com;
# Redirect to HTTPS
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name api.skillseekers.example.com;
ssl_certificate /etc/letsencrypt/live/api.skillseekers.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/api.skillseekers.example.com/privkey.pem;
# Security headers
add_header Strict-Transport-Security "max-age=31536000" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
# Rate limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
limit_req zone=api burst=20 nodelay;
location / {
proxy_pass http://localhost:8765;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
}
}
```
### 3. TLS/SSL
**Let's Encrypt (free certificates):**
```bash
# Install certbot
sudo apt install certbot python3-certbot-nginx
# Obtain certificate
sudo certbot --nginx -d api.skillseekers.example.com
# Auto-renewal (cron)
0 12 * * * /usr/bin/certbot renew --quiet
```
### 4. Authentication & Authorization
**API Key Authentication (optional):**
```python
# Add to MCP server
from fastapi import Security, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
security = HTTPBearer()
async def verify_token(credentials: HTTPAuthorizationCredentials = Security(security)):
token = credentials.credentials
if token != os.getenv("API_SECRET_KEY"):
raise HTTPException(status_code=401, detail="Invalid token")
return token
```
## Scaling
### 1. Vertical Scaling
**Increase resources:**
```yaml
# Kubernetes resource limits
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "4"
memory: "8Gi"
```
### 2. Horizontal Scaling
**Deploy multiple instances:**
```bash
# Kubernetes HPA (Horizontal Pod Autoscaler)
kubectl autoscale deployment skillseekers-mcp \
--cpu-percent=70 \
--min=2 \
--max=10
```
**Load Balancing:**
```nginx
# Nginx load balancer
upstream skillseekers {
least_conn;
server 10.0.0.1:8765;
server 10.0.0.2:8765;
server 10.0.0.3:8765;
}
server {
listen 80;
location / {
proxy_pass http://skillseekers;
}
}
```
### 3. Database/Storage Scaling
**Distributed caching:**
```python
# Redis for distributed cache
import redis
cache = redis.Redis(host='redis.example.com', port=6379, db=0)
```
**Object storage:**
- Use S3/GCS/Azure Blob for skill packages
- Enable CDN for static assets
- Use read replicas for databases
### 4. Rate Limit Management
**Multiple GitHub tokens:**
```bash
# Configure multiple profiles
skill-seekers config --github
# Automatic token rotation on rate limit
# (handled by rate_limit_handler.py)
```
## Backup & Disaster Recovery
### 1. Data Backup
**What to backup:**
- Configuration files (`~/.config/skill-seekers/`)
- Generated skills (`output/`)
- Database/cache (if applicable)
- Logs (for forensics)
**Backup script:**
```bash
#!/bin/bash
# /opt/skillseekers/scripts/backup.sh
BACKUP_DIR="/backups/skillseekers"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
# Create backup
tar -czf "$BACKUP_DIR/backup_$TIMESTAMP.tar.gz" \
~/.config/skill-seekers \
/opt/skillseekers/output \
/opt/skillseekers/.env
# Retain last 30 days
find "$BACKUP_DIR" -name "backup_*.tar.gz" -mtime +30 -delete
# Upload to S3 (optional)
aws s3 cp "$BACKUP_DIR/backup_$TIMESTAMP.tar.gz" \
s3://backups/skillseekers/
```
**Schedule backups:**
```bash
# Crontab
0 2 * * * /opt/skillseekers/scripts/backup.sh
```
### 2. Disaster Recovery Plan
**Recovery steps:**
1. **Provision new infrastructure**
```bash
# Deploy from backup
terraform apply
```
2. **Restore configuration**
```bash
tar -xzf backup_20250207.tar.gz -C /
```
3. **Verify services**
```bash
skill-seekers config --test
systemctl status skillseekers-mcp
```
4. **Test functionality**
```bash
skill-seekers create --config configs/test.json --max-pages 10
```
**RTO/RPO targets:**
- **RTO (Recovery Time Objective):** < 2 hours
- **RPO (Recovery Point Objective):** < 24 hours
## Troubleshooting
### Common Issues
#### 1. High Memory Usage
**Symptoms:**
- OOM kills
- Slow performance
- Swapping
**Solutions:**
```bash
# Check memory usage
ps aux --sort=-%mem | head -10
# Reduce parallel workers
skill-seekers create --config config.json --workers 2
# Enable memory limits
docker run --memory=4g skillseekers:latest
```
#### 2. GitHub Rate Limits
**Symptoms:**
- `403 Forbidden` errors
- "API rate limit exceeded" messages
**Solutions:**
```bash
# Check rate limit
curl -H "Authorization: token $GITHUB_TOKEN" \
https://api.github.com/rate_limit
# Add more tokens
skill-seekers config --github
# Use rate limit strategy
# (automatic with multi-token config)
```
#### 3. Slow Scraping
**Symptoms:**
- Long scraping times
- Timeouts
**Solutions:**
```bash
# Enable async scraping (2-3x faster)
skill-seekers create --config config.json --async
# Increase concurrency
# (adjust in config: "concurrency": 10)
# Re-runs reuse cached data automatically (use --fresh to discard)
skill-seekers create --config config.json
```
#### 4. API Errors
**Symptoms:**
- `401 Unauthorized`
- `429 Too Many Requests`
**Solutions:**
```bash
# Verify API keys
skill-seekers config --test
# Check API key validity
# Claude API: https://console.anthropic.com/
# OpenAI: https://platform.openai.com/api-keys
# Google: https://console.cloud.google.com/apis/credentials
# Rotate keys if compromised
```
#### 5. Service Won't Start
**Symptoms:**
- systemd service fails
- Container exits immediately
**Solutions:**
```bash
# Check logs
journalctl -u skillseekers-mcp -n 100
# Or for Docker
docker logs skillseekers-mcp
# Common causes:
# - Missing environment variables
# - Port already in use
# - Permission issues
# Verify config
skill-seekers config --show
```
### Debug Mode
Enable detailed logging:
```bash
# Set debug level
export LOG_LEVEL=DEBUG
# Run with verbose output
skill-seekers create --config config.json --verbose
```
### Getting Help
**Community Support:**
- GitHub Issues: https://github.com/yusufkaraaslan/Skill_Seekers/issues
- Documentation: https://skillseekersweb.com/
**Log Collection:**
```bash
# Collect diagnostic info
tar -czf skillseekers-debug.tar.gz \
/var/log/skillseekers/ \
~/.config/skill-seekers/configs/ \
/opt/skillseekers/.env
```
## Performance Tuning
### 1. Scraping Performance
**Optimization techniques:**
```python
# Enable async scraping
"async_scraping": true,
"concurrency": 20, # Adjust based on resources
# Optimize selectors
"selectors": {
"main_content": "article", # More specific = faster
"code_blocks": "pre code"
}
# Enable caching
"use_cache": true,
"cache_ttl": 86400 # 24 hours
```
### 2. Embedding Performance
**GPU acceleration (if available):**
```python
# Use GPU for sentence-transformers
pip install sentence-transformers[gpu]
# Configure
export CUDA_VISIBLE_DEVICES=0
```
**Batch processing:**
```python
# Generate embeddings in batches
generator.generate_batch(texts, batch_size=32)
```
### 3. Storage Performance
**Use SSD for:**
- SQLite databases
- Cache directories
- Log files
**Use object storage for:**
- Skill packages
- Backup archives
- Large datasets
## Next Steps
1. **Review** deployment option that fits your infrastructure
2. **Configure** monitoring and alerting
3. **Set up** backups and disaster recovery
4. **Test** failover procedures
5. **Document** your specific deployment
6. **Train** your team on operations
---
**Need help?** See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) or open an issue on GitHub.
+231
View File
@@ -0,0 +1,231 @@
# Skill Seekers Documentation
> **Complete documentation for Skill Seekers v3.7.0**
---
## Welcome!
This is the official documentation for **Skill Seekers** - the universal tool for converting **18 source types** (documentation sites, GitHub repos, PDFs, videos, Word docs, EPUB books, Jupyter notebooks, local HTML, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence, Notion, Slack/Discord, and local codebases) into AI-ready skills for 21+ platforms.
---
## Where Should I Start?
### 🚀 I'm New Here
Start with our **Getting Started** guides:
1. [Installation](getting-started/01-installation.md) - Install Skill Seekers
2. [Quick Start](getting-started/02-quick-start.md) - Create your first skill in 3 commands
3. [Your First Skill](getting-started/03-your-first-skill.md) - Complete walkthrough
4. [Next Steps](getting-started/04-next-steps.md) - Where to go from here
5. [Scan a Project](getting-started/05-scan-a-project.md) - Bootstrap configs from a codebase
### 📖 I Want to Learn
Explore our **User Guides**:
- [Core Concepts](user-guide/01-core-concepts.md) - How Skill Seekers works
- [Scraping Guide](user-guide/02-scraping.md) - All scraping options
- [Enhancement Guide](user-guide/03-enhancement.md) - AI enhancement explained
- [Packaging Guide](user-guide/04-packaging.md) - Export to platforms
- [Workflows Guide](user-guide/05-workflows.md) - Enhancement workflows
- [Troubleshooting](user-guide/06-troubleshooting.md) - Common issues
### 📚 I Need Reference
Look up specific information:
- [CLI Reference](reference/CLI_REFERENCE.md) - All 19 commands
- [MCP Reference](reference/MCP_REFERENCE.md) - 40 MCP tools
- [Config Format](reference/CONFIG_FORMAT.md) - JSON specification
- [Environment Variables](reference/ENVIRONMENT_VARIABLES.md) - All env vars
- [FAQ](FAQ.md) - Frequently asked questions
- [Troubleshooting](TROUBLESHOOTING.md) - Full troubleshooting reference
### 🚀 I'm Ready for Advanced Topics
Power user features:
- [MCP Server Setup](advanced/mcp-server.md) - MCP integration
- [MCP Tools Deep Dive](advanced/mcp-server.md) - Advanced MCP usage
- [Custom Workflows](advanced/custom-workflows.md) - Create workflows
- [Multi-Source Scraping](advanced/multi-source.md) - Combine sources
---
## Quick Reference
### The 3 Commands
```bash
# 1. Install
pip install skill-seekers
# 2. Create skill
skill-seekers create https://docs.django.com/
# 3. Package for Claude
skill-seekers package output/django --target claude
```
### Common Commands
```bash
# Create from any source (auto-detects type)
skill-seekers create https://docs.django.com/
skill-seekers create facebook/react
skill-seekers create manual.pdf
skill-seekers create notebook.ipynb
# Scan a project for tech stack — emits one config per framework
skill-seekers scan ./my-react-app --out ./configs/scanned/
# Enhance skill
skill-seekers enhance output/my-skill/
# Package for platform
skill-seekers package output/my-skill/ --target claude
# Upload
skill-seekers upload output/my-skill-claude.zip
# Install complete workflow
skill-seekers install --config react --target claude
# Doctor / diagnostics
skill-seekers doctor
```
---
## Documentation Structure
```
docs/
├── README.md # This file - start here
├── ARCHITECTURE.md # How docs are organized
├── UML_ARCHITECTURE.md # Software architecture (UML diagrams)
├── UNIFICATION_PLAN.md # Grand Unification refactor plan + phase results
├── BUG_AUDIT.md # Full-codebase bug audit (historical record)
├── getting-started/ # For new users
│ ├── 01-installation.md
│ ├── 02-quick-start.md
│ ├── 03-your-first-skill.md
│ ├── 04-next-steps.md
│ └── 05-scan-a-project.md
├── user-guide/ # Common tasks
│ ├── 01-core-concepts.md
│ ├── 02-scraping.md
│ ├── 03-enhancement.md
│ ├── 04-packaging.md
│ ├── 05-workflows.md
│ └── 06-troubleshooting.md
├── guides/ # How-to guides
│ ├── MCP_SETUP.md
│ ├── MIGRATION_GUIDE.md
│ ├── TESTING_GUIDE.md
│ └── UPLOAD_GUIDE.md
├── integrations/ # Platform integrations
│ ├── LANGCHAIN.md
│ ├── LLAMA_INDEX.md
│ ├── CURSOR.md
│ └── ...
├── features/ # Feature deep-dives
│ ├── BOOTSTRAP_SKILL.md
│ ├── UNIFIED_SCRAPING.md
│ └── ENHANCEMENT.md
├── reference/ # Technical reference
│ ├── CLI_REFERENCE.md # 19 commands
│ ├── MCP_REFERENCE.md # 40 MCP tools
│ ├── CONFIG_FORMAT.md # JSON spec
│ └── ENVIRONMENT_VARIABLES.md
├── advanced/ # Power user topics
│ ├── mcp-server.md
│ ├── custom-workflows.md
│ └── multi-source.md
├── archive/ # Legacy docs
├── blog/ # Blog posts
├── case-studies/ # Case studies
├── plans/ # Feature plans
├── roadmap/ # Roadmap
├── strategy/ # Strategy docs
└── zh-CN/ # Chinese translations
```
---
## By Use Case
### I Want to Build AI Skills
For Claude, Gemini, ChatGPT:
1. [Quick Start](getting-started/02-quick-start.md)
2. [Enhancement Guide](user-guide/03-enhancement.md)
3. [Workflows Guide](user-guide/05-workflows.md)
### I Want to Build RAG Pipelines
For LangChain, LlamaIndex, vector DBs:
1. [Core Concepts](user-guide/01-core-concepts.md)
2. [Packaging Guide](user-guide/04-packaging.md)
3. [MCP Reference](reference/MCP_REFERENCE.md)
### I Want AI Coding Assistance
For Cursor, Windsurf, Cline, Roo, Aider, Bolt, Kilo, Continue, Kimi Code:
1. [Your First Skill](getting-started/03-your-first-skill.md)
2. [Local Codebase Analysis](user-guide/02-scraping.md#local-codebase-analysis)
3. `skill-seekers install-agent --agent cursor`
---
## Version Information
- **Current Version:** 3.7.0
- **Last Updated:** 2026-06-11
- **Source Types:** 18
- **Python Required:** 3.10+
---
## Contributing to Documentation
Found an issue? Want to improve docs?
1. Edit files in the `docs/` directory
2. Follow the existing structure
3. Submit a PR
See [Contributing Guide](../CONTRIBUTING.md) for details.
---
## External Links
- **Main Repository:** https://github.com/yusufkaraaslan/Skill_Seekers
- **Website:** https://skillseekersweb.com/
- **PyPI:** https://pypi.org/project/skill-seekers/
- **Issues:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
---
## License
MIT License - see [LICENSE](../LICENSE) file.
---
*Happy skill building! 🚀*
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 141 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 568 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 750 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 458 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 341 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 491 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 214 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 261 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 156 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 264 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

Some files were not shown because too many files have changed in this diff Show More