chore: import upstream snapshot with attribution
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers CLI - Convert documentation to AI skills dockerfile:Dockerfile name:skill-seekers]) (push) Waiting to run
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers MCP Server - 25 tools for AI assistants dockerfile:Dockerfile.mcp name:skill-seekers-mcp]) (push) Waiting to run
Docker Publish / Test Docker Images (push) Blocked by required conditions
Test Vector Database Adaptors / Test chroma Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test faiss Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test qdrant Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test weaviate Adaptor (push) Waiting to run
Test Vector Database Adaptors / Test MCP Vector DB Tools (push) Waiting to run
Tests / Code Quality (Ruff & Mypy) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (macos-latest, 3.11) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (macos-latest, 3.12) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.10) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.11) (push) Waiting to run
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.12) (push) Waiting to run
Tests / Tests (push) Blocked by required conditions
Tests / Serial / Integration / E2E Tests (push) Blocked by required conditions
Tests / MCP Server Tests (push) Blocked by required conditions
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"skill-seeker": {
|
||||
"type": "stdio",
|
||||
"command": "/path/to/your/Skill_Seekers/.venv/bin/python3",
|
||||
"args": [
|
||||
"-m",
|
||||
"skill_seekers.mcp.server_fastmcp"
|
||||
],
|
||||
"cwd": "/path/to/your/Skill_Seekers",
|
||||
"env": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "skill-seekers",
|
||||
"version": "3.5.0",
|
||||
"description": "Convert docs and repos into AI skills via MCP",
|
||||
"author": {
|
||||
"name": "yusufkaraaslan",
|
||||
"url": "https://github.com/yusufkaraaslan/Skill_Seekers"
|
||||
},
|
||||
"homepage": "https://skillseekersweb.com/",
|
||||
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
|
||||
"keywords": ["mcp", "codex", "skills", "documentation"],
|
||||
"mcpServers": "./.mcp.json",
|
||||
"skills": "./skills/",
|
||||
"interface": {
|
||||
"displayName": "Skill Seekers",
|
||||
"shortDescription": "Convert docs and repos into AI skills via MCP",
|
||||
"longDescription": "Convert documentation websites, GitHub repositories, and PDFs into AI skills with automatic conflict detection.",
|
||||
"category": "Development",
|
||||
"websiteURL": "https://skillseekersweb.com/"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
# Python artifacts
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
.venv
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
.coverage.*
|
||||
htmlcov/
|
||||
.tox/
|
||||
.hypothesis/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitattributes
|
||||
|
||||
# Documentation
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# CI/CD
|
||||
.github/
|
||||
.gitlab-ci.yml
|
||||
.travis.yml
|
||||
|
||||
# Output directories
|
||||
output/
|
||||
data/
|
||||
*.zip
|
||||
*.tar.gz
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
logs/
|
||||
|
||||
# Environment files
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# Test files
|
||||
tests/
|
||||
test_*.py
|
||||
*_test.py
|
||||
|
||||
# Docker
|
||||
Dockerfile*
|
||||
docker-compose*.yml
|
||||
.dockerignore
|
||||
@@ -0,0 +1,46 @@
|
||||
# Skill Seekers Docker Environment Configuration
|
||||
# Copy this file to .env and fill in your API keys
|
||||
|
||||
# Claude AI / Anthropic API
|
||||
# Required for AI enhancement features
|
||||
# Get your key from: https://console.anthropic.com/
|
||||
ANTHROPIC_API_KEY=sk-ant-your-key-here
|
||||
|
||||
# Google Gemini API (Optional)
|
||||
# Required for Gemini platform support
|
||||
# Get your key from: https://makersuite.google.com/app/apikey
|
||||
GOOGLE_API_KEY=
|
||||
|
||||
# OpenAI API (Optional)
|
||||
# Required for OpenAI/ChatGPT platform support
|
||||
# Get your key from: https://platform.openai.com/api-keys
|
||||
OPENAI_API_KEY=
|
||||
|
||||
# GitHub Token (Optional, but recommended)
|
||||
# Increases rate limits from 60/hour to 5000/hour
|
||||
# Create token at: https://github.com/settings/tokens
|
||||
# Required scopes: public_repo (for public repos)
|
||||
GITHUB_TOKEN=
|
||||
|
||||
# MCP Server Configuration
|
||||
MCP_TRANSPORT=http
|
||||
MCP_PORT=8765
|
||||
|
||||
# Docker Resource Limits (Optional)
|
||||
# Uncomment to set custom limits
|
||||
# DOCKER_CPU_LIMIT=2.0
|
||||
# DOCKER_MEMORY_LIMIT=4g
|
||||
|
||||
# Pinecone API (Optional)
|
||||
# Required for Pinecone vector database upload
|
||||
# Get your key from: https://app.pinecone.io/
|
||||
PINECONE_API_KEY=
|
||||
|
||||
# Vector Database Ports (Optional - change if needed)
|
||||
# WEAVIATE_PORT=8080
|
||||
# QDRANT_PORT=6333
|
||||
# CHROMA_PORT=8000
|
||||
|
||||
# Logging (Optional)
|
||||
# SKILL_SEEKERS_LOG_LEVEL=INFO
|
||||
# SKILL_SEEKERS_LOG_FILE=/data/logs/skill-seekers.log
|
||||
@@ -0,0 +1,4 @@
|
||||
# GitHub Sponsors configuration
|
||||
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository
|
||||
|
||||
buy_me_a_coffee: yusufkaraaslan
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
name: Bug Report
|
||||
about: Report a bug or issue with Skill Seekers
|
||||
title: '[BUG] '
|
||||
labels: 'type: bug'
|
||||
assignees: ''
|
||||
---
|
||||
|
||||
## 🐛 Bug Description
|
||||
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
## 🔄 Steps to Reproduce
|
||||
|
||||
1. Go to '...'
|
||||
2. Run command '...'
|
||||
3. See error
|
||||
|
||||
## ✅ Expected Behavior
|
||||
|
||||
What you expected to happen.
|
||||
|
||||
## ❌ Actual Behavior
|
||||
|
||||
What actually happened.
|
||||
|
||||
## 📋 Environment
|
||||
|
||||
- **OS:** [e.g., macOS 14.0, Ubuntu 22.04, Windows 11]
|
||||
- **Python Version:** [e.g., 3.10, 3.11]
|
||||
- **Skill Seekers Version:** [e.g., v1.0.0]
|
||||
- **Installation Method:** [pip, git clone, etc.]
|
||||
|
||||
## 📊 Error Output
|
||||
|
||||
```
|
||||
Paste the full error message or traceback here
|
||||
```
|
||||
|
||||
## 📸 Screenshots
|
||||
|
||||
If applicable, add screenshots to help explain the problem.
|
||||
|
||||
## 🔍 Additional Context
|
||||
|
||||
- Config file used (if applicable)
|
||||
- Documentation URL being scraped
|
||||
- Any custom modifications made
|
||||
|
||||
## 🎯 Possible Solution
|
||||
|
||||
If you have an idea of how to fix this, please share!
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: Documentation Improvement
|
||||
about: Suggest improvements to documentation
|
||||
title: '[DOCS] '
|
||||
labels: 'type: documentation'
|
||||
assignees: ''
|
||||
---
|
||||
|
||||
## 📚 Documentation Issue
|
||||
|
||||
What documentation needs to be improved, added, or fixed?
|
||||
|
||||
## 📍 Location
|
||||
|
||||
- **File:** [e.g., README.md, docs/CLAUDE.md]
|
||||
- **Section:** [e.g., Installation, Configuration]
|
||||
- **URL:** [if applicable]
|
||||
|
||||
## ❌ Current State
|
||||
|
||||
Describe what's currently unclear, missing, or incorrect.
|
||||
|
||||
## ✅ Proposed Improvement
|
||||
|
||||
How should the documentation be changed?
|
||||
|
||||
## 🎯 Target Audience
|
||||
|
||||
Who would benefit from this documentation improvement?
|
||||
- [ ] New users
|
||||
- [ ] Advanced users
|
||||
- [ ] Contributors
|
||||
- [ ] API users
|
||||
|
||||
## 📝 Additional Context
|
||||
|
||||
Any additional context or examples that would help.
|
||||
|
||||
## 🔗 Related Issues
|
||||
|
||||
Link to any related issues or PRs.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
name: Feature Request
|
||||
about: Suggest a new feature for Skill Seekers
|
||||
title: '[FEATURE] '
|
||||
labels: 'type: feature'
|
||||
assignees: ''
|
||||
---
|
||||
|
||||
## 🚀 Feature Description
|
||||
|
||||
A clear and concise description of the feature you'd like to see.
|
||||
|
||||
## 💡 Use Case
|
||||
|
||||
Describe the problem this feature would solve. What is the user trying to accomplish?
|
||||
|
||||
## 📋 Proposed Solution
|
||||
|
||||
Describe how you envision this feature working.
|
||||
|
||||
## 🔄 Alternatives Considered
|
||||
|
||||
Have you considered any alternative solutions or workarounds?
|
||||
|
||||
## 📊 Expected Impact
|
||||
|
||||
- **Priority:** Low / Medium / High / Critical
|
||||
- **Effort:** XS / S / M / L / XL
|
||||
- **Users Affected:** Describe who would benefit
|
||||
|
||||
## 📝 Additional Context
|
||||
|
||||
Add any other context, screenshots, or examples about the feature request.
|
||||
|
||||
## ✅ Acceptance Criteria
|
||||
|
||||
- [ ] Criteria 1
|
||||
- [ ] Criteria 2
|
||||
- [ ] Criteria 3
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: MCP Tool Request
|
||||
about: Suggest a new tool for the MCP server
|
||||
title: '[MCP] Add tool: '
|
||||
labels: mcp, enhancement
|
||||
assignees: ''
|
||||
---
|
||||
|
||||
## Tool Name
|
||||
<!-- e.g., auto_detect_selectors -->
|
||||
|
||||
## Tool Description
|
||||
<!-- What does this tool do? -->
|
||||
|
||||
## Input Parameters
|
||||
```json
|
||||
{
|
||||
"param1": {
|
||||
"type": "string",
|
||||
"description": "...",
|
||||
"required": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Expected Output
|
||||
<!-- What should the tool return? -->
|
||||
|
||||
## Use Case Example
|
||||
<!-- How would users interact with this tool? -->
|
||||
```
|
||||
User: "Auto-detect selectors for https://docs.example.com"
|
||||
Tool: Analyzes page structure and suggests optimal selectors
|
||||
```
|
||||
|
||||
## CLI Integration
|
||||
<!-- Which CLI tool does this wrap? Or is it new logic? -->
|
||||
- [ ] Wraps existing CLI tool: `cli/tool_name.py`
|
||||
- [ ] New functionality
|
||||
|
||||
## Implementation Notes
|
||||
<!-- Technical details, dependencies, etc. -->
|
||||
@@ -0,0 +1,49 @@
|
||||
# Pull Request
|
||||
|
||||
## 📋 Description
|
||||
|
||||
Brief description of changes made.
|
||||
|
||||
## 🔗 Related Issues
|
||||
|
||||
Closes #(issue number)
|
||||
Relates to #(issue number)
|
||||
|
||||
## 🎯 Type of Change
|
||||
|
||||
- [ ] 🐛 Bug fix (non-breaking change which fixes an issue)
|
||||
- [ ] ✨ New feature (non-breaking change which adds functionality)
|
||||
- [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
|
||||
- [ ] 📚 Documentation update
|
||||
- [ ] ♻️ Code refactoring
|
||||
- [ ] ⚡ Performance improvement
|
||||
- [ ] 🧪 Test update
|
||||
|
||||
## ✅ Checklist
|
||||
|
||||
- [ ] My code follows the style guidelines of this project
|
||||
- [ ] I have performed a self-review of my own code
|
||||
- [ ] I have commented my code, particularly in hard-to-understand areas
|
||||
- [ ] I have made corresponding changes to the documentation
|
||||
- [ ] Documentation updated (README, examples, AGENTS.md if CLI changed)
|
||||
- [ ] My changes generate no new warnings
|
||||
- [ ] I have added tests that prove my fix is effective or that my feature works
|
||||
- [ ] New and existing unit tests pass locally with my changes
|
||||
- [ ] Any dependent changes have been merged and published
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
Describe the tests you ran to verify your changes.
|
||||
|
||||
**Test Configuration:**
|
||||
- Python version:
|
||||
- OS:
|
||||
- Dependencies installed:
|
||||
|
||||
## 📸 Screenshots (if applicable)
|
||||
|
||||
Add screenshots to demonstrate visual changes.
|
||||
|
||||
## 📝 Additional Notes
|
||||
|
||||
Any additional information reviewers should know.
|
||||
@@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
# Script to create GitHub issues via web browser
|
||||
# Since gh CLI is not available, we'll open browser to create issues
|
||||
|
||||
REPO="yusufkaraaslan/Skill_Seekers"
|
||||
BASE_URL="https://github.com/${REPO}/issues/new"
|
||||
|
||||
echo "🚀 Creating GitHub Issues for Skill Seeker MCP Development"
|
||||
echo "=========================================================="
|
||||
echo ""
|
||||
echo "Opening browser to create issues..."
|
||||
echo "Please copy the content from .github/ISSUES_TO_CREATE.md"
|
||||
echo ""
|
||||
|
||||
# Issue 1: Fix test failures
|
||||
echo "📝 Issue 1: Fix 3 test failures"
|
||||
echo "URL: ${BASE_URL}?labels=bug,tests,good+first+issue&title=Fix+3+test+failures+(warnings+vs+errors+handling)"
|
||||
echo ""
|
||||
|
||||
# Issue 2: MCP setup guide
|
||||
echo "📝 Issue 2: Create MCP setup guide"
|
||||
echo "URL: ${BASE_URL}?labels=documentation,mcp,enhancement&title=Create+comprehensive+MCP+setup+guide+for+Claude+Code"
|
||||
echo ""
|
||||
|
||||
# Issue 3: Test MCP server
|
||||
echo "📝 Issue 3: Test MCP server"
|
||||
echo "URL: ${BASE_URL}?labels=testing,mcp,priority-high&title=Test+MCP+server+with+actual+Claude+Code+instance"
|
||||
echo ""
|
||||
|
||||
# Issue 4: Update documentation
|
||||
echo "📝 Issue 4: Update documentation"
|
||||
echo "URL: ${BASE_URL}?labels=documentation,breaking-change&title=Update+all+documentation+for+new+monorepo+structure"
|
||||
echo ""
|
||||
|
||||
echo "=========================================================="
|
||||
echo "📋 Instructions:"
|
||||
echo "1. Click each URL above (or copy to browser)"
|
||||
echo "2. Copy the issue body from .github/ISSUES_TO_CREATE.md"
|
||||
echo "3. Paste into the issue description"
|
||||
echo "4. Click 'Submit new issue'"
|
||||
echo ""
|
||||
echo "Or use this quick link to view all templates:"
|
||||
echo "cat .github/ISSUES_TO_CREATE.md"
|
||||
@@ -0,0 +1,139 @@
|
||||
# Docker Image Publishing - Automated builds and pushes to Docker Hub
|
||||
# Security Note: Uses secrets for Docker Hub credentials. Matrix values are hardcoded.
|
||||
# Triggers: push/pull_request/workflow_dispatch only. No untrusted input.
|
||||
|
||||
name: Docker Publish
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main ]
|
||||
tags:
|
||||
- 'v*'
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- 'Dockerfile*'
|
||||
- 'docker-compose.yml'
|
||||
- 'src/**'
|
||||
- 'pyproject.toml'
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
DOCKER_REGISTRY: docker.io
|
||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
name: Build and Push Docker Images
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
image:
|
||||
- name: skill-seekers
|
||||
dockerfile: Dockerfile
|
||||
description: "Skill Seekers CLI - Convert documentation to AI skills"
|
||||
- name: skill-seekers-mcp
|
||||
dockerfile: Dockerfile.mcp
|
||||
description: "Skill Seekers MCP Server - 25 tools for AI assistants"
|
||||
|
||||
env:
|
||||
IMAGE_NAME: ${{ matrix.image.name }}
|
||||
IMAGE_DOCKERFILE: ${{ matrix.image.dockerfile }}
|
||||
IMAGE_DESCRIPTION: ${{ matrix.image.description }}
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v2
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v2
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v4
|
||||
with:
|
||||
images: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_USERNAME }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=ref,event=pr
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}}
|
||||
type=semver,pattern={{major}}
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v4
|
||||
with:
|
||||
context: .
|
||||
file: ${{ env.IMAGE_DOCKERFILE }}
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
platforms: linux/amd64,linux/arm64
|
||||
|
||||
- name: Create image summary
|
||||
run: |
|
||||
echo "## 🐳 Docker Image: $IMAGE_NAME" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Description:** $IMAGE_DESCRIPTION" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Tags:**" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "${{ steps.meta.outputs.tags }}" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
test-images:
|
||||
name: Test Docker Images
|
||||
needs: build-and-push
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'pull_request'
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Build CLI image
|
||||
run: |
|
||||
docker build -t skill-seekers:test -f Dockerfile .
|
||||
|
||||
- name: Test CLI image
|
||||
run: |
|
||||
echo "🧪 Testing CLI image..."
|
||||
docker run --rm skill-seekers:test skill-seekers --version
|
||||
docker run --rm skill-seekers:test skill-seekers --help
|
||||
|
||||
- name: Build MCP image
|
||||
run: |
|
||||
docker build -t skill-seekers-mcp:test -f Dockerfile.mcp .
|
||||
|
||||
- name: Test MCP image
|
||||
run: |
|
||||
echo "🧪 Testing MCP server image..."
|
||||
# Start MCP server in background
|
||||
docker run -d --name mcp-test -p 8765:8765 skill-seekers-mcp:test
|
||||
|
||||
# Wait for server to start
|
||||
sleep 10
|
||||
|
||||
# Check health
|
||||
curl -f http://localhost:8765/health || exit 1
|
||||
|
||||
# Stop container
|
||||
docker stop mcp-test
|
||||
docker rm mcp-test
|
||||
|
||||
- name: Test Docker Compose
|
||||
run: |
|
||||
echo "🧪 Testing Docker Compose..."
|
||||
docker compose config
|
||||
echo "✅ Docker Compose configuration valid"
|
||||
@@ -0,0 +1,171 @@
|
||||
# Security Note: This workflow uses workflow_dispatch inputs and pull_request events.
|
||||
# All untrusted inputs are accessed via environment variables (env:) as recommended.
|
||||
# No direct usage of github.event.issue/comment/review content in run: commands.
|
||||
|
||||
name: Quality Metrics Dashboard
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
skill_dir:
|
||||
description: 'Path to skill directory to analyze (e.g., output/react)'
|
||||
required: true
|
||||
type: string
|
||||
fail_threshold:
|
||||
description: 'Minimum quality score to pass (default: 70)'
|
||||
required: false
|
||||
default: '70'
|
||||
type: string
|
||||
pull_request:
|
||||
paths:
|
||||
- 'output/**'
|
||||
- 'configs/**'
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: Quality Metrics Analysis
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
env:
|
||||
SKILL_DIR_INPUT: ${{ github.event.inputs.skill_dir }}
|
||||
FAIL_THRESHOLD_INPUT: ${{ github.event.inputs.fail_threshold }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -e .
|
||||
|
||||
- name: Find skill directories
|
||||
id: find_skills
|
||||
run: |
|
||||
if [ -n "$SKILL_DIR_INPUT" ]; then
|
||||
# Manual trigger with specific directory
|
||||
echo "dirs=$SKILL_DIR_INPUT" >> $GITHUB_OUTPUT
|
||||
else
|
||||
# PR trigger - find all skill directories
|
||||
DIRS=$(find output -maxdepth 1 -type d -name "*" ! -name "output" | tr '\n' ' ' || echo "")
|
||||
if [ -z "$DIRS" ]; then
|
||||
echo "No skill directories found"
|
||||
echo "dirs=" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "dirs=$DIRS" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
fi
|
||||
|
||||
- name: Analyze quality metrics
|
||||
id: quality
|
||||
run: |
|
||||
DIRS="${{ steps.find_skills.outputs.dirs }}"
|
||||
THRESHOLD="${FAIL_THRESHOLD_INPUT:-70}"
|
||||
|
||||
if [ -z "$DIRS" ]; then
|
||||
echo "No directories to analyze"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
ALL_PASSED=true
|
||||
SUMMARY_FILE="quality_summary.md"
|
||||
|
||||
echo "# 📊 Quality Metrics Dashboard" > $SUMMARY_FILE
|
||||
echo "" >> $SUMMARY_FILE
|
||||
echo "**Threshold:** $THRESHOLD/100" >> $SUMMARY_FILE
|
||||
echo "" >> $SUMMARY_FILE
|
||||
|
||||
for skill_dir in $DIRS; do
|
||||
if [ ! -d "$skill_dir" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
SKILL_NAME=$(basename "$skill_dir")
|
||||
echo "🔍 Analyzing $SKILL_NAME..."
|
||||
|
||||
# Run quality analysis
|
||||
python3 -c "
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from skill_seekers.cli.quality_metrics import QualityAnalyzer
|
||||
|
||||
skill_dir = Path('$skill_dir')
|
||||
threshold = float('$THRESHOLD')
|
||||
skill_name = '$SKILL_NAME'
|
||||
|
||||
analyzer = QualityAnalyzer(skill_dir)
|
||||
report = analyzer.generate_report()
|
||||
|
||||
formatted = analyzer.format_report(report)
|
||||
print(formatted)
|
||||
|
||||
with open(f'quality_{skill_name}.txt', 'w') as f:
|
||||
f.write(formatted)
|
||||
|
||||
score = report.overall_score.total_score
|
||||
grade = report.overall_score.grade
|
||||
status = 'PASS' if score >= threshold else 'FAIL'
|
||||
|
||||
summary_line = f'{status} **{skill_name}**: {grade} ({score:.1f}/100)'
|
||||
print(f'\n{summary_line}')
|
||||
|
||||
with open('quality_summary.md', 'a') as f:
|
||||
f.write(f'{summary_line}\n')
|
||||
|
||||
if score < threshold:
|
||||
print(f'::error file={skill_dir}/SKILL.md::Quality score {score:.1f} is below threshold {threshold}')
|
||||
sys.exit(1)
|
||||
elif score < 80:
|
||||
print(f'::warning file={skill_dir}/SKILL.md::Quality score {score:.1f} could be improved')
|
||||
else:
|
||||
print(f'::notice file={skill_dir}/SKILL.md::Quality score {score:.1f} - Excellent!')
|
||||
"
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
ALL_PASSED=false
|
||||
fi
|
||||
|
||||
echo "" >> $SUMMARY_FILE
|
||||
done
|
||||
|
||||
if [ "$ALL_PASSED" = false ]; then
|
||||
echo "❌ Some skills failed quality thresholds"
|
||||
exit 1
|
||||
else
|
||||
echo "✅ All skills passed quality thresholds"
|
||||
fi
|
||||
|
||||
- name: Upload quality reports
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: quality-metrics-reports
|
||||
path: quality_*.txt
|
||||
retention-days: 30
|
||||
continue-on-error: true
|
||||
|
||||
- name: Post summary to PR
|
||||
if: github.event_name == 'pull_request'
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const summary = fs.readFileSync('quality_summary.md', 'utf8');
|
||||
|
||||
github.rest.issues.createComment({
|
||||
issue_number: context.issue.number,
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
body: summary
|
||||
});
|
||||
continue-on-error: true
|
||||
|
||||
- name: Create dashboard summary
|
||||
run: |
|
||||
if [ -f "quality_summary.md" ]; then
|
||||
cat quality_summary.md >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
@@ -0,0 +1,125 @@
|
||||
name: Release
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: 'recursive'
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
if [ -f skill_seeker_mcp/requirements.txt ]; then pip install -r skill_seeker_mcp/requirements.txt; fi
|
||||
# Install package in editable mode for tests (required for src/ layout)
|
||||
pip install -e .
|
||||
|
||||
- name: Run tests
|
||||
timeout-minutes: 30
|
||||
run: |
|
||||
pip install pytest-timeout
|
||||
python -m pytest tests/ -v \
|
||||
-m "not slow and not integration and not e2e and not network and not serial and not mcp_only" \
|
||||
--timeout=120
|
||||
|
||||
- name: Extract version from tag
|
||||
id: get_version
|
||||
run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Verify Python version
|
||||
run: |
|
||||
python --version
|
||||
python -c "import sys; assert sys.version_info >= (3, 10), f'Python {sys.version} is not >= 3.10'"
|
||||
|
||||
- name: Verify version consistency
|
||||
run: |
|
||||
TAG_VERSION="${{ steps.get_version.outputs.VERSION }}"
|
||||
PKG_VERSION=$(python -c "import skill_seekers; print(skill_seekers.__version__)")
|
||||
TOML_VERSION=$(grep -m1 '^version' pyproject.toml | sed 's/version *= *"\(.*\)"/\1/')
|
||||
echo "Tag version: $TAG_VERSION"
|
||||
echo "Package version: $PKG_VERSION"
|
||||
echo "TOML version: $TOML_VERSION"
|
||||
if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
|
||||
echo "::error::Version mismatch! Tag=$TAG_VERSION but package reports=$PKG_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$TAG_VERSION" != "$TOML_VERSION" ]; then
|
||||
echo "::error::Version mismatch! Tag=$TAG_VERSION but pyproject.toml has=$TOML_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ All versions match: $TAG_VERSION"
|
||||
|
||||
- name: Create Release Notes
|
||||
id: release_notes
|
||||
run: |
|
||||
if [ -f CHANGELOG.md ]; then
|
||||
# Extract changelog for this version (escape dots for exact match)
|
||||
VERSION="${{ steps.get_version.outputs.VERSION }}"
|
||||
ESCAPED_VERSION=$(echo "$VERSION" | sed 's/\./\\./g')
|
||||
sed -n "/## \[${ESCAPED_VERSION}\]/,/## \[/p" CHANGELOG.md | sed '$d' > release_notes.md
|
||||
fi
|
||||
# Fallback if extraction produced empty file or CHANGELOG.md missing
|
||||
if [ ! -s release_notes.md ]; then
|
||||
echo "Release v${{ steps.get_version.outputs.VERSION }}" > release_notes.md
|
||||
fi
|
||||
|
||||
- name: Check if release exists
|
||||
id: check_release
|
||||
run: |
|
||||
if gh release view ${{ github.ref_name }} > /dev/null 2>&1; then
|
||||
echo "exists=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "exists=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Create GitHub Release
|
||||
if: steps.check_release.outputs.exists == 'false'
|
||||
uses: softprops/action-gh-release@v1
|
||||
with:
|
||||
name: v${{ steps.get_version.outputs.VERSION }}
|
||||
tag_name: ${{ github.ref_name }}
|
||||
body_path: release_notes.md
|
||||
draft: false
|
||||
prerelease: false
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Skip Release Creation
|
||||
if: steps.check_release.outputs.exists == 'true'
|
||||
run: |
|
||||
echo "ℹ️ Release ${{ github.ref_name }} already exists, skipping creation"
|
||||
echo "View at: https://github.com/${{ github.repository }}/releases/tag/${{ github.ref_name }}"
|
||||
|
||||
- name: Build package
|
||||
run: |
|
||||
uv build
|
||||
|
||||
- name: Publish to PyPI
|
||||
env:
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
uv publish --token $UV_PUBLISH_TOKEN
|
||||
@@ -0,0 +1,197 @@
|
||||
# Automated Skill Updates - Runs weekly to refresh documentation
|
||||
# Security Note: Schedule triggers with hardcoded constants. Workflow_dispatch input
|
||||
# accessed via FRAMEWORKS_INPUT env variable (safe pattern).
|
||||
|
||||
name: Scheduled Skill Updates
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Run every Sunday at 3 AM UTC
|
||||
- cron: '0 3 * * 0'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
frameworks:
|
||||
description: 'Frameworks to update (comma-separated or "all")'
|
||||
required: false
|
||||
default: 'all'
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
update-skills:
|
||||
name: Update ${{ matrix.framework }}
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# Popular frameworks to keep updated
|
||||
framework:
|
||||
- react
|
||||
- django
|
||||
- fastapi
|
||||
- godot
|
||||
- vue
|
||||
- flask
|
||||
|
||||
env:
|
||||
FRAMEWORK: ${{ matrix.framework }}
|
||||
FRAMEWORKS_INPUT: ${{ github.event.inputs.frameworks }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -e .
|
||||
|
||||
- name: Check if framework should be updated
|
||||
id: should_update
|
||||
run: |
|
||||
FRAMEWORKS_INPUT="${FRAMEWORKS_INPUT:-all}"
|
||||
|
||||
if [ "$FRAMEWORKS_INPUT" = "all" ] || [ -z "$FRAMEWORKS_INPUT" ]; then
|
||||
echo "update=true" >> $GITHUB_OUTPUT
|
||||
elif echo "$FRAMEWORKS_INPUT" | grep -q "$FRAMEWORK"; then
|
||||
echo "update=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "update=false" >> $GITHUB_OUTPUT
|
||||
echo "⏭️ Skipping $FRAMEWORK (not in update list)"
|
||||
fi
|
||||
|
||||
- name: Check for existing skill
|
||||
if: steps.should_update.outputs.update == 'true'
|
||||
id: check_existing
|
||||
run: |
|
||||
SKILL_DIR="output/$FRAMEWORK"
|
||||
if [ -d "$SKILL_DIR" ]; then
|
||||
echo "exists=true" >> $GITHUB_OUTPUT
|
||||
echo "📦 Found existing skill at $SKILL_DIR"
|
||||
else
|
||||
echo "exists=false" >> $GITHUB_OUTPUT
|
||||
echo "🆕 No existing skill found"
|
||||
fi
|
||||
|
||||
- name: Incremental update (if exists)
|
||||
if: steps.should_update.outputs.update == 'true' && steps.check_existing.outputs.exists == 'true'
|
||||
run: |
|
||||
echo "⚡ Performing incremental update for $FRAMEWORK..."
|
||||
|
||||
SKILL_DIR="output/$FRAMEWORK"
|
||||
|
||||
# Detect changes using incremental updater
|
||||
python3 -c "
|
||||
import sys, os
|
||||
from pathlib import Path
|
||||
|
||||
from skill_seekers.cli.incremental_updater import IncrementalUpdater
|
||||
|
||||
framework = os.environ['FRAMEWORK']
|
||||
skill_dir = Path(f'output/{framework}')
|
||||
|
||||
updater = IncrementalUpdater(skill_dir)
|
||||
changes = updater.detect_changes()
|
||||
|
||||
if changes.has_changes:
|
||||
print(f'Changes detected:')
|
||||
print(f' Added: {len(changes.added)}')
|
||||
print(f' Modified: {len(changes.modified)}')
|
||||
print(f' Deleted: {len(changes.deleted)}')
|
||||
updater.current_versions = updater._scan_documents()
|
||||
updater.save_current_versions()
|
||||
else:
|
||||
print('No changes detected, skill is up to date')
|
||||
"
|
||||
|
||||
- name: Full scrape (if new or manual)
|
||||
if: steps.should_update.outputs.update == 'true' && steps.check_existing.outputs.exists == 'false'
|
||||
run: |
|
||||
echo "📥 Performing full scrape for $FRAMEWORK..."
|
||||
|
||||
CONFIG_FILE="configs/${FRAMEWORK}.json"
|
||||
|
||||
if [ ! -f "$CONFIG_FILE" ]; then
|
||||
echo "⚠️ Config not found: $CONFIG_FILE"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Use streaming ingestion for large docs
|
||||
skill-seekers create "$CONFIG_FILE" --max-pages 200
|
||||
|
||||
- name: Generate quality report
|
||||
if: steps.should_update.outputs.update == 'true'
|
||||
run: |
|
||||
SKILL_DIR="output/$FRAMEWORK"
|
||||
|
||||
if [ ! -d "$SKILL_DIR" ]; then
|
||||
echo "⚠️ Skill directory not found"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "📊 Generating quality metrics..."
|
||||
|
||||
python3 -c "
|
||||
import sys, os
|
||||
from pathlib import Path
|
||||
|
||||
from skill_seekers.cli.quality_metrics import QualityAnalyzer
|
||||
|
||||
framework = os.environ['FRAMEWORK']
|
||||
skill_dir = Path(f'output/{framework}')
|
||||
|
||||
analyzer = QualityAnalyzer(skill_dir)
|
||||
report = analyzer.generate_report()
|
||||
|
||||
print(f'Quality Score: {report.overall_score.grade} ({report.overall_score.total_score:.1f}/100)')
|
||||
print(f' Completeness: {report.overall_score.completeness:.1f}%')
|
||||
print(f' Accuracy: {report.overall_score.accuracy:.1f}%')
|
||||
print(f' Coverage: {report.overall_score.coverage:.1f}%')
|
||||
print(f' Health: {report.overall_score.health:.1f}%')
|
||||
"
|
||||
|
||||
- name: Package for Claude
|
||||
if: steps.should_update.outputs.update == 'true'
|
||||
run: |
|
||||
SKILL_DIR="output/$FRAMEWORK"
|
||||
|
||||
if [ -d "$SKILL_DIR" ]; then
|
||||
echo "📦 Packaging $FRAMEWORK for Claude AI..."
|
||||
skill-seekers package "$SKILL_DIR" --target claude
|
||||
fi
|
||||
|
||||
- name: Upload updated skill
|
||||
if: steps.should_update.outputs.update == 'true'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ env.FRAMEWORK }}-skill-updated
|
||||
path: output/${{ env.FRAMEWORK }}.zip
|
||||
retention-days: 90
|
||||
|
||||
summary:
|
||||
name: Update Summary
|
||||
needs: update-skills
|
||||
runs-on: ubuntu-latest
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Create summary
|
||||
run: |
|
||||
echo "## 🔄 Scheduled Skills Update" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Date:** $(date -u '+%Y-%m-%d %H:%M UTC')" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "### Updated Frameworks" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- React" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- Django" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- FastAPI" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- Godot" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- Vue" >> $GITHUB_STEP_SUMMARY
|
||||
echo "- Flask" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "Updated skills available in workflow artifacts." >> $GITHUB_STEP_SUMMARY
|
||||
@@ -0,0 +1,112 @@
|
||||
# Security Note: This workflow uses only push/pull_request/workflow_dispatch triggers.
|
||||
# Matrix values are hardcoded constants. No untrusted input is used in run: commands.
|
||||
|
||||
name: Test Vector Database Adaptors
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main, development ]
|
||||
paths:
|
||||
- 'src/skill_seekers/cli/adaptors/**'
|
||||
- 'src/skill_seekers/mcp/tools/vector_db_tools.py'
|
||||
- 'tests/test_*adaptor*.py'
|
||||
- 'tests/test_mcp_vector_dbs.py'
|
||||
- 'tests/test_docker_smoke.py'
|
||||
- 'tests/test_upload_integration.py'
|
||||
pull_request:
|
||||
branches: [ main, development ]
|
||||
paths:
|
||||
- 'src/skill_seekers/cli/adaptors/**'
|
||||
- 'src/skill_seekers/mcp/tools/vector_db_tools.py'
|
||||
- 'tests/test_*adaptor*.py'
|
||||
- 'tests/test_mcp_vector_dbs.py'
|
||||
- 'tests/test_docker_smoke.py'
|
||||
- 'tests/test_upload_integration.py'
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
test-adaptors:
|
||||
name: Test ${{ matrix.adaptor }} Adaptor
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
adaptor: [weaviate, chroma, faiss, qdrant]
|
||||
python-version: ['3.10', '3.12']
|
||||
|
||||
env:
|
||||
ADAPTOR_NAME: ${{ matrix.adaptor }}
|
||||
PYTHON_VERSION: ${{ matrix.python-version }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install pytest
|
||||
pip install -e .
|
||||
|
||||
- name: Run adaptor tests
|
||||
run: |
|
||||
echo "🧪 Testing $ADAPTOR_NAME adaptor..."
|
||||
python -m pytest "tests/test_adaptors/test_${ADAPTOR_NAME}_adaptor.py" -v --tb=short
|
||||
|
||||
- name: Test adaptor integration
|
||||
run: |
|
||||
echo "🔗 Testing $ADAPTOR_NAME integration..."
|
||||
|
||||
# Create test skill
|
||||
mkdir -p test_skill/references
|
||||
echo "# Test Skill" > test_skill/SKILL.md
|
||||
echo "Test content" >> test_skill/SKILL.md
|
||||
echo "# Reference" > test_skill/references/ref.md
|
||||
|
||||
# Test adaptor packaging
|
||||
python3 -c "
|
||||
import os
|
||||
from pathlib import Path
|
||||
from skill_seekers.cli.adaptors import get_adaptor
|
||||
adaptor_name = os.environ['ADAPTOR_NAME']
|
||||
adaptor = get_adaptor(adaptor_name)
|
||||
package_path = adaptor.package(Path('test_skill'), Path('.'))
|
||||
print(f'Package created: {package_path}')
|
||||
assert package_path.exists(), 'Package file not created'
|
||||
print(f'Package size: {package_path.stat().st_size} bytes')
|
||||
"
|
||||
|
||||
- name: Upload test package
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: test-package-${{ env.ADAPTOR_NAME }}-py${{ env.PYTHON_VERSION }}
|
||||
path: test_skill-${{ env.ADAPTOR_NAME }}.json
|
||||
retention-days: 7
|
||||
|
||||
test-mcp-tools:
|
||||
name: Test MCP Vector DB Tools
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install pytest
|
||||
pip install -e .
|
||||
|
||||
- name: Run MCP vector DB tests
|
||||
run: |
|
||||
echo "🧪 Testing MCP vector database tools..."
|
||||
python -m pytest tests/test_mcp_vector_dbs.py -v --tb=short
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
name: Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main, development ]
|
||||
pull_request:
|
||||
branches: [ main, development ]
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Code Quality (Ruff & Mypy)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install "ruff==0.15.8" mypy
|
||||
pip install -e .
|
||||
- name: Run ruff linter
|
||||
run: ruff check src/ tests/ --output-format=github
|
||||
- name: Run ruff formatter check
|
||||
run: ruff format --check src/ tests/
|
||||
- name: Run mypy type checker
|
||||
run: mypy src/skill_seekers --show-error-codes --pretty
|
||||
continue-on-error: true
|
||||
|
||||
test-fast:
|
||||
name: Fast Unit Tests (parallel)
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, macos-latest]
|
||||
python-version: ['3.10', '3.11', '3.12']
|
||||
exclude:
|
||||
- os: macos-latest
|
||||
python-version: '3.10'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
- name: Cache pip packages
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/pip
|
||||
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-pip-
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
pip install pytest-xdist pytest-timeout
|
||||
pip install -e .
|
||||
- name: Run fast tests (xdist parallel)
|
||||
run: |
|
||||
python -m pytest tests/ -n auto --dist=loadfile \
|
||||
-m "not slow and not integration and not e2e and not network and not serial and not mcp_only" \
|
||||
-q --timeout=120 --cov=src/skill_seekers --cov-report=xml --cov-report=term
|
||||
timeout-minutes: 15
|
||||
- name: Upload coverage to Codecov
|
||||
uses: codecov/codecov-action@v4
|
||||
with:
|
||||
file: ./coverage.xml
|
||||
flags: unittests
|
||||
name: codecov-umbrella
|
||||
fail_ci_if_error: false
|
||||
|
||||
test-serial:
|
||||
name: Serial / Integration / E2E Tests
|
||||
runs-on: ubuntu-latest
|
||||
needs: [test-fast]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
pip install pytest-timeout
|
||||
pip install -e .
|
||||
- name: Run serial/integration/E2E tests
|
||||
run: |
|
||||
python -m pytest tests/ \
|
||||
--ignore=tests/test_bootstrap_skill_e2e.py \
|
||||
--ignore=tests/test_bootstrap_skill.py \
|
||||
-m "(integration or e2e or slow or network or serial) and not mcp_only" \
|
||||
-v --timeout=300
|
||||
timeout-minutes: 25
|
||||
|
||||
test-mcp:
|
||||
name: MCP Server Tests
|
||||
runs-on: ubuntu-latest
|
||||
needs: [test-fast]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
pip install pytest-timeout
|
||||
pip install -e ".[mcp]"
|
||||
pip install -e .
|
||||
- name: Run MCP tests
|
||||
run: |
|
||||
python -m pytest tests/ \
|
||||
-m "mcp_only and not network" \
|
||||
-v --timeout=180
|
||||
timeout-minutes: 10
|
||||
|
||||
tests-complete:
|
||||
name: Tests
|
||||
needs: [lint, test-fast, test-serial, test-mcp]
|
||||
runs-on: ubuntu-latest
|
||||
if: always()
|
||||
steps:
|
||||
- name: Check all results
|
||||
run: |
|
||||
if [ "${{ needs.lint.result }}" != "success" ]; then
|
||||
echo "❌ Code quality checks failed!"
|
||||
exit 1
|
||||
fi
|
||||
if [ "${{ needs.test-fast.result }}" != "success" ]; then
|
||||
echo "❌ Fast tests failed!"
|
||||
exit 1
|
||||
fi
|
||||
if [ "${{ needs.test-serial.result }}" == "failure" ]; then
|
||||
echo "❌ Serial/integration tests failed!"
|
||||
exit 1
|
||||
fi
|
||||
if [ "${{ needs.test-mcp.result }}" == "failure" ]; then
|
||||
echo "❌ MCP tests failed!"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ All checks passed!"
|
||||
@@ -0,0 +1,188 @@
|
||||
name: Vector Database Export
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
skill_name:
|
||||
description: 'Skill name to export (e.g., react, django, godot)'
|
||||
required: true
|
||||
type: string
|
||||
targets:
|
||||
description: 'Vector databases to export (comma-separated: weaviate,chroma,faiss,qdrant or "all")'
|
||||
required: true
|
||||
default: 'all'
|
||||
type: string
|
||||
config_path:
|
||||
description: 'Path to config file (optional, auto-detected from skill_name if not provided)'
|
||||
required: false
|
||||
type: string
|
||||
schedule:
|
||||
# Run weekly on Sunday at 2 AM UTC for popular frameworks
|
||||
- cron: '0 2 * * 0'
|
||||
|
||||
jobs:
|
||||
export:
|
||||
name: Export to Vector Databases
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# For scheduled runs, export popular frameworks
|
||||
skill: ${{ github.event_name == 'schedule' && fromJson('["react", "django", "godot", "fastapi"]') || fromJson(format('["{0}"]', github.event.inputs.skill_name)) }}
|
||||
|
||||
env:
|
||||
SKILL_NAME: ${{ matrix.skill }}
|
||||
TARGETS_INPUT: ${{ github.event.inputs.targets }}
|
||||
CONFIG_PATH_INPUT: ${{ github.event.inputs.config_path }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: recursive
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -e .
|
||||
|
||||
- name: Determine config path
|
||||
id: config
|
||||
run: |
|
||||
if [ -n "$CONFIG_PATH_INPUT" ]; then
|
||||
echo "path=$CONFIG_PATH_INPUT" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "path=configs/$SKILL_NAME.json" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Check if config exists
|
||||
id: check_config
|
||||
run: |
|
||||
CONFIG_FILE="${{ steps.config.outputs.path }}"
|
||||
if [ -f "$CONFIG_FILE" ]; then
|
||||
echo "exists=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "exists=false" >> $GITHUB_OUTPUT
|
||||
echo "⚠️ Config not found: $CONFIG_FILE"
|
||||
fi
|
||||
|
||||
- name: Scrape documentation
|
||||
if: steps.check_config.outputs.exists == 'true'
|
||||
run: |
|
||||
echo "📥 Scraping documentation for $SKILL_NAME..."
|
||||
skill-seekers create "${{ steps.config.outputs.path }}" --max-pages 100
|
||||
continue-on-error: true
|
||||
|
||||
- name: Determine export targets
|
||||
id: targets
|
||||
run: |
|
||||
TARGETS="${TARGETS_INPUT:-all}"
|
||||
if [ "$TARGETS" = "all" ]; then
|
||||
echo "list=weaviate chroma faiss qdrant" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "list=$(echo "$TARGETS" | tr ',' ' ')" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Export to vector databases
|
||||
if: steps.check_config.outputs.exists == 'true'
|
||||
env:
|
||||
EXPORT_TARGETS: ${{ steps.targets.outputs.list }}
|
||||
run: |
|
||||
SKILL_DIR="output/$SKILL_NAME"
|
||||
|
||||
if [ ! -d "$SKILL_DIR" ]; then
|
||||
echo "❌ Skill directory not found: $SKILL_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "📦 Exporting $SKILL_NAME to vector databases..."
|
||||
|
||||
for target in $EXPORT_TARGETS; do
|
||||
echo ""
|
||||
echo "🔹 Exporting to $target..."
|
||||
|
||||
# Use adaptor directly via CLI
|
||||
python3 -c "
|
||||
from pathlib import Path
|
||||
from skill_seekers.cli.adaptors import get_adaptor
|
||||
adaptor = get_adaptor('$target')
|
||||
package_path = adaptor.package(Path('$SKILL_DIR'), Path('output'))
|
||||
print(f'Exported to {package_path}')
|
||||
"
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "✅ $target export complete"
|
||||
else
|
||||
echo "❌ $target export failed"
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Generate quality report
|
||||
if: steps.check_config.outputs.exists == 'true'
|
||||
run: |
|
||||
SKILL_DIR="output/$SKILL_NAME"
|
||||
|
||||
if [ -d "$SKILL_DIR" ]; then
|
||||
echo "📊 Generating quality metrics..."
|
||||
|
||||
python3 -c "
|
||||
from pathlib import Path
|
||||
from skill_seekers.cli.quality_metrics import QualityAnalyzer
|
||||
analyzer = QualityAnalyzer(Path('$SKILL_DIR'))
|
||||
report = analyzer.generate_report()
|
||||
formatted = analyzer.format_report(report)
|
||||
print(formatted)
|
||||
with open('quality_report_${SKILL_NAME}.txt', 'w') as f:
|
||||
f.write(formatted)
|
||||
"
|
||||
fi
|
||||
continue-on-error: true
|
||||
|
||||
- name: Upload vector database exports
|
||||
if: steps.check_config.outputs.exists == 'true'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ env.SKILL_NAME }}-vector-exports
|
||||
path: |
|
||||
output/${{ env.SKILL_NAME }}-*.json
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload quality report
|
||||
if: steps.check_config.outputs.exists == 'true'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ env.SKILL_NAME }}-quality-report
|
||||
path: quality_report_${{ env.SKILL_NAME }}.txt
|
||||
retention-days: 30
|
||||
continue-on-error: true
|
||||
|
||||
- name: Create export summary
|
||||
if: steps.check_config.outputs.exists == 'true'
|
||||
env:
|
||||
EXPORT_TARGETS: ${{ steps.targets.outputs.list }}
|
||||
run: |
|
||||
echo "## 📦 Vector Database Export Summary: $SKILL_NAME" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
for target in $EXPORT_TARGETS; do
|
||||
FILE="output/${SKILL_NAME}-${target}.json"
|
||||
if [ -f "$FILE" ]; then
|
||||
SIZE=$(du -h "$FILE" | cut -f1)
|
||||
echo "✅ **$target**: $SIZE" >> $GITHUB_STEP_SUMMARY
|
||||
else
|
||||
echo "❌ **$target**: Export failed" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
done
|
||||
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f "quality_report_${SKILL_NAME}.txt" ]; then
|
||||
echo "### 📊 Quality Metrics" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
head -30 "quality_report_${SKILL_NAME}.txt" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
@@ -0,0 +1,79 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Environment variables (secrets)
|
||||
.env
|
||||
.env.local
|
||||
.env.*.local
|
||||
|
||||
# Virtual Environment
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
|
||||
# Output directory
|
||||
output/
|
||||
*.zip
|
||||
|
||||
# Skill Seekers cache (intermediate files)
|
||||
.skillseeker-cache/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Backups
|
||||
*.backup
|
||||
|
||||
# Testing artifacts
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
*.cover
|
||||
.hypothesis/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
|
||||
# Build artifacts
|
||||
.build/
|
||||
skill-seekers-configs/
|
||||
.claude/skills
|
||||
.mcp.json
|
||||
!/.mcp.json
|
||||
!distribution/claude-plugin/.mcp.json
|
||||
settings.json
|
||||
USER_GUIDE.md
|
||||
|
||||
# Local tooling artifacts (per-developer state, not project state)
|
||||
# chromadb's default persistence directory
|
||||
chroma_db/
|
||||
# Superpowers SDK output
|
||||
docs/superpowers/
|
||||
@@ -0,0 +1,3 @@
|
||||
[submodule "api/configs_repo"]
|
||||
path = api/configs_repo
|
||||
url = https://github.com/yusufkaraaslan/skill-seekers-configs.git
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"skill-seekers": {
|
||||
"command": "python",
|
||||
"args": ["-m", "skill_seekers.mcp.server_fastmcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,302 @@
|
||||
# AGENTS.md - Skill Seekers
|
||||
|
||||
Comprehensive reference for AI coding agents. Skill Seekers is a Python CLI tool (v3.6.0) that converts documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more into AI-ready skills for 21+ LLM platforms and RAG pipelines.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Skill Seekers** is a universal preprocessing layer that transforms raw documentation and code into structured knowledge assets. It supports 17+ source types and exports to 21+ AI platforms including Claude, Gemini, OpenAI, LangChain, LlamaIndex, and various vector databases.
|
||||
|
||||
### Key Capabilities
|
||||
- **Source Types (17):** Documentation websites, GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML, OpenAPI specs, AsciiDoc, PowerPoint, Confluence, Notion, RSS feeds, man pages, chat exports
|
||||
- **Export Targets (21):** Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown, LangChain, LlamaIndex, Haystack, Weaviate, ChromaDB, FAISS, Qdrant, Pinecone
|
||||
- **MCP Server:** FastMCP-based Model Context Protocol server for AI assistant integration
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# REQUIRED before running tests (src/ layout — tests hard-exit if package not installed)
|
||||
pip install -e .
|
||||
|
||||
# With dev tools (pytest, ruff, mypy, coverage)
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# With specific LLM platform support
|
||||
pip install -e ".[gemini]" # Google Gemini
|
||||
pip install -e ".[openai]" # OpenAI ChatGPT
|
||||
pip install -e ".[all-llms]" # All LLM platforms
|
||||
|
||||
# With all optional dependencies (except video-full)
|
||||
pip install -e ".[all]"
|
||||
|
||||
# Full video processing (heavy dependencies)
|
||||
pip install -e ".[video-full]"
|
||||
```
|
||||
|
||||
Note: `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not. Always install in editable mode first.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Create a `.env` file or export these variables:
|
||||
```bash
|
||||
ANTHROPIC_API_KEY # For Claude AI enhancement
|
||||
GOOGLE_API_KEY # For Gemini support
|
||||
OPENAI_API_KEY # For OpenAI support
|
||||
GITHUB_TOKEN # For GitHub repo scraping (higher rate limits)
|
||||
```
|
||||
|
||||
## Build / Test / Lint
|
||||
|
||||
```bash
|
||||
# Full suite (never skip — all must pass)
|
||||
pytest tests/ -v
|
||||
|
||||
# Fast iteration (skip slow, integration, E2E, network, MCP)
|
||||
pytest tests/ -m "not slow and not integration and not e2e and not network and not serial and not mcp_only" -q
|
||||
|
||||
# Fast parallel (install pytest-xdist first)
|
||||
pytest tests/ -n auto --dist=loadfile -m "not slow and not integration and not e2e and not network and not serial and not mcp_only" -q
|
||||
|
||||
# 3-phase runner script (recommended for local dev)
|
||||
bash scripts/run_tests_fast.sh
|
||||
|
||||
# Single test
|
||||
pytest tests/test_scraper_features.py::test_detect_language -v
|
||||
|
||||
# Skip slow/integration
|
||||
pytest tests/ -v -m "not slow and not integration"
|
||||
|
||||
# With coverage
|
||||
pytest tests/ --cov=src/skill_seekers --cov-report=term
|
||||
|
||||
# Lint + format check (matches CI)
|
||||
ruff check src/ tests/
|
||||
ruff format --check src/ tests/
|
||||
|
||||
# Type check (non-blocking — mypy is continue-on-error in CI)
|
||||
mypy src/skill_seekers --show-error-codes --pretty
|
||||
```
|
||||
|
||||
**Pytest config:** `asyncio_mode = "auto"`, so `@pytest.mark.asyncio` is implicit. Test markers: `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`, `serial`, `network`, `mcp_only`.
|
||||
|
||||
**CI note:** CI pins `ruff==0.15.8` (not the `>=0.14.13` dev dep). If formatting behaves differently locally, check the CI version.
|
||||
|
||||
**CI test phases:** Tests are split into 3 parallel jobs:
|
||||
- `test-fast` — 3386 unit tests with xdist across OS/Python matrix
|
||||
- `test-serial` — 69 serial/integration/E2E/network tests
|
||||
- `test-mcp` — 193 MCP tests (requires `[mcp]` extras)
|
||||
|
||||
## Code Style
|
||||
|
||||
### Formatting Rules (ruff — from pyproject.toml)
|
||||
- **Line length:** 100 characters
|
||||
- **Target Python:** 3.10+
|
||||
- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM
|
||||
- **Ignored rules:** E501 (line length handled by formatter), F541 (f-string style), ARG002 (unused method args for interface compliance), B007 (intentional unused loop vars), I001 (formatter handles imports), SIM114 (readability preference)
|
||||
|
||||
### Imports
|
||||
- Sort with isort (via ruff); `skill_seekers` is first-party
|
||||
- Standard library → third-party → first-party, separated by blank lines
|
||||
- Use `from __future__ import annotations` only if needed for forward refs
|
||||
- Guard optional imports with try/except ImportError (see `adaptors/__init__.py` pattern):
|
||||
```python
|
||||
try:
|
||||
from .claude import ClaudeAdaptor
|
||||
from .minimax import MiniMaxAdaptor
|
||||
except ImportError:
|
||||
ClaudeAdaptor = None
|
||||
MiniMaxAdaptor = None
|
||||
```
|
||||
|
||||
### Naming Conventions
|
||||
- **Files:** `snake_case.py` (e.g., `source_detector.py`, `config_validator.py`)
|
||||
- **Classes:** `PascalCase` (e.g., `SkillAdaptor`, `ClaudeAdaptor`, `SourceDetector`)
|
||||
- **Functions/methods:** `snake_case` (e.g., `get_adaptor()`, `detect_language()`)
|
||||
- **Constants:** `UPPER_CASE` (e.g., `ADAPTORS`, `DEFAULT_CHUNK_TOKENS`, `VALID_SOURCE_TYPES`)
|
||||
- **Private:** prefix with `_` (e.g., `_read_existing_content()`, `_validate_unified()`)
|
||||
|
||||
### Type Hints
|
||||
- Gradual typing — add hints where practical, not enforced everywhere
|
||||
- Use modern syntax: `str | None` not `Optional[str]`, `list[str]` not `List[str]`
|
||||
- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`, `ignore_missing_imports = true`
|
||||
- Tests are excluded from strict type checking (`disallow_untyped_defs = false`, `check_untyped_defs = false` for `tests.*`)
|
||||
|
||||
### Docstrings
|
||||
- Module-level docstring on every file (triple-quoted, describes purpose)
|
||||
- Google-style docstrings for public functions/classes
|
||||
- Include `Args:`, `Returns:`, `Raises:` sections where useful
|
||||
|
||||
### Error Handling
|
||||
- Use specific exceptions, never bare `except:`
|
||||
- Provide helpful error messages with context
|
||||
- Use `raise ValueError(...)` for invalid arguments, `raise RuntimeError(...)` for state errors
|
||||
- Guard optional dependency imports with try/except and give clear install instructions on failure
|
||||
- Chain exceptions with `raise ... from e` when wrapping
|
||||
|
||||
### Suppressing Lint Warnings
|
||||
- Use inline `# noqa: XXXX` comments (e.g., `# noqa: F401` for re-exports, `# noqa: ARG001` for required but unused params)
|
||||
|
||||
## Project Layout
|
||||
|
||||
```
|
||||
src/skill_seekers/ # Main package (src/ layout)
|
||||
cli/ # CLI commands and entry points (100+ files)
|
||||
adaptors/ # Platform adaptors (Strategy pattern, inherit SkillAdaptor)
|
||||
arguments/ # CLI argument definitions (one per source type)
|
||||
parsers/ # Subcommand parsers (one per source type)
|
||||
storage/ # Cloud storage (inherit BaseStorageAdaptor)
|
||||
main.py # Unified CLI entry point (COMMAND_MODULES dict)
|
||||
source_detector.py # Auto-detects source type from user input
|
||||
create_command.py # Unified `create` command routing
|
||||
config_validator.py # VALID_SOURCE_TYPES set + per-type validation
|
||||
unified_scraper.py # Multi-source orchestrator (scraped_data + dispatch)
|
||||
unified_skill_builder.py # Pairwise synthesis + generic merge
|
||||
mcp/ # MCP server (FastMCP + legacy)
|
||||
tools/ # MCP tool implementations by category (10 files)
|
||||
server_fastmcp.py # FastMCP server implementation
|
||||
server_legacy.py # Legacy MCP server
|
||||
sync/ # Sync monitoring (Pydantic models)
|
||||
benchmark/ # Benchmarking framework
|
||||
embedding/ # FastAPI embedding server
|
||||
workflows/ # 67 YAML workflow presets
|
||||
_version.py # Reads version from pyproject.toml
|
||||
tests/ # 160 test files (pytest)
|
||||
test_adaptors/ # 22 adaptor-specific test files
|
||||
conftest.py # Test configuration with package check
|
||||
configs/ # Preset JSON scraping configs
|
||||
docs/ # Documentation (guides, integrations, architecture)
|
||||
```
|
||||
|
||||
## Key Patterns
|
||||
|
||||
**Adaptor (Strategy) pattern** — all platform logic in `cli/adaptors/`. Inherit `SkillAdaptor`, implement `format_skill_md()`, `package()`, `upload()`. Register in `adaptors/__init__.py` ADAPTORS dict.
|
||||
|
||||
**Scraper pattern** — each source type has: `cli/<type>_scraper.py` (with `<Type>ToSkillConverter` class + `main()`), `arguments/<type>.py`, `parsers/<type>_parser.py`. Register in `parsers/__init__.py` PARSERS list, `main.py` COMMAND_MODULES dict, `config_validator.py` VALID_SOURCE_TYPES set.
|
||||
|
||||
**Unified pipeline** — `unified_scraper.py` dispatches to per-type `_scrape_<type>()` methods. `unified_skill_builder.py` uses pairwise synthesis for docs+github+pdf combos and `_generic_merge()` for all other combinations.
|
||||
|
||||
**MCP tools** — grouped in `mcp/tools/` by category. `scrape_generic_tool` handles all new source types.
|
||||
|
||||
**CLI subcommands** — git-style in `cli/main.py`. Each delegates to a module's `main()` function.
|
||||
|
||||
**Supported source types (17):** documentation (web), github, pdf, local, word, video, epub, jupyter, html, openapi, asciidoc, pptx, confluence, notion, rss, manpage, chat. Each detected automatically by `source_detector.py`.
|
||||
|
||||
**Supported platforms (21):** claude, gemini, openai, minimax, opencode, kimi, deepseek, qwen, openrouter, together, fireworks, markdown, langchain, llama-index, haystack, weaviate, chroma, faiss, qdrant, pinecone.
|
||||
|
||||
## CLI Commands
|
||||
|
||||
```bash
|
||||
# Core commands
|
||||
skill-seekers create <source> # Create skill from any source (auto-detects type)
|
||||
skill-seekers scan <dir> # AI-detect a project's tech stack and emit per-framework configs
|
||||
skill-seekers enhance <directory> # AI-powered enhancement
|
||||
skill-seekers package <directory> # Package skill for target platform
|
||||
skill-seekers upload <file> # Upload skill to target platform
|
||||
skill-seekers install <source> # One-command workflow (scrape + enhance + package + upload)
|
||||
|
||||
# Utilities
|
||||
skill-seekers estimate <source> # Estimate page count before scraping
|
||||
skill-seekers doctor # Health check for dependencies
|
||||
skill-seekers config # Configure API keys and settings
|
||||
skill-seekers workflows # List and apply workflow presets
|
||||
skill-seekers resume <job_id> # Resume interrupted scraping
|
||||
|
||||
# Advanced
|
||||
skill-seekers stream <source> # Streaming ingestion
|
||||
skill-seekers update <directory> # Incremental update
|
||||
skill-seekers multilang <directory> # Multi-language support
|
||||
```
|
||||
|
||||
## Testing Instructions
|
||||
|
||||
### Test Structure
|
||||
- Unit tests: `tests/test_*.py` — test individual modules
|
||||
- Adaptor tests: `tests/test_adaptors/test_*_adaptor.py` — test platform adaptors
|
||||
- E2E tests: `tests/test_*_e2e.py` — end-to-end integration tests
|
||||
|
||||
### Running Tests
|
||||
```bash
|
||||
# Fast test run (skip slow/integration tests)
|
||||
pytest tests/ -v -m "not slow and not integration"
|
||||
|
||||
# Full test suite
|
||||
pytest tests/ -v
|
||||
|
||||
# With coverage report
|
||||
pytest tests/ --cov=src/skill_seekers --cov-report=term-missing
|
||||
|
||||
# Specific test categories
|
||||
pytest tests/ -v -m "slow" # Only slow tests
|
||||
pytest tests/ -v -m "integration" # Only integration tests
|
||||
pytest tests/ -v -m "e2e" # Only E2E tests
|
||||
```
|
||||
|
||||
### Test Fixtures
|
||||
Test fixtures are located in `tests/fixtures/` and include sample configs, HTML files, and mock data.
|
||||
|
||||
## Git Workflow
|
||||
|
||||
- **`main`** — production, protected
|
||||
- **`development`** — default PR target, active dev
|
||||
- Feature branches created from `development`
|
||||
|
||||
## Pre-commit Checklist
|
||||
|
||||
```bash
|
||||
ruff check src/ tests/
|
||||
ruff format --check src/ tests/
|
||||
pytest tests/ -v -x # stop on first failure
|
||||
```
|
||||
|
||||
Never commit API keys. Use env vars: `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `GITHUB_TOKEN`.
|
||||
|
||||
## CI/CD
|
||||
|
||||
GitHub Actions (7 workflows in `.github/workflows/`):
|
||||
- **tests.yml** — ruff + mypy lint job, then pytest matrix (Ubuntu + macOS, Python 3.10-3.12) with Codecov upload
|
||||
- **release.yml** — tag-triggered: tests → version verification → PyPI publish via `uv build`
|
||||
- **test-vector-dbs.yml** — tests vector DB adaptors (weaviate, chroma, faiss, qdrant)
|
||||
- **docker-publish.yml** — multi-platform Docker builds (amd64, arm64) for CLI + MCP images
|
||||
- **quality-metrics.yml** — quality analysis with configurable threshold
|
||||
- **scheduled-updates.yml** — weekly skill updates for popular frameworks
|
||||
- **vector-db-export.yml** — weekly vector DB exports
|
||||
|
||||
## Deployment
|
||||
|
||||
### Docker
|
||||
Multi-stage Dockerfile with Python 3.12 slim base:
|
||||
```bash
|
||||
# Build image
|
||||
docker build -t skill-seekers .
|
||||
|
||||
# Run CLI
|
||||
docker run -v $(pwd)/output:/output skill-seekers create https://docs.example.com
|
||||
|
||||
# Run MCP server
|
||||
docker run -p 8765:8765 skill-seekers skill-seekers-mcp
|
||||
```
|
||||
|
||||
### MCP Server
|
||||
The MCP server provides Model Context Protocol integration:
|
||||
```bash
|
||||
# Start FastMCP server
|
||||
skill-seekers-mcp
|
||||
|
||||
# Or use the Python module
|
||||
python -m skill_seekers.mcp.server_fastmcp
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **API Keys:** Never commit API keys to version control. Use environment variables or `.env` files (already in `.gitignore`)
|
||||
- **Docker:** Runs as non-root user (`skillseeker`, UID 1000)
|
||||
- **Dependencies:** Regular security updates via `pip audit` or `safety check`
|
||||
- **Sandboxing:** Video processing uses optional dependencies that can be heavy; install `[video-full]` only when needed
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- **Website:** https://skillseekersweb.com/
|
||||
- **Documentation:** https://skillseekersweb.com/
|
||||
- **PyPI:** https://pypi.org/project/skill-seekers/
|
||||
- **Repository:** https://github.com/yusufkaraaslan/Skill_Seekers
|
||||
- **Config Browser:** https://skillseekersweb.com/
|
||||
- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2
|
||||
@@ -0,0 +1,617 @@
|
||||
# Bulletproof Quick Start Guide
|
||||
|
||||
**Target Audience:** Complete beginners | Never used Python/git before? Start here!
|
||||
|
||||
**Time:** 15-30 minutes total (including all installations)
|
||||
|
||||
**Result:** Working Skill Seeker installation + your first Claude skill created
|
||||
|
||||
---
|
||||
|
||||
## 📋 What You'll Need
|
||||
|
||||
Before starting, you need:
|
||||
- A computer (macOS, Linux, or Windows with WSL)
|
||||
- Internet connection
|
||||
- 30 minutes of time
|
||||
|
||||
That's it! We'll install everything else together.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Install Python (5 minutes)
|
||||
|
||||
### Check if You Already Have Python
|
||||
|
||||
Open Terminal (macOS/Linux) or Command Prompt (Windows) and type:
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
```
|
||||
|
||||
**✅ If you see:** `Python 3.10.x` or `Python 3.11.x` or higher → **Skip to Step 2!**
|
||||
|
||||
**❌ If you see:** `command not found` or version less than 3.10 → **Continue below**
|
||||
|
||||
### Install Python
|
||||
|
||||
#### macOS:
|
||||
```bash
|
||||
# Install Homebrew (if not installed)
|
||||
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
|
||||
|
||||
# Install Python
|
||||
brew install python3
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
python3 --version
|
||||
# Should show: Python 3.11.x or similar
|
||||
```
|
||||
|
||||
#### Linux (Ubuntu/Debian):
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install python3 python3-pip
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
python3 --version
|
||||
pip3 --version
|
||||
```
|
||||
|
||||
#### Windows:
|
||||
1. Download Python from: https://www.python.org/downloads/
|
||||
2. Run installer
|
||||
3. **IMPORTANT:** Check "Add Python to PATH" during installation
|
||||
4. Open Command Prompt and verify:
|
||||
```bash
|
||||
python --version
|
||||
```
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
Python 3.11.5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Install Git (3 minutes)
|
||||
|
||||
### Check if You Have Git
|
||||
|
||||
```bash
|
||||
git --version
|
||||
```
|
||||
|
||||
**✅ If you see:** `git version 2.x.x` → **Skip to Step 3!**
|
||||
|
||||
**❌ If not installed:**
|
||||
|
||||
#### macOS:
|
||||
```bash
|
||||
brew install git
|
||||
```
|
||||
|
||||
#### Linux:
|
||||
```bash
|
||||
sudo apt install git
|
||||
```
|
||||
|
||||
#### Windows:
|
||||
Download from: https://git-scm.com/download/win
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
git --version
|
||||
# Should show: git version 2.x.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Get Skill Seeker (2 minutes)
|
||||
|
||||
### Choose Where to Put It
|
||||
|
||||
Pick a location for the project. Good choices:
|
||||
- macOS/Linux: `~/Projects/` or `~/Documents/`
|
||||
- Note: `~` means your home directory (`$HOME` or `/Users/yourname` on macOS, `/home/yourname` on Linux)
|
||||
- Windows: `C:\Users\YourName\Projects\`
|
||||
|
||||
### Clone the Repository
|
||||
|
||||
```bash
|
||||
# Create Projects directory (if it doesn't exist)
|
||||
mkdir -p ~/Projects
|
||||
cd ~/Projects
|
||||
|
||||
# Clone Skill Seeker
|
||||
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git
|
||||
|
||||
# Enter the directory
|
||||
cd Skill_Seekers
|
||||
```
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
Cloning into 'Skill_Seekers'...
|
||||
remote: Enumerating objects: 245, done.
|
||||
remote: Counting objects: 100% (245/245), done.
|
||||
```
|
||||
|
||||
**Verify you're in the right place:**
|
||||
```bash
|
||||
pwd
|
||||
# Should show something like:
|
||||
# macOS: /Users/yourname/Projects/Skill_Seekers
|
||||
# Linux: /home/yourname/Projects/Skill_Seekers
|
||||
# (Replace 'yourname' with YOUR actual username)
|
||||
|
||||
ls
|
||||
# Should show: README.md, cli/, mcp/, configs/, etc.
|
||||
```
|
||||
|
||||
**❌ If `git clone` fails:**
|
||||
```bash
|
||||
# Check internet connection
|
||||
ping google.com
|
||||
|
||||
# Or download ZIP manually:
|
||||
# https://github.com/yusufkaraaslan/Skill_Seekers/archive/refs/heads/main.zip
|
||||
# Then unzip and cd into it
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Setup Virtual Environment & Install Skill Seekers (3 minutes)
|
||||
|
||||
A virtual environment keeps Skill Seeker's dependencies isolated and prevents conflicts.
|
||||
|
||||
```bash
|
||||
# Make sure you're in the Skill_Seekers directory
|
||||
cd ~/Projects/Skill_Seekers # ~ means your home directory ($HOME)
|
||||
# Adjust if you chose a different location
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
|
||||
# Activate it
|
||||
source venv/bin/activate # macOS/Linux
|
||||
# Windows users: venv\Scripts\activate
|
||||
```
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
(venv) username@computer Skill_Seekers %
|
||||
```
|
||||
Notice `(venv)` appears in your prompt - this means the virtual environment is active!
|
||||
|
||||
```bash
|
||||
# Now install Skill Seekers package (this installs all dependencies automatically)
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
Successfully installed skill-seekers-2.7.4 requests-2.32.5 beautifulsoup4-4.14.2 anthropic-0.76.0 ...
|
||||
Obtaining file:///path/to/Skill_Seekers
|
||||
Installing collected packages: skill-seekers
|
||||
Successfully installed skill-seekers
|
||||
```
|
||||
|
||||
**What just happened?**
|
||||
- `pip install -e .` installs the package in "editable" mode
|
||||
- The `.` means "current directory" (where pyproject.toml is)
|
||||
- This automatically installs ALL required dependencies
|
||||
- This registers the `skill-seekers` command so you can use it from anywhere
|
||||
- The `-e` flag means changes to the code take effect immediately (useful for development)
|
||||
|
||||
**Important Notes:**
|
||||
- **Every time** you open a new terminal to use Skill Seeker, run `source venv/bin/activate` first (Windows: `venv\Scripts\activate`)
|
||||
- You'll know it's active when you see `(venv)` in your terminal prompt
|
||||
- To deactivate later: just type `deactivate`
|
||||
|
||||
**❌ If python3 not found:**
|
||||
```bash
|
||||
# Try without the 3
|
||||
python -m venv venv
|
||||
```
|
||||
|
||||
**❌ If permission denied:**
|
||||
```bash
|
||||
# Virtual environment approach doesn't need sudo - you might have the wrong path
|
||||
# Make sure you're in the Skill_Seekers directory:
|
||||
pwd
|
||||
# Should show something like:
|
||||
# macOS: /Users/yourname/Projects/Skill_Seekers
|
||||
# Linux: /home/yourname/Projects/Skill_Seekers
|
||||
# (Replace 'yourname' with YOUR actual username)
|
||||
```
|
||||
|
||||
**❌ If "pip: command not found":**
|
||||
```bash
|
||||
# Try with python -m pip instead
|
||||
python3 -m pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Test Your Installation (1 minute)
|
||||
|
||||
Let's make sure everything works:
|
||||
|
||||
```bash
|
||||
# Test the CLI is installed correctly
|
||||
skill-seekers create --help
|
||||
```
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
usage: skill-seekers create [-h] ...
|
||||
```
|
||||
|
||||
**❌ If you see "command not found":**
|
||||
```bash
|
||||
# Ensure the package is installed
|
||||
pip install -e .
|
||||
|
||||
# Verify installation
|
||||
skill-seekers --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Create Your First Skill! (5-10 minutes)
|
||||
|
||||
Let's create a simple skill using a preset configuration.
|
||||
|
||||
### Option A: Small Test (Recommended First Time)
|
||||
|
||||
```bash
|
||||
# Create a config for a small site first
|
||||
cat > configs/test.json << 'EOF'
|
||||
{
|
||||
"name": "test-skill",
|
||||
"description": "Test skill creation",
|
||||
"base_url": "https://tailwindcss.com/docs/installation",
|
||||
"selectors": {
|
||||
"main_content": "#content-wrapper",
|
||||
"title": "h1, h2, h3",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"max_pages": 5,
|
||||
"rate_limit": 0.5
|
||||
}
|
||||
EOF
|
||||
|
||||
# Run the scraper
|
||||
skill-seekers create --config configs/test.json
|
||||
```
|
||||
|
||||
**Note for Windows users:** The `cat > file << 'EOF'` syntax doesn't work in PowerShell. Instead, create the file manually:
|
||||
|
||||
```powershell
|
||||
# In PowerShell, create configs/test.json with this content:
|
||||
@"
|
||||
{
|
||||
"name": "test-skill",
|
||||
"description": "Test skill creation",
|
||||
"base_url": "https://tailwindcss.com/docs/installation",
|
||||
"selectors": {
|
||||
"main_content": "#content-wrapper",
|
||||
"title": "h1, h2, h3",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"max_pages": 5,
|
||||
"rate_limit": 0.5
|
||||
}
|
||||
"@ | Out-File -FilePath configs/test.json -Encoding utf8
|
||||
|
||||
# Then run the scraper
|
||||
skill-seekers create --config configs/test.json
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Scrapes 5 pages from Tailwind CSS docs
|
||||
2. Creates `output/test-skill/` directory
|
||||
3. Generates SKILL.md and reference files
|
||||
|
||||
**⏱️ Time:** ~30 seconds
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
Scraping: https://tailwindcss.com/docs/installation
|
||||
Page 1/5: Installation
|
||||
Page 2/5: Editor Setup
|
||||
...
|
||||
✅ Skill created at: output/test-skill/
|
||||
```
|
||||
|
||||
### Option B: Full Example (React Docs)
|
||||
|
||||
```bash
|
||||
# Use the React preset
|
||||
skill-seekers create --config configs/react.json --max-pages 50
|
||||
```
|
||||
|
||||
**⏱️ Time:** ~5 minutes
|
||||
|
||||
**What you get:**
|
||||
- `output/react/SKILL.md` - Main skill file
|
||||
- `output/react/references/` - Organized documentation
|
||||
|
||||
### Verify It Worked
|
||||
|
||||
```bash
|
||||
# Check the output
|
||||
ls output/test-skill/
|
||||
# Should show: SKILL.md, references/, scripts/, assets/
|
||||
|
||||
# Look at the generated skill
|
||||
head output/test-skill/SKILL.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Package for Claude (30 seconds)
|
||||
|
||||
```bash
|
||||
# Package the skill
|
||||
skill-seekers package output/test-skill/
|
||||
```
|
||||
|
||||
**✅ Success looks like:**
|
||||
```
|
||||
✅ Skill packaged successfully!
|
||||
📦 Created: output/test-skill.zip
|
||||
📏 Size: 45.2 KB
|
||||
|
||||
Ready to upload to Claude AI!
|
||||
```
|
||||
|
||||
**Now you have:** `output/test-skill.zip` ready to upload to Claude!
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Upload to Claude (2 minutes)
|
||||
|
||||
1. Go to https://claude.ai
|
||||
2. Click your profile → Settings
|
||||
3. Click "Knowledge" or "Skills"
|
||||
4. Click "Upload Skill"
|
||||
5. Select `output/test-skill.zip`
|
||||
6. Done! Claude can now use this skill
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Success! What's Next?
|
||||
|
||||
You now have a working Skill Seeker installation! Here's what you can do:
|
||||
|
||||
### Try Other Presets
|
||||
|
||||
```bash
|
||||
# See all available presets
|
||||
ls configs/
|
||||
|
||||
# Try Vue.js
|
||||
skill-seekers create --config configs/vue.json --max-pages 50
|
||||
|
||||
# Try Django
|
||||
skill-seekers create --config configs/django.json --max-pages 50
|
||||
```
|
||||
|
||||
### Try Other Source Types (17 Supported!)
|
||||
|
||||
```bash
|
||||
# Auto-detect source type with the `create` command
|
||||
skill-seekers create https://docs.example.com # Documentation
|
||||
skill-seekers create facebook/react # GitHub repo
|
||||
skill-seekers create manual.pdf # PDF
|
||||
skill-seekers create report.docx # Word document
|
||||
skill-seekers create book.epub # EPUB book
|
||||
skill-seekers create analysis.ipynb # Jupyter Notebook
|
||||
skill-seekers create spec.yaml # OpenAPI/Swagger spec
|
||||
skill-seekers create slides.pptx # PowerPoint
|
||||
|
||||
# More source types
|
||||
skill-seekers create --video-url https://youtube.com/watch?v=abc # Video
|
||||
skill-seekers create --space-key DOCS # Confluence wiki
|
||||
skill-seekers create --database-id DB_ID # Notion
|
||||
skill-seekers create feed.rss # RSS feed
|
||||
skill-seekers create grep.1 # Man page
|
||||
skill-seekers create --chat-export-path ./slack-export # Slack/Discord
|
||||
```
|
||||
|
||||
### Create Custom Skills
|
||||
|
||||
```bash
|
||||
# Interactive mode - answer questions
|
||||
skill-seekers create --interactive
|
||||
|
||||
# Or create config for any website
|
||||
skill-seekers scrape \
|
||||
--name myframework \
|
||||
--url https://docs.myframework.com/ \
|
||||
--description "My favorite framework"
|
||||
```
|
||||
|
||||
### Where to Save Custom Configs
|
||||
|
||||
You have three options for placing your custom config files:
|
||||
|
||||
**Option 1: User Config Directory (Recommended)**
|
||||
|
||||
```bash
|
||||
# Create config in your home directory
|
||||
mkdir -p ~/.config/skill-seekers/configs
|
||||
cat > ~/.config/skill-seekers/configs/myproject.json << 'EOF'
|
||||
{
|
||||
"name": "myproject",
|
||||
"base_url": "https://docs.myproject.com/",
|
||||
"max_pages": 50
|
||||
}
|
||||
EOF
|
||||
|
||||
# Use it
|
||||
skill-seekers create --config myproject.json
|
||||
```
|
||||
|
||||
**Option 2: Current Directory (Project-Specific)**
|
||||
|
||||
```bash
|
||||
# Create config in your project
|
||||
mkdir -p configs
|
||||
nano configs/myproject.json
|
||||
|
||||
# Use it
|
||||
skill-seekers create --config configs/myproject.json
|
||||
```
|
||||
|
||||
**Option 3: Absolute Path**
|
||||
|
||||
```bash
|
||||
# Use any file path
|
||||
skill-seekers create --config /full/path/to/config.json
|
||||
```
|
||||
|
||||
The tool searches in this order: exact path → `./configs/` → `~/.config/skill-seekers/configs/` → API presets
|
||||
|
||||
### Use with Claude Code (Advanced)
|
||||
|
||||
If you have Claude Code installed:
|
||||
|
||||
```bash
|
||||
# One-time setup
|
||||
./setup_mcp.sh
|
||||
|
||||
# Then use natural language in Claude Code:
|
||||
# "Generate a skill for Svelte docs"
|
||||
# "Package the skill at output/svelte/"
|
||||
```
|
||||
|
||||
**See:** [docs/guides/MCP_SETUP.md](docs/guides/MCP_SETUP.md) for full MCP setup
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "Command not found" errors
|
||||
|
||||
**Problem:** `python3: command not found`
|
||||
|
||||
**Solution:** Python not installed or not in PATH
|
||||
- macOS/Linux: Reinstall Python with brew/apt
|
||||
- Windows: Reinstall Python, check "Add to PATH"
|
||||
- Try `python` instead of `python3`
|
||||
|
||||
### "Permission denied" errors
|
||||
|
||||
**Problem:** Can't install packages
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Use --user flag for pip
|
||||
pip3 install --user skill-seekers
|
||||
|
||||
# Or use a virtual environment (recommended)
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### "No such file or directory"
|
||||
|
||||
**Problem:** CLI not found or config file path incorrect
|
||||
|
||||
**Solution:** Ensure you're using the CLI command correctly
|
||||
```bash
|
||||
# Verify installation
|
||||
skill-seekers --version
|
||||
# Should show version 3.6.0+
|
||||
|
||||
# Verify config file exists at expected path
|
||||
ls configs/
|
||||
```
|
||||
|
||||
### "ModuleNotFoundError" or "command not found: skill-seekers"
|
||||
|
||||
**Problem:** Package not installed or virtual environment not activated
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Install the package in editable mode
|
||||
pip install -e .
|
||||
|
||||
# Or if you need dev tools
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
### Scraping is slow or fails
|
||||
|
||||
**Problem:** Takes forever or gets errors
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Use smaller max_pages for testing
|
||||
skill-seekers create --config configs/react.json --max-pages 10
|
||||
|
||||
# Check internet connection
|
||||
ping google.com
|
||||
|
||||
# Check the website is accessible
|
||||
curl -I https://docs.yoursite.com
|
||||
```
|
||||
|
||||
### Still stuck?
|
||||
|
||||
1. **Check our detailed troubleshooting guide:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md)
|
||||
2. **Open an issue:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
3. **Include this info:**
|
||||
- Operating system (macOS 13, Ubuntu 22.04, Windows 11, etc.)
|
||||
- Python version (`python3 --version`)
|
||||
- Full error message
|
||||
- What command you ran
|
||||
|
||||
---
|
||||
|
||||
## 📚 Next Steps
|
||||
|
||||
- **Read the full README:** [README.md](README.md)
|
||||
- **Learn about presets:** [configs/](configs/)
|
||||
- **Try MCP integration:** [docs/guides/MCP_SETUP.md](docs/guides/MCP_SETUP.md)
|
||||
- **Advanced usage:** [docs/](docs/)
|
||||
|
||||
---
|
||||
|
||||
## ✅ Quick Reference
|
||||
|
||||
```bash
|
||||
# Your typical workflow:
|
||||
|
||||
# 1. Create/use a config
|
||||
skill-seekers create --config configs/react.json --max-pages 50
|
||||
|
||||
# 2. Package it
|
||||
skill-seekers package output/react/
|
||||
|
||||
# 3. Upload output/react.zip to Claude
|
||||
|
||||
# Done! 🎉
|
||||
```
|
||||
|
||||
**Common locations:**
|
||||
- **Configs:** `configs/*.json`
|
||||
- **Output:** `output/skill-name/`
|
||||
- **Packaged skills:** `output/skill-name.zip`
|
||||
|
||||
**Time estimates:**
|
||||
- Small skill (5-10 pages): 30 seconds
|
||||
- Medium skill (50-100 pages): 3-5 minutes
|
||||
- Large skill (500+ pages): 15-30 minutes
|
||||
|
||||
---
|
||||
|
||||
**Still confused?** That's okay! Open an issue and we'll help you get started: https://github.com/yusufkaraaslan/Skill_Seekers/issues/new
|
||||
@@ -0,0 +1,285 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Skill Seekers** converts documentation from 18 source types into production-ready formats for 21+ AI platforms (LLM platforms, RAG frameworks, vector databases, AI coding assistants). Published on PyPI as `skill-seekers`.
|
||||
|
||||
**Version:** 3.7.0 | **Python:** 3.10+ | **Website:** https://skillseekersweb.com/
|
||||
|
||||
**Architecture:** See `docs/UML_ARCHITECTURE.md` for UML diagrams and module overview. StarUML project at `docs/UML/skill_seekers.mdj`. Refactor state/history: `docs/UNIFICATION_PLAN.md` (Grand Unification — all 5 phases done; remaining cosmetic items listed there).
|
||||
|
||||
## Essential Commands
|
||||
|
||||
```bash
|
||||
# REQUIRED before running tests or CLI (src/ layout)
|
||||
pip install -e .
|
||||
|
||||
# Run all tests (NEVER skip - all must pass before commits)
|
||||
pytest tests/ -v
|
||||
|
||||
# Fast iteration (skip slow MCP tests ~20min)
|
||||
pytest tests/ --ignore=tests/test_mcp_fastmcp.py --ignore=tests/test_mcp_server.py --ignore=tests/test_install_skill_e2e.py -q
|
||||
|
||||
# Single test
|
||||
pytest tests/test_scraper_features.py::test_detect_language -vv -s
|
||||
|
||||
# Code quality (must pass before push - matches CI)
|
||||
uvx ruff check src/ tests/
|
||||
uvx ruff format --check src/ tests/
|
||||
mypy src/skill_seekers # continue-on-error in CI
|
||||
|
||||
# Auto-fix lint/format issues
|
||||
uvx ruff check --fix --unsafe-fixes src/ tests/
|
||||
uvx ruff format src/ tests/
|
||||
|
||||
# Build & publish
|
||||
uv build
|
||||
uv publish
|
||||
```
|
||||
|
||||
## CI Matrix
|
||||
|
||||
Runs on push/PR to `main` or `development`. Lint job (Python 3.12, Ubuntu) + Test job (Ubuntu + macOS, Python 3.10/3.11/3.12, excludes macOS+3.10). Both must pass for merge.
|
||||
|
||||
## Git Workflow
|
||||
|
||||
- **Main branch:** `main` (requires tests + 1 review)
|
||||
- **Development branch:** `development` (default PR target, requires tests)
|
||||
- **Feature branches:** `feature/{task-id}-{description}` from `development`
|
||||
- PRs always target `development`, never `main` directly
|
||||
|
||||
## Architecture
|
||||
|
||||
### CLI: Unified create command
|
||||
|
||||
Entry point `src/skill_seekers/cli/main.py`. The `create` command is the **primary** entry point for skill creation — it auto-detects source type and routes to the appropriate `SkillConverter`. The `scan` command (added in #327) is a separate discovery step for projects with multiple frameworks; it emits one config file per detected framework and you then run `create` on each.
|
||||
|
||||
```
|
||||
skill-seekers create <source> # Auto-detect: URL, owner/repo, ./path, file.pdf, etc.
|
||||
skill-seekers scan <dir> # AI-driven discovery → emits one config per detected framework + <project>-codebase.json
|
||||
skill-seekers package <dir> # Package for platform (--target claude/gemini/openai/markdown/minimax/opencode/kimi/deepseek/qwen/openrouter/together/fireworks/langchain/llama-index/haystack/chroma/faiss/weaviate/qdrant/pinecone/ibm-bob)
|
||||
```
|
||||
|
||||
### Scan command (issue #327)
|
||||
|
||||
`skill-seekers scan <dir>` is an AI-driven project knowledge-base bootstrapper. Pipeline in `src/skill_seekers/cli/scan_command.py`:
|
||||
|
||||
1. `collect_signals()` in `signal_collectors.py` — deterministic, bounded gathering of manifests + README + Dockerfile/CI + sampled source files + git remote. **Per-kind byte budgets** (24 KB manifest / 6 KB README / 6 KB CI / 28 KB samples, total 64 KB) so a fat package.json can't crowd out other kinds. `_SOURCE_DIRS` covers ~14 layouts (Go `cmd/`, Rust `crates/`, JS monorepo `apps/packages/`, Maven `source/`, Django at root); also walks root one level deep for flat-layout Python.
|
||||
2. `detect_with_ai(bundle, AgentClient)` — one LLM call, structured JSON output. **Source signals are first-2-KB of each file** (whole-file sampling, no regex parsing — added in WS4 because regex missed Go multi-line imports + Rust `mod`/`extern crate`). Canonical-slug prompt + the canonical-name resolver are coupled — change one, update the other.
|
||||
3. `resolve_or_generate_with_status()` — for each detection: try `out_dir/<slug>.json` (cache from prior run), then `resolve_config_path` from `config_fetcher` with multiple canonical name candidates (`_canonical_name_candidates` handles `"Godot Engine"` → `"godot"`, plus CJK / European suffixes like `"Godot 引擎"`, `"React フレームワーク"`, `"Lodash Bibliothek"`), then `generate_config_with_ai` as the last resort. Always appends `.json` to lookup names so local-disk and user-dir resolution actually finds files. Always stamps `metadata.detected_version` (nested, not top-level — `metadata.version` already exists and means config-schema version).
|
||||
4. `emit_codebase_config()` — always writes `<project>-codebase.json` (a `type: local` source pointed at the project root).
|
||||
5. `diff_against_existing()` — keyed by **filename slug** (not internal `data["name"]`) so re-scans don't churn when the AI returns a display name vs the registry canonical slug.
|
||||
6. `_archive_removed()` — when a config disappears from detections, MOVE (not delete — user may have hand-edited) to `out_dir/.archived/<UTC-timestamp>/`. Runs after diff, before fresh writes.
|
||||
7. `maybe_publish()` — **native async** (WS11). Opt-in submission of freshly AI-generated configs to the community registry. Pre-checks `GITHUB_TOKEN`. Idempotency guard: `_find_existing_issue` queries GitHub Search API for an existing open issue with the same config name before submitting. Retries transient failures (rate limit, 5xx) with 0s/5s/15s backoff. `_prompt_async` wraps `input()` via `asyncio.to_thread` so the event loop isn't blocked.
|
||||
|
||||
**CLI dispatch** uses the `COMMAND_CLASSES` table in `main.py` (added in WS1). `scan` and `doctor` are dispatched as `Cls(args).execute()` consuming the parsed argparse namespace directly — no `_reconstruct_argv` hack, no duplicate argparse. `ScanCommand.execute()` is the single `asyncio.run` boundary wrapping `run_scan` (sync) + `maybe_publish` (async). Remaining ~14 commands still use the legacy `COMMAND_MODULES` dispatch; they're flagged for migration.
|
||||
|
||||
**Cost guardrails**: `--max-ai-generations N` (default 10) caps unbounded AI generation; `--dry-run` previews without writing or invoking AI; `--probe-urls` HEAD-checks AI-generated URLs with retry-on-404 and stamps `metadata._url_unverified` on confirmed-bad URLs.
|
||||
|
||||
**Safety**: All writes use `_atomic_write_json` (`os.replace` after writing to `.tmp`) so a `KeyboardInterrupt` mid-write can't corrupt configs. `_safe_size` guards `stat()` so broken symlinks don't crash the scan. `ScanCommand.execute` calls `logging.basicConfig` so `logger.warning`/`error` is visible; exit code is non-zero when no configs and no codebase config were emitted.
|
||||
|
||||
**Public constant**: `SourceDetector.CODE_PROJECT_MARKERS` (was `_CODE_PROJECT_MARKERS`) — shared between source_detector + signal_collectors. ~50 manifest types now (Pipfile, environment.yml, deno.json, flake.nix, Chart.yaml, deps.edn, dune-project, BUILD.bazel, …). Public so cross-module access doesn't reach into a private attribute.
|
||||
|
||||
### SkillConverter Pattern (Template Method + Factory)
|
||||
|
||||
All 18 source types implement the `SkillConverter` base class (`skill_converter.py`):
|
||||
|
||||
```python
|
||||
converter = get_converter("web", config) # Factory lookup
|
||||
converter.run() # Template: extract() → build_skill()
|
||||
```
|
||||
|
||||
Registry in `CONVERTER_REGISTRY` maps source type → (module, class). `create_command.py` builds config from `ExecutionContext`, calls `get_converter()`, then runs centralized enhancement. `get_converter("config", {...})` constructs `UnifiedScraper` from the same factory-shaped dict (no special cases in create_command/MCP). The base resolves `skill_dir` once (strips trailing separators) and derives `data_file` via `data_file_for()` — subclasses must not re-derive paths.
|
||||
|
||||
### DocumentSkillBuilder (build side of 9 document scrapers)
|
||||
|
||||
`cli/document_skill_builder.py:DocumentSkillBuilder` sits between `SkillConverter` and the 9 document scrapers (epub, word, pptx, html, pdf, jupyter, man, rss, chat). It owns `categorize_content`, reference-file writing (tables, truncation, image guard), `index.md` + `SKILL.md` generation, and `load_extracted_data`. Variation points are class attrs (`DOC_NOUN`, `SOURCE_LABEL`, `LOAD_TOTAL_KEY`, `PATTERN_KEYWORDS`, `RANGE_LABEL`, …) and small hook methods (`category_stem`, `_write_reference_section`, `_write_skill_md_metadata`). Output is pinned **byte-identical** by golden trees in `tests/golden/phase2/` — `UPDATE_GOLDENS=1` rewrites them, only do that deliberately. Surviving full-method overrides are domain-shaped and commented per scraper.
|
||||
|
||||
### UnifiedScraper (multi-source configs)
|
||||
|
||||
`unified_scraper.py` dispatches via the class-level `SOURCE_DISPATCH` table; `_scrape_with_converter()` is the shared engine for the 13 mechanical source types (`get_converter()` + public `converter.extract()` + cache copy + sub-skill build), so **new types registered in `CONVERTER_REGISTRY` work in unified configs automatically**. documentation/github/local stay bespoke (commented why). `run()` deliberately does NOT follow the base template (TestRunOrchestration pins that run() triggers workflows).
|
||||
|
||||
### Data Flow (5 phases)
|
||||
|
||||
1. **Scrape** - Source-specific scraper extracts content to `output/{name}_data/pages/*.json`
|
||||
2. **Build** - `build_skill()` categorizes pages, extracts patterns, generates `output/{name}/SKILL.md`
|
||||
3. **Enhance** (optional) - LLM rewrites SKILL.md (`--enhance-level 0-3`, auto-detects API vs LOCAL mode)
|
||||
4. **Package** - Platform adaptor formats output (`.zip`, `.tar.gz`, JSON, vector index)
|
||||
5. **Upload** (optional) - Platform API upload
|
||||
|
||||
### Platform Adaptor Pattern (Strategy + Factory)
|
||||
|
||||
Factory: `get_adaptor(platform, config)` in `adaptors/__init__.py` returns a `SkillAdaptor` instance. Base class `SkillAdaptor` + `SkillMetadata` in `adaptors/base.py`.
|
||||
|
||||
```
|
||||
src/skill_seekers/cli/adaptors/
|
||||
├── __init__.py # Factory: get_adaptor(platform, config), ADAPTORS registry
|
||||
├── base.py # Abstract base: SkillAdaptor, SkillMetadata
|
||||
├── openai_compatible.py # Shared base for OpenAI-compatible platforms
|
||||
├── claude.py # --target claude
|
||||
├── gemini.py # --target gemini
|
||||
├── openai.py # --target openai
|
||||
├── markdown.py # --target markdown
|
||||
├── minimax.py # --target minimax
|
||||
├── opencode.py # --target opencode
|
||||
├── kimi.py # --target kimi
|
||||
├── deepseek.py # --target deepseek
|
||||
├── qwen.py # --target qwen
|
||||
├── openrouter.py # --target openrouter
|
||||
├── together.py # --target together
|
||||
├── fireworks.py # --target fireworks
|
||||
├── langchain.py # --target langchain
|
||||
├── llama_index.py # --target llama-index
|
||||
├── haystack.py # --target haystack
|
||||
├── chroma.py # --target chroma
|
||||
├── faiss_helpers.py # --target faiss
|
||||
├── qdrant.py # --target qdrant
|
||||
├── weaviate.py # --target weaviate
|
||||
├── pinecone_adaptor.py # --target pinecone
|
||||
└── streaming_adaptor.py # --target streaming
|
||||
```
|
||||
|
||||
All adaptors use `--target`. All adaptors are imported with `try/except ImportError` so missing optional deps don't break the registry.
|
||||
|
||||
### 18 Source Type Converters
|
||||
|
||||
Each in `src/skill_seekers/cli/{type}_scraper.py` as a `SkillConverter` subclass (no `main()`). The `create_command.py` uses `source_detector.py` to auto-detect, then calls `get_converter()`. Converters: web (doc_scraper), github, pdf, word, epub, video, local (codebase_scraper), jupyter, html, openapi, asciidoc, pptx, rss, manpage, confluence, notion, chat, config (unified_scraper).
|
||||
|
||||
### CLI Argument System (single-definition parsers)
|
||||
|
||||
```
|
||||
src/skill_seekers/cli/
|
||||
├── parsers/ # Central SubcommandParser classes — the ONLY definition of each command's flags
|
||||
│ └── create_parser.py # Progressive help disclosure (--help-web, --help-github, etc.)
|
||||
├── arguments/ # Argument definitions
|
||||
│ ├── common.py # add_all_standard_arguments() - shared across all scrapers
|
||||
│ └── create.py # UNIVERSAL_ARGUMENTS, WEB_ARGUMENTS, GITHUB_ARGUMENTS, etc.
|
||||
├── exit_codes.py # EXIT_SUCCESS/ERROR/VALIDATION/INTERRUPT
|
||||
└── source_detector.py # Auto-detect source type from input string
|
||||
```
|
||||
|
||||
Command modules' standalone `main(args=None)` paths build their parser FROM the central `SubcommandParser` class — **add/change a flag in `parsers/*.py` only**. Drift guards (`tests/test_cli_parsers.py::TestCentralModuleParserSync` and `TestCentralParserSingleSource`) fail CI on any divergence of dests/defaults/option strings.
|
||||
|
||||
`ExecutionContext.override()` is context-local (a `ContextVar` layered over the unchanged base singleton) — thread/async safe for the MCP server; propagate to worker threads via `copy_context`.
|
||||
|
||||
### C3.x Codebase Analysis Pipeline
|
||||
|
||||
Local codebase analysis features, all opt-out (`--skip-*` flags):
|
||||
- C3.1 `pattern_recognizer.py` - Design pattern detection (10 GoF patterns, 9 languages)
|
||||
- C3.2 `test_example_extractor.py` - Usage examples from tests
|
||||
- C3.3 `how_to_guide_builder.py` - AI-enhanced educational guides
|
||||
- C3.4 `config_extractor.py` - Configuration pattern extraction
|
||||
- C3.5 `generate_router.py` - Architecture overview generation
|
||||
- C3.10 `signal_flow_analyzer.py` - Godot signal flow analysis
|
||||
|
||||
### MCP Server
|
||||
|
||||
`src/skill_seekers/mcp/server_fastmcp.py` - 40 tools via FastMCP. Transport: stdio (Claude Code) or HTTP (Cursor/Windsurf). Optional dependency: `pip install -e ".[mcp]"`
|
||||
|
||||
- **Tools run in-process** via `run_cli_main()` in `mcp/tools/_common.py`: same argv parsed by the command's REAL parser (sys.argv patch under a lock), stdout/stderr capture + contextvar log capture, identical `(stdout, stderr, returncode)` contract. No subprocess startup; old hard timeouts are advisory.
|
||||
- **Exceptions BY DESIGN**: `enhance_skill` (LOCAL agent) and `install_skill`'s enhancement step stay subprocess — the agent must be a real child process for the fork-bomb-guard env semantics (`SKILL_SEEKER_ENHANCE_ACTIVE`). Never make these in-process.
|
||||
- **Domain logic lives in `skill_seekers.services/`** (marketplace_manager, marketplace_publisher, config_publisher, source_manager, git_repo) — importable by CLI without the `[mcp]` extra; old `skill_seekers.mcp.*` paths are back-compat shims. No `sys.path` hacks anywhere in `mcp/`.
|
||||
|
||||
### Enhancement (AgentClient is the single AI transport)
|
||||
|
||||
Every text-based AI call goes through `AgentClient` (`src/skill_seekers/cli/agent_client.py`): central truncation gate, timeout policy, error classification. `API_PROVIDERS` (provider registry) and `AGENT_PRESETS` (local-agent command templates) live ONLY there. Adaptors declare provider/endpoint/model/prompt and route through `SkillAdaptor._enhance_skill_md_via_client` (atomic save with backup). `video_visual` frame classification is the documented multimodal exception (AgentClient is text-only).
|
||||
|
||||
- **API mode** (if API key set): Anthropic, Google Gemini, OpenAI, Moonshot/Kimi — detected in registry order; `SKILL_SEEKER_PROVIDER` forces one. Models: `SKILL_SEEKER_MODEL` (global) or `ANTHROPIC_MODEL`/`GOOGLE_MODEL`/`OPENAI_MODEL`/`MOONSHOT_MODEL`; `ANTHROPIC_BASE_URL` for compatible endpoints.
|
||||
- **LOCAL mode** (fallback): Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — command built by `build_local_agent_command()`.
|
||||
- Control: `--enhance-level 0` (off) / `1` (SKILL.md only) / `2` (default, balanced) / `3` (full)
|
||||
- Agent selection: `--agent claude|codex|copilot|opencode|kimi|custom`
|
||||
|
||||
## Key Implementation Details
|
||||
|
||||
### Smart Categorization (`doc_scraper.py:smart_categorize()`)
|
||||
|
||||
Scores pages against category keywords: 3 points for URL match, 2 for title, 1 for content. Threshold of 2+ required. Falls back to "other".
|
||||
|
||||
### Content Extraction (`doc_scraper.py`)
|
||||
|
||||
`FALLBACK_MAIN_SELECTORS` constant + `_find_main_content()` helper handle CSS selector fallback. Links are extracted from the full page before early return (not just main content). `body` is deliberately excluded from fallbacks.
|
||||
|
||||
### Three-Stream GitHub Architecture (`unified_codebase_analyzer.py`)
|
||||
|
||||
Stream 1: Code Analysis (AST, patterns, tests, guides). Stream 2: Documentation (README, docs/, wiki). Stream 3: Community (issues, PRs, metadata). Depth control: `basic` (1-2 min) or `c3x` (20-60 min).
|
||||
|
||||
## Testing
|
||||
|
||||
### Test markers (pytest.ini)
|
||||
|
||||
```bash
|
||||
pytest tests/ -v # Default: fast tests only
|
||||
pytest tests/ -v -m slow # Include slow tests (>5s)
|
||||
pytest tests/ -v -m integration # External services required
|
||||
pytest tests/ -v -m e2e # Resource-intensive
|
||||
pytest tests/ -v -m "not slow and not integration" # Fastest subset
|
||||
```
|
||||
|
||||
### Known legitimate skips (~11)
|
||||
|
||||
- 2: chromadb incompatible with Python 3.14 (pydantic v1)
|
||||
- 2: weaviate-client not installed
|
||||
- 2: Qdrant not running (requires docker)
|
||||
- 2: langchain/llama_index not installed
|
||||
- 3: GITHUB_TOKEN not set
|
||||
|
||||
### sys.modules gotcha
|
||||
|
||||
`test_swift_detection.py` deletes `skill_seekers.cli` modules from `sys.modules`. It must save and restore both `sys.modules` entries AND parent package attributes (`setattr`). See the test file for the pattern.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Core deps include `langchain`, `llama-index`, `anthropic`, `httpx`, `PyMuPDF`, `pydantic`. Platform-specific deps are optional:
|
||||
|
||||
```bash
|
||||
pip install -e ".[mcp]" # MCP server
|
||||
pip install -e ".[gemini]" # Google Gemini
|
||||
pip install -e ".[openai]" # OpenAI
|
||||
pip install -e ".[docx]" # Word documents
|
||||
pip install -e ".[epub]" # EPUB books
|
||||
pip install -e ".[video]" # Video (lightweight)
|
||||
pip install -e ".[video-full]"# Video (Whisper + visual)
|
||||
pip install -e ".[jupyter]" # Jupyter notebooks
|
||||
pip install -e ".[pptx]" # PowerPoint
|
||||
pip install -e ".[rss]" # RSS/Atom feeds
|
||||
pip install -e ".[confluence]"# Confluence wiki
|
||||
pip install -e ".[notion]" # Notion pages
|
||||
pip install -e ".[chroma]" # ChromaDB
|
||||
pip install -e ".[all]" # Everything (except video-full)
|
||||
```
|
||||
|
||||
Dev dependencies use PEP 735 `[dependency-groups]` in pyproject.toml.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
ANTHROPIC_API_KEY=sk-ant-... # Claude AI (or compatible endpoint)
|
||||
ANTHROPIC_BASE_URL=https://... # Optional: Claude-compatible API endpoint
|
||||
GOOGLE_API_KEY=AIza... # Google Gemini (optional)
|
||||
OPENAI_API_KEY=sk-... # OpenAI (optional)
|
||||
GITHUB_TOKEN=ghp_... # Higher GitHub rate limits
|
||||
```
|
||||
|
||||
## Adding New Features
|
||||
|
||||
### New platform adaptor
|
||||
1. Create `src/skill_seekers/cli/adaptors/{platform}.py` inheriting `SkillAdaptor` from `base.py`
|
||||
2. Register in `adaptors/__init__.py` (add try/except import + add to `ADAPTORS` dict)
|
||||
3. Add optional dep to `pyproject.toml`
|
||||
4. Add tests in `tests/`
|
||||
|
||||
### New source type converter
|
||||
1. Create `src/skill_seekers/cli/{type}_scraper.py` — for document-shaped sources inherit `DocumentSkillBuilder` (categorization/references/index/SKILL.md come free; implement `extract()` + hooks), otherwise inherit `SkillConverter` and implement `extract()` and `build_skill()`. Set `SOURCE_TYPE`.
|
||||
2. Register in `CONVERTER_REGISTRY` in `skill_converter.py` — this also makes the type work in unified configs automatically (UnifiedScraper engine)
|
||||
3. Add source type config building in `create_command.py:_build_config()`
|
||||
4. Add auto-detection in `source_detector.py`
|
||||
5. Add optional dep if needed
|
||||
6. Add tests
|
||||
|
||||
### New CLI argument
|
||||
- Subcommand flag: define ONLY in the central parser class (`parsers/{cmd}_parser.py`) — module `main()` builds from it; the drift-guard test fails otherwise
|
||||
- Universal: `UNIVERSAL_ARGUMENTS` in `arguments/create.py`
|
||||
- Source-specific: appropriate dict (`WEB_ARGUMENTS`, `GITHUB_ARGUMENTS`, etc.)
|
||||
- Shared across scrapers: `add_all_standard_arguments()` in `arguments/common.py`
|
||||
@@ -0,0 +1,550 @@
|
||||
# Contributing to Skill Seeker
|
||||
|
||||
First off, thank you for considering contributing to Skill Seeker! It's people like you that make Skill Seeker such a great tool.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Branch Workflow](#branch-workflow)
|
||||
- [Code of Conduct](#code-of-conduct)
|
||||
- [How Can I Contribute?](#how-can-i-contribute)
|
||||
- [Development Setup](#development-setup)
|
||||
- [Pull Request Process](#pull-request-process)
|
||||
- [Coding Standards](#coding-standards)
|
||||
- [Testing](#testing)
|
||||
- [Documentation](#documentation)
|
||||
|
||||
---
|
||||
|
||||
## Branch Workflow
|
||||
|
||||
**⚠️ IMPORTANT:** Skill Seekers uses a two-branch workflow.
|
||||
|
||||
### Branch Structure
|
||||
|
||||
```
|
||||
main (production)
|
||||
↑
|
||||
│ (only maintainer merges)
|
||||
│
|
||||
development (integration) ← default branch for PRs
|
||||
↑
|
||||
│ (all contributor PRs go here)
|
||||
│
|
||||
feature branches
|
||||
```
|
||||
|
||||
### Branches
|
||||
|
||||
- **`main`** - Production branch
|
||||
- Always stable
|
||||
- Only receives merges from `development` by maintainers
|
||||
- Protected: requires tests + 1 review
|
||||
|
||||
- **`development`** - Integration branch
|
||||
- **Default branch for all PRs**
|
||||
- Active development happens here
|
||||
- Protected: requires tests to pass
|
||||
- Gets merged to `main` by maintainers
|
||||
|
||||
- **Feature branches** - Your work
|
||||
- Created from `development`
|
||||
- Named descriptively (e.g., `feature/123-add-github-scraping`)
|
||||
- Merged back to `development` via PR
|
||||
|
||||
### Workflow Example
|
||||
|
||||
```bash
|
||||
# 1. Fork and clone
|
||||
git clone https://github.com/YOUR_USERNAME/Skill_Seekers.git
|
||||
cd Skill_Seekers
|
||||
|
||||
# 2. Add upstream
|
||||
git remote add upstream https://github.com/yusufkaraaslan/Skill_Seekers.git
|
||||
|
||||
# 3. Create feature branch from development
|
||||
git checkout development
|
||||
git pull upstream development
|
||||
git checkout -b my-feature
|
||||
|
||||
# 4. Make changes, commit, push
|
||||
git add .
|
||||
git commit -m "Add my feature"
|
||||
git push origin my-feature
|
||||
|
||||
# 5. Create PR targeting 'development' branch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Repositories
|
||||
|
||||
Skill Seekers spans multiple repositories. Make sure you're contributing to the right one:
|
||||
|
||||
| What you want to work on | Repository |
|
||||
|--------------------------|-----------|
|
||||
| Core CLI, scrapers, MCP tools, adaptors | [Skill_Seekers](https://github.com/yusufkaraaslan/Skill_Seekers) (this repo) |
|
||||
| Website, docs, UI/UX | [skillseekersweb](https://github.com/yusufkaraaslan/skillseekersweb) |
|
||||
| Preset configs, community configs | [skill-seekers-configs](https://github.com/yusufkaraaslan/skill-seekers-configs) |
|
||||
| GitHub Action integration | [skill-seekers-action](https://github.com/yusufkaraaslan/skill-seekers-action) |
|
||||
| Claude Code plugin | [skill-seekers-plugin](https://github.com/yusufkaraaslan/skill-seekers-plugin) |
|
||||
| Homebrew formula | [homebrew-skill-seekers](https://github.com/yusufkaraaslan/homebrew-skill-seekers) |
|
||||
|
||||
---
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
This project and everyone participating in it is governed by our commitment to fostering an open and welcoming environment. Please be respectful and constructive in all interactions.
|
||||
|
||||
---
|
||||
|
||||
## How Can I Contribute?
|
||||
|
||||
### Reporting Bugs
|
||||
|
||||
Before creating bug reports, please check the [existing issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues) to avoid duplicates.
|
||||
|
||||
When creating a bug report, include:
|
||||
- **Clear title and description**
|
||||
- **Steps to reproduce** the issue
|
||||
- **Expected behavior** vs actual behavior
|
||||
- **Screenshots** if applicable
|
||||
- **Environment details** (OS, Python version, etc.)
|
||||
- **Error messages** and stack traces
|
||||
|
||||
**Example:**
|
||||
```markdown
|
||||
**Bug:** MCP tool fails when config has no categories
|
||||
|
||||
**Steps to Reproduce:**
|
||||
1. Create config with empty categories: `"categories": {}`
|
||||
2. Run `skill-seekers create --config configs/test.json`
|
||||
3. See error
|
||||
|
||||
**Expected:** Should use auto-inferred categories
|
||||
**Actual:** Crashes with KeyError
|
||||
|
||||
**Environment:**
|
||||
- OS: Ubuntu 22.04
|
||||
- Python: 3.10.5
|
||||
- Version: 1.0.0
|
||||
```
|
||||
|
||||
### Suggesting Enhancements
|
||||
|
||||
Enhancement suggestions are tracked as [GitHub issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues).
|
||||
|
||||
Include:
|
||||
- **Clear title** describing the enhancement
|
||||
- **Detailed description** of the proposed functionality
|
||||
- **Use cases** that would benefit from this enhancement
|
||||
- **Examples** of how it would work
|
||||
- **Alternatives considered**
|
||||
|
||||
### Adding New Framework Configs
|
||||
|
||||
We welcome new framework configurations! To add one:
|
||||
|
||||
1. Create a config file in `configs/`
|
||||
2. Test it thoroughly with different page counts
|
||||
3. Submit a PR with:
|
||||
- The config file
|
||||
- Brief description of the framework
|
||||
- Test results (number of pages scraped, categories found)
|
||||
|
||||
**Example PR:**
|
||||
```markdown
|
||||
**Add Svelte Documentation Config**
|
||||
|
||||
Adds configuration for Svelte documentation (https://svelte.dev/docs).
|
||||
|
||||
- Config: `configs/svelte.json`
|
||||
- Tested with max_pages: 100
|
||||
- Successfully categorized: getting_started, components, api, advanced
|
||||
- Total pages available: ~150
|
||||
```
|
||||
|
||||
### Pull Requests
|
||||
|
||||
We actively welcome your pull requests!
|
||||
|
||||
**⚠️ IMPORTANT:** All PRs must target the `development` branch, not `main`.
|
||||
|
||||
1. Fork the repo and create your branch from `development`
|
||||
2. If you've added code, add tests
|
||||
3. If you've changed APIs, update the documentation
|
||||
4. Ensure the test suite passes
|
||||
5. Make sure your code follows our coding standards
|
||||
6. Issue that pull request to `development` branch!
|
||||
|
||||
---
|
||||
|
||||
## Development Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.10 or higher (required for MCP integration)
|
||||
- Git
|
||||
|
||||
### Setup Steps
|
||||
|
||||
1. **Fork and clone the repository**
|
||||
```bash
|
||||
git clone https://github.com/YOUR_USERNAME/Skill_Seekers.git
|
||||
cd Skill_Seekers
|
||||
```
|
||||
|
||||
2. **Install dependencies**
|
||||
```bash
|
||||
pip install -e .
|
||||
pip install -e ".[dev]"
|
||||
pip install -e ".[all]"
|
||||
```
|
||||
|
||||
3. **Create a feature branch from development**
|
||||
```bash
|
||||
git checkout development
|
||||
git pull upstream development
|
||||
git checkout -b feature/my-awesome-feature
|
||||
```
|
||||
|
||||
4. **Make your changes**
|
||||
```bash
|
||||
# Edit files...
|
||||
```
|
||||
|
||||
5. **Run tests**
|
||||
```bash
|
||||
python -m pytest tests/ -v
|
||||
```
|
||||
|
||||
6. **Commit your changes**
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "Add awesome feature"
|
||||
```
|
||||
|
||||
7. **Push to your fork**
|
||||
```bash
|
||||
git push origin feature/my-awesome-feature
|
||||
```
|
||||
|
||||
8. **Create a Pull Request**
|
||||
|
||||
---
|
||||
|
||||
## Pull Request Process
|
||||
|
||||
### Before Submitting
|
||||
|
||||
- [ ] Tests pass locally (`python -m pytest tests/ -v`)
|
||||
- [ ] Code follows PEP 8 style guidelines
|
||||
- [ ] Documentation is updated if needed
|
||||
- [ ] CHANGELOG.md is updated (if applicable)
|
||||
- [ ] Commit messages are clear and descriptive
|
||||
|
||||
### PR Template
|
||||
|
||||
```markdown
|
||||
## Description
|
||||
Brief description of what this PR does.
|
||||
|
||||
## Type of Change
|
||||
- [ ] Bug fix (non-breaking change which fixes an issue)
|
||||
- [ ] New feature (non-breaking change which adds functionality)
|
||||
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
|
||||
- [ ] Documentation update
|
||||
|
||||
## How Has This Been Tested?
|
||||
Describe the tests you ran to verify your changes.
|
||||
|
||||
## Checklist
|
||||
- [ ] My code follows the style guidelines of this project
|
||||
- [ ] I have performed a self-review of my own code
|
||||
- [ ] I have commented my code, particularly in hard-to-understand areas
|
||||
- [ ] I have made corresponding changes to the documentation
|
||||
- [ ] My changes generate no new warnings
|
||||
- [ ] I have added tests that prove my fix is effective or that my feature works
|
||||
- [ ] New and existing unit tests pass locally with my changes
|
||||
```
|
||||
|
||||
### Review Process
|
||||
|
||||
1. A maintainer will review your PR within 3-5 business days
|
||||
2. Address any feedback or requested changes
|
||||
3. Once approved, a maintainer will merge your PR
|
||||
4. Your contribution will be included in the next release!
|
||||
|
||||
### Code Review Guidelines
|
||||
|
||||
**Fix both, don't follow precedent.** When a reviewer flags an anti-pattern in
|
||||
your PR, do not defend it by pointing at another file that does the same thing.
|
||||
"`X.py` already does this" is not a justification — it's evidence that two
|
||||
places need fixing, not that the smell is sanctioned.
|
||||
|
||||
The correct response is one of:
|
||||
- "Good catch — I'll fix both `new_file.py` and `existing_file.py` in this PR."
|
||||
- "Out of scope here, but I'll file a follow-up to fix `existing_file.py`."
|
||||
|
||||
The wrong response is:
|
||||
- "But `existing_file.py` does the same thing, so this matches the convention."
|
||||
|
||||
This rule cuts both ways: maintainers calling out an anti-pattern should be
|
||||
willing to either accept the broader fix or open the follow-up issue
|
||||
themselves. Bad-precedent-as-convention is how codebases ossify.
|
||||
|
||||
---
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### Python Style Guide
|
||||
|
||||
We follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) with some modifications:
|
||||
|
||||
- **Line length:** 100 characters (not 79)
|
||||
- **Indentation:** 4 spaces
|
||||
- **Quotes:** Double quotes for strings
|
||||
- **Naming:**
|
||||
- Functions/variables: `snake_case`
|
||||
- Classes: `PascalCase`
|
||||
- Constants: `UPPER_SNAKE_CASE`
|
||||
|
||||
### Code Organization
|
||||
|
||||
```python
|
||||
# 1. Standard library imports
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 2. Third-party imports
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
# 3. Local application imports
|
||||
from cli.utils import open_folder
|
||||
|
||||
# 4. Constants
|
||||
MAX_PAGES = 1000
|
||||
DEFAULT_RATE_LIMIT = 0.5
|
||||
|
||||
# 5. Functions and classes
|
||||
def my_function():
|
||||
"""Docstring describing what this function does."""
|
||||
pass
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
- All functions should have docstrings
|
||||
- Use type hints where appropriate
|
||||
- Add comments for complex logic
|
||||
|
||||
```python
|
||||
def scrape_page(url: str, selectors: dict) -> dict:
|
||||
"""
|
||||
Scrape a single page and extract content.
|
||||
|
||||
Args:
|
||||
url: The URL to scrape
|
||||
selectors: Dictionary of CSS selectors
|
||||
|
||||
Returns:
|
||||
Dictionary containing extracted content
|
||||
|
||||
Raises:
|
||||
RequestException: If page cannot be fetched
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
### Code Quality Tools
|
||||
|
||||
We use **Ruff** for linting and code formatting. Ruff is a fast Python linter that combines multiple tools (Flake8, isort, Black, etc.) into one.
|
||||
|
||||
**Running Ruff:**
|
||||
|
||||
```bash
|
||||
# Check for linting errors
|
||||
uvx ruff check src/ tests/
|
||||
|
||||
# Auto-fix issues
|
||||
uvx ruff check --fix src/ tests/
|
||||
|
||||
# Format code
|
||||
uvx ruff format src/ tests/
|
||||
```
|
||||
|
||||
**Common Ruff Rules:**
|
||||
- **SIM102** - Simplify nested if statements (use `and` instead)
|
||||
- **SIM117** - Combine multiple `with` statements
|
||||
- **B904** - Use `from e` for proper exception chaining
|
||||
- **SIM113** - Use enumerate instead of manual counters
|
||||
- **B007** - Use `_` for unused loop variables
|
||||
- **ARG002** - Remove unused function arguments
|
||||
|
||||
**CI/CD Integration:**
|
||||
|
||||
All pull requests automatically run:
|
||||
1. `ruff check` - Linting validation
|
||||
2. `ruff format --check` - Format validation
|
||||
3. `pytest` - Test suite
|
||||
|
||||
Make sure all checks pass before submitting your PR:
|
||||
|
||||
```bash
|
||||
# Run the same checks as CI
|
||||
uvx ruff check src/ tests/
|
||||
uvx ruff format --check src/ tests/
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python -m pytest tests/ -v
|
||||
|
||||
# Run specific test file
|
||||
python -m pytest tests/test_mcp_server.py -v
|
||||
|
||||
# Run with coverage
|
||||
python -m pytest tests/ --cov=src/skill_seekers --cov-report=term
|
||||
```
|
||||
|
||||
### Writing Tests
|
||||
|
||||
- Tests go in the `tests/` directory
|
||||
- Test files should start with `test_`
|
||||
- Use descriptive test names
|
||||
|
||||
```python
|
||||
def test_config_validation_with_missing_fields():
|
||||
"""Test that config validation fails when required fields are missing."""
|
||||
config = {"name": "test"} # Missing base_url
|
||||
result = validate_config(config)
|
||||
assert result is False
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
|
||||
- Aim for >80% code coverage
|
||||
- Critical paths should have 100% coverage
|
||||
- Add tests for bug fixes to prevent regressions
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
### Where to Document
|
||||
|
||||
- **README.md** - Overview, quick start, basic usage
|
||||
- **docs/** - Detailed guides and tutorials
|
||||
- **CHANGELOG.md** - All notable changes
|
||||
- **Code comments** - Complex logic and non-obvious decisions
|
||||
|
||||
### Documentation Style
|
||||
|
||||
- Use clear, simple language
|
||||
- Include code examples
|
||||
- Add screenshots for UI-related features
|
||||
- Keep it up to date with code changes
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
Skill_Seekers/
|
||||
├── src/skill_seekers/ # Main package (src/ layout)
|
||||
│ ├── cli/ # CLI commands and entry points
|
||||
│ │ ├── main.py # Unified CLI entry (COMMAND_MODULES dict)
|
||||
│ │ ├── source_detector.py # Auto-detects source type
|
||||
│ │ ├── create_command.py # Unified `create` command routing
|
||||
│ │ ├── config_validator.py # VALID_SOURCE_TYPES set
|
||||
│ │ ├── unified_scraper.py # Multi-source orchestrator
|
||||
│ │ ├── unified_skill_builder.py # Pairwise synthesis + generic merge
|
||||
│ │ ├── doc_scraper.py # Documentation (web)
|
||||
│ │ ├── github_scraper.py # GitHub repos
|
||||
│ │ ├── pdf_scraper.py # PDF files
|
||||
│ │ ├── word_scraper.py # Word (.docx)
|
||||
│ │ ├── epub_scraper.py # EPUB books
|
||||
│ │ ├── video_scraper.py # Video (YouTube, Vimeo, local)
|
||||
│ │ ├── codebase_scraper.py # Local codebases
|
||||
│ │ ├── jupyter_scraper.py # Jupyter Notebooks
|
||||
│ │ ├── html_scraper.py # Local HTML files
|
||||
│ │ ├── openapi_scraper.py # OpenAPI/Swagger specs
|
||||
│ │ ├── asciidoc_scraper.py # AsciiDoc files
|
||||
│ │ ├── pptx_scraper.py # PowerPoint files
|
||||
│ │ ├── rss_scraper.py # RSS/Atom feeds
|
||||
│ │ ├── manpage_scraper.py # Man pages
|
||||
│ │ ├── confluence_scraper.py # Confluence wikis
|
||||
│ │ ├── notion_scraper.py # Notion pages
|
||||
│ │ ├── chat_scraper.py # Slack/Discord exports
|
||||
│ │ ├── adaptors/ # Platform adaptors (Strategy pattern)
|
||||
│ │ ├── arguments/ # CLI argument definitions (one per source)
|
||||
│ │ ├── parsers/ # Subcommand parsers (one per source)
|
||||
│ │ └── storage/ # Cloud storage adaptors
|
||||
│ ├── services/ # Shared domain logic (marketplace, config publishing, git sources) — used by CLI and MCP
|
||||
│ ├── mcp/ # MCP server + tools (in-process; thin layer over cli/ + services/)
|
||||
│ └── sync/ # Sync monitoring
|
||||
├── configs/ # Preset JSON scraping configs
|
||||
├── docs/ # Documentation
|
||||
├── tests/ # 115+ test files (pytest)
|
||||
└── .github/ # GitHub config
|
||||
└── workflows/ # CI/CD workflows
|
||||
```
|
||||
|
||||
**Scraper pattern (18 source types):** Each source type is a `SkillConverter` subclass in `cli/<type>_scraper.py` (document-shaped sources subclass `DocumentSkillBuilder`, which provides the whole build side), reached via `skill-seekers create` auto-detection — there is no per-type `main()`. Register new types in: `CONVERTER_REGISTRY` in `skill_converter.py` (this also enables them in unified multi-source configs), `create_command.py:_build_config()`, `source_detector.py`, and `config_validator.py` VALID_SOURCE_TYPES. CLI flags are defined once in the central `parsers/*.py` classes (a drift-guard test enforces this).
|
||||
|
||||
### UML Architecture
|
||||
|
||||
Full UML class diagrams are maintained in StarUML and synced from source code:
|
||||
|
||||
- **[docs/UML_ARCHITECTURE.md](docs/UML_ARCHITECTURE.md)** - Overview with embedded PNG diagrams
|
||||
- **[docs/UML/skill_seekers.mdj](docs/UML/skill_seekers.mdj)** - StarUML project (open with [StarUML](https://staruml.io/))
|
||||
- **[docs/UML/exports/](docs/UML/exports/)** - 14 PNG exports (package overview + 13 class diagrams)
|
||||
- **[docs/UML/html/](docs/UML/html/index.html/index.html)** - HTML API reference
|
||||
|
||||
**Key design patterns documented in UML:**
|
||||
- Strategy + Factory in Adaptors (SkillAdaptor ABC + 20+ implementations)
|
||||
- Strategy + Factory in Storage (BaseStorageAdaptor + S3/GCS/Azure)
|
||||
- Template Method in Parsers (SubcommandParser + 28 subclasses)
|
||||
- Template Method in Analysis (BasePatternDetector + 10 GoF detectors)
|
||||
- Command pattern in CLI (CLIDispatcher + COMMAND_MODULES lazy dispatch)
|
||||
|
||||
When adding new classes or modules, please update the corresponding UML diagram to keep architecture docs in sync.
|
||||
|
||||
---
|
||||
|
||||
## Release Process
|
||||
|
||||
Releases are managed by maintainers:
|
||||
|
||||
1. Update version in relevant files
|
||||
2. Update CHANGELOG.md
|
||||
3. Create and push version tag
|
||||
4. GitHub Actions will create the release
|
||||
5. Announce on relevant channels
|
||||
|
||||
---
|
||||
|
||||
## Questions?
|
||||
|
||||
- 💬 [Open a discussion](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)
|
||||
- 🐛 [Report a bug](https://github.com/yusufkaraaslan/Skill_Seekers/issues)
|
||||
- 📧 Contact: yusufkaraaslan.yk@pm.me
|
||||
|
||||
---
|
||||
|
||||
## Recognition
|
||||
|
||||
Contributors will be recognized in:
|
||||
- README.md contributors section
|
||||
- CHANGELOG.md for each release
|
||||
- GitHub contributors page
|
||||
|
||||
Thank you for contributing to Skill Seeker! 🎉
|
||||
@@ -0,0 +1,531 @@
|
||||
# Skill Seekers Documentation Audit Report
|
||||
**Date:** 2026-05-30
|
||||
**Project Version:** 3.6.0 (from `pyproject.toml`)
|
||||
**Auditor:** Agent Swarm Analysis
|
||||
**Scope:** All non-generated `.md` files (excludes `output/`, `.skillseeker-cache/`, virtualenvs)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The Skill Seekers project has **~163 non-generated markdown files** across the repository. While the codebase is actively maintained at **v3.6.0**, the documentation ecosystem suffers from **severe version drift**, **widespread stale CLI references**, **broken internal links**, **incomplete translations**, **near-duplicate files**, and **significant gaps between claimed and actual features**.
|
||||
|
||||
### Severity Distribution
|
||||
| Severity | Count | Description |
|
||||
|----------|-------|-------------|
|
||||
| **CRITICAL** | 45+ | Wrong commands, broken links, missing sections, incorrect versions |
|
||||
| **WARNING** | 60+ | Stale counts, truncated content, minor inaccuracies, deprecated paths |
|
||||
| **INFO** | 35+ | Missing features from docs, structural suggestions, cosmetic issues |
|
||||
|
||||
### Key Finding
|
||||
> **The English README.md itself is outdated (claims v3.5.0, shows 14 removed CLI subcommands, has 11 broken internal links), and ALL 11 translations lag even further behind (claiming v3.2.0). The documentation as a whole gives new users a systematically incorrect understanding of how to use the tool.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Version Consistency Crisis
|
||||
|
||||
Version numbers are **completely inconsistent** across the documentation ecosystem:
|
||||
|
||||
| Location | Claims | Actual | Status |
|
||||
|----------|--------|--------|--------|
|
||||
| `pyproject.toml` | — | **3.6.0** | ✅ Source of truth |
|
||||
| `AGENTS.md` | 3.6.0 | 3.6.0 | ✅ Correct |
|
||||
| `README.md` (English) | 3.5.0 | 3.6.0 | ❌ Stale |
|
||||
| **All 11 README translations** | 3.2.0 | 3.6.0 | ❌ **Very stale** |
|
||||
| `docs/README.md` | 3.2.0 | 3.6.0 | ❌ Stale |
|
||||
| `docs/getting-started/*.md` | 3.1.0–3.2.0 | 3.6.0 | ❌ Stale |
|
||||
| `docs/user-guide/*.md` | 3.1.0–3.2.0 | 3.6.0 | ❌ Stale |
|
||||
| `docs/guides/MCP_SETUP.md` | 2.4.0 | 3.6.0 | ❌ Very stale |
|
||||
| `docs/reference/MCP_REFERENCE.md` | 3.5.0 | 3.6.0 | ❌ Slightly stale |
|
||||
| `docs/advanced/mcp-server.md` | 3.2.0 | 3.6.0 | ❌ Stale |
|
||||
| `ROADMAP.md` | 3.2.0 | 3.6.0 | ❌ Stale |
|
||||
| `QWEN.md` | 3.3.0 | 3.6.0 | ❌ Stale |
|
||||
| `integrations/INTEGRATIONS.md` | 2.10.0+ | 3.6.0 | ❌ Very stale |
|
||||
| `api/configs_repo/STATUS.md` | 2.11.0+ | 3.6.0 | ❌ Very stale |
|
||||
|
||||
**Impact:** New users cannot determine which version of the tool they are reading documentation for. Features documented at v3.1.0 may have changed significantly by v3.6.0.
|
||||
|
||||
---
|
||||
|
||||
## 2. README.md (English) — Deep Analysis
|
||||
|
||||
### 2.1 CRITICAL Issues
|
||||
|
||||
#### A. Broken / Outdated Internal Links (11 links)
|
||||
| Link in README | Actual Location | Status |
|
||||
|---|---|---|
|
||||
| `ASYNC_SUPPORT.md` | **Does not exist anywhere** | ❌ Dead |
|
||||
| `docs/ENHANCEMENT_MODES.md` | `docs/features/ENHANCEMENT_MODES.md` | ❌ Wrong path |
|
||||
| `docs/FEATURE_MATRIX.md` | `docs/reference/FEATURE_MATRIX.md` | ❌ Wrong path |
|
||||
| `docs/GIT_CONFIG_SOURCES.md` | `docs/reference/GIT_CONFIG_SOURCES.md` | ❌ Wrong path |
|
||||
| `docs/HOW_TO_GUIDES.md#ai-enhancement-new` | `docs/features/HOW_TO_GUIDES.md` | ❌ Wrong path |
|
||||
| `docs/IMPLEMENTATION_SUMMARY_THREE_STREAM.md` | `docs/archive/historical/IMPLEMENTATION_SUMMARY_THREE_STREAM.md` | ❌ Wrong path |
|
||||
| `docs/LARGE_DOCUMENTATION.md` | `docs/reference/LARGE_DOCUMENTATION.md` | ❌ Wrong path |
|
||||
| `docs/MCP_SETUP.md` | `docs/guides/MCP_SETUP.md` | ❌ Wrong path |
|
||||
| `docs/QUICK_REFERENCE.md` | `docs/archive/legacy/QUICK_REFERENCE.md` | ❌ Wrong path |
|
||||
| `docs/UNIFIED_SCRAPING.md` | `docs/features/UNIFIED_SCRAPING.md` | ❌ Wrong path |
|
||||
| `QUICKSTART.md` | `docs/archive/legacy/QUICKSTART.md` | ❌ Wrong path |
|
||||
|
||||
#### B. Non-Existent CLI Subcommands (14 removed commands still shown)
|
||||
The README displays **legacy subcommands that were removed** in the unified CLI redesign:
|
||||
|
||||
| Invalid Command Shown | Correct Modern Command |
|
||||
|---|---|
|
||||
| `skill-seekers scrape --config ...` | `skill-seekers create --config ...` |
|
||||
| `skill-seekers scrape --url ...` | `skill-seekers create https://...` |
|
||||
| `skill-seekers video --url ...` | `skill-seekers create ... --video-url ...` |
|
||||
| `skill-seekers video --setup` | `skill-seekers create --setup` |
|
||||
| `skill-seekers pdf --pdf ...` | `skill-seekers create ... --pdf ...` |
|
||||
| `skill-seekers github --repo ...` | `skill-seekers create ...` (auto-detect) |
|
||||
| `skill-seekers unified --config ...` | `skill-seekers create configs/..._unified.json` |
|
||||
| `skill-seekers confluence --space ...` | `skill-seekers create ... --space-key ...` |
|
||||
| `skill-seekers notion --database-id ...` | `skill-seekers create ... --database-id ...` |
|
||||
| `skill-seekers chat --export-dir ...` | `skill-seekers create ... --chat-export-path ...` |
|
||||
| `skill-seekers list-configs` | **Does not exist** |
|
||||
| `skill-seekers analyze --directory ...` | **Does not exist** |
|
||||
|
||||
#### C. Wrong CLI Flags in Examples
|
||||
- `skill-seekers video --url ...` → should use `--video-url`
|
||||
- `skill-seekers video --playlist ...` → should use `--video-playlist`
|
||||
- `skill-seekers confluence --space TEAM` → should use `--space-key`
|
||||
- `skill-seekers chat --export-dir ./slack-export` → should use `--chat-export-path`
|
||||
- `skill-seekers package ... --format chroma/faiss/qdrant` → should use `--target`
|
||||
|
||||
#### D. Inconsistent Test Counts
|
||||
- Badge: `3194+ Passing`
|
||||
- "Quality Assurance" prose: `2,540+ tests`
|
||||
- Actual codebase: ~3,462 test functions
|
||||
- **Three different numbers cited in different places**
|
||||
|
||||
#### E. Missing Major Features
|
||||
The README **omits** these significant v3.x features:
|
||||
- `--preset` (`quick` / `standard` / `comprehensive`)
|
||||
- `--dry-run`, `--fresh`, `--resume`, `--skip-scrape`
|
||||
- `--chunk-for-rag`, `--chunk-tokens`, `--chunk-overlap-tokens`
|
||||
- `--streaming`, `--marketplace`, `--marketplace-category`
|
||||
- `skill-seekers doctor` (health check)
|
||||
- `skill-seekers sync-config` (config drift detection)
|
||||
- `skill-seekers stream` (streaming ingestion)
|
||||
- `skill-seekers update` (incremental updater)
|
||||
- `skill-seekers multilang` (multi-language support)
|
||||
- `skill-seekers quality` (quality scoring)
|
||||
- `skill-seekers extract-test-examples`
|
||||
|
||||
### 2.2 WARNING Issues
|
||||
- MCP tool count: README says 26, AGENTS.md says 40, actual is 40
|
||||
- LLM platform comparison table shows only 5 of ~21 supported platforms
|
||||
- Install-agent names: README says `Kimi` but CLI uses `kimi-code`
|
||||
- `--target claude` description misleadingly claims auto-copy for Cursor/Windsurf
|
||||
|
||||
---
|
||||
|
||||
## 3. Translation Analysis (All 11 READMEs)
|
||||
|
||||
### 3.1 Summary Table
|
||||
|
||||
| Translation | Headings | Version | Test Badge | Architecture Section | Scan Section | IBM Bob | MiniMax | Status |
|
||||
|-------------|----------|---------|------------|----------------------|--------------|---------|---------|--------|
|
||||
| English | 210 | 3.5.0 | 3194+ | ✅ | ✅ | ✅ | ✅ | ⚠️ Stale |
|
||||
| **zh-CN** | 165 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **ja** | 165 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **ko** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **de** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **es** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **fr** | 196 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **pt-BR** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **ru** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **tr** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **ar** | 166 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
| **hi** | 195 | **3.2.0** | **2540+** | ❌ Missing | ❌ Missing | ❌ Missing | ❌ Missing | 🔴 Very stale |
|
||||
|
||||
### 3.2 Universal Translation Deficiencies
|
||||
**Every single translation** is missing these major sections that exist in English:
|
||||
- `## 📚 Documentation` (early quick-link table)
|
||||
- `## Architecture` (module overview + UML links)
|
||||
- `### 🛰️ AI-driven project scan (new)`
|
||||
- `### 🤖 Agent-Agnostic Skill Generation`
|
||||
- `### 📦 Marketplace Pipeline`
|
||||
- `IBM Bob` support (agent count says 18 instead of 19)
|
||||
- `MiniMax AI` platform comparison row and examples
|
||||
- `pepy.tech` and `Trendshift` badges
|
||||
|
||||
### 3.3 Per-Translation Nuances
|
||||
- **zh-CN, ja, ko** (CJK): Most severely truncated. Installation table has 13 rows vs English 15. Performance table missing Video rows. Smart Rate Limit Management heavily truncated.
|
||||
- **ko** is the *best* CJK translation: includes Security section at bottom, has correct 15-row install table, includes Video performance rows.
|
||||
- **fr, tr** (European): Include the early Documentation table (most others don't). Otherwise similar gaps.
|
||||
- **hi** (Devanagari): Most complete non-European translation. Has all 5 print statements in Three-Stream example. Bootstrap "What you get" bullets present. Good technical transliteration.
|
||||
- **ar** (Arabic): Most truncated. Three-Stream example has only 2 print statements. Missing Rate Limit Strategies Explained. Some bidi rendering issues in code comments.
|
||||
- **ru** (Cyrillic): Has some untranslated English fragments ("OpenAPI/Swagger", "README"). Uses Latin "API"/"LOCAL" inconsistently.
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Project Documentation
|
||||
|
||||
### 4.1 AGENTS.md — ✅ MOSTLY ACCURATE
|
||||
- Version 3.6.0 ✅
|
||||
- Source types count (17 + config) ✅
|
||||
- 40 MCP tools ✅
|
||||
- Minor: Claims "~14 legacy commands" but actual is 16
|
||||
|
||||
### 4.2 CLAUDE.md — ✅ ACCURATE
|
||||
- Architecture descriptions match codebase
|
||||
- File paths correct
|
||||
- 24 adaptor files match actual
|
||||
- Minor: Uses `uvx ruff` while AGENTS.md uses `ruff`
|
||||
|
||||
### 4.3 CHANGELOG.md — ✅ ACCURATE
|
||||
- Latest version [3.6.0] correctly listed
|
||||
- `[Unreleased]` section holds post-3.6.0 changes appropriately
|
||||
- Format consistent
|
||||
|
||||
### 4.4 ROADMAP.md — 🔴 SEVERELY OUTDATED
|
||||
- Claims `v3.2.0` is current (actual: 3.6.0)
|
||||
- Lists `v2.7.0`, `v2.8.0`, `v2.9.0` as future releases
|
||||
- **Completed tasks NOT checked off**:
|
||||
- A1.3 `submit_config` MCP tool — EXISTS
|
||||
- C3.3 "Build how-to guides" — `how_to_guide_builder.py` EXISTS
|
||||
- C3.4 "Extract configuration patterns" — `config_extractor.py` EXISTS
|
||||
- C3.5 "Create architectural overview" — `generate_router.py` EXISTS
|
||||
- E1.1 `fetch_config` MCP tool — EXISTS
|
||||
- Wrong metrics: "1,880+ tests" (actual: ~3,445), "24 preset configs" (actual: 12), "5 bundled workflows" (actual: 68 YAML files)
|
||||
|
||||
### 4.5 TROUBLESHOOTING.md — 🔴 BROKEN PATHS
|
||||
- Uses **pre-`src/` layout paths** that no longer exist:
|
||||
- `cli/doc_scraper.py` (×3)
|
||||
- `python3 mcp/server.py` (×2)
|
||||
- `pip3 install -r mcp/requirements.txt` — file does not exist
|
||||
- **Missing critical step**: Never mentions `pip install -e .` which `conftest.py` hard-requires
|
||||
|
||||
### 4.6 CONTRIBUTING.md — 🔴 WRONG SETUP
|
||||
- Tells contributors: `pip install requests beautifulsoup4` and `pip install -r mcp/requirements.txt`
|
||||
- **Should say**: `pip install -e .` / `pip install -e ".[dev]"` (as AGENTS.md states)
|
||||
- Coverage paths wrong: `--cov=cli --cov=mcp` should be `--cov=src/skill_seekers`
|
||||
- **Contradicts AGENTS.md**: Describes pre-commit hook setup, but AGENTS.md says "**No pre-commit hooks, no Makefile**"
|
||||
- Branch naming: uses `feature/my-awesome-feature` without task ID; AGENTS.md specifies `feature/{task-id}-{description}`
|
||||
|
||||
### 4.7 QWEN.md — 🔴 REDUNDANT & OUTDATED
|
||||
- Version says `v3.3.0` (actual: 3.6.0)
|
||||
- Claims 26+ MCP tools (actual: 40)
|
||||
- Shows `skill-seekers package --target langchain` — `langchain` is a `--format` value, not `--target`
|
||||
- Shows `--target cursor` — not a valid standalone `--target`
|
||||
- **Omits `scan` and `doctor` entirely**
|
||||
- Stale numbers: 67 YAML presets (actual: 68), 123 test files (actual: 143)
|
||||
- **Recommendation**: Update to 3.6.0 or remove (largely duplicates CLAUDE.md/AGENTS.md)
|
||||
|
||||
---
|
||||
|
||||
## 5. docs/ Directory Analysis
|
||||
|
||||
### 5.1 Structural Issues
|
||||
|
||||
#### A. docs/README.md Omits ~40% of Actual Content
|
||||
The docs hub README lists only a subset of directories. It **omits**:
|
||||
- `guides/` (7 files)
|
||||
- `integrations/` (15 files)
|
||||
- `features/` (5 files)
|
||||
- `archive/`, `blog/`, `case-studies/`, `plans/`, `roadmap/`, `strategy/`, `agents/`, `superpowers/`
|
||||
- `zh-CN/` (entire translation tree)
|
||||
- `UML/`
|
||||
|
||||
#### B. Phantom Link
|
||||
`docs/README.md` links to `advanced/mcp-tools.md` — **does not exist**. Only `advanced/mcp-server.md` exists.
|
||||
|
||||
### 5.2 Near-Duplicate File Pairs
|
||||
|
||||
**Pair 1:** `DOCKER_GUIDE.md` (575 lines) ↔ `DOCKER_DEPLOYMENT.md` (762 lines)
|
||||
- Both cover Docker deployment, Compose, volumes, networking, monitoring, troubleshooting
|
||||
- `DOCKER_DEPLOYMENT.md` is more detailed (adds embedding server, sync monitor, Prometheus/Grafana/Loki)
|
||||
- `DOCKER_GUIDE.md` has broken link: `[Vector Database Integration](docs/strategy/WEEK2_COMPLETE.md)` — `WEEK2_COMPLETE.md` does not exist
|
||||
- **Recommendation**: Merge into single file
|
||||
|
||||
**Pair 2:** `KUBERNETES_GUIDE.md` (957 lines) ↔ `KUBERNETES_DEPLOYMENT.md` (933 lines)
|
||||
- Both cover Helm charts, manual deployment, scaling, HA, monitoring, security
|
||||
- `KUBERNETES_DEPLOYMENT.md` adds Velero backup/restore, cost optimization
|
||||
- `KUBERNETES_GUIDE.md` uses `skillseekers` namespace; `KUBERNETES_DEPLOYMENT.md` uses `skill-seekers`
|
||||
- **Recommendation**: Merge into single file
|
||||
|
||||
### 5.3 Partial Duplication: Troubleshooting
|
||||
- `user-guide/06-troubleshooting.md` (~200 lines, v3.1.0)
|
||||
- Top-level `TROUBLESHOOTING.md` (1094 lines, no version stamp)
|
||||
- **Significant overlap** in installation, configuration, scraping, enhancement sections
|
||||
- **Recommendation**: Consolidate or clearly differentiate scope
|
||||
|
||||
### 5.4 Stale MCP Documentation
|
||||
Three different MCP docs claim three different tool counts:
|
||||
|
||||
| File | Claims | Actual |
|
||||
|------|--------|--------|
|
||||
| `guides/MCP_SETUP.md` | 26 tools / v2.4.0 | 40 tools / v3.6.0 |
|
||||
| `advanced/mcp-server.md` | 27 tools / v3.2.0 | 40 tools / v3.6.0 |
|
||||
| `reference/MCP_REFERENCE.md` | 40 tools / v3.5.0 | 40 tools / v3.6.0 |
|
||||
|
||||
`guides/MCP_SETUP.md` also claims MCP SDK v1.25.0 and miscounts tool categories (says 27 but lists 28).
|
||||
|
||||
### 5.5 Stale Guide Content
|
||||
- `guides/MIGRATION_GUIDE.md` header says `v3.1.0-dev` but discusses migrating **to v2.7.0** as "latest"
|
||||
- `integrations/INTEGRATIONS.md` says "Skill Seekers Version: v2.10.0+"
|
||||
- `integrations/LANGCHAIN.md` says "Skill Seekers Version: v2.9.0+"
|
||||
- `features/BOOTSTRAP_SKILL.md` says version 2.7.0 in frontmatter example
|
||||
|
||||
### 5.6 FEATURE_MATRIX Undercount
|
||||
Lists only 12 platforms but CLI supports 21+ packaging targets. Missing: DeepSeek, Qwen, OpenRouter, Together, Fireworks, IBM Bob, OpenCode, Kimi, and others.
|
||||
|
||||
---
|
||||
|
||||
## 6. docs/zh-CN/ Translation Quality — 🔴 CRITICAL FINDING
|
||||
|
||||
### 6.1 The Shocking Truth
|
||||
**The `docs/zh-CN/` directory does NOT contain Chinese translations.**
|
||||
|
||||
Out of **21 files audited**:
|
||||
- **0 files (0%)** are fully translated to Chinese
|
||||
- **1 file (~15%)** has partial Chinese (`advanced/mcp-server.md` — just section headers and some tool descriptions)
|
||||
- **20 files (95%)** are **English copies** with minor divergence
|
||||
|
||||
### 6.2 Content Drift Pattern
|
||||
The zh-CN files fall into three categories:
|
||||
|
||||
| Category | Files | Description |
|
||||
|----------|-------|-------------|
|
||||
| **zh-CN LAGS** | README, CLI_REFERENCE, API_REFERENCE, packaging | English docs are newer (v3.5.0 vs v3.2.0) |
|
||||
| **zh-CN LEADS** | ARCHITECTURE, core-concepts, scraping, CONFIG_FORMAT, workflows, MCP_REFERENCE, multi-source | zh-CN documents features not yet in English docs (17 sources, unified config, new platforms) |
|
||||
| **Identical** | 03-your-first-skill, 06-troubleshooting, SKILL_ARCHITECTURE, custom-workflows | Perfect copies |
|
||||
|
||||
### 6.3 Broken Link
|
||||
- `zh-CN/getting-started/04-next-steps.md` links to `05-scan-a-project.md` — **does not exist in zh-CN/** (only in `docs/getting-started/`)
|
||||
|
||||
### 6.4 Recommendation
|
||||
If the goal is Chinese-language documentation: **entire directory needs professional translation**.
|
||||
If the goal is parallel English documentation: **rename directory** (e.g., `docs/extended/`) to avoid misleading users.
|
||||
|
||||
---
|
||||
|
||||
## 7. Archive, Plans, Strategy & Roadmap Relevance
|
||||
|
||||
### 7.1 docs/archive/ — Mostly Appropriate
|
||||
|
||||
**Historical (8 files):** All are completed verification reports. Correctly archived. Could compress further.
|
||||
**Legacy (4 files + index):** Self-aware deprecated docs with redirects. Correctly archived.
|
||||
**Plans (2 files):** Active skills design from Oct 2025. Status unclear — **verify if implemented**.
|
||||
**Research (4 files):** PDF research from Oct 2025. Informed current implementation. Keep as historical record.
|
||||
|
||||
### 7.2 docs/plans/video/ — 🔴 MISLEADING
|
||||
All 8 files still say "Status: Planning" but **video support IS implemented** (`video_scraper.py`, `video_models.py`, `video_setup.py`, 60 tests).
|
||||
**Action:** Update headers to "Implemented" and verify code alignment. Consider moving to `docs/features/`.
|
||||
|
||||
### 7.3 docs/strategy/ — Mixed
|
||||
|
||||
| File | Verdict |
|
||||
|------|---------|
|
||||
| `README.md` | Update — integration guides WERE created (18 files) but doc still says "to be created" |
|
||||
| `ACTION_PLAN.md` | Archive — historical 4-week plan, partially executed |
|
||||
| `ARBITRARY_LIMITS_AND_DEAD_CODE_PLAN.md` | Update — Stage 1 completed, Stages 2-3 status unknown |
|
||||
| `INTEGRATION_STRATEGY.md` | Update — much executed, should reflect success |
|
||||
| `KIMI_ANALYSIS_COMPARISON.md` | Archive — analysis complete, strategy adopted |
|
||||
| `STAGE_1_*.md` (3 files) | Archive deeper — historical implementation records |
|
||||
| `DEEPWIKI_ANALYSIS.md`, `INTEGRATION_TEMPLATES.md` | Keep — still valuable |
|
||||
|
||||
### 7.4 docs/roadmap/ — ✅ APPROPRIATE (Future Design)
|
||||
Intelligence System roadmap (4 files, Jan 2026). No `skill_seekers/intelligence/` package exists yet. These are **future design docs** — correctly kept as research/planning.
|
||||
|
||||
### 7.5 docs/agents/ — Archive
|
||||
EPUB implementation plans (Mar 2026). EPUB scraper EXISTS and is tested (107 tests). These are completed agent-driven development records — **archive deeper**.
|
||||
|
||||
### 7.6 docs/superpowers/ — Verify
|
||||
Scrape-count and SPA detection plan (Mar 2026). References `doc_scraper.py` line numbers. **Verify if implemented** and update status.
|
||||
|
||||
---
|
||||
|
||||
## 8. Distribution & Examples
|
||||
|
||||
### 8.1 distribution/ — 🔴 BROKEN EXAMPLES
|
||||
|
||||
#### github-action/README.md
|
||||
- Lists CLI commands: `scrape`, `github`, `pdf`, `video`, `analyze`, `unified`
|
||||
- **None exist in v3.6.0**
|
||||
- Users copying these workflow examples will get `error: argument command: invalid choice`
|
||||
|
||||
#### claude-plugin/commands/install-skill.md
|
||||
- Lists targets: `cursor`, `windsurf`, `continue`, `cline`
|
||||
- **Not registered adaptors** — will fail
|
||||
|
||||
#### claude-plugin/README.md & skills/skill-builder/SKILL.md
|
||||
- Claims "35 MCP tools" — actual is **40**
|
||||
|
||||
### 8.2 examples/ — 🔴 ALL EXAMPLES USE REMOVED COMMANDS
|
||||
|
||||
**Every example README** (chroma, cline, continue-dev, cursor, haystack, langchain, llama-index, pinecone, weaviate, windsurf) references:
|
||||
- `skill-seekers scrape --config ...` → should be `skill-seekers create --config ...`
|
||||
- `skill-seekers github --repo ...` → should be `skill-seekers create ...`
|
||||
|
||||
#### windsurf-fastapi-context/
|
||||
- References `--split-rules` and `--max-chars` flags on `skill-seekers package`
|
||||
- **These flags do not exist**
|
||||
|
||||
#### Other Example Issues
|
||||
- Many reference `skill-seekers v2.10.0` — current is 3.6.0
|
||||
- `cursor-react-skill/README.md`: Uses `--target claude` for Cursor. Cursor uses `.cursorrules` (plain markdown); `--target markdown` would be more appropriate
|
||||
- `continue-dev-universal/README.md`: References `~/.continue/config.json` for HTTP context providers. Continue.dev has shifted toward YAML config and MCP servers
|
||||
|
||||
### 8.3 api/configs_repo/ — 🔴 SEVERELY STALE
|
||||
|
||||
#### STATUS.md
|
||||
- "Last Updated: 2024-12-21"
|
||||
- "Total Configs: 90 unified format"
|
||||
- "Format Version: Unified (v2.11.0+)"
|
||||
- Actual: v3.6.0, submodule shows 22+ official categories
|
||||
|
||||
#### Divergence Between Copies
|
||||
- `api/configs_repo/README.md`: Claims **178 configs / 21 categories / v3.1.0+**
|
||||
- `skill-seekers-configs/README.md` (root copy): Claims **24 configs / 7 categories**
|
||||
- **These wildly diverge**
|
||||
|
||||
#### CONTRIBUTING.md (configs_repo)
|
||||
- References `skill-seekers scrape --config` — command removed
|
||||
|
||||
### 8.4 skill-seekers-configs/ — 🔴 COUNTS DON'T ADD UP
|
||||
|
||||
All TODO files claim `✅ COMPLETE` but counts are contradictory:
|
||||
|
||||
| File | Claims | Actually Lists |
|
||||
|------|--------|----------------|
|
||||
| `TODO-ai-ml.md` | 40 configs | 34 items |
|
||||
| `TODO-build-tools.md` | 9 configs | ~15 items |
|
||||
| `TODO-cloud.md` | 10 configs | ~15 items |
|
||||
| `TODO-databases.md` | 9 configs | ~20 items |
|
||||
| `TODO-development-tools.md` | 6 configs | ~12 items |
|
||||
| `TODO-devops.md` | 7 configs | ~12 items |
|
||||
| `TODO-game-engines.md` | 7 configs | ~20 items |
|
||||
| `TODO-testing.md` | 4 configs | ~10 items |
|
||||
| `TODO-web-frameworks.md` | 12 configs | ~18 items |
|
||||
|
||||
---
|
||||
|
||||
## 9. Tests, Scripts, Skills, src docs
|
||||
|
||||
### 9.1 tests/mcp_integration_test.md — 🔴 BROKEN PATHS
|
||||
- `python3 mcp/server.py` → should be `python -m skill_seekers.mcp.server_fastmcp`
|
||||
- `pip3 install -r mcp/requirements.txt` → file doesn't exist
|
||||
- `python3 cli/doc_scraper.py`, `python3 cli/package_skill.py` → pre-`src/` layout
|
||||
|
||||
### 9.2 scripts/skill_header.md — 🔴 DEPRECATED FLAGS
|
||||
- References `--depth surface/deep/full` → **deprecated** (use `--preset`)
|
||||
- References `--ai-mode none/api/local` → **does not exist** in current CLI
|
||||
|
||||
### 9.3 skills/skill-seekers/SKILL.md — ⚠️ STALE COUNT
|
||||
- Claims **35 MCP tools** — actual is **40**
|
||||
- Otherwise well-structured
|
||||
|
||||
### 9.4 src/skill_seekers/mcp/README.md — 🔴 INCONSISTENT & BROKEN LINKS
|
||||
- Header says "34 tools", later says "40 tools" — inconsistent
|
||||
- References `docs/MCP_SETUP.md`, `docs/USAGE.md`, `docs/TESTING.md` — **none exist**
|
||||
- References `cli/doc_scraper.py`, `cli/estimate_pages.py`, `cli/package_skill.py` — pre-`src/` layout
|
||||
- Claims "34 tests | Pass rate: 100%" and "25 tests" in different sections
|
||||
|
||||
---
|
||||
|
||||
## 10. Missing Documentation
|
||||
|
||||
These features exist in the codebase but are **under-documented or undocumented** in user-facing docs:
|
||||
|
||||
| Feature | Code Evidence | Doc Status |
|
||||
|---------|---------------|------------|
|
||||
| `doctor` command | `doctor_command.py` | Not mentioned in README, getting-started, or user-guide |
|
||||
| `--preset` flag | Major UX feature in `create` | Mentioned in CLI_REFERENCE but not in README or getting-started |
|
||||
| `--dry-run`, `--fresh`, `--resume` | Lifecycle management | Undocumented in user guides |
|
||||
| `--chunk-for-rag`, `--chunk-tokens` | RAG chunking on `package` | Undocumented in packaging guides |
|
||||
| `--streaming` | Memory-efficient packaging | Undocumented |
|
||||
| `--marketplace`, `--marketplace-category` | Marketplace publishing | Undocumented |
|
||||
| `scan` command deep guide | `scan_command.py` | Brief mention only; no deep guide |
|
||||
| `video` source type deep guide | 7 `video_*.py` files | Mentioned but not deeply documented |
|
||||
| `epub` source type | `epub_scraper.py` (107 tests) | Unclear coverage in user guides |
|
||||
| `browser` extra dependency | Playwright for SPA sites | Not in installation table |
|
||||
| `embedding` extra | Embedding server support | Not in installation table |
|
||||
| Cloud extras (`s3`, `gcs`, `azure`) | Cloud storage upload | Not in installation table |
|
||||
| All 68 YAML workflow presets | `workflows/` directory | Only mentioned as "67" or "68" |
|
||||
|
||||
---
|
||||
|
||||
## 11. Unneeded / Redundant Documentation
|
||||
|
||||
| File(s) | Issue | Recommendation |
|
||||
|---------|-------|----------------|
|
||||
| `DOCKER_GUIDE.md` + `DOCKER_DEPLOYMENT.md` | Near-duplicates | Merge into one |
|
||||
| `KUBERNETES_GUIDE.md` + `KUBERNETES_DEPLOYMENT.md` | Near-duplicates | Merge into one |
|
||||
| `user-guide/06-troubleshooting.md` + `TROUBLESHOOTING.md` | Heavy overlap | Consolidate or differentiate scope |
|
||||
| `QWEN.md` | Duplicates CLAUDE.md/AGENTS.md, less accurate | Update or remove |
|
||||
| `docs/archive/historical/*` (8 files) | Historical artifacts | Compress or move to external wiki |
|
||||
| `docs/strategy/STAGE_1_*.md` (3 files) | Implementation records | Merge into one summary |
|
||||
| `docs/agents/*` (2 files) | Completed EPUB plans | Archive deeper |
|
||||
| `skill-seekers-configs/README.md` (root copy) | Wildly diverges from submodule | Remove or sync with `api/configs_repo/` |
|
||||
|
||||
---
|
||||
|
||||
## 12. Recommendations (Prioritized)
|
||||
|
||||
### P0 — Critical (Fix Immediately)
|
||||
1. **Update English README.md** to v3.6.0 and remove all 14 legacy CLI subcommand examples
|
||||
2. **Fix all 11 broken internal links** in README.md
|
||||
3. **Update or remove ALL 11 README translations** — they are 4 minor versions behind and missing major sections
|
||||
4. **Update ALL example READMEs** to use `skill-seekers create` instead of removed `scrape`/`github`/`video`/`pdf`/`unified` commands
|
||||
5. **Fix `distribution/github-action/README.md`** to use valid CLI commands
|
||||
6. **Update `ROADMAP.md`** version header, check off completed tasks, remove obsolete v2.x release planning
|
||||
7. **Rewrite `TROUBLESHOOTING.md`** for `src/` layout and modern CLI paths
|
||||
8. **Fix `CONTRIBUTING.md`** setup instructions to match AGENTS.md (`pip install -e .`)
|
||||
|
||||
### P1 — High (Fix Soon)
|
||||
9. **Unify version stamps** to 3.6.0 across all docs (or automate via CI build step)
|
||||
10. **Consolidate Docker and K8s duplicate files**
|
||||
11. **Update MCP tool counts** to consistently say **40** across all docs
|
||||
12. **Update `guides/MCP_SETUP.md`** to match current v3.6.0 reality
|
||||
13. **Update `integrations/INTEGRATIONS.md`** from v2.10.0+ to v3.6.0
|
||||
14. **Fix `tests/mcp_integration_test.md`** paths to `server_fastmcp.py`
|
||||
15. **Fix `src/skill_seekers/mcp/README.md`** broken links and inconsistent counts
|
||||
16. **Update `FEATURE_MATRIX.md`** to include all 21+ packaging targets
|
||||
17. **Update `scripts/skill_header.md`** to remove deprecated `--depth` and non-existent `--ai-mode`
|
||||
18. **Sync or remove `skill-seekers-configs/README.md`** (root copy)
|
||||
|
||||
### P2 — Medium (Quality Improvements)
|
||||
19. **Update `docs/README.md`** to reflect actual directory structure (add missing directories)
|
||||
20. **Update `docs/plans/video/*.md`** from "Planning" to "Implemented"
|
||||
21. **Update `docs/strategy/README.md`** and `INTEGRATION_STRATEGY.md` to show executed deliverables
|
||||
22. **Add missing features to README**: `--preset`, `--dry-run`, `doctor`, `sync-config`, `stream`, `update`, `multilang`, `quality`
|
||||
23. **Add missing optional deps** to installation table: `browser`, `embedding`, `s3`, `gcs`, `azure`, `rag-upload`
|
||||
24. **Add `scan` and `doctor` deep guides** to getting-started or user-guide
|
||||
25. **Update `guides/MIGRATION_GUIDE.md`** to discuss actual v3.5→v3.6 migration
|
||||
|
||||
### P3 — Low (Nice to Have)
|
||||
26. **Automate version stamping** in CI to prevent future drift
|
||||
27. **Archive completed agent plans** (EPUB, scrape-count) deeper or compress
|
||||
28. **Add documentation update checklist** to `.github/PULL_REQUEST_TEMPLATE.md`
|
||||
29. **Regenerate UML diagrams** to reflect scan feature and other recent changes
|
||||
30. **Consider professional translation** of `docs/zh-CN/` or rename to `docs/extended/` if not actually Chinese
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Files Audited by Area
|
||||
|
||||
| Area | Files Count | Key Finding |
|
||||
|------|-------------|-------------|
|
||||
| Root READMEs | 12 | All translations 4 versions behind |
|
||||
| Core docs | 7 | ROADMAP, TROUBLESHOOTING, CONTRIBUTING severely outdated |
|
||||
| docs/ main | ~86 | Version inconsistency, duplicate files, missing structure |
|
||||
| docs/zh-CN/ | 21 | **0% actually translated to Chinese** |
|
||||
| docs/archive/ | 18 | Appropriately archived, some could compress deeper |
|
||||
| docs/plans/video/ | 8 | Misleadingly labeled "Planning" — already implemented |
|
||||
| docs/strategy/ | 11 | Partially executed, needs update |
|
||||
| docs/roadmap/ | 4 | Future design docs — appropriately kept |
|
||||
| docs/integrations/ | 18 | Successfully created strategy deliverables |
|
||||
| distribution/ | 6 | GitHub Action uses removed commands |
|
||||
| examples/ | 13 | ALL use removed CLI commands |
|
||||
| api/configs_repo/ | 6 | STATUS.md severely stale |
|
||||
| skill-seekers-configs/ | 13 | Counts don't add up |
|
||||
| .github/ | 5 | Templates functional but could reference `doctor` |
|
||||
| tests/ | 1 | Broken paths to old MCP server |
|
||||
| scripts/ | 1 | Deprecated flags |
|
||||
| skills/ | 1 | Stale MCP tool count |
|
||||
| src/ | 1 | Broken links, inconsistent counts |
|
||||
|
||||
---
|
||||
|
||||
*End of Report*
|
||||
@@ -0,0 +1,75 @@
|
||||
# Skill Seekers - Multi-stage Docker Build
|
||||
# Optimized for production deployment with minimal image size
|
||||
|
||||
# Stage 1: Builder - Install dependencies and build
|
||||
FROM python:3.12-slim as builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Install build dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
gcc \
|
||||
g++ \
|
||||
git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy dependency files
|
||||
COPY pyproject.toml README.md ./
|
||||
COPY src/ src/
|
||||
|
||||
# Install dependencies and build package
|
||||
RUN pip install --no-cache-dir --upgrade pip uv && \
|
||||
uv pip install --system --no-cache -e . && \
|
||||
uv pip install --system --no-cache ".[all-llms]"
|
||||
|
||||
# Stage 2: Runtime - Minimal production image
|
||||
FROM python:3.12-slim
|
||||
|
||||
LABEL maintainer="Skill Seekers <noreply@skillseekers.dev>"
|
||||
LABEL description="Skill Seekers - Convert documentation to AI skills"
|
||||
LABEL version="2.9.0"
|
||||
|
||||
# Install runtime dependencies only
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
git \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Create non-root user
|
||||
RUN useradd -m -u 1000 -s /bin/bash skillseeker && \
|
||||
mkdir -p /app /data /configs /output && \
|
||||
chown -R skillseeker:skillseeker /app /data /configs /output
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy Python packages from builder
|
||||
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
|
||||
COPY --from=builder /usr/local/bin/skill-seekers* /usr/local/bin/
|
||||
|
||||
# Copy application code
|
||||
COPY --chown=skillseeker:skillseeker src/ src/
|
||||
COPY --chown=skillseeker:skillseeker configs/ configs/
|
||||
COPY --chown=skillseeker:skillseeker pyproject.toml README.md ./
|
||||
|
||||
# Switch to non-root user
|
||||
USER skillseeker
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PATH="/home/skillseeker/.local/bin:$PATH" \
|
||||
SKILL_SEEKERS_HOME=/data \
|
||||
SKILL_SEEKERS_OUTPUT=/output
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
|
||||
CMD skill-seekers --version || exit 1
|
||||
|
||||
# Default volumes
|
||||
VOLUME ["/data", "/configs", "/output"]
|
||||
|
||||
# Expose MCP server port (HTTP mode)
|
||||
EXPOSE 8765
|
||||
|
||||
# Default command - show help
|
||||
CMD ["skill-seekers", "--help"]
|
||||
@@ -0,0 +1,57 @@
|
||||
# Skill Seekers MCP Server - Docker Image
|
||||
# Optimized for MCP server deployment (stdio + HTTP modes)
|
||||
|
||||
FROM python:3.12-slim
|
||||
|
||||
LABEL maintainer="Skill Seekers <noreply@skillseekers.dev>"
|
||||
LABEL description="Skill Seekers MCP Server - 35 tools for AI skills generation"
|
||||
LABEL version="3.3.0"
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install runtime dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
git \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Create non-root user
|
||||
RUN useradd -m -u 1000 -s /bin/bash mcp && \
|
||||
mkdir -p /app /data /configs /output && \
|
||||
chown -R mcp:mcp /app /data /configs /output
|
||||
|
||||
# Copy application files
|
||||
COPY --chown=mcp:mcp src/ src/
|
||||
COPY --chown=mcp:mcp configs/ configs/
|
||||
COPY --chown=mcp:mcp pyproject.toml README.md ./
|
||||
|
||||
# Install dependencies
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e ".[all-llms]" && \
|
||||
pip install --no-cache-dir mcp
|
||||
|
||||
# Switch to non-root user
|
||||
USER mcp
|
||||
|
||||
# Environment variables
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
MCP_TRANSPORT=http \
|
||||
MCP_PORT=8765 \
|
||||
SKILL_SEEKERS_HOME=/data \
|
||||
SKILL_SEEKERS_OUTPUT=/output
|
||||
|
||||
# Health check for HTTP mode
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=10s --retries=3 \
|
||||
CMD curl -f http://localhost:${MCP_PORT}/health || exit 1
|
||||
|
||||
# Volumes
|
||||
VOLUME ["/data", "/configs", "/output"]
|
||||
|
||||
# Expose MCP server port (default 8765, overridden by $PORT on cloud platforms)
|
||||
EXPOSE ${MCP_PORT:-8765}
|
||||
|
||||
# Start MCP server in HTTP mode by default
|
||||
# Uses shell form so $PORT/$MCP_PORT env vars are expanded at runtime
|
||||
# Cloud platforms (Render, Railway, etc.) set $PORT automatically
|
||||
CMD python -m skill_seekers.mcp.server_fastmcp --http --host 0.0.0.0 --port ${PORT:-${MCP_PORT:-8765}}
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Yusuf Karaaslan
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,719 @@
|
||||
# QWEN.md - Skill Seekers
|
||||
|
||||
Comprehensive context file for AI coding agents working with the Skill Seekers project.
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Skill Seekers** (v3.6.0) is a Python CLI tool and MCP server that converts documentation sites, GitHub repositories, PDFs, videos, notebooks, wikis, and 17+ source types into structured AI-ready skills for 21+ LLM platforms and RAG pipelines.
|
||||
|
||||
**Tagline:** "The data layer for AI systems" — sits between raw documentation and every AI system that consumes it (Claude, Gemini, LangChain, LlamaIndex, Cursor, etc.).
|
||||
|
||||
### Key Capabilities
|
||||
|
||||
- **18 source types:** documentation (web), GitHub, PDF, Word (.docx), EPUB, video, local codebase, Jupyter, HTML, OpenAPI, AsciiDoc, PowerPoint, RSS/Atom, man pages, Confluence, Notion, Slack/Discord chat
|
||||
- **21+ export targets:** Claude, Gemini, OpenAI, DeepSeek, Qwen, Fireworks, Together, OpenRouter, IBM BoB, Kimi, MiniMax, OpenCode, LangChain, LlamaIndex, Haystack, Pinecone, Chroma, Weaviate, Qdrant, FAISS, Markdown, and more
|
||||
- **Unified pipeline:** One scraping command → export to any platform without re-scraping
|
||||
- **MCP server:** 40 tools for AI assistants to scrape, package, and manage skills
|
||||
- **AI enhancement:** Optional Claude-powered enhancement for better skill quality
|
||||
|
||||
### Project Links
|
||||
|
||||
| Resource | Link |
|
||||
|----------|------|
|
||||
| Website | https://skillseekersweb.com/ |
|
||||
| PyPI | https://pypi.org/project/skill-seekers/ |
|
||||
| GitHub | https://github.com/yusufkaraaslan/Skill_Seekers |
|
||||
| Configs | https://github.com/yusufkaraaslan/skill-seekers-configs |
|
||||
| MCP | https://modelcontextprotocol.io |
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
Skill_Seekers/
|
||||
├── src/skill_seekers/ # Main package (src/ layout)
|
||||
│ ├── cli/ # CLI commands (97 files)
|
||||
│ │ ├── adaptors/ # Platform adaptors (Strategy pattern)
|
||||
│ │ ├── arguments/ # CLI argument definitions
|
||||
│ │ ├── parsers/ # Subcommand parsers
|
||||
│ │ ├── storage/ # Cloud storage adaptors
|
||||
│ │ ├── main.py # Unified CLI entry point
|
||||
│ │ ├── source_detector.py # Auto-detects source type
|
||||
│ │ ├── create_command.py # Unified `create` command
|
||||
│ │ ├── config_validator.py # Config validation
|
||||
│ │ ├── unified_scraper.py # Multi-source orchestrator
|
||||
│ │ └── unified_skill_builder.py # Skill merging
|
||||
│ ├── mcp/ # MCP server
|
||||
│ │ ├── server.py # Main MCP server
|
||||
│ │ ├── server_fastmcp.py # FastMCP implementation
|
||||
│ │ └── tools/ # MCP tools (10 files)
|
||||
│ ├── sync/ # Sync monitoring (Pydantic)
|
||||
│ ├── benchmark/ # Benchmarking framework
|
||||
│ ├── embedding/ # FastAPI embedding server
|
||||
│ └── workflows/ # 68 YAML workflow presets
|
||||
├── tests/ # ~143 test files (pytest)
|
||||
├── configs/ # Preset JSON scraping configs
|
||||
├── docs/ # Documentation
|
||||
├── templates/ # GitHub Actions, etc.
|
||||
├── scripts/ # Utility scripts
|
||||
└── pyproject.toml # Project configuration
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup & Installation
|
||||
|
||||
### Required: Install in Editable Mode
|
||||
|
||||
```bash
|
||||
# ALWAYS run this first — tests hard-exit if package not installed
|
||||
pip install -e .
|
||||
|
||||
# With dev tools (pytest, ruff, mypy, coverage)
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# With all optional dependencies
|
||||
pip install -e ".[all]"
|
||||
```
|
||||
|
||||
**Note:** `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Copy `.env.example` to `.env` and configure:
|
||||
|
||||
```bash
|
||||
# Required for AI enhancement
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# Optional: LLM platforms
|
||||
GOOGLE_API_KEY=... # Gemini
|
||||
OPENAI_API_KEY=... # OpenAI/ChatGPT
|
||||
|
||||
# Optional: GitHub (increases rate limits)
|
||||
GITHUB_TOKEN=...
|
||||
|
||||
# MCP Server config
|
||||
MCP_TRANSPORT=http
|
||||
MCP_PORT=8765
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Build / Test / Lint Commands
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run ALL tests (required before commits)
|
||||
pytest tests/ -v
|
||||
|
||||
# Run single test file
|
||||
pytest tests/test_scraper_features.py -v
|
||||
|
||||
# Run single test function
|
||||
pytest tests/test_scraper_features.py::test_detect_language -v
|
||||
|
||||
# Run single test class method
|
||||
pytest tests/test_adaptors/test_claude_adaptor.py::TestClaudeAdaptor::test_package -v
|
||||
|
||||
# Skip slow/integration tests
|
||||
pytest tests/ -v -m "not slow and not integration"
|
||||
|
||||
# With coverage
|
||||
pytest tests/ --cov=src/skill_seekers --cov-report=term
|
||||
```
|
||||
|
||||
### Linting & Formatting
|
||||
|
||||
```bash
|
||||
# Lint (ruff)
|
||||
ruff check src/ tests/
|
||||
ruff check src/ tests/ --fix
|
||||
|
||||
# Format (ruff)
|
||||
ruff format --check src/ tests/
|
||||
ruff format src/ tests/
|
||||
|
||||
# Type check (mypy)
|
||||
mypy src/skill_seekers --show-error-codes --pretty
|
||||
```
|
||||
|
||||
### Pytest Configuration
|
||||
|
||||
From `pyproject.toml`:
|
||||
- `addopts = "-v --tb=short --strict-markers"`
|
||||
- `asyncio_mode = "auto"`
|
||||
- `asyncio_default_fixture_loop_scope = "function"`
|
||||
|
||||
**Test markers:** `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`
|
||||
|
||||
**Test count:** 123 test files (107 in `tests/`, 16 in `tests/test_adaptors/`)
|
||||
|
||||
---
|
||||
|
||||
## CLI Usage
|
||||
|
||||
### Core Commands
|
||||
|
||||
```bash
|
||||
# Unified create command (auto-detects source type)
|
||||
skill-seekers create https://docs.react.dev/
|
||||
skill-seekers create facebook/react
|
||||
skill-seekers create manual.pdf
|
||||
skill-seekers create notebook.ipynb
|
||||
|
||||
# Package for specific platform
|
||||
skill-seekers package output/react --target claude # Claude AI (ZIP)
|
||||
skill-seekers package output/react --target gemini # Gemini (tar.gz)
|
||||
skill-seekers package output/react --target openai # OpenAI
|
||||
skill-seekers package output/react --target cursor # .cursorrules
|
||||
|
||||
# Multi-source unified scraping
|
||||
skill-seekers create configs/react_unified.json
|
||||
```
|
||||
|
||||
### All 20+ Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `create` | Unified create (auto-detects source) |
|
||||
| `scan` | AI-detect project tech stack and emit configs |
|
||||
| `doctor` | Health check for dependencies and configuration |
|
||||
| `scrape` | Scrape documentation website |
|
||||
| `github` | Scrape GitHub repository |
|
||||
| `pdf` | Extract from PDF |
|
||||
| `word` | Extract from Word (.docx) |
|
||||
| `epub` | Extract from EPUB |
|
||||
| `video` | Extract from video |
|
||||
| `jupyter` | Extract from Jupyter notebook |
|
||||
| `html` | Extract from local HTML |
|
||||
| `openapi` | Extract from OpenAPI spec |
|
||||
| `asciidoc` | Extract from AsciiDoc |
|
||||
| `pptx` | Extract from PowerPoint |
|
||||
| `rss` | Extract from RSS/Atom feed |
|
||||
| `manpage` | Extract from man pages |
|
||||
| `confluence` | Extract from Confluence |
|
||||
| `notion` | Extract from Notion |
|
||||
| `chat` | Extract from Slack/Discord |
|
||||
| `unified` | Multi-source scraping |
|
||||
| `analyze` | Analyze local codebase |
|
||||
| `enhance` | AI enhancement |
|
||||
| `package` | Package skill |
|
||||
| `upload` | Upload to platform |
|
||||
| `install-agent` | Install to AI agent |
|
||||
|
||||
---
|
||||
|
||||
## Code Style & Conventions
|
||||
|
||||
### Formatting Rules (ruff)
|
||||
|
||||
- **Line length:** 100 characters
|
||||
- **Target Python:** 3.10+
|
||||
- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM
|
||||
- **Ignored rules:** E501, F541, ARG002, B007, I001, SIM114
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
| Type | Convention | Example |
|
||||
|------|------------|---------|
|
||||
| Files | `snake_case.py` | `source_detector.py` |
|
||||
| Classes | `PascalCase` | `SkillAdaptor`, `ClaudeAdaptor` |
|
||||
| Functions | `snake_case` | `get_adaptor()`, `detect_language()` |
|
||||
| Constants | `UPPER_CASE` | `ADAPTORS`, `DEFAULT_CHUNK_TOKENS` |
|
||||
| Private | `_prefix` | `_read_existing_content()` |
|
||||
|
||||
### Type Hints
|
||||
|
||||
- Gradual typing with modern syntax
|
||||
- Use `str | None` not `Optional[str]`
|
||||
- Use `list[str]` not `List[str]`
|
||||
- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`
|
||||
|
||||
### Docstrings
|
||||
|
||||
- Module-level docstring on every file
|
||||
- Google-style for public functions/classes
|
||||
- Include `Args:`, `Returns:`, `Raises:` sections
|
||||
|
||||
### Error Handling
|
||||
|
||||
```python
|
||||
# Use specific exceptions
|
||||
raise ValueError("Invalid config: missing 'sources'")
|
||||
raise RuntimeError("Scraping failed after 3 retries")
|
||||
|
||||
# Chain exceptions
|
||||
try:
|
||||
...
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to process {source}") from e
|
||||
|
||||
# Guard optional imports
|
||||
try:
|
||||
from .claude import ClaudeAdaptor
|
||||
except ImportError:
|
||||
ClaudeAdaptor = None
|
||||
```
|
||||
|
||||
### Import Patterns
|
||||
|
||||
```python
|
||||
# Standard library → third-party → first-party
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from beautifulsoup4 import BeautifulSoup
|
||||
|
||||
from skill_seekers.cli.adaptors import ClaudeAdaptor
|
||||
from skill_seekers.cli.source_detector import SourceDetector
|
||||
|
||||
# Guard optional imports
|
||||
try:
|
||||
from .gemini import GeminiAdaptor
|
||||
except ImportError:
|
||||
GeminiAdaptor = None
|
||||
|
||||
# Re-exports (use noqa)
|
||||
from .base import SkillAdaptor, SkillMetadata # noqa: F401
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Architectural Patterns
|
||||
|
||||
### 1. Adaptor (Strategy) Pattern
|
||||
|
||||
All platform logic in `cli/adaptors/`. Each adaptor inherits `SkillAdaptor`:
|
||||
|
||||
```python
|
||||
from skill_seekers.cli.adaptors.base import SkillAdaptor, SkillMetadata
|
||||
|
||||
class ClaudeAdaptor(SkillAdaptor):
|
||||
PLATFORM = "claude"
|
||||
PLATFORM_NAME = "Claude AI (Anthropic)"
|
||||
|
||||
def format_skill_md(self, skill_dir: Path, metadata: SkillMetadata) -> str:
|
||||
"""Format SKILL.md with YAML frontmatter"""
|
||||
...
|
||||
|
||||
def package(self, skill_dir: Path, output_path: Path, ...) -> Path:
|
||||
"""Package as ZIP with SKILL.md, references/, scripts/"""
|
||||
...
|
||||
|
||||
def upload(self, package_path: Path, api_key: str) -> str:
|
||||
"""Upload to Claude API"""
|
||||
...
|
||||
```
|
||||
|
||||
**Registered in:** `cli/adaptors/__init__.py` → `ADAPTORS` dict
|
||||
|
||||
### 2. Scraper Pattern
|
||||
|
||||
Each source type has 3 files:
|
||||
|
||||
```
|
||||
cli/<type>_scraper.py # Main scraper class + main()
|
||||
arguments/<type>.py # CLI argument definitions
|
||||
parsers/<type>_parser.py # ArgumentParser setup
|
||||
```
|
||||
|
||||
**Example:** `pdf_scraper.py` → `PdfToSkillConverter` class
|
||||
|
||||
**Registered in:**
|
||||
- `parsers/__init__.py` → `PARSERS` list
|
||||
- `main.py` → `COMMAND_MODULES` dict
|
||||
- `config_validator.py` → `VALID_SOURCE_TYPES` set
|
||||
|
||||
### 3. Unified Pipeline
|
||||
|
||||
**`unified_scraper.py`** orchestrates multi-source scraping:
|
||||
|
||||
```python
|
||||
class UnifiedScraper:
|
||||
def __init__(self, config_path: str, merge_mode: str = "rule-based"):
|
||||
self.config = load_config(config_path)
|
||||
self.scraped_data = {
|
||||
"documentation": [],
|
||||
"github": [],
|
||||
"pdf": [],
|
||||
# ... 18 source types
|
||||
}
|
||||
|
||||
def run(self) -> Path:
|
||||
# 1. Scrape all sources
|
||||
for source in self.config["sources"]:
|
||||
self._scrape_source(source)
|
||||
|
||||
# 2. Merge (pairwise synthesis or generic)
|
||||
merged = self._merge_sources()
|
||||
|
||||
# 3. Build unified skill
|
||||
return self._build_skill(merged)
|
||||
```
|
||||
|
||||
**`unified_skill_builder.py`** uses:
|
||||
- **Pairwise synthesis** for docs+github+pdf combos
|
||||
- **`_generic_merge()`** for other combinations
|
||||
|
||||
### 4. Source Detection
|
||||
|
||||
**`source_detector.py`** auto-detects from user input:
|
||||
|
||||
```python
|
||||
class SourceDetector:
|
||||
@classmethod
|
||||
def detect(cls, source: str) -> SourceInfo:
|
||||
# Check file extensions
|
||||
if source.endswith(".pdf"):
|
||||
return cls._detect_pdf(source)
|
||||
if source.endswith(".ipynb"):
|
||||
return cls._detect_jupyter(source)
|
||||
|
||||
# Check GitHub patterns
|
||||
if cls.GITHUB_REPO_PATTERN.match(source):
|
||||
return cls._detect_github(source)
|
||||
|
||||
# Check URLs
|
||||
parsed = urlparse(source)
|
||||
if parsed.scheme in ("http", "https"):
|
||||
return cls._detect_web(source)
|
||||
|
||||
# Check local directories
|
||||
if os.path.isdir(source):
|
||||
return cls._detect_local(source)
|
||||
```
|
||||
|
||||
### 5. MCP Tools
|
||||
|
||||
**`mcp/tools/`** grouped by category:
|
||||
|
||||
- `scrape_tools.py` — Scraping tools
|
||||
- `package_tools.py` — Packaging tools
|
||||
- `enhance_tools.py` — Enhancement tools
|
||||
- `install_tools.py` — Installation tools
|
||||
- `vector_db_tools.py` — Vector DB tools
|
||||
- `workflow_tools.py` — Workflow tools
|
||||
|
||||
**`scrape_generic_tool`** handles all new source types dynamically.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Unified Config Format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "react-skill",
|
||||
"description": "React documentation skill",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"url": "https://react.dev/",
|
||||
"config": {
|
||||
"max_pages": 100,
|
||||
"include_patterns": ["**/*.md"],
|
||||
"exclude_patterns": ["**/blog/**"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "facebook/react",
|
||||
"config": {
|
||||
"include": ["src/", "packages/"],
|
||||
"exclude": ["**/*.test.tsx"]
|
||||
}
|
||||
}
|
||||
],
|
||||
"merge_mode": "rule-based",
|
||||
"output": "output/react"
|
||||
}
|
||||
```
|
||||
|
||||
### Valid Source Types
|
||||
|
||||
```python
|
||||
VALID_SOURCE_TYPES = {
|
||||
"documentation", "github", "pdf", "local", "word",
|
||||
"video", "epub", "jupyter", "html", "openapi",
|
||||
"asciidoc", "pptx", "confluence", "notion", "rss",
|
||||
"manpage", "chat"
|
||||
}
|
||||
```
|
||||
|
||||
### Merge Modes
|
||||
|
||||
- **`rule-based`** — Deterministic merging with conflict resolution rules
|
||||
- **`claude-enhanced`** — AI-powered merging (requires `ANTHROPIC_API_KEY`)
|
||||
|
||||
---
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Branch Structure
|
||||
|
||||
```
|
||||
main (production, protected)
|
||||
↑
|
||||
│ (maintainer merges only)
|
||||
│
|
||||
development (integration, default PR target)
|
||||
↑
|
||||
│ (all contributor PRs)
|
||||
│
|
||||
feature branches
|
||||
```
|
||||
|
||||
### PR Process
|
||||
|
||||
1. Fork and clone
|
||||
2. Create feature branch from `development`
|
||||
3. Make changes, commit, push
|
||||
4. Create PR targeting **`development`** (NOT `main`)
|
||||
5. Wait for tests + review
|
||||
|
||||
```bash
|
||||
git checkout development
|
||||
git pull upstream development
|
||||
git checkout -b my-feature
|
||||
# ... make changes
|
||||
git commit -m "Add feature X"
|
||||
git push origin my-feature
|
||||
# Create PR → development
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Practices
|
||||
|
||||
### Test Organization
|
||||
|
||||
```
|
||||
tests/
|
||||
├── conftest.py # Fixtures, setup
|
||||
├── test_adaptors/ # Adaptor tests (16 files)
|
||||
├── test_scraper_features.py # Core scraper tests
|
||||
├── test_source_detector.py # Source detection tests
|
||||
├── test_config_validation.py # Config validation
|
||||
├── test_mcp_*.py # MCP tests
|
||||
├── test_*_e2e.py # End-to-end tests
|
||||
└── fixtures/ # Test fixtures
|
||||
```
|
||||
|
||||
### Test Patterns
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from skill_seekers.cli.source_detector import SourceDetector
|
||||
|
||||
class TestSourceDetector:
|
||||
"""Test source detection"""
|
||||
|
||||
def test_detect_pdf(self, tmp_path):
|
||||
"""Test PDF detection"""
|
||||
pdf_file = tmp_path / "test.pdf"
|
||||
pdf_file.touch()
|
||||
|
||||
result = SourceDetector.detect(str(pdf_file))
|
||||
assert result.type == "pdf"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_scraping(self):
|
||||
"""Test async scraping"""
|
||||
# asyncio_mode = "auto" — decorator often implicit
|
||||
...
|
||||
|
||||
@pytest.mark.slow
|
||||
def test_slow_operation(self):
|
||||
"""Mark slow tests for optional skipping"""
|
||||
...
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_external_api(self):
|
||||
"""Mark integration tests requiring external services"""
|
||||
...
|
||||
```
|
||||
|
||||
### Fixtures
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
@pytest.fixture
|
||||
def sample_config(tmp_path):
|
||||
"""Create sample config file"""
|
||||
config = {
|
||||
"name": "test-skill",
|
||||
"sources": [{"type": "documentation", "url": "https://example.com"}]
|
||||
}
|
||||
config_file = tmp_path / "config.json"
|
||||
config_file.write_text(json.dumps(config))
|
||||
return str(config_file)
|
||||
|
||||
@pytest.fixture
|
||||
def mock_response():
|
||||
"""Mock HTTP response"""
|
||||
class MockResponse:
|
||||
status_code = 200
|
||||
text = "<html><body>Test</body></html>"
|
||||
return MockResponse()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Guidelines
|
||||
|
||||
### Before Commits
|
||||
|
||||
```bash
|
||||
# 1. Lint
|
||||
ruff check src/ tests/
|
||||
ruff format --check src/ tests/
|
||||
|
||||
# 2. Type check
|
||||
mypy src/skill_seekers
|
||||
|
||||
# 3. Test (ALL must pass)
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
### Adding New Source Types
|
||||
|
||||
1. Create scraper: `cli/<type>_scraper.py` with `<Type>ToSkillConverter` class
|
||||
2. Create arguments: `arguments/<type>.py`
|
||||
3. Create parser: `parsers/<type>_parser.py`
|
||||
4. Register in `parsers/__init__.py` → `PARSERS`
|
||||
5. Register in `main.py` → `COMMAND_MODULES`
|
||||
6. Add to `config_validator.py` → `VALID_SOURCE_TYPES`
|
||||
7. Add detection to `source_detector.py`
|
||||
8. Add to `unified_scraper.py` → `scraped_data` dict
|
||||
9. Write tests in `tests/test_<type>_scraper.py`
|
||||
|
||||
### Adding New Platform Adaptors
|
||||
|
||||
1. Create adaptor: `cli/adaptors/<platform>.py` inheriting `SkillAdaptor`
|
||||
2. Implement: `format_skill_md()`, `package()`, `upload()`
|
||||
3. Register in `cli/adaptors/__init__.py` → `ADAPTORS` dict
|
||||
4. Add to `package_skill.py` → target mapping
|
||||
5. Write tests in `tests/test_adaptors/test_<platform>.py`
|
||||
|
||||
### Adding New MCP Tools
|
||||
|
||||
1. Create tool in `mcp/tools/<category>_tools.py`
|
||||
2. Use `@mcp.tool()` decorator
|
||||
3. Register in `mcp/server.py` or `mcp/server_fastmcp.py`
|
||||
4. Write tests in `tests/test_mcp_*.py`
|
||||
|
||||
---
|
||||
|
||||
## Common Issues & Solutions
|
||||
|
||||
### Tests Fail with Import Error
|
||||
|
||||
**Problem:** `ModuleNotFoundError: No module named 'skill_seekers'`
|
||||
|
||||
**Solution:** Install in editable mode first:
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Optional Dependency Missing
|
||||
|
||||
**Problem:** `ImportError: No module named 'mammoth'`
|
||||
|
||||
**Solution:** Install optional dependency:
|
||||
```bash
|
||||
pip install "skill-seekers[docx]"
|
||||
# or
|
||||
pip install "skill-seekers[all]"
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
**Problem:** GitHub API rate limited (60/hour anonymous)
|
||||
|
||||
**Solution:** Set `GITHUB_TOKEN` in `.env`:
|
||||
```bash
|
||||
GITHUB_TOKEN=ghp_...
|
||||
```
|
||||
|
||||
### Async Scraping Issues
|
||||
|
||||
**Problem:** Event loop errors in async tests
|
||||
|
||||
**Solution:** Use `@pytest.mark.asyncio` decorator (auto mode enabled)
|
||||
|
||||
---
|
||||
|
||||
## Key Files Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `pyproject.toml` | Project config, dependencies, tool settings |
|
||||
| `src/skill_seekers/cli/main.py` | Unified CLI entry point |
|
||||
| `src/skill_seekers/cli/source_detector.py` | Auto-detect source types |
|
||||
| `src/skill_seekers/cli/config_validator.py` | Config validation |
|
||||
| `src/skill_seekers/cli/unified_scraper.py` | Multi-source orchestrator |
|
||||
| `src/skill_seekers/cli/adaptors/base.py` | Adaptor interface |
|
||||
| `src/skill_seekers/cli/adaptors/__init__.py` | Adaptor registry |
|
||||
| `src/skill_seekers/mcp/server.py` | MCP server |
|
||||
| `tests/conftest.py` | Test fixtures |
|
||||
| `AGENTS.md` | Quick reference for AI agents |
|
||||
|
||||
---
|
||||
|
||||
## Version & Release
|
||||
|
||||
- **Current version:** 3.3.0 (from `pyproject.toml`)
|
||||
- **Version source:** `src/skill_seekers/_version.py` reads from `pyproject.toml`
|
||||
- **Release process:** Tag → GitHub Actions → PyPI publish
|
||||
- **Changelog:** `CHANGELOG.md` (Keep a Changelog format)
|
||||
|
||||
---
|
||||
|
||||
## Related Repositories
|
||||
|
||||
| Repository | Purpose |
|
||||
|------------|---------|
|
||||
| [Skill_Seekers](https://github.com/yusufkaraaslan/Skill_Seekers) | Core CLI & MCP (this repo) |
|
||||
| [skillseekersweb](https://github.com/yusufkaraaslan/skillseekersweb) | Website & docs |
|
||||
| [skill-seekers-configs](https://github.com/yusufkaraaslan/skill-seekers-configs) | Community configs |
|
||||
| [skill-seekers-action](https://github.com/yusufkaraaslan/skill-seekers-action) | GitHub Action |
|
||||
| [skill-seekers-plugin](https://github.com/yusufkaraaslan/skill-seekers-plugin) | Claude Code plugin |
|
||||
| [homebrew-skill-seekers](https://github.com/yusufkaraaslan/homebrew-skill-seekers) | Homebrew tap |
|
||||
|
||||
---
|
||||
|
||||
## Quick Commands Cheat Sheet
|
||||
|
||||
```bash
|
||||
# Setup
|
||||
pip install -e ".[dev]"
|
||||
cp .env.example .env
|
||||
# Edit .env with API keys
|
||||
|
||||
# Development
|
||||
ruff check src/ tests/ --fix
|
||||
ruff format src/ tests/
|
||||
mypy src/skill_seekers
|
||||
pytest tests/ -v
|
||||
|
||||
# Test subsets
|
||||
pytest tests/test_adaptors/ -v
|
||||
pytest tests/ -m "not slow and not integration"
|
||||
pytest tests/ --cov=src/skill_seekers
|
||||
|
||||
# Usage
|
||||
skill-seekers create https://docs.python.org/
|
||||
skill-seekers package output/python --target claude
|
||||
skill-seekers create configs/react_unified.json
|
||||
```
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目:`yusufkaraaslan/Skill_Seekers`
|
||||
- 原始仓库:https://github.com/yusufkaraaslan/Skill_Seekers
|
||||
- 导入方式:上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源,不代表 WeHub 是原项目作者
|
||||
@@ -0,0 +1,433 @@
|
||||
# Skill Seekers Roadmap
|
||||
|
||||
Transform Skill Seekers into the easiest way to create Claude AI skills from **any knowledge source** - documentation websites, PDFs, codebases, GitHub repos, Office docs, and more - with both CLI and MCP interfaces.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Current Status: v3.6.0 ✅
|
||||
|
||||
**Latest Release:** v3.6.0 (May 2026)
|
||||
|
||||
**What Works:**
|
||||
- ✅ **18 source types** — documentation, GitHub, PDF, video, Word, EPUB, Jupyter, local HTML, OpenAPI, AsciiDoc, PowerPoint, RSS/Atom, man pages, Confluence, Notion, Slack/Discord, local codebase
|
||||
- ✅ Unified multi-source scraping with generic merge for any source combination
|
||||
- ✅ 40 MCP tools fully functional
|
||||
- ✅ Multi-platform support (21 platforms: Claude, Gemini, OpenAI, DeepSeek, Qwen, Fireworks, Together, OpenRouter, IBM BoB, Kimi, MiniMax, OpenCode, LangChain, LlamaIndex, Haystack, Pinecone, ChromaDB, FAISS, Weaviate, Qdrant, Markdown)
|
||||
- ✅ Auto-upload to all platforms
|
||||
- ✅ 12 preset configs (including unified configs)
|
||||
- ✅ Large docs support (40K+ pages with router skills)
|
||||
- ✅ C3.x codebase analysis suite (C3.1-C3.10)
|
||||
- ✅ Bootstrap skill feature - self-hosting capability
|
||||
- ✅ 3,445+ tests passing
|
||||
- ✅ Unified `create` command with auto-detection for all 18 source types
|
||||
- ✅ 68 YAML workflow presets
|
||||
- ✅ Cloud storage integration (S3, GCS, Azure)
|
||||
- ✅ Source auto-detection via `source_detector.py`
|
||||
|
||||
**Recent Improvements (v3.2.0):**
|
||||
- ✅ **10 new source types**: Word, EPUB, video, Jupyter, local HTML, OpenAPI, AsciiDoc, PowerPoint, RSS/Atom, man pages, Confluence, Notion, Slack/Discord
|
||||
- ✅ **Generic merge system**: `_generic_merge()` in `unified_skill_builder.py` handles arbitrary source combinations
|
||||
- ✅ **Unified CLI**: `create` command auto-detects all 18 source types
|
||||
- ✅ **Workflow Presets**: YAML-based enhancement presets with CLI management
|
||||
- ✅ **Progressive Disclosure**: Default help shows 13 universal flags, detailed help per source
|
||||
- ✅ **Bug Fixes**: Markdown parser h1 filtering, paragraph length filtering
|
||||
- ✅ **Docs Cleanup**: Removed 47 stale planning/QA/release markdown files
|
||||
|
||||
---
|
||||
|
||||
## 🧭 Development Philosophy
|
||||
|
||||
**Small tasks → Pick one → Complete → Move on**
|
||||
|
||||
Instead of rigid milestones, we use a **flexible task-based approach**:
|
||||
- 136 small, independent tasks across 10 categories
|
||||
- Pick any task, any order
|
||||
- Start small, ship often
|
||||
- No deadlines, just continuous progress
|
||||
|
||||
**Philosophy:** Small steps → Consistent progress → Compound results
|
||||
|
||||
---
|
||||
|
||||
## 📋 Task-Based Roadmap (136 Tasks, 10 Categories)
|
||||
|
||||
### 🌐 **Category A: Community & Sharing**
|
||||
Small tasks that build community features incrementally
|
||||
|
||||
#### A1: Config Sharing (Website Feature)
|
||||
- [x] **Task A1.1:** Create simple JSON API endpoint to list configs ✅ **COMPLETE**
|
||||
- **Status:** Live at https://api.skillseekersweb.com
|
||||
- **Features:** 6 REST endpoints, auto-categorization, auto-tags, filtering, SSL enabled
|
||||
- [x] **Task A1.2:** Add MCP tool `fetch_config` to download from website ✅ **COMPLETE**
|
||||
- **Features:** List 24 configs, filter by category, download by name
|
||||
- [x] **Task A1.3:** Add MCP tool `submit_config` to submit custom configs ✅ **COMPLETE**
|
||||
- **Purpose:** Allow users to submit custom configs via MCP (creates GitHub issue)
|
||||
- **Completed:** May 2026
|
||||
- [ ] **Task A1.4:** Create static config catalog website (GitHub Pages)
|
||||
- **Purpose:** Read-only catalog to browse/search configs
|
||||
- **Time:** 2-3 hours
|
||||
- [ ] **Task A1.5:** Add config rating/voting system
|
||||
- **Purpose:** Community feedback on config quality
|
||||
- **Time:** 3-4 hours
|
||||
- [ ] **Task A1.6:** Admin review queue for submitted configs
|
||||
- **Approach:** Use GitHub Issues with labels
|
||||
- **Time:** 1-2 hours
|
||||
- [x] **Task A1.7:** Add MCP tool `install_skill` for one-command workflow ✅ **COMPLETE**
|
||||
- **Features:** fetch → scrape → enhance → package → upload
|
||||
- **Completed:** December 21, 2025
|
||||
- [ ] **Task A1.8:** Add smart skill detection and auto-install
|
||||
- **Purpose:** Auto-detect missing skills from user queries
|
||||
- **Time:** 4-6 hours
|
||||
|
||||
**Start Next:** Pick A1.4 (static config catalog website)
|
||||
|
||||
#### A2: Knowledge Sharing (Website Feature)
|
||||
- [ ] **Task A2.1:** Design knowledge database schema
|
||||
- [ ] **Task A2.2:** Create API endpoint to upload knowledge (.zip files)
|
||||
- [ ] **Task A2.3:** Add MCP tool `fetch_knowledge` to download from site
|
||||
- [ ] **Task A2.4:** Add knowledge preview/description
|
||||
- [ ] **Task A2.5:** Add knowledge categorization (by framework/topic)
|
||||
- [ ] **Task A2.6:** Add knowledge search functionality
|
||||
|
||||
**Start Small:** Pick A2.1 first (schema design, no coding)
|
||||
|
||||
#### A3: Simple Website Foundation
|
||||
- [ ] **Task A3.1:** Create single-page static site (GitHub Pages)
|
||||
- [ ] **Task A3.2:** Add config gallery view
|
||||
- [ ] **Task A3.3:** Add "Submit Config" link
|
||||
- [ ] **Task A3.4:** Add basic stats
|
||||
- [ ] **Task A3.5:** Add simple blog using GitHub Issues
|
||||
- [ ] **Task A3.6:** Add RSS feed for updates
|
||||
|
||||
**Start Small:** Pick A3.1 first (single HTML page)
|
||||
|
||||
---
|
||||
|
||||
### 🛠️ **Category B: New Input Formats**
|
||||
Add support for non-HTML documentation sources
|
||||
|
||||
#### B1: PDF Documentation Support ✅ **COMPLETE (v3.0.0)**
|
||||
- [x] **Task B1.1:** Research PDF parsing libraries ✅
|
||||
- [x] **Task B1.2:** Create simple PDF text extractor (POC) ✅
|
||||
- [x] **Task B1.3:** Add PDF page detection and chunking ✅
|
||||
- [x] **Task B1.4:** Extract code blocks from PDFs ✅
|
||||
- [x] **Task B1.5:** Add PDF image extraction ✅
|
||||
- [x] **Task B1.6:** Create `pdf_scraper.py` CLI tool ✅
|
||||
- [x] **Task B1.7:** Add MCP tool `scrape_pdf` ✅
|
||||
- [x] **Task B1.8:** Create PDF config format ✅
|
||||
|
||||
#### B2: Microsoft Word (.docx) Support ✅ **COMPLETE (v3.2.0)**
|
||||
- [x] **Task B2.1-B2.7:** Word document parsing and scraping ✅
|
||||
|
||||
#### B3: Excel/Spreadsheet (.xlsx) Support
|
||||
- [ ] **Task B3.1-B3.6:** Spreadsheet parsing and API extraction
|
||||
|
||||
#### B4: Markdown Files Support ✅ **COMPLETE (v3.1.0)**
|
||||
- [x] **Task B4.1-B4.6:** Local markdown directory scraping ✅
|
||||
|
||||
#### B5: Additional Source Types ✅ **COMPLETE (v3.2.0)**
|
||||
- [x] **EPUB** - `epub_scraper.py` ✅
|
||||
- [x] **Video** - `video_scraper.py` (YouTube, Vimeo, local files) ✅
|
||||
- [x] **Jupyter Notebook** - `jupyter_scraper.py` ✅
|
||||
- [x] **Local HTML** - `html_scraper.py` ✅
|
||||
- [x] **OpenAPI/Swagger** - `openapi_scraper.py` ✅
|
||||
- [x] **AsciiDoc** - `asciidoc_scraper.py` ✅
|
||||
- [x] **PowerPoint** - `pptx_scraper.py` ✅
|
||||
- [x] **RSS/Atom** - `rss_scraper.py` ✅
|
||||
- [x] **Man pages** - `manpage_scraper.py` ✅
|
||||
- [x] **Confluence** - `confluence_scraper.py` ✅
|
||||
- [x] **Notion** - `notion_scraper.py` ✅
|
||||
- [x] **Slack/Discord** - `chat_scraper.py` ✅
|
||||
|
||||
---
|
||||
|
||||
### 💻 **Category C: Codebase Knowledge**
|
||||
Generate skills from actual code repositories
|
||||
|
||||
#### C1: GitHub Repository Scraping
|
||||
- [ ] **Task C1.1-C1.12:** GitHub API integration and code analysis
|
||||
|
||||
#### C2: Local Codebase Scraping
|
||||
- [ ] **Task C2.1-C2.8:** Local directory analysis and API extraction
|
||||
|
||||
#### C3: Code Pattern Recognition
|
||||
- [x] **Task C3.1:** Detect common patterns (singleton, factory, etc.) ✅ **v2.6.0**
|
||||
- 10 GoF patterns, 9 languages, 87% precision
|
||||
- [x] **Task C3.2:** Extract usage examples from test files ✅ **v2.6.0**
|
||||
- 5 categories, 9 languages, 80%+ high-confidence examples
|
||||
- [x] **Task C3.3:** Build "how to" guides from code ✅ **COMPLETE**
|
||||
- [x] **Task C3.4:** Extract configuration patterns ✅ **COMPLETE**
|
||||
- [x] **Task C3.5:** Create architectural overview ✅ **COMPLETE**
|
||||
- [x] **Task C3.6:** AI Enhancement for Pattern Detection ✅ **v2.6.0**
|
||||
- Claude API integration for enhanced insights
|
||||
- [x] **Task C3.7:** Architectural Pattern Detection ✅ **v2.6.0**
|
||||
- Detects 8 architectural patterns, framework-aware
|
||||
|
||||
**Start Next:** Pick C1.1 (GitHub API client)
|
||||
|
||||
---
|
||||
|
||||
### 🔌 **Category D: Context7 Integration**
|
||||
- [ ] **Task D1.1-D1.4:** Research and planning
|
||||
- [ ] **Task D2.1-D2.5:** Basic integration
|
||||
|
||||
---
|
||||
|
||||
### 🚀 **Category E: MCP Enhancements**
|
||||
Small improvements to existing MCP tools
|
||||
|
||||
#### E1: New MCP Tools
|
||||
- [x] **Task E1.3:** Add `scrape_pdf` MCP tool ✅
|
||||
- [x] **Task E1.1:** Add `fetch_config` MCP tool ✅ **COMPLETE**
|
||||
- [ ] **Task E1.2:** Add `fetch_knowledge` MCP tool
|
||||
- [ ] **Task E1.4-E1.9:** Additional format scrapers
|
||||
|
||||
#### E2: MCP Quality Improvements
|
||||
- [ ] **Task E2.1:** Add error handling to all tools
|
||||
- [ ] **Task E2.2:** Add structured logging
|
||||
- [ ] **Task E2.3:** Add progress indicators
|
||||
- [ ] **Task E2.4:** Add validation for all inputs
|
||||
- [ ] **Task E2.5:** Add helpful error messages
|
||||
- [x] **Task E2.6:** Add retry logic for network failures ✅ **Utilities ready**
|
||||
|
||||
---
|
||||
|
||||
### ⚡ **Category F: Performance & Reliability**
|
||||
Technical improvements to existing features
|
||||
|
||||
#### F1: Core Scraper Improvements
|
||||
- [ ] **Task F1.1:** Add URL normalization
|
||||
- [ ] **Task F1.2:** Add duplicate page detection
|
||||
- [ ] **Task F1.3:** Add memory-efficient streaming
|
||||
- [ ] **Task F1.4:** Add HTML parser fallback
|
||||
- [x] **Task F1.5:** Add network retry with exponential backoff ✅
|
||||
- [ ] **Task F1.6:** Fix package path output bug
|
||||
|
||||
#### F2: Incremental Updates
|
||||
- [ ] **Task F2.1-F2.5:** Track modifications, update only changed content
|
||||
|
||||
---
|
||||
|
||||
### 🎨 **Category G: Tools & Utilities**
|
||||
Small standalone tools that add value
|
||||
|
||||
#### G1: Config Tools
|
||||
- [ ] **Task G1.1:** Create `validate_config.py`
|
||||
- [ ] **Task G1.2:** Create `test_selectors.py`
|
||||
- [ ] **Task G1.3:** Create `auto_detect_selectors.py` (AI-powered)
|
||||
- [ ] **Task G1.4:** Create `compare_configs.py`
|
||||
- [ ] **Task G1.5:** Create `optimize_config.py`
|
||||
|
||||
#### G2: Skill Quality Tools
|
||||
- [ ] **Task G2.1-G2.5:** Quality analysis and reporting
|
||||
|
||||
---
|
||||
|
||||
### 📚 **Category H: Community Response**
|
||||
- [ ] **Task H1.1-H1.5:** Address open GitHub issues
|
||||
|
||||
---
|
||||
|
||||
### 🎓 **Category I: Content & Documentation**
|
||||
- [ ] **Task I1.1-I1.6:** Video tutorials
|
||||
- [ ] **Task I2.1-I2.5:** Written guides
|
||||
|
||||
---
|
||||
|
||||
### 🧪 **Category J: Testing & Quality**
|
||||
- [ ] **Task J1.1-J1.6:** Test expansion and coverage
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Recommended Starting Tasks
|
||||
|
||||
### Quick Wins (1-2 hours each):
|
||||
1. **H1.1** - Respond to Issue #8
|
||||
2. **J1.1** - Install MCP package
|
||||
3. **A3.1** - Create GitHub Pages site
|
||||
4. **B1.1** - Research PDF parsing
|
||||
5. **F1.1** - Add URL normalization
|
||||
|
||||
### Medium Tasks (3-5 hours each):
|
||||
6. ✅ **A1.1** - JSON API for configs (COMPLETE)
|
||||
7. **G1.1** - Config validator script
|
||||
8. **C1.1** - GitHub API client
|
||||
9. **I1.1** - Video script writing
|
||||
10. **E2.1** - Error handling for MCP tools
|
||||
|
||||
---
|
||||
|
||||
## 📊 Release History
|
||||
|
||||
### ✅ v2.6.0 - C3.x Codebase Analysis Suite (January 14, 2026)
|
||||
**Focus:** Complete codebase analysis with multi-platform support
|
||||
|
||||
**Completed Features:**
|
||||
- C3.x suite (C3.1-C3.8): Pattern detection, test extraction, architecture analysis
|
||||
- Multi-platform support: Claude, Gemini, OpenAI, Markdown
|
||||
- Platform adaptor architecture
|
||||
- 18 MCP tools (up from 9)
|
||||
- 700+ tests passing
|
||||
- Unified multi-source scraping maturity
|
||||
|
||||
### ✅ v2.1.0 - Test Coverage & Quality (November 29, 2025)
|
||||
**Focus:** Test coverage and unified scraping
|
||||
|
||||
**Completed Features:**
|
||||
- Fixed 12 unified scraping tests
|
||||
- GitHub repository scraping with unlimited local analysis
|
||||
- PDF extraction and conversion
|
||||
- 427 tests passing
|
||||
|
||||
### ✅ v1.0.0 - Production Release (October 19, 2025)
|
||||
**First stable release**
|
||||
|
||||
**Core Features:**
|
||||
- Documentation scraping with BFS
|
||||
- Smart categorization
|
||||
- Language detection
|
||||
- Pattern extraction
|
||||
- 12 preset configurations
|
||||
- MCP server with 9 tools
|
||||
- Large documentation support (40K+ pages)
|
||||
- Auto-upload functionality
|
||||
|
||||
---
|
||||
|
||||
## 📅 Release Planning
|
||||
|
||||
### Release: v3.7.0 (Estimated: Q3 2026)
|
||||
**Focus:** Developer Experience & Integrations
|
||||
|
||||
**Planned Features:**
|
||||
- CI/CD integration examples
|
||||
- Docker containerization
|
||||
- Enhanced scraping formats (Sphinx, Docusaurus detection)
|
||||
- Performance optimizations
|
||||
- Real-time documentation monitoring
|
||||
|
||||
---
|
||||
|
||||
## 🔮 Long-term Vision (v3.0+)
|
||||
|
||||
### Major Features Under Consideration
|
||||
|
||||
#### Advanced Scraping
|
||||
- Real-time documentation monitoring
|
||||
- Automatic skill updates
|
||||
- Change notifications
|
||||
- Multi-language documentation support
|
||||
|
||||
#### Collaboration
|
||||
- Collaborative skill curation
|
||||
- Shared skill repositories
|
||||
- Community ratings and reviews
|
||||
- Skill marketplace
|
||||
|
||||
#### AI & Intelligence
|
||||
- Enhanced AI analysis
|
||||
- Better conflict detection algorithms
|
||||
- Automatic documentation quality scoring
|
||||
- Semantic understanding and natural language queries
|
||||
|
||||
#### Ecosystem
|
||||
- VS Code extension
|
||||
- IntelliJ/PyCharm plugin
|
||||
- Interactive TUI mode
|
||||
- Skill diff and merge tools
|
||||
|
||||
---
|
||||
|
||||
## 📈 Metrics & Goals
|
||||
|
||||
### Current State (v3.6.0) ✅
|
||||
- ✅ 18 source types supported (17 + config)
|
||||
- ✅ 12 preset configs
|
||||
- ✅ 3,445+ tests (excellent coverage)
|
||||
- ✅ 40 MCP tools
|
||||
- ✅ 21 platform adaptors
|
||||
- ✅ C3.x codebase analysis suite complete
|
||||
- ✅ Multi-source synthesis with generic merge for any combination
|
||||
|
||||
### Goals for v3.7+
|
||||
- 🎯 Professional website live
|
||||
- 🎯 50+ preset configs
|
||||
- 🎯 Video tutorial series (5+ videos)
|
||||
- 🎯 100+ GitHub stars
|
||||
- 🎯 Community contributions flowing
|
||||
|
||||
### Goals for v3.0+
|
||||
- 🎯 Auto-detection for 80%+ of sites
|
||||
- 🎯 <1 minute skill generation
|
||||
- 🎯 Active community marketplace
|
||||
- 🎯 Quality scoring system
|
||||
- 🎯 Real-time monitoring
|
||||
|
||||
---
|
||||
|
||||
## 🤝 How to Influence the Roadmap
|
||||
|
||||
### Priority System
|
||||
|
||||
Features are prioritized based on:
|
||||
1. **User impact** - How many users will benefit?
|
||||
2. **Technical feasibility** - How complex is the implementation?
|
||||
3. **Community interest** - How many upvotes/requests?
|
||||
4. **Strategic alignment** - Does it fit our vision?
|
||||
|
||||
### Ways to Contribute
|
||||
|
||||
1. **Vote on Features** - ⭐ Star feature request issues
|
||||
2. **Contribute Code** - Pick any task from the 136 available
|
||||
3. **Share Feedback** - Open issues, share success stories
|
||||
4. **Help with Documentation** - Write tutorials, improve docs
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Flexibility Rules
|
||||
|
||||
1. **Pick any task, any order** - No rigid dependencies
|
||||
2. **Start small** - Research tasks before implementation
|
||||
3. **One task at a time** - Focus, complete, move on
|
||||
4. **Switch anytime** - Not enjoying it? Pick another!
|
||||
5. **Document as you go** - Each task should update docs
|
||||
6. **Test incrementally** - Each task should have a quick test
|
||||
7. **Ship early** - Don't wait for "complete" features
|
||||
|
||||
---
|
||||
|
||||
## 📊 Progress Tracking
|
||||
|
||||
**Completed Tasks:** 15+ (C3.1, C3.2, C3.3, C3.4, C3.5, C3.6, C3.7, A1.1, A1.2, A1.3, A1.7, E1.1, E1.3, E2.6, F1.5)
|
||||
**In Progress:** v3.7.0 planning
|
||||
**Total Available Tasks:** 136
|
||||
|
||||
**No pressure, no deadlines, just progress!** ✨
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Related Projects
|
||||
|
||||
- [Model Context Protocol](https://modelcontextprotocol.io/)
|
||||
- [Claude Code](https://claude.ai/code)
|
||||
- [Anthropic Claude](https://claude.ai)
|
||||
- Documentation frameworks we support: Docusaurus, GitBook, VuePress, Sphinx, MkDocs
|
||||
|
||||
---
|
||||
|
||||
## 📚 Learn More
|
||||
|
||||
- **Project Board**: https://github.com/users/yusufkaraaslan/projects/2
|
||||
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)
|
||||
- **Contributing**: [CONTRIBUTING.md](CONTRIBUTING.md)
|
||||
- **Discussions**: https://github.com/yusufkaraaslan/Skill_Seekers/discussions
|
||||
- **Issues**: https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** May 30, 2026
|
||||
**Philosophy:** Small steps → Consistent progress → Compound results
|
||||
|
||||
**Together, we're building the future of documentation-to-AI skill conversion!** 🚀
|
||||
@@ -0,0 +1,485 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
Common issues and solutions when using Skill Seeker.
|
||||
|
||||
---
|
||||
|
||||
## Installation Issues
|
||||
|
||||
### Python Not Found
|
||||
|
||||
**Error:**
|
||||
```
|
||||
python3: command not found
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. **Check if Python is installed:**
|
||||
```bash
|
||||
which python3
|
||||
python --version # Try without the 3
|
||||
```
|
||||
|
||||
2. **Install Python:**
|
||||
- **macOS:** `brew install python3`
|
||||
- **Linux:** `sudo apt install python3 python3-pip`
|
||||
- **Windows:** Download from python.org, check "Add to PATH"
|
||||
|
||||
3. **Use python instead of python3:**
|
||||
```bash
|
||||
python -m skill_seekers --help
|
||||
```
|
||||
|
||||
### Module Not Found
|
||||
|
||||
**Error:**
|
||||
```
|
||||
ModuleNotFoundError: No module named 'requests'
|
||||
ModuleNotFoundError: No module named 'bs4'
|
||||
ModuleNotFoundError: No module named 'mcp'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. **Install the package in editable mode (critical first step):**
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
2. **Install MCP extras if needed:**
|
||||
```bash
|
||||
pip install -e ".[mcp]"
|
||||
```
|
||||
|
||||
3. **Use --user flag if permission denied:**
|
||||
```bash
|
||||
pip install --user -e .
|
||||
```
|
||||
|
||||
4. **Check pip is working:**
|
||||
```bash
|
||||
pip3 --version
|
||||
```
|
||||
|
||||
### Permission Denied
|
||||
|
||||
**Error:**
|
||||
```
|
||||
Permission denied: '/usr/local/lib/python3.x/...'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. **Use --user flag:**
|
||||
```bash
|
||||
pip3 install --user -e .
|
||||
```
|
||||
|
||||
2. **Use sudo (not recommended):**
|
||||
```bash
|
||||
sudo pip install -e .
|
||||
```
|
||||
|
||||
3. **Use virtual environment (best practice):**
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Runtime Issues
|
||||
|
||||
### File Not Found
|
||||
|
||||
**Error:**
|
||||
```
|
||||
FileNotFoundError: [Errno 2] No such file or directory: 'src/skill_seekers/cli/main.py'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
1. **Check you're in the Skill_Seekers directory:**
|
||||
```bash
|
||||
pwd
|
||||
# Should show: .../Skill_Seekers
|
||||
|
||||
ls
|
||||
# Should show: README.md, src/, configs/, tests/
|
||||
```
|
||||
|
||||
2. **Change to the correct directory:**
|
||||
```bash
|
||||
cd ~/Projects/Skill_Seekers # Adjust path
|
||||
```
|
||||
|
||||
3. **Ensure the package is installed:**
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Config File Not Found
|
||||
|
||||
**Error:**
|
||||
```
|
||||
❌ Error: Config file not found: configs/myconfig.json
|
||||
```
|
||||
|
||||
**Understanding Config Locations:**
|
||||
|
||||
The tool searches for configs in this order:
|
||||
1. Exact path as provided
|
||||
2. `./configs/` (current directory)
|
||||
3. `~/.config/skill-seekers/configs/` (user config directory)
|
||||
4. SkillSeekersWeb.com API (preset configs)
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Place config in user directory (recommended for custom configs):**
|
||||
```bash
|
||||
mkdir -p ~/.config/skill-seekers/configs
|
||||
cp myconfig.json ~/.config/skill-seekers/configs/
|
||||
|
||||
# Now you can use it from anywhere
|
||||
skill-seekers create --config myconfig.json
|
||||
```
|
||||
|
||||
2. **Place config in current directory (project-specific):**
|
||||
```bash
|
||||
mkdir -p configs
|
||||
cp myconfig.json configs/
|
||||
|
||||
skill-seekers create --config configs/myconfig.json
|
||||
```
|
||||
|
||||
3. **Use absolute path:**
|
||||
```bash
|
||||
skill-seekers create --config /full/path/to/myconfig.json
|
||||
```
|
||||
|
||||
4. **Check if it's a preset config (auto-downloads):**
|
||||
```bash
|
||||
# List all available presets
|
||||
skill-seekers estimate --all
|
||||
|
||||
# Use preset (auto-fetched from API)
|
||||
skill-seekers create --config react.json
|
||||
```
|
||||
|
||||
5. **Create new config interactively:**
|
||||
```bash
|
||||
skill-seekers create --interactive
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Setup Issues
|
||||
|
||||
### MCP Server Not Loading
|
||||
|
||||
**Symptoms:**
|
||||
- Tools don't appear in Claude Code
|
||||
- "List all available configs" doesn't work
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check configuration file:**
|
||||
```bash
|
||||
cat ~/.config/claude-code/mcp.json
|
||||
```
|
||||
|
||||
2. **Verify paths are ABSOLUTE (not placeholders):**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"skill-seeker": {
|
||||
"command": "python",
|
||||
"args": [
|
||||
"-m",
|
||||
"skill_seekers.mcp.server_fastmcp"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
❌ **Bad:** `$REPO_PATH` or `/path/to/Skill_Seekers`
|
||||
✅ **Good:** `/Users/john/Projects/Skill_Seekers`
|
||||
|
||||
3. **Test server manually:**
|
||||
```bash
|
||||
cd ~/Projects/Skill_Seekers
|
||||
python -m skill_seekers.mcp.server_fastmcp
|
||||
# Should start without errors (Ctrl+C to stop)
|
||||
```
|
||||
|
||||
4. **Re-run setup script:**
|
||||
```bash
|
||||
./setup_mcp.sh
|
||||
# Select "y" for auto-configure
|
||||
```
|
||||
|
||||
5. **RESTART Claude Code completely:**
|
||||
- Quit (don't just close window)
|
||||
- Reopen
|
||||
|
||||
### Placeholder Paths in Config
|
||||
|
||||
**Problem:** Config has `$REPO_PATH` or `/Users/username/` instead of real paths
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Get your actual path
|
||||
cd ~/Projects/Skill_Seekers
|
||||
pwd
|
||||
# Copy this path
|
||||
|
||||
# Edit config
|
||||
nano ~/.config/claude-code/mcp.json
|
||||
|
||||
# Replace ALL instances of placeholders with your actual path
|
||||
# Save (Ctrl+O, Enter, Ctrl+X)
|
||||
|
||||
# Restart Claude Code
|
||||
```
|
||||
|
||||
### Tools Appear But Don't Work
|
||||
|
||||
**Symptoms:**
|
||||
- Tools listed but commands fail
|
||||
- "Error executing tool" messages
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check working directory:**
|
||||
```json
|
||||
{
|
||||
"cwd": "/FULL/PATH/TO/Skill_Seekers"
|
||||
}
|
||||
```
|
||||
|
||||
2. **Verify package is installed:**
|
||||
```bash
|
||||
pip list | grep skill-seekers
|
||||
python -c "import skill_seekers; print(skill_seekers.__version__)"
|
||||
```
|
||||
|
||||
3. **Test CLI tools directly:**
|
||||
```bash
|
||||
skill-seekers create --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scraping Issues
|
||||
|
||||
### Slow or Hanging
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check network connection:**
|
||||
```bash
|
||||
ping google.com
|
||||
curl -I https://docs.yoursite.com
|
||||
```
|
||||
|
||||
2. **Use smaller max_pages for testing:**
|
||||
```bash
|
||||
skill-seekers create --config configs/test.json --max-pages 5
|
||||
```
|
||||
|
||||
3. **Increase rate_limit in config:**
|
||||
```json
|
||||
{
|
||||
"rate_limit": 1.0 // Increase from 0.5
|
||||
}
|
||||
```
|
||||
|
||||
### No Content Extracted
|
||||
|
||||
**Problem:** Pages scraped but content is empty
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Check selector in config:**
|
||||
```bash
|
||||
# Test with browser dev tools
|
||||
# Look for: article, main, div[role="main"], div.content
|
||||
```
|
||||
|
||||
2. **Verify website is accessible:**
|
||||
```bash
|
||||
curl https://docs.example.com
|
||||
```
|
||||
|
||||
3. **Try different selectors:**
|
||||
```json
|
||||
{
|
||||
"selectors": {
|
||||
"main_content": "article" // Try: main, div.content, etc.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Rate Limiting / 429 Errors
|
||||
|
||||
**Error:**
|
||||
```
|
||||
HTTP Error 429: Too Many Requests
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Increase rate_limit:**
|
||||
```json
|
||||
{
|
||||
"rate_limit": 2.0 // Wait 2 seconds between requests
|
||||
}
|
||||
```
|
||||
|
||||
2. **Reduce max_pages:**
|
||||
```json
|
||||
{
|
||||
"max_pages": 50 // Scrape fewer pages
|
||||
}
|
||||
```
|
||||
|
||||
3. **Try again later:**
|
||||
```bash
|
||||
# Wait an hour and retry
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Platform-Specific Issues
|
||||
|
||||
### macOS
|
||||
|
||||
**Issue:** Can't run `./setup_mcp.sh`
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
chmod +x setup_mcp.sh
|
||||
./setup_mcp.sh
|
||||
```
|
||||
|
||||
**Issue:** Homebrew not installed
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
|
||||
```
|
||||
|
||||
### Linux
|
||||
|
||||
**Issue:** pip3 not found
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install python3-pip
|
||||
```
|
||||
|
||||
**Issue:** Permission errors
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Use --user flag
|
||||
pip3 install --user -e .
|
||||
```
|
||||
|
||||
### Windows (WSL)
|
||||
|
||||
**Issue:** Python not in PATH
|
||||
|
||||
**Solution:**
|
||||
1. Reinstall Python
|
||||
2. Check "Add Python to PATH"
|
||||
3. Or add manually to PATH
|
||||
|
||||
**Issue:** Line ending errors
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
dos2unix setup_mcp.sh
|
||||
./setup_mcp.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Use these to check your setup:
|
||||
|
||||
```bash
|
||||
# 1. Check Python
|
||||
python3 --version # Should be 3.10+
|
||||
|
||||
# 2. Check package is installed
|
||||
pip list | grep skill-seekers
|
||||
python -c "import skill_seekers; print(skill_seekers.__version__)"
|
||||
|
||||
# 3. Check source layout
|
||||
ls src/skill_seekers/cli/
|
||||
ls src/skill_seekers/mcp/
|
||||
ls configs/
|
||||
|
||||
# 4. Check MCP config
|
||||
cat ~/.config/claude-code/mcp.json
|
||||
|
||||
# 5. Test scraper
|
||||
skill-seekers create --help
|
||||
|
||||
# 6. Test MCP server
|
||||
timeout 3 python -m skill_seekers.mcp.server_fastmcp || echo "Server OK"
|
||||
|
||||
# 7. Check git repo
|
||||
git status
|
||||
git log --oneline -5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
If none of these solutions work:
|
||||
|
||||
1. **Check existing issues:**
|
||||
https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
|
||||
2. **Open a new issue with:**
|
||||
- Your OS (macOS 13, Ubuntu 22.04, etc.)
|
||||
- Python version (`python3 --version`)
|
||||
- Full error message
|
||||
- What command you ran
|
||||
- Output of verification commands above
|
||||
|
||||
3. **Include this debug info:**
|
||||
```bash
|
||||
# System info
|
||||
uname -a
|
||||
python3 --version
|
||||
pip3 --version
|
||||
|
||||
# Skill Seeker info
|
||||
cd ~/Projects/Skill_Seekers # Your path
|
||||
pwd
|
||||
git log --oneline -1
|
||||
ls -la cli/ mcp/ configs/
|
||||
|
||||
# MCP config (if using MCP)
|
||||
cat ~/.config/claude-code/mcp.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Fixes Checklist
|
||||
|
||||
- [ ] In the Skill_Seekers directory? (`pwd`)
|
||||
- [ ] Python 3.10+ installed? (`python3 --version`)
|
||||
- [ ] Package installed? (`pip list | grep skill-seekers`)
|
||||
- [ ] Config file exists? (`ls configs/yourconfig.json`)
|
||||
- [ ] Internet connection working? (`ping google.com`)
|
||||
- [ ] For MCP: Config uses absolute paths? (not `$REPO_PATH`)
|
||||
- [ ] For MCP: Claude Code restarted? (quit and reopen)
|
||||
|
||||
---
|
||||
|
||||
**Still stuck?** Open an issue: https://github.com/yusufkaraaslan/Skill_Seekers/issues/new
|
||||
@@ -0,0 +1,2 @@
|
||||
# configs_repo is now a git submodule, tracked in .gitmodules
|
||||
# configs_repo/
|
||||
@@ -0,0 +1,267 @@
|
||||
# Skill Seekers Config API
|
||||
|
||||
FastAPI backend for discovering and downloading Skill Seekers configuration files.
|
||||
|
||||
## 🚀 Endpoints
|
||||
|
||||
### Base URL
|
||||
- **Production**: `https://skillseekersweb.com`
|
||||
- **Local**: `http://localhost:8000`
|
||||
|
||||
### Available Endpoints
|
||||
|
||||
#### 1. **GET /** - API Information
|
||||
Returns API metadata and available endpoints.
|
||||
|
||||
```bash
|
||||
curl https://skillseekersweb.com/
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"name": "Skill Seekers Config API",
|
||||
"version": "1.0.0",
|
||||
"endpoints": {
|
||||
"/api/configs": "List all available configs",
|
||||
"/api/configs/{name}": "Get specific config details",
|
||||
"/api/categories": "List all categories",
|
||||
"/docs": "API documentation"
|
||||
},
|
||||
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
|
||||
"website": "https://skillseekersweb.com"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 2. **GET /api/configs** - List All Configs
|
||||
Returns list of all available configs with metadata.
|
||||
|
||||
**Query Parameters:**
|
||||
- `category` (optional) - Filter by category (e.g., `web-frameworks`)
|
||||
- `tag` (optional) - Filter by tag (e.g., `javascript`)
|
||||
- `type` (optional) - Filter by type (`single-source` or `unified`)
|
||||
|
||||
```bash
|
||||
# Get all configs
|
||||
curl https://skillseekersweb.com/api/configs
|
||||
|
||||
# Filter by category
|
||||
curl https://skillseekersweb.com/api/configs?category=web-frameworks
|
||||
|
||||
# Filter by tag
|
||||
curl https://skillseekersweb.com/api/configs?tag=javascript
|
||||
|
||||
# Filter by type
|
||||
curl https://skillseekersweb.com/api/configs?type=unified
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"total": 24,
|
||||
"filters": null,
|
||||
"configs": [
|
||||
{
|
||||
"name": "react",
|
||||
"description": "React framework for building user interfaces...",
|
||||
"type": "single-source",
|
||||
"category": "web-frameworks",
|
||||
"tags": ["javascript", "frontend", "documentation"],
|
||||
"primary_source": "https://react.dev/",
|
||||
"max_pages": 300,
|
||||
"file_size": 1055,
|
||||
"last_updated": "2025-11-30T09:26:07+00:00",
|
||||
"download_url": "https://skillseekersweb.com/api/download/react.json",
|
||||
"config_file": "react.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 3. **GET /api/configs/{name}** - Get Specific Config
|
||||
Returns detailed information about a specific config.
|
||||
|
||||
```bash
|
||||
curl https://skillseekersweb.com/api/configs/react
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"name": "react",
|
||||
"description": "React framework for building user interfaces...",
|
||||
"type": "single-source",
|
||||
"category": "web-frameworks",
|
||||
"tags": ["javascript", "frontend", "documentation"],
|
||||
"primary_source": "https://react.dev/",
|
||||
"max_pages": 300,
|
||||
"file_size": 1055,
|
||||
"last_updated": "2025-11-30T09:26:07+00:00",
|
||||
"download_url": "https://skillseekersweb.com/api/download/react.json",
|
||||
"config_file": "react.json"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 4. **GET /api/categories** - List Categories
|
||||
Returns all available categories with config counts.
|
||||
|
||||
```bash
|
||||
curl https://skillseekersweb.com/api/categories
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"total_categories": 5,
|
||||
"categories": {
|
||||
"web-frameworks": 7,
|
||||
"game-engines": 2,
|
||||
"devops": 2,
|
||||
"css-frameworks": 1,
|
||||
"uncategorized": 12
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 5. **GET /api/download/{config_name}** - Download Config File
|
||||
Downloads the actual config JSON file.
|
||||
|
||||
```bash
|
||||
# Download react config
|
||||
curl -O https://skillseekersweb.com/api/download/react.json
|
||||
|
||||
# Download with just name (auto-adds .json)
|
||||
curl -O https://skillseekersweb.com/api/download/react
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 6. **GET /health** - Health Check
|
||||
Health check endpoint for monitoring.
|
||||
|
||||
```bash
|
||||
curl https://skillseekersweb.com/health
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"service": "skill-seekers-api"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 7. **GET /docs** - API Documentation
|
||||
Interactive OpenAPI documentation (Swagger UI).
|
||||
|
||||
Visit: `https://skillseekersweb.com/docs`
|
||||
|
||||
---
|
||||
|
||||
## 📦 Metadata Fields
|
||||
|
||||
Each config includes the following metadata:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `name` | string | Config identifier (e.g., "react") |
|
||||
| `description` | string | What the config is used for |
|
||||
| `type` | string | "single-source" or "unified" |
|
||||
| `category` | string | Auto-categorized (e.g., "web-frameworks") |
|
||||
| `tags` | array | Relevant tags (e.g., ["javascript", "frontend"]) |
|
||||
| `primary_source` | string | Main documentation URL or repo |
|
||||
| `max_pages` | int | Estimated page count for scraping |
|
||||
| `file_size` | int | Config file size in bytes |
|
||||
| `last_updated` | string | ISO 8601 date of last update |
|
||||
| `download_url` | string | Direct download link |
|
||||
| `config_file` | string | Filename (e.g., "react.json") |
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Categories
|
||||
|
||||
Configs are auto-categorized into:
|
||||
|
||||
- **web-frameworks** - Web development frameworks (React, Django, FastAPI, etc.)
|
||||
- **game-engines** - Game development engines (Godot, Unity, etc.)
|
||||
- **devops** - DevOps tools (Kubernetes, Ansible, etc.)
|
||||
- **css-frameworks** - CSS frameworks (Tailwind, etc.)
|
||||
- **development-tools** - Dev tools (Claude Code, etc.)
|
||||
- **gaming** - Gaming platforms (Steam, etc.)
|
||||
- **uncategorized** - Other configs
|
||||
|
||||
---
|
||||
|
||||
## 🏷️ Tags
|
||||
|
||||
Common tags include:
|
||||
|
||||
- **Language**: `javascript`, `python`, `php`
|
||||
- **Domain**: `frontend`, `backend`, `devops`, `game-development`
|
||||
- **Type**: `documentation`, `github`, `pdf`, `multi-source`
|
||||
- **Tech**: `css`, `testing`, `api`
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Local Development
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
cd api
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Run server
|
||||
python main.py
|
||||
```
|
||||
|
||||
API will be available at `http://localhost:8000`
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Test health check
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# List all configs
|
||||
curl http://localhost:8000/api/configs
|
||||
|
||||
# Get specific config
|
||||
curl http://localhost:8000/api/configs/react
|
||||
|
||||
# Download config
|
||||
curl -O http://localhost:8000/api/download/react.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Deployment
|
||||
|
||||
### Render
|
||||
|
||||
This API is configured for Render deployment via `render.yaml`.
|
||||
|
||||
1. Push to GitHub
|
||||
2. Connect repository to Render
|
||||
3. Render auto-deploys from `render.yaml`
|
||||
4. Configure custom domain: `skillseekersweb.com`
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
- **API Documentation**: https://skillseekersweb.com/docs
|
||||
- **GitHub Repository**: https://github.com/yusufkaraaslan/Skill_Seekers
|
||||
- **Main Project**: https://github.com/yusufkaraaslan/Skill_Seekers#readme
|
||||
@@ -0,0 +1,6 @@
|
||||
"""
|
||||
Skill Seekers Config API
|
||||
FastAPI backend for discovering and downloading config files
|
||||
"""
|
||||
|
||||
__version__ = "1.0.0"
|
||||
@@ -0,0 +1,366 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Config Analyzer - Extract metadata from Skill Seekers config files
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
class ConfigAnalyzer:
|
||||
"""Analyzes Skill Seekers config files and extracts metadata"""
|
||||
|
||||
# Category mapping based on config content
|
||||
CATEGORY_MAPPING = {
|
||||
"web-frameworks": ["react", "vue", "django", "fastapi", "laravel", "astro", "hono"],
|
||||
"game-engines": ["godot", "unity", "unreal"],
|
||||
"devops": ["kubernetes", "ansible", "docker", "terraform"],
|
||||
"css-frameworks": ["tailwind", "bootstrap", "bulma"],
|
||||
"development-tools": ["claude-code", "vscode", "git"],
|
||||
"gaming": ["steam"],
|
||||
"testing": ["pytest", "jest", "test"],
|
||||
}
|
||||
|
||||
# Tag extraction keywords
|
||||
TAG_KEYWORDS = {
|
||||
"javascript": ["react", "vue", "astro", "hono", "javascript", "js", "node"],
|
||||
"python": ["django", "fastapi", "ansible", "python", "flask"],
|
||||
"php": ["laravel", "php"],
|
||||
"frontend": ["react", "vue", "astro", "tailwind", "frontend", "ui"],
|
||||
"backend": ["django", "fastapi", "laravel", "backend", "server", "api"],
|
||||
"css": ["tailwind", "css", "styling"],
|
||||
"game-development": ["godot", "unity", "unreal", "game"],
|
||||
"devops": ["kubernetes", "ansible", "docker", "k8s", "devops"],
|
||||
"documentation": ["docs", "documentation"],
|
||||
"testing": ["test", "testing", "pytest", "jest"],
|
||||
}
|
||||
|
||||
def __init__(self, config_dir: Path, base_url: str = "https://api.skillseekersweb.com"):
|
||||
"""
|
||||
Initialize config analyzer
|
||||
|
||||
Args:
|
||||
config_dir: Path to configs directory
|
||||
base_url: Base URL for download links
|
||||
"""
|
||||
self.config_dir = Path(config_dir)
|
||||
self.base_url = base_url
|
||||
|
||||
if not self.config_dir.exists():
|
||||
raise ValueError(f"Config directory not found: {self.config_dir}")
|
||||
|
||||
def analyze_all_configs(self) -> list[dict[str, Any]]:
|
||||
"""
|
||||
Analyze all config files and extract metadata
|
||||
|
||||
Returns:
|
||||
List of config metadata dicts
|
||||
"""
|
||||
configs = []
|
||||
|
||||
# Find all JSON files recursively in configs directory and subdirectories
|
||||
for config_file in sorted(self.config_dir.rglob("*.json")):
|
||||
# Skip test/example configs in test-examples directory
|
||||
if "test-examples" in config_file.parts:
|
||||
continue
|
||||
|
||||
try:
|
||||
metadata = self.analyze_config(config_file)
|
||||
if metadata: # Skip invalid configs
|
||||
configs.append(metadata)
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to analyze {config_file.name}: {e}")
|
||||
continue
|
||||
|
||||
return configs
|
||||
|
||||
def analyze_config(self, config_path: Path) -> dict[str, Any] | None:
|
||||
"""
|
||||
Analyze a single config file and extract metadata
|
||||
|
||||
Args:
|
||||
config_path: Path to config JSON file
|
||||
|
||||
Returns:
|
||||
Config metadata dict or None if invalid
|
||||
"""
|
||||
try:
|
||||
# Read config file
|
||||
with open(config_path) as f:
|
||||
config_data = json.load(f)
|
||||
|
||||
# Skip if no name field
|
||||
if "name" not in config_data:
|
||||
return None
|
||||
|
||||
name = config_data["name"]
|
||||
description = config_data.get("description", "")
|
||||
|
||||
# Determine config type
|
||||
config_type = self._determine_type(config_data)
|
||||
|
||||
# Get primary source (base_url or repo)
|
||||
primary_source = self._get_primary_source(config_data, config_type)
|
||||
|
||||
# Use directory name as category (official/{category}/{name}.json)
|
||||
# Fall back to keyword-based categorization if not in a named subdirectory
|
||||
category = self._categorize_config(name, description, config_data, config_path)
|
||||
|
||||
# Extract tags
|
||||
tags = self._extract_tags(name, description, config_data)
|
||||
|
||||
# Get file metadata
|
||||
file_size = config_path.stat().st_size
|
||||
last_updated = self._get_last_updated(config_path)
|
||||
|
||||
# Generate download URL
|
||||
download_url = f"{self.base_url}/api/download/{config_path.name}"
|
||||
|
||||
# Get max_pages (for estimation)
|
||||
max_pages = self._get_max_pages(config_data)
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"description": description,
|
||||
"type": config_type,
|
||||
"category": category,
|
||||
"tags": tags,
|
||||
"primary_source": primary_source,
|
||||
"max_pages": max_pages,
|
||||
"file_size": file_size,
|
||||
"last_updated": last_updated,
|
||||
"download_url": download_url,
|
||||
"config_file": config_path.name,
|
||||
}
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Invalid JSON in {config_path.name}: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
print(f"Error analyzing {config_path.name}: {e}")
|
||||
return None
|
||||
|
||||
def get_config_by_name(self, name: str) -> dict[str, Any] | None:
|
||||
"""
|
||||
Get config metadata by name
|
||||
|
||||
Args:
|
||||
name: Config name (e.g., "react", "django")
|
||||
|
||||
Returns:
|
||||
Config metadata or None if not found
|
||||
"""
|
||||
configs = self.analyze_all_configs()
|
||||
for config in configs:
|
||||
if config["name"] == name:
|
||||
return config
|
||||
return None
|
||||
|
||||
def _determine_type(self, config_data: dict[str, Any]) -> str:
|
||||
"""
|
||||
Determine if config is single-source or unified
|
||||
|
||||
Args:
|
||||
config_data: Config JSON data
|
||||
|
||||
Returns:
|
||||
"single-source" or "unified"
|
||||
"""
|
||||
# Unified configs have "sources" array
|
||||
if "sources" in config_data:
|
||||
return "unified"
|
||||
|
||||
# Check for merge_mode (another indicator of unified configs)
|
||||
if "merge_mode" in config_data:
|
||||
return "unified"
|
||||
|
||||
return "single-source"
|
||||
|
||||
def _get_primary_source(self, config_data: dict[str, Any], config_type: str) -> str:
|
||||
"""
|
||||
Get primary source URL/repo
|
||||
|
||||
Args:
|
||||
config_data: Config JSON data
|
||||
config_type: "single-source" or "unified"
|
||||
|
||||
Returns:
|
||||
Primary source URL or repo name
|
||||
"""
|
||||
if config_type == "unified":
|
||||
# Get first source
|
||||
sources = config_data.get("sources", [])
|
||||
if sources:
|
||||
first_source = sources[0]
|
||||
if first_source.get("type") == "documentation":
|
||||
return first_source.get("base_url", "")
|
||||
elif first_source.get("type") == "github":
|
||||
return f"github.com/{first_source.get('repo', '')}"
|
||||
elif first_source.get("type") == "pdf":
|
||||
return first_source.get("pdf_url", "PDF file")
|
||||
return "Multiple sources"
|
||||
|
||||
# Single-source configs
|
||||
if "base_url" in config_data:
|
||||
return config_data["base_url"]
|
||||
elif "repo" in config_data:
|
||||
return f"github.com/{config_data['repo']}"
|
||||
elif "pdf_url" in config_data or "pdf" in config_data:
|
||||
return "PDF file"
|
||||
|
||||
return "Unknown"
|
||||
|
||||
def _categorize_config(
|
||||
self,
|
||||
name: str,
|
||||
description: str,
|
||||
config_data: dict[str, Any],
|
||||
config_path: Path | None = None,
|
||||
) -> str:
|
||||
"""
|
||||
Categorize config using directory structure first, then keyword fallback.
|
||||
|
||||
The configs_repo organizes files as official/{category}/{name}.json so the
|
||||
parent directory name is the authoritative category.
|
||||
|
||||
Args:
|
||||
name: Config name
|
||||
description: Config description
|
||||
config_data: Full config data
|
||||
config_path: Path to config file (used to read directory-based category)
|
||||
|
||||
Returns:
|
||||
Category name
|
||||
"""
|
||||
# Primary: use directory structure (official/{category}/{name}.json)
|
||||
if config_path is not None:
|
||||
parent = config_path.parent.name
|
||||
# Exclude generic/root directories from being used as categories
|
||||
if parent not in ("official", "community", "configs", "configs_repo", "."):
|
||||
return parent
|
||||
|
||||
# Fallback: keyword matching against config name
|
||||
name_lower = name.lower()
|
||||
for category, keywords in self.CATEGORY_MAPPING.items():
|
||||
if any(keyword in name_lower for keyword in keywords):
|
||||
return category
|
||||
|
||||
# Fallback: description hints
|
||||
desc_lower = description.lower()
|
||||
if "framework" in desc_lower or "library" in desc_lower:
|
||||
if any(word in desc_lower for word in ["web", "frontend", "backend", "api"]):
|
||||
return "web-frameworks"
|
||||
|
||||
if "game" in desc_lower or "engine" in desc_lower:
|
||||
return "game-engines"
|
||||
|
||||
if "devops" in desc_lower or "deployment" in desc_lower or "infrastructure" in desc_lower:
|
||||
return "devops"
|
||||
|
||||
return "uncategorized"
|
||||
|
||||
def _extract_tags(self, name: str, description: str, config_data: dict[str, Any]) -> list[str]:
|
||||
"""
|
||||
Extract relevant tags from config
|
||||
|
||||
Args:
|
||||
name: Config name
|
||||
description: Config description
|
||||
config_data: Full config data
|
||||
|
||||
Returns:
|
||||
List of tags
|
||||
"""
|
||||
tags = set()
|
||||
name_lower = name.lower()
|
||||
desc_lower = description.lower()
|
||||
|
||||
# Check against tag keywords
|
||||
for tag, keywords in self.TAG_KEYWORDS.items():
|
||||
if any(keyword in name_lower or keyword in desc_lower for keyword in keywords):
|
||||
tags.add(tag)
|
||||
|
||||
# Add config type as tag
|
||||
config_type = self._determine_type(config_data)
|
||||
if config_type == "unified":
|
||||
tags.add("multi-source")
|
||||
|
||||
# Add source type tags
|
||||
if "base_url" in config_data or (
|
||||
config_type == "unified"
|
||||
and any(s.get("type") == "documentation" for s in config_data.get("sources", []))
|
||||
):
|
||||
tags.add("documentation")
|
||||
|
||||
if "repo" in config_data or (
|
||||
config_type == "unified"
|
||||
and any(s.get("type") == "github" for s in config_data.get("sources", []))
|
||||
):
|
||||
tags.add("github")
|
||||
|
||||
if (
|
||||
"pdf" in config_data
|
||||
or "pdf_url" in config_data
|
||||
or (
|
||||
config_type == "unified"
|
||||
and any(s.get("type") == "pdf" for s in config_data.get("sources", []))
|
||||
)
|
||||
):
|
||||
tags.add("pdf")
|
||||
|
||||
return sorted(list(tags))
|
||||
|
||||
def _get_max_pages(self, config_data: dict[str, Any]) -> int | None:
|
||||
"""
|
||||
Get max_pages value from config
|
||||
|
||||
Args:
|
||||
config_data: Config JSON data
|
||||
|
||||
Returns:
|
||||
max_pages value or None
|
||||
"""
|
||||
# Single-source configs
|
||||
if "max_pages" in config_data:
|
||||
return config_data["max_pages"]
|
||||
|
||||
# Unified configs - get from first documentation source
|
||||
if "sources" in config_data:
|
||||
for source in config_data["sources"]:
|
||||
if source.get("type") == "documentation" and "max_pages" in source:
|
||||
return source["max_pages"]
|
||||
|
||||
return None
|
||||
|
||||
def _get_last_updated(self, config_path: Path) -> str:
|
||||
"""
|
||||
Get last updated date from git history
|
||||
|
||||
Args:
|
||||
config_path: Path to config file
|
||||
|
||||
Returns:
|
||||
ISO format date string
|
||||
"""
|
||||
try:
|
||||
# Try to get last commit date for this file
|
||||
result = subprocess.run(
|
||||
["git", "log", "-1", "--format=%cI", str(config_path)],
|
||||
cwd=config_path.parent.parent,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
|
||||
if result.returncode == 0 and result.stdout.strip():
|
||||
return result.stdout.strip()
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Fallback to file modification time
|
||||
mtime = config_path.stat().st_mtime
|
||||
return datetime.fromtimestamp(mtime).isoformat()
|
||||
@@ -0,0 +1,204 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Skill Seekers Config API
|
||||
FastAPI backend for listing available skill configs
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from config_analyzer import ConfigAnalyzer
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
app = FastAPI(
|
||||
title="Skill Seekers Config API",
|
||||
description="API for discovering and downloading Skill Seekers configuration files",
|
||||
version="1.0.0",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
)
|
||||
|
||||
# CORS middleware - allow all origins for public API
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Initialize config analyzer
|
||||
# Try configs_repo first (production), fallback to configs (local development)
|
||||
CONFIG_DIR = Path(__file__).parent / "configs_repo" / "official"
|
||||
if not CONFIG_DIR.exists():
|
||||
CONFIG_DIR = Path(__file__).parent.parent / "configs"
|
||||
|
||||
analyzer = ConfigAnalyzer(CONFIG_DIR)
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def root():
|
||||
"""Root endpoint - API information"""
|
||||
return {
|
||||
"name": "Skill Seekers Config API",
|
||||
"version": "1.0.0",
|
||||
"endpoints": {
|
||||
"/api/configs": "List all available configs",
|
||||
"/api/configs/{name}": "Get specific config details",
|
||||
"/api/categories": "List all categories",
|
||||
"/api/download/{name}": "Download config file",
|
||||
"/docs": "API documentation",
|
||||
},
|
||||
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
|
||||
"configs_repository": "https://github.com/yusufkaraaslan/skill-seekers-configs",
|
||||
"website": "https://api.skillseekersweb.com",
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/configs")
|
||||
async def list_configs(
|
||||
category: str | None = None, tag: str | None = None, type: str | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
List all available configs with metadata
|
||||
|
||||
Query Parameters:
|
||||
- category: Filter by category (e.g., "web-frameworks")
|
||||
- tag: Filter by tag (e.g., "javascript")
|
||||
- type: Filter by type ("single-source" or "unified")
|
||||
|
||||
Returns:
|
||||
- version: API version
|
||||
- total: Total number of configs
|
||||
- filters: Applied filters
|
||||
- configs: List of config metadata
|
||||
"""
|
||||
try:
|
||||
# Get all configs
|
||||
all_configs = analyzer.analyze_all_configs()
|
||||
|
||||
# Apply filters
|
||||
configs = all_configs
|
||||
filters_applied = {}
|
||||
|
||||
if category:
|
||||
configs = [c for c in configs if c.get("category") == category]
|
||||
filters_applied["category"] = category
|
||||
|
||||
if tag:
|
||||
configs = [c for c in configs if tag in c.get("tags", [])]
|
||||
filters_applied["tag"] = tag
|
||||
|
||||
if type:
|
||||
configs = [c for c in configs if c.get("type") == type]
|
||||
filters_applied["type"] = type
|
||||
|
||||
return {
|
||||
"version": "1.0.0",
|
||||
"total": len(configs),
|
||||
"filters": filters_applied if filters_applied else None,
|
||||
"configs": configs,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Error analyzing configs: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/api/configs/{name}")
|
||||
async def get_config(name: str) -> dict[str, Any]:
|
||||
"""
|
||||
Get detailed information about a specific config
|
||||
|
||||
Path Parameters:
|
||||
- name: Config name (e.g., "react", "django")
|
||||
|
||||
Returns:
|
||||
- Full config metadata including all fields
|
||||
"""
|
||||
try:
|
||||
config = analyzer.get_config_by_name(name)
|
||||
|
||||
if not config:
|
||||
raise HTTPException(status_code=404, detail=f"Config '{name}' not found")
|
||||
|
||||
return config
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Error loading config: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/api/categories")
|
||||
async def list_categories() -> dict[str, Any]:
|
||||
"""
|
||||
List all available categories with config counts
|
||||
|
||||
Returns:
|
||||
- categories: Dict of category names to config counts
|
||||
- total_categories: Total number of categories
|
||||
"""
|
||||
try:
|
||||
configs = analyzer.analyze_all_configs()
|
||||
|
||||
# Count configs per category
|
||||
category_counts = {}
|
||||
for config in configs:
|
||||
cat = config.get("category", "uncategorized")
|
||||
category_counts[cat] = category_counts.get(cat, 0) + 1
|
||||
|
||||
return {"total_categories": len(category_counts), "categories": category_counts}
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Error analyzing categories: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/api/download/{config_name}")
|
||||
async def download_config(config_name: str):
|
||||
"""
|
||||
Download a specific config file
|
||||
|
||||
Path Parameters:
|
||||
- config_name: Config filename (e.g., "react.json", "django.json")
|
||||
|
||||
Returns:
|
||||
- JSON file for download
|
||||
"""
|
||||
try:
|
||||
# Validate filename (prevent directory traversal)
|
||||
if ".." in config_name or "/" in config_name or "\\" in config_name:
|
||||
raise HTTPException(status_code=400, detail="Invalid config name")
|
||||
|
||||
# Ensure .json extension
|
||||
if not config_name.endswith(".json"):
|
||||
config_name = f"{config_name}.json"
|
||||
|
||||
# Search recursively in all subdirectories
|
||||
config_path = None
|
||||
for found_path in CONFIG_DIR.rglob(config_name):
|
||||
config_path = found_path
|
||||
break
|
||||
|
||||
if not config_path or not config_path.exists():
|
||||
raise HTTPException(status_code=404, detail=f"Config file '{config_name}' not found")
|
||||
|
||||
return FileResponse(path=config_path, media_type="application/json", filename=config_name)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Error downloading config: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
"""Health check endpoint for monitoring"""
|
||||
return {"status": "healthy", "service": "skill-seekers-api"}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
@@ -0,0 +1,3 @@
|
||||
fastapi==0.115.0
|
||||
uvicorn[standard]==0.32.0
|
||||
python-multipart==0.0.12
|
||||
@@ -0,0 +1,25 @@
|
||||
coverage:
|
||||
status:
|
||||
project:
|
||||
default:
|
||||
target: auto
|
||||
threshold: 1%
|
||||
informational: false
|
||||
patch:
|
||||
default:
|
||||
# Patch coverage is informational only — doesn't fail CI.
|
||||
# Rationale: version bumps, docs, config, and test-only commits have
|
||||
# naturally low or zero patch coverage and shouldn't block releases.
|
||||
informational: true
|
||||
|
||||
comment:
|
||||
layout: "reach, diff, flags, files"
|
||||
behavior: default
|
||||
require_changes: false
|
||||
|
||||
# Ignore files that don't need coverage
|
||||
ignore:
|
||||
- "tests/**/*"
|
||||
- "docs/**/*"
|
||||
- "**/__init__.py"
|
||||
- "src/skill_seekers/_version.py"
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "astrovalley",
|
||||
"description": "Space farming/automation game with combat and exploration - GitHub repo with deep codebase analysis",
|
||||
"version": "1.0.0",
|
||||
"target": "claude",
|
||||
"sources": [
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "yusufkaraaslan/AstroValley",
|
||||
"clone_path": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/AstroValley",
|
||||
"enable_codebase_analysis": true,
|
||||
"analysis_depth": "deep",
|
||||
"include_issues": true,
|
||||
"include_pull_requests": true,
|
||||
"include_discussions": false,
|
||||
"max_issues": 100,
|
||||
"codebase_options": {
|
||||
"build_api_reference": true,
|
||||
"build_dependency_graph": true,
|
||||
"detect_patterns": true,
|
||||
"extract_test_examples": true,
|
||||
"extract_config_patterns": true,
|
||||
"detect_architecture": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"synthesis_strategy": "comprehensive",
|
||||
"ai_enhancement": {
|
||||
"mode": "auto",
|
||||
"enable": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
{
|
||||
"name": "blender",
|
||||
"description": "Complete Blender 3D creation suite knowledge base combining official documentation and source code analysis. Use for comprehensive understanding of 3D modeling, animation, rendering, compositing, video editing, game development, Python scripting, and Blender's internal architecture.",
|
||||
"merge_mode": "claude-enhanced",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://docs.blender.org/manual/en/latest/",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "article[role='main']",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code, div.highlight pre"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/getting_started/",
|
||||
"/interface/",
|
||||
"/editors/",
|
||||
"/modeling/",
|
||||
"/sculpt_paint/",
|
||||
"/grease_pencil/",
|
||||
"/animation/",
|
||||
"/physics/",
|
||||
"/render/",
|
||||
"/scene_layout/",
|
||||
"/compositing/",
|
||||
"/video_editing/",
|
||||
"/files/",
|
||||
"/addons/",
|
||||
"/advanced/",
|
||||
"/troubleshooting/"
|
||||
],
|
||||
"exclude": [
|
||||
"/_static/",
|
||||
"/_images/",
|
||||
"/search.html",
|
||||
"/genindex.html",
|
||||
"/glossary.html",
|
||||
"/index.html$"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"getting_started",
|
||||
"installing",
|
||||
"configuration",
|
||||
"introduction",
|
||||
"quickstart",
|
||||
"about"
|
||||
],
|
||||
"interface": [
|
||||
"interface",
|
||||
"window_system",
|
||||
"keymap",
|
||||
"controls",
|
||||
"operators",
|
||||
"tools",
|
||||
"ui",
|
||||
"navigation"
|
||||
],
|
||||
"modeling": [
|
||||
"modeling",
|
||||
"mesh",
|
||||
"curve",
|
||||
"surface",
|
||||
"metaball",
|
||||
"text",
|
||||
"volume",
|
||||
"geometry_nodes",
|
||||
"modifiers",
|
||||
"mesh_tools",
|
||||
"edit_mode"
|
||||
],
|
||||
"sculpting": [
|
||||
"sculpt",
|
||||
"sculpting",
|
||||
"brush",
|
||||
"texture_paint",
|
||||
"vertex_paint",
|
||||
"weight_paint",
|
||||
"dynamic_paint"
|
||||
],
|
||||
"grease_pencil": [
|
||||
"grease_pencil",
|
||||
"2d_animation",
|
||||
"drawing",
|
||||
"stroke"
|
||||
],
|
||||
"animation": [
|
||||
"animation",
|
||||
"keyframe",
|
||||
"rigging",
|
||||
"armature",
|
||||
"constraints",
|
||||
"drivers",
|
||||
"shape_keys",
|
||||
"motion_paths",
|
||||
"timeline",
|
||||
"dope_sheet",
|
||||
"graph_editor",
|
||||
"nla"
|
||||
],
|
||||
"physics": [
|
||||
"physics",
|
||||
"simulation",
|
||||
"particles",
|
||||
"hair",
|
||||
"fluid",
|
||||
"cloth",
|
||||
"soft_body",
|
||||
"rigid_body",
|
||||
"dynamic_paint",
|
||||
"force_fields"
|
||||
],
|
||||
"shading": [
|
||||
"shading",
|
||||
"shader",
|
||||
"material",
|
||||
"texture",
|
||||
"nodes",
|
||||
"shader_nodes",
|
||||
"lighting",
|
||||
"world"
|
||||
],
|
||||
"rendering": [
|
||||
"render",
|
||||
"eevee",
|
||||
"cycles",
|
||||
"workbench",
|
||||
"freestyle",
|
||||
"camera",
|
||||
"output",
|
||||
"color_management",
|
||||
"optimization"
|
||||
],
|
||||
"compositing": [
|
||||
"compositing",
|
||||
"compositor",
|
||||
"nodes",
|
||||
"color_correction",
|
||||
"filters",
|
||||
"matte"
|
||||
],
|
||||
"video_editing": [
|
||||
"video_editing",
|
||||
"vse",
|
||||
"sequencer",
|
||||
"strips",
|
||||
"effects",
|
||||
"preview"
|
||||
],
|
||||
"scene_layout": [
|
||||
"scene",
|
||||
"object",
|
||||
"collection",
|
||||
"properties",
|
||||
"outliner",
|
||||
"view_layers"
|
||||
],
|
||||
"files_assets": [
|
||||
"files",
|
||||
"import",
|
||||
"export",
|
||||
"asset",
|
||||
"library",
|
||||
"data_blocks",
|
||||
"linking",
|
||||
"append"
|
||||
],
|
||||
"addons": [
|
||||
"addon",
|
||||
"plugin",
|
||||
"extension",
|
||||
"import_export"
|
||||
],
|
||||
"scripting": [
|
||||
"scripting",
|
||||
"python",
|
||||
"api",
|
||||
"bpy",
|
||||
"operator",
|
||||
"custom",
|
||||
"automation"
|
||||
],
|
||||
"advanced": [
|
||||
"advanced",
|
||||
"command_line",
|
||||
"app_templates",
|
||||
"extensions",
|
||||
"limits"
|
||||
],
|
||||
"troubleshooting": [
|
||||
"troubleshooting",
|
||||
"crash",
|
||||
"recover",
|
||||
"gpu",
|
||||
"graphics"
|
||||
]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 1500
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "blender/blender",
|
||||
"github_token": null,
|
||||
"code_analysis_depth": "deep",
|
||||
"include_code": true,
|
||||
"include_issues": true,
|
||||
"max_issues": 200,
|
||||
"include_changelog": true,
|
||||
"include_releases": true,
|
||||
"include_wiki": true,
|
||||
"file_patterns": [
|
||||
"source/blender/blenkernel/**/*.h",
|
||||
"source/blender/blenkernel/**/*.c",
|
||||
"source/blender/blenkernel/**/*.cc",
|
||||
"source/blender/blenlib/**/*.h",
|
||||
"source/blender/blenlib/**/*.c",
|
||||
"source/blender/blenlib/**/*.cc",
|
||||
"source/blender/editors/**/*.h",
|
||||
"source/blender/editors/**/*.c",
|
||||
"source/blender/editors/**/*.cc",
|
||||
"source/blender/makesdna/**/*.h",
|
||||
"source/blender/makesrna/**/*.c",
|
||||
"source/blender/makesrna/**/*.cc",
|
||||
"source/blender/render/**/*.h",
|
||||
"source/blender/render/**/*.c",
|
||||
"source/blender/render/**/*.cc",
|
||||
"source/blender/python/**/*.h",
|
||||
"source/blender/python/**/*.c",
|
||||
"source/blender/python/**/*.cc",
|
||||
"source/blender/python/**/*.py",
|
||||
"source/blender/depsgraph/**/*.h",
|
||||
"source/blender/depsgraph/**/*.cc",
|
||||
"source/blender/draw/**/*.h",
|
||||
"source/blender/draw/**/*.c",
|
||||
"source/blender/draw/**/*.cc",
|
||||
"source/blender/gpu/**/*.h",
|
||||
"source/blender/gpu/**/*.c",
|
||||
"source/blender/gpu/**/*.cc",
|
||||
"source/blender/nodes/**/*.h",
|
||||
"source/blender/nodes/**/*.c",
|
||||
"source/blender/nodes/**/*.cc",
|
||||
"source/blender/windowmanager/**/*.h",
|
||||
"source/blender/windowmanager/**/*.c",
|
||||
"source/blender/windowmanager/**/*.cc",
|
||||
"intern/cycles/**/*.h",
|
||||
"intern/cycles/**/*.cpp",
|
||||
"scripts/startup/bl_ui/**/*.py",
|
||||
"scripts/modules/**/*.py",
|
||||
"release/scripts/startup/**/*.py",
|
||||
"README.md",
|
||||
"CONTRIBUTING.md",
|
||||
"BUILD.md",
|
||||
"CODE_OF_CONDUCT.md"
|
||||
],
|
||||
"exclude_patterns": [
|
||||
"**/tests/**",
|
||||
"**/__pycache__/**",
|
||||
"build_files/**",
|
||||
"doc/**"
|
||||
],
|
||||
"analysis_features": {
|
||||
"detect_patterns": true,
|
||||
"extract_tests": true,
|
||||
"build_guides": true,
|
||||
"extract_config": true,
|
||||
"build_api_reference": true,
|
||||
"analyze_dependencies": true,
|
||||
"detect_architecture": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
{
|
||||
"name": "claude-code-unified",
|
||||
"description": "Claude Code CLI - unified knowledge base combining official documentation and leaked source code analysis. Use for understanding Claude Code internals, architecture, tools, commands, IDE integrations, MCP, plugins, skills, and development workflows.",
|
||||
"merge_mode": "claude-enhanced",
|
||||
"workflows": [
|
||||
"complex-merge",
|
||||
"cli-tooling",
|
||||
"architecture-comprehensive",
|
||||
"api-documentation",
|
||||
"security-focus",
|
||||
"ai-assistant-internals"
|
||||
],
|
||||
"workflow_vars": {
|
||||
"depth": "comprehensive",
|
||||
"merge_strategy": "priority",
|
||||
"source_priority_order": "official_docs,code,community"
|
||||
},
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://code.claude.com/docs/en/",
|
||||
"start_urls": [
|
||||
"https://code.claude.com/docs/en/overview",
|
||||
"https://code.claude.com/docs/en/quickstart",
|
||||
"https://code.claude.com/docs/en/common-workflows",
|
||||
"https://code.claude.com/docs/en/claude-code-on-the-web",
|
||||
"https://code.claude.com/docs/en/desktop",
|
||||
"https://code.claude.com/docs/en/chrome",
|
||||
"https://code.claude.com/docs/en/vs-code",
|
||||
"https://code.claude.com/docs/en/jetbrains",
|
||||
"https://code.claude.com/docs/en/github-actions",
|
||||
"https://code.claude.com/docs/en/gitlab-ci-cd",
|
||||
"https://code.claude.com/docs/en/slack",
|
||||
"https://code.claude.com/docs/en/sub-agents",
|
||||
"https://code.claude.com/docs/en/plugins",
|
||||
"https://code.claude.com/docs/en/discover-plugins",
|
||||
"https://code.claude.com/docs/en/skills",
|
||||
"https://code.claude.com/docs/en/output-styles",
|
||||
"https://code.claude.com/docs/en/hooks-guide",
|
||||
"https://code.claude.com/docs/en/headless",
|
||||
"https://code.claude.com/docs/en/mcp",
|
||||
"https://code.claude.com/docs/en/third-party-integrations",
|
||||
"https://code.claude.com/docs/en/amazon-bedrock",
|
||||
"https://code.claude.com/docs/en/google-vertex-ai",
|
||||
"https://code.claude.com/docs/en/microsoft-foundry",
|
||||
"https://code.claude.com/docs/en/network-config",
|
||||
"https://code.claude.com/docs/en/llm-gateway",
|
||||
"https://code.claude.com/docs/en/devcontainer",
|
||||
"https://code.claude.com/docs/en/sandboxing",
|
||||
"https://code.claude.com/docs/en/setup",
|
||||
"https://code.claude.com/docs/en/iam",
|
||||
"https://code.claude.com/docs/en/security",
|
||||
"https://code.claude.com/docs/en/data-usage",
|
||||
"https://code.claude.com/docs/en/monitoring-usage",
|
||||
"https://code.claude.com/docs/en/costs",
|
||||
"https://code.claude.com/docs/en/analytics",
|
||||
"https://code.claude.com/docs/en/plugin-marketplaces",
|
||||
"https://code.claude.com/docs/en/settings",
|
||||
"https://code.claude.com/docs/en/terminal-config",
|
||||
"https://code.claude.com/docs/en/model-config",
|
||||
"https://code.claude.com/docs/en/memory",
|
||||
"https://code.claude.com/docs/en/statusline",
|
||||
"https://code.claude.com/docs/en/cli-reference",
|
||||
"https://code.claude.com/docs/en/interactive-mode",
|
||||
"https://code.claude.com/docs/en/slash-commands",
|
||||
"https://code.claude.com/docs/en/checkpointing",
|
||||
"https://code.claude.com/docs/en/hooks",
|
||||
"https://code.claude.com/docs/en/plugins-reference",
|
||||
"https://code.claude.com/docs/en/troubleshooting",
|
||||
"https://code.claude.com/docs/en/legal-and-compliance"
|
||||
],
|
||||
"selectors": {
|
||||
"main_content": "#content-area, #content-container, article, main",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/docs/en/"
|
||||
],
|
||||
"exclude": [
|
||||
"/docs/fr/",
|
||||
"/docs/de/",
|
||||
"/docs/it/",
|
||||
"/docs/ja/",
|
||||
"/docs/es/",
|
||||
"/docs/ko/",
|
||||
"/docs/zh-CN/",
|
||||
"/docs/zh-TW/",
|
||||
"/docs/ru/",
|
||||
"/docs/id/",
|
||||
"/docs/pt/",
|
||||
"/changelog",
|
||||
"github.com"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"overview",
|
||||
"quickstart",
|
||||
"common-workflows"
|
||||
],
|
||||
"ide_integrations": [
|
||||
"vs-code",
|
||||
"jetbrains",
|
||||
"desktop",
|
||||
"chrome",
|
||||
"claude-code-on-the-web",
|
||||
"slack"
|
||||
],
|
||||
"ci_cd": [
|
||||
"github-actions",
|
||||
"gitlab-ci-cd"
|
||||
],
|
||||
"building": [
|
||||
"sub-agents",
|
||||
"subagent",
|
||||
"plugins",
|
||||
"discover-plugins",
|
||||
"skills",
|
||||
"output-styles",
|
||||
"hooks-guide",
|
||||
"headless",
|
||||
"programmatic"
|
||||
],
|
||||
"mcp": [
|
||||
"mcp",
|
||||
"model-context-protocol"
|
||||
],
|
||||
"deployment": [
|
||||
"third-party-integrations",
|
||||
"amazon-bedrock",
|
||||
"google-vertex-ai",
|
||||
"microsoft-foundry",
|
||||
"network-config",
|
||||
"llm-gateway",
|
||||
"devcontainer",
|
||||
"sandboxing"
|
||||
],
|
||||
"administration": [
|
||||
"setup",
|
||||
"iam",
|
||||
"security",
|
||||
"data-usage",
|
||||
"monitoring-usage",
|
||||
"costs",
|
||||
"analytics",
|
||||
"plugin-marketplaces"
|
||||
],
|
||||
"configuration": [
|
||||
"settings",
|
||||
"terminal-config",
|
||||
"model-config",
|
||||
"memory",
|
||||
"statusline"
|
||||
],
|
||||
"reference": [
|
||||
"cli-reference",
|
||||
"interactive-mode",
|
||||
"slash-commands",
|
||||
"checkpointing",
|
||||
"hooks",
|
||||
"plugins-reference"
|
||||
],
|
||||
"troubleshooting": [
|
||||
"troubleshooting"
|
||||
],
|
||||
"legal": [
|
||||
"legal-and-compliance"
|
||||
]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 250
|
||||
},
|
||||
{
|
||||
"type": "local",
|
||||
"path": "/Users/yusufkaraaslan/Github/cli-template",
|
||||
"name": "claude-code-source",
|
||||
"description": "Leaked Claude Code CLI TypeScript source code (main.tsx, tools, commands, components, Ink renderer, bridge, coordinator, plugins, skills, MCP, vim, voice, remote, server, memdir, tasks, state, schemas, entrypoints)",
|
||||
"weight": 0.6,
|
||||
"enhance_level": 3,
|
||||
"include_code": true,
|
||||
"file_patterns": [
|
||||
"*.ts",
|
||||
"*.tsx",
|
||||
"*.md",
|
||||
"*.json"
|
||||
],
|
||||
"skip_patterns": [
|
||||
".git/",
|
||||
"node_modules/",
|
||||
"__pycache__/",
|
||||
"*.map",
|
||||
"*.js",
|
||||
"dist/",
|
||||
"build/"
|
||||
],
|
||||
"categories": {
|
||||
"core": [
|
||||
"main.tsx",
|
||||
"commands.ts",
|
||||
"tools.ts",
|
||||
"Tool.ts",
|
||||
"QueryEngine.ts",
|
||||
"context.ts",
|
||||
"cost-tracker.ts"
|
||||
],
|
||||
"commands": [
|
||||
"commands/"
|
||||
],
|
||||
"tools": [
|
||||
"tools/"
|
||||
],
|
||||
"ui": [
|
||||
"components/",
|
||||
"hooks/",
|
||||
"screens/",
|
||||
"ink/"
|
||||
],
|
||||
"integrations": [
|
||||
"bridge/",
|
||||
"coordinator/",
|
||||
"remote/",
|
||||
"server/",
|
||||
"services/"
|
||||
],
|
||||
"features": [
|
||||
"plugins/",
|
||||
"skills/",
|
||||
"vim/",
|
||||
"voice/",
|
||||
"tasks/",
|
||||
"memdir/",
|
||||
"state/",
|
||||
"buddy/"
|
||||
],
|
||||
"infrastructure": [
|
||||
"entrypoints/",
|
||||
"migrations/",
|
||||
"schemas/",
|
||||
"query/",
|
||||
"upstreamproxy/",
|
||||
"native-ts/",
|
||||
"outputStyles/",
|
||||
"utils/",
|
||||
"types/",
|
||||
"constants/"
|
||||
]
|
||||
},
|
||||
"analysis_features": {
|
||||
"detect_patterns": true,
|
||||
"extract_tests": true,
|
||||
"build_guides": true,
|
||||
"extract_config": true,
|
||||
"build_api_reference": true,
|
||||
"analyze_dependencies": true,
|
||||
"detect_architecture": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
{
|
||||
"name": "claude-code",
|
||||
"description": "Claude Code CLI and development environment. Use for Claude Code features, tools, workflows, MCP integration, plugins, hooks, configuration, deployment, and AI-assisted development.",
|
||||
"merge_mode": "rule-based",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"_migration_note": "TODO: Migrate to external skill-seekers-configs repo. Kept temporarily to preserve PR #244 work.",
|
||||
"base_url": "https://code.claude.com/docs/en/",
|
||||
"start_urls": [
|
||||
"https://code.claude.com/docs/en/overview",
|
||||
"https://code.claude.com/docs/en/quickstart",
|
||||
"https://code.claude.com/docs/en/common-workflows",
|
||||
"https://code.claude.com/docs/en/claude-code-on-the-web",
|
||||
"https://code.claude.com/docs/en/desktop",
|
||||
"https://code.claude.com/docs/en/chrome",
|
||||
"https://code.claude.com/docs/en/vs-code",
|
||||
"https://code.claude.com/docs/en/jetbrains",
|
||||
"https://code.claude.com/docs/en/github-actions",
|
||||
"https://code.claude.com/docs/en/gitlab-ci-cd",
|
||||
"https://code.claude.com/docs/en/slack",
|
||||
"https://code.claude.com/docs/en/sub-agents",
|
||||
"https://code.claude.com/docs/en/plugins",
|
||||
"https://code.claude.com/docs/en/discover-plugins",
|
||||
"https://code.claude.com/docs/en/skills",
|
||||
"https://code.claude.com/docs/en/output-styles",
|
||||
"https://code.claude.com/docs/en/hooks-guide",
|
||||
"https://code.claude.com/docs/en/headless",
|
||||
"https://code.claude.com/docs/en/mcp",
|
||||
"https://code.claude.com/docs/en/third-party-integrations",
|
||||
"https://code.claude.com/docs/en/amazon-bedrock",
|
||||
"https://code.claude.com/docs/en/google-vertex-ai",
|
||||
"https://code.claude.com/docs/en/microsoft-foundry",
|
||||
"https://code.claude.com/docs/en/network-config",
|
||||
"https://code.claude.com/docs/en/llm-gateway",
|
||||
"https://code.claude.com/docs/en/devcontainer",
|
||||
"https://code.claude.com/docs/en/sandboxing",
|
||||
"https://code.claude.com/docs/en/setup",
|
||||
"https://code.claude.com/docs/en/iam",
|
||||
"https://code.claude.com/docs/en/security",
|
||||
"https://code.claude.com/docs/en/data-usage",
|
||||
"https://code.claude.com/docs/en/monitoring-usage",
|
||||
"https://code.claude.com/docs/en/costs",
|
||||
"https://code.claude.com/docs/en/analytics",
|
||||
"https://code.claude.com/docs/en/plugin-marketplaces",
|
||||
"https://code.claude.com/docs/en/settings",
|
||||
"https://code.claude.com/docs/en/terminal-config",
|
||||
"https://code.claude.com/docs/en/model-config",
|
||||
"https://code.claude.com/docs/en/memory",
|
||||
"https://code.claude.com/docs/en/statusline",
|
||||
"https://code.claude.com/docs/en/cli-reference",
|
||||
"https://code.claude.com/docs/en/interactive-mode",
|
||||
"https://code.claude.com/docs/en/slash-commands",
|
||||
"https://code.claude.com/docs/en/checkpointing",
|
||||
"https://code.claude.com/docs/en/hooks",
|
||||
"https://code.claude.com/docs/en/plugins-reference",
|
||||
"https://code.claude.com/docs/en/troubleshooting",
|
||||
"https://code.claude.com/docs/en/legal-and-compliance"
|
||||
],
|
||||
"selectors": {
|
||||
"main_content": "#content-area, #content-container, article, main",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/docs/en/"
|
||||
],
|
||||
"exclude": [
|
||||
"/docs/fr/",
|
||||
"/docs/de/",
|
||||
"/docs/it/",
|
||||
"/docs/ja/",
|
||||
"/docs/es/",
|
||||
"/docs/ko/",
|
||||
"/docs/zh-CN/",
|
||||
"/docs/zh-TW/",
|
||||
"/docs/ru/",
|
||||
"/docs/id/",
|
||||
"/docs/pt/",
|
||||
"/changelog",
|
||||
"github.com"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"overview",
|
||||
"quickstart",
|
||||
"common-workflows"
|
||||
],
|
||||
"ide_integrations": [
|
||||
"vs-code",
|
||||
"jetbrains",
|
||||
"desktop",
|
||||
"chrome",
|
||||
"claude-code-on-the-web",
|
||||
"slack"
|
||||
],
|
||||
"ci_cd": [
|
||||
"github-actions",
|
||||
"gitlab-ci-cd"
|
||||
],
|
||||
"building": [
|
||||
"sub-agents",
|
||||
"subagent",
|
||||
"plugins",
|
||||
"discover-plugins",
|
||||
"skills",
|
||||
"output-styles",
|
||||
"hooks-guide",
|
||||
"headless",
|
||||
"programmatic"
|
||||
],
|
||||
"mcp": [
|
||||
"mcp",
|
||||
"model-context-protocol"
|
||||
],
|
||||
"deployment": [
|
||||
"third-party-integrations",
|
||||
"amazon-bedrock",
|
||||
"google-vertex-ai",
|
||||
"microsoft-foundry",
|
||||
"network-config",
|
||||
"llm-gateway",
|
||||
"devcontainer",
|
||||
"sandboxing"
|
||||
],
|
||||
"administration": [
|
||||
"setup",
|
||||
"iam",
|
||||
"security",
|
||||
"data-usage",
|
||||
"monitoring-usage",
|
||||
"costs",
|
||||
"analytics",
|
||||
"plugin-marketplaces"
|
||||
],
|
||||
"configuration": [
|
||||
"settings",
|
||||
"terminal-config",
|
||||
"model-config",
|
||||
"memory",
|
||||
"statusline"
|
||||
],
|
||||
"reference": [
|
||||
"cli-reference",
|
||||
"interactive-mode",
|
||||
"slash-commands",
|
||||
"checkpointing",
|
||||
"hooks",
|
||||
"plugins-reference"
|
||||
],
|
||||
"troubleshooting": [
|
||||
"troubleshooting"
|
||||
],
|
||||
"legal": [
|
||||
"legal-and-compliance"
|
||||
]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 250
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"name": "godot",
|
||||
"description": "Complete Godot Engine knowledge base combining official documentation and source code analysis",
|
||||
"merge_mode": "claude-enhanced",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://docs.godotengine.org/en/stable/",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "div[role='main']",
|
||||
"title": "title",
|
||||
"code_blocks": "pre"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [],
|
||||
"exclude": ["/search.html", "/_static/", "/_images/"]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": ["introduction", "getting_started", "step_by_step"],
|
||||
"scripting": ["scripting", "gdscript", "c_sharp"],
|
||||
"2d": ["2d", "canvas", "sprite", "animation"],
|
||||
"3d": ["3d", "spatial", "mesh", "shader"],
|
||||
"physics": ["physics", "collision", "rigidbody"],
|
||||
"api": ["api", "class", "reference", "method"]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 500
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "godotengine/godot",
|
||||
"enable_codebase_analysis": true,
|
||||
"code_analysis_depth": "deep",
|
||||
"fetch_issues": true,
|
||||
"max_issues": 100,
|
||||
"fetch_changelog": true,
|
||||
"fetch_releases": true,
|
||||
"file_patterns": [
|
||||
"core/**/*.h",
|
||||
"core/**/*.cpp",
|
||||
"scene/**/*.h",
|
||||
"scene/**/*.cpp",
|
||||
"servers/**/*.h",
|
||||
"servers/**/*.cpp"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
{
|
||||
"name": "godot",
|
||||
"description": "Godot Engine 4.x - Complete open source game engine (documentation + source code + signal flow analysis)",
|
||||
"output_dir": "output/godot-unified/",
|
||||
|
||||
"sources": [
|
||||
{
|
||||
"type": "local",
|
||||
"path": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/godot-docs",
|
||||
"name": "documentation",
|
||||
"description": "Official Godot 4.x documentation (RST + Markdown)",
|
||||
"weight": 0.4,
|
||||
"enhance_level": 3,
|
||||
"file_patterns": ["*.rst", "*.md"],
|
||||
"skip_patterns": [
|
||||
"build/",
|
||||
"_build/",
|
||||
".git/",
|
||||
"node_modules/",
|
||||
"__pycache__/"
|
||||
],
|
||||
"categories": {
|
||||
"getting_started": ["getting_started", "introduction", "tutorial"],
|
||||
"core_concepts": ["classes", "nodes", "scenes", "signals"],
|
||||
"api": ["api", "reference", "class_reference"],
|
||||
"tutorials": ["tutorials", "how_to", "examples"],
|
||||
"advanced": ["advanced", "performance", "optimization"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "local",
|
||||
"path": "/mnt/1ece809a-2821-4f10-aecb-fcdf34760c0b/Git/godot",
|
||||
"name": "source_code",
|
||||
"description": "Godot Engine C++ source code + GDScript core",
|
||||
"weight": 0.6,
|
||||
"enhance_level": 3,
|
||||
"languages": ["C++", "GDScript", "Python", "GodotShader"],
|
||||
"skip_patterns": [
|
||||
".git/",
|
||||
"thirdparty/",
|
||||
"tests/",
|
||||
"doc/",
|
||||
"misc/",
|
||||
"drivers/",
|
||||
"platform/android/",
|
||||
"platform/ios/",
|
||||
"platform/web/",
|
||||
"*.obj",
|
||||
"*.o",
|
||||
"*.a",
|
||||
"*.so"
|
||||
],
|
||||
"focus_dirs": [
|
||||
"core/",
|
||||
"scene/",
|
||||
"servers/",
|
||||
"modules/gdscript/",
|
||||
"editor/"
|
||||
],
|
||||
"analysis_depth": "full",
|
||||
"extract_patterns": true,
|
||||
"extract_tests": true,
|
||||
"extract_signals": true,
|
||||
"extract_config": true
|
||||
}
|
||||
],
|
||||
|
||||
"merge_strategy": "unified",
|
||||
"conflict_resolution": "code_first",
|
||||
"detect_conflicts": true,
|
||||
|
||||
"analysis_features": {
|
||||
"pattern_detection": true,
|
||||
"test_extraction": true,
|
||||
"how_to_guides": true,
|
||||
"config_extraction": true,
|
||||
"architecture_overview": true,
|
||||
"signal_flow_analysis": true,
|
||||
"api_reference": true,
|
||||
"dependency_graph": true
|
||||
},
|
||||
|
||||
"enhancement": {
|
||||
"enabled": true,
|
||||
"level": 3,
|
||||
"mode": "LOCAL"
|
||||
},
|
||||
|
||||
"chunking": {
|
||||
"enabled": true,
|
||||
"chunk_size": 1000,
|
||||
"chunk_overlap": 200
|
||||
},
|
||||
|
||||
"output_formats": [
|
||||
"claude",
|
||||
"markdown"
|
||||
],
|
||||
|
||||
"metadata": {
|
||||
"version": "4.x",
|
||||
"framework": "godot",
|
||||
"language": "cpp+gdscript",
|
||||
"tags": ["game-engine", "godot", "cpp", "gdscript", "signals", "nodes"],
|
||||
"documentation_url": "https://docs.godotengine.org/",
|
||||
"repository_url": "https://github.com/godotengine/godot"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
{
|
||||
"name": "httpx",
|
||||
"description": "Use this skill when working with HTTPX, a fully featured HTTP client for Python 3 with sync and async APIs. HTTPX provides a familiar requests-like interface with support for HTTP/2, connection pooling, and comprehensive middleware capabilities.",
|
||||
"version": "1.0.0",
|
||||
"base_url": "https://www.python-httpx.org/",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://www.python-httpx.org/",
|
||||
"selectors": {
|
||||
"main_content": "article.md-content__inner",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "encode/httpx",
|
||||
"code_analysis_depth": "deep",
|
||||
"enable_codebase_analysis": true,
|
||||
"fetch_issues": true,
|
||||
"fetch_changelog": true,
|
||||
"fetch_releases": true,
|
||||
"max_issues": 50
|
||||
}
|
||||
],
|
||||
"selectors": {
|
||||
"main_content": "article.md-content__inner",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code",
|
||||
"navigation": "nav.md-tabs",
|
||||
"sidebar": "nav.md-nav--primary"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/quickstart/",
|
||||
"/advanced/",
|
||||
"/api/",
|
||||
"/async/",
|
||||
"/http2/",
|
||||
"/compatibility/"
|
||||
],
|
||||
"exclude": [
|
||||
"/changelog/",
|
||||
"/contributing/",
|
||||
"/exceptions/"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"quickstart",
|
||||
"install",
|
||||
"introduction",
|
||||
"overview"
|
||||
],
|
||||
"core_concepts": [
|
||||
"client",
|
||||
"request",
|
||||
"response",
|
||||
"timeout",
|
||||
"pool"
|
||||
],
|
||||
"async": [
|
||||
"async",
|
||||
"asyncio",
|
||||
"trio",
|
||||
"concurrent"
|
||||
],
|
||||
"http2": [
|
||||
"http2",
|
||||
"http/2",
|
||||
"multiplexing"
|
||||
],
|
||||
"advanced": [
|
||||
"authentication",
|
||||
"middleware",
|
||||
"transport",
|
||||
"proxy",
|
||||
"ssl",
|
||||
"streaming"
|
||||
],
|
||||
"api_reference": [
|
||||
"api",
|
||||
"reference",
|
||||
"client",
|
||||
"request",
|
||||
"response"
|
||||
],
|
||||
"compatibility": [
|
||||
"requests",
|
||||
"migration",
|
||||
"compatibility"
|
||||
]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 100,
|
||||
"metadata": {
|
||||
"author": "Encode",
|
||||
"language": "Python",
|
||||
"framework_type": "HTTP Client",
|
||||
"use_cases": [
|
||||
"Making HTTP requests",
|
||||
"REST API clients",
|
||||
"Async HTTP operations",
|
||||
"HTTP/2 support",
|
||||
"Connection pooling"
|
||||
],
|
||||
"related_skills": [
|
||||
"requests",
|
||||
"aiohttp",
|
||||
"urllib3"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
{
|
||||
"name": "medusa-mercurjs",
|
||||
"description": "Complete Medusa v2 + MercurJS multi-vendor e-commerce framework knowledge. Use when building headless commerce applications, implementing multi-vendor marketplaces, or understanding Medusa modules/workflows.",
|
||||
"merge_mode": "rule-based",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://docs.medusajs.com",
|
||||
"llms_txt_url": "https://docs.medusajs.com/llms-full.txt",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "main, article, .content",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/learn",
|
||||
"/resources"
|
||||
],
|
||||
"exclude": []
|
||||
},
|
||||
"categories": {
|
||||
"installation": ["installation", "install", "docker", "update"],
|
||||
"fundamentals": ["fundamentals", "api-routes", "data-models", "modules", "module-links", "workflows", "events-and-subscribers", "scheduled-jobs", "custom-cli-scripts", "admin", "environment-variables"],
|
||||
"customization": ["customization", "custom-features", "extend-features", "integrate-systems", "customize-admin"],
|
||||
"debugging_testing": ["debugging-and-testing", "logging", "testing", "test-tools", "instrumentation", "feature-flags", "debug-workflows"],
|
||||
"deployment": ["deployment", "production", "deploy", "general"],
|
||||
"commerce_modules": ["commerce-modules", "product", "cart", "order", "payment", "pricing", "tax", "inventory", "fulfillment", "customer", "promotion", "auth", "region", "currency", "sales-channel", "stock-location", "api-key", "user"],
|
||||
"infrastructure_modules": ["infrastructure-modules", "caching", "event", "file", "locking", "notification", "workflow-engine", "analytics"],
|
||||
"storefront": ["storefront-development", "publishable-api-keys", "checkout", "products", "customers", "regions"],
|
||||
"integrations": ["integrations", "sanity", "contentful", "stripe", "paypal", "shipstation", "sentry"],
|
||||
"cli_tools": ["medusa-cli", "commands", "build", "develop", "plugin", "db"],
|
||||
"references": ["references", "medusa-workflows", "helper-steps", "service-factory-reference", "data-model-repository-reference", "test-tools-reference", "fulfillment", "auth", "notification-provider", "file-provider", "locking-service", "caching-service"],
|
||||
"recipes": ["recipes", "erp", "marketplace", "b2b", "subscriptions", "digital-products", "bundled-products"],
|
||||
"admin_components": ["admin-components", "widgets", "ui-routes"],
|
||||
"examples": ["examples", "guides", "how-to-tutorials", "tutorials"]
|
||||
},
|
||||
"rate_limit": 0.3,
|
||||
"max_pages": 500
|
||||
},
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://docs.mercurjs.com/",
|
||||
"llms_txt_url": "https://docs.mercurjs.com/llms-full.txt",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "main, article",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": ["/"],
|
||||
"exclude": []
|
||||
},
|
||||
"categories": {
|
||||
"quick_start": ["introduction", "get-started"],
|
||||
"components": ["components", "backend", "admin-panel", "vendor-panel", "storefront"],
|
||||
"core_concepts": ["core-concepts", "seller", "commission", "payouts", "order-splitting", "reviews", "requests", "notifications", "marketplace-settings"],
|
||||
"product": ["product", "core-commerce-modules", "core-infrastructure-modules", "framework"],
|
||||
"integrations": ["integrations", "algolia", "resend", "stripe"],
|
||||
"api_admin": ["api-reference/admin", "admin-algolia", "admin-api-keys", "admin-attributes", "admin-auth", "admin-campaigns", "admin-claims", "admin-collections", "admin-commission", "admin-currencies", "admin-customers", "admin-draft-orders", "admin-exchanges", "admin-fulfillment", "admin-inventory", "admin-invites", "admin-notifications", "admin-orders", "admin-payments", "admin-price-lists", "admin-products", "admin-promotions", "admin-regions", "admin-reservations", "admin-returns", "admin-sales-channels", "admin-sellers", "admin-shipping", "admin-stock-locations", "admin-stores", "admin-tax", "admin-uploads", "admin-users"],
|
||||
"api_store": ["api-reference/store", "store-auth", "store-carts", "store-collections", "store-currencies", "store-customers", "store-fulfillment", "store-orders", "store-payment", "store-products", "store-regions", "store-returns"],
|
||||
"api_vendor": ["api-reference/vendor", "vendor-auth", "vendor-fulfillment", "vendor-inventory", "vendor-orders", "vendor-payouts", "vendor-products", "vendor-returns", "vendor-sellers", "vendor-shipping", "vendor-stock-locations", "vendor-uploads"],
|
||||
"help": ["help", "llm", "mcp", "support"]
|
||||
},
|
||||
"rate_limit": 0.3,
|
||||
"max_pages": 300
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"name": "react",
|
||||
"description": "Complete React knowledge base combining official documentation and React codebase insights. Use when working with React, understanding API changes, or debugging React internals.",
|
||||
"version": "1.1.0",
|
||||
"merge_mode": "rule-based",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://react.dev/",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "article",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [],
|
||||
"exclude": [
|
||||
"/blog/",
|
||||
"/community/"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"learn",
|
||||
"installation",
|
||||
"quick-start"
|
||||
],
|
||||
"components": [
|
||||
"components",
|
||||
"props",
|
||||
"state"
|
||||
],
|
||||
"hooks": [
|
||||
"hooks",
|
||||
"usestate",
|
||||
"useeffect",
|
||||
"usecontext"
|
||||
],
|
||||
"api": [
|
||||
"api",
|
||||
"reference"
|
||||
],
|
||||
"advanced": [
|
||||
"context",
|
||||
"refs",
|
||||
"portals",
|
||||
"suspense"
|
||||
]
|
||||
},
|
||||
"rate_limit": 0.5
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "facebook/react",
|
||||
"enable_codebase_analysis": true,
|
||||
"code_analysis_depth": "deep",
|
||||
"fetch_issues": true,
|
||||
"max_issues": 100,
|
||||
"fetch_changelog": true,
|
||||
"fetch_releases": true,
|
||||
"file_patterns": [
|
||||
"packages/react/src/**/*.js",
|
||||
"packages/react-dom/src/**/*.js"
|
||||
]
|
||||
}
|
||||
],
|
||||
"base_url": "https://react.dev/"
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"name": "unity-addressables",
|
||||
"description": "Unity Addressables asset management system - Use when implementing asset loading, remote content delivery, asset bundles, memory management, or dynamic asset loading in Unity projects.",
|
||||
"version": "1.0.0",
|
||||
"merge_mode": "claude-enhanced",
|
||||
"base_url": "https://docs.unity3d.com/Packages/com.unity.addressables@2.3/manual/index.html",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://docs.unity3d.com/Packages/com.unity.addressables@2.3/manual/index.html",
|
||||
"browser": true,
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": ".content-wrap, .section, article, main",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"com.unity.addressables"
|
||||
],
|
||||
"exclude": [
|
||||
"/changelog/",
|
||||
"/license/"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"index",
|
||||
"getting-started",
|
||||
"installation",
|
||||
"quickstart",
|
||||
"AddressableAssetsGettingStarted"
|
||||
],
|
||||
"concepts": [
|
||||
"AddressableAssetsDevelopmentCycle",
|
||||
"AddressableAssetsOverview",
|
||||
"asset-references",
|
||||
"labels",
|
||||
"groups",
|
||||
"profiles"
|
||||
],
|
||||
"loading": [
|
||||
"load-assets",
|
||||
"LoadingAddressableAssets",
|
||||
"MemoryManagement",
|
||||
"AsyncOperationHandle",
|
||||
"UnloadingAddressableAssets",
|
||||
"synchronous-addressables"
|
||||
],
|
||||
"api": [
|
||||
"api",
|
||||
"reference",
|
||||
"Addressables",
|
||||
"AddressablesAPI"
|
||||
],
|
||||
"building": [
|
||||
"Builds",
|
||||
"BuildLayoutReport",
|
||||
"ContentUpdateWorkflow",
|
||||
"AddressableAssetSettings",
|
||||
"build-scripting"
|
||||
],
|
||||
"advanced": [
|
||||
"remote-content-distribution",
|
||||
"content-catalogs",
|
||||
"diagnostic-tools",
|
||||
"CustomOperations",
|
||||
"TransformInternalId",
|
||||
"ccd"
|
||||
]
|
||||
},
|
||||
"rate_limit": 0.5
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "Unity-Technologies/Addressables-Sample",
|
||||
"include_code": true,
|
||||
"language": "C#",
|
||||
"enable_codebase_analysis": true,
|
||||
"code_analysis_depth": "deep",
|
||||
"fetch_issues": false,
|
||||
"fetch_changelog": false,
|
||||
"fetch_releases": false,
|
||||
"file_patterns": [
|
||||
"**/*.cs"
|
||||
]
|
||||
}
|
||||
],
|
||||
"analysis_features": {
|
||||
"pattern_detection": true,
|
||||
"test_extraction": true,
|
||||
"how_to_guides": true,
|
||||
"config_extraction": true,
|
||||
"architecture_overview": true,
|
||||
"api_reference": true,
|
||||
"dependency_graph": true
|
||||
},
|
||||
|
||||
"enhancement": {
|
||||
"enabled": true,
|
||||
"level": 3,
|
||||
"mode": "LOCAL"
|
||||
},
|
||||
|
||||
"chunking": {
|
||||
"enabled": true,
|
||||
"chunk_size": 1000,
|
||||
"chunk_overlap": 200
|
||||
},
|
||||
|
||||
"output_formats": [
|
||||
"claude",
|
||||
"markdown"
|
||||
],
|
||||
|
||||
"metadata": {
|
||||
"version": "2.3",
|
||||
"framework": "unity",
|
||||
"language": "csharp",
|
||||
"tags": ["unity", "addressables", "asset-management", "asset-bundles", "content-delivery", "csharp"],
|
||||
"documentation_url": "https://docs.unity3d.com/Packages/com.unity.addressables@latest/",
|
||||
"repository_url": "https://github.com/Unity-Technologies/Addressables-Sample"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
{
|
||||
"name": "dotween",
|
||||
"description": "DOTween (HOTween v2) animation engine for Unity - Use when implementing tweening animations, sequences, easing, UI animations, or any programmatic animation in Unity C# projects.",
|
||||
"version": "1.0.0",
|
||||
"merge_mode": "claude-enhanced",
|
||||
"base_url": "https://dotween.demigiant.com/documentation.php",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "https://dotween.demigiant.com/documentation.php",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "#content, .documentation, article, main",
|
||||
"title": "h1, h2",
|
||||
"code_blocks": "pre code, .code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/documentation",
|
||||
"/getstarted",
|
||||
"/pro",
|
||||
"/support"
|
||||
],
|
||||
"exclude": [
|
||||
"/download",
|
||||
"/credits"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"getstarted",
|
||||
"setup",
|
||||
"installation",
|
||||
"initialization"
|
||||
],
|
||||
"core_api": [
|
||||
"DOTween",
|
||||
"Tweener",
|
||||
"Sequence",
|
||||
"Tween",
|
||||
"TweenParams",
|
||||
"DOVirtual"
|
||||
],
|
||||
"shortcuts": [
|
||||
"shortcuts",
|
||||
"transform",
|
||||
"material",
|
||||
"rigidbody",
|
||||
"camera",
|
||||
"audio",
|
||||
"light",
|
||||
"spriterenderer"
|
||||
],
|
||||
"easing": [
|
||||
"ease",
|
||||
"easing",
|
||||
"animationcurve",
|
||||
"custom-ease"
|
||||
],
|
||||
"sequences": [
|
||||
"sequence",
|
||||
"append",
|
||||
"insert",
|
||||
"join",
|
||||
"prepend",
|
||||
"callbacks"
|
||||
],
|
||||
"advanced": [
|
||||
"paths",
|
||||
"blendable",
|
||||
"DOTweenAnimation",
|
||||
"DOTweenVisualManager",
|
||||
"pro"
|
||||
]
|
||||
},
|
||||
"rate_limit": 1.0
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "Demigiant/dotween",
|
||||
"include_code": true,
|
||||
"language": "C#",
|
||||
"enable_codebase_analysis": true,
|
||||
"code_analysis_depth": "deep",
|
||||
"fetch_issues": true,
|
||||
"max_issues": 50,
|
||||
"fetch_changelog": false,
|
||||
"fetch_releases": true
|
||||
}
|
||||
],
|
||||
"analysis_features": {
|
||||
"pattern_detection": true,
|
||||
"test_extraction": true,
|
||||
"how_to_guides": true,
|
||||
"config_extraction": true,
|
||||
"architecture_overview": true,
|
||||
"api_reference": true,
|
||||
"dependency_graph": true
|
||||
},
|
||||
|
||||
"enhancement": {
|
||||
"enabled": true,
|
||||
"level": 3,
|
||||
"mode": "LOCAL",
|
||||
"agent": "kimi",
|
||||
"timeout": "unlimited"
|
||||
},
|
||||
|
||||
"chunking": {
|
||||
"enabled": true,
|
||||
"chunk_size": 1000,
|
||||
"chunk_overlap": 200
|
||||
},
|
||||
|
||||
"output_formats": [
|
||||
"claude",
|
||||
"markdown"
|
||||
],
|
||||
|
||||
"metadata": {
|
||||
"version": "1.2",
|
||||
"framework": "unity",
|
||||
"language": "csharp",
|
||||
"tags": ["unity", "dotween", "tweening", "animation", "easing", "csharp"],
|
||||
"documentation_url": "https://dotween.demigiant.com/documentation.php",
|
||||
"repository_url": "https://github.com/Demigiant/dotween"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
{
|
||||
"name": "spine-unity",
|
||||
"description": "Spine 2D skeletal animation runtime for Unity - Use when implementing Spine animations, SkeletonAnimation components, skin management, or spine-unity integration in Unity projects.",
|
||||
"version": "1.0.0",
|
||||
"merge_mode": "claude-enhanced",
|
||||
"base_url": "http://en.esotericsoftware.com/spine-unity",
|
||||
"sources": [
|
||||
{
|
||||
"type": "documentation",
|
||||
"base_url": "http://en.esotericsoftware.com/spine-unity",
|
||||
"extract_api": true,
|
||||
"selectors": {
|
||||
"main_content": "#wiki_page_content, .wiki-content, article, main",
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": [
|
||||
"/spine-unity",
|
||||
"/spine-runtime",
|
||||
"/spine-api",
|
||||
"/spine-applying-animations",
|
||||
"/spine-attachments",
|
||||
"/spine-events",
|
||||
"/spine-skeleton",
|
||||
"/spine-skins",
|
||||
"/spine-slots"
|
||||
],
|
||||
"exclude": [
|
||||
"/spine-corona",
|
||||
"/spine-cocos2d",
|
||||
"/spine-godot",
|
||||
"/spine-unreal",
|
||||
"/spine-flutter",
|
||||
"/spine-sfml",
|
||||
"/spine-monogame",
|
||||
"/spine-love",
|
||||
"/spine-haxe",
|
||||
"/spine-ts",
|
||||
"/spine-phaser",
|
||||
"/spine-pixi",
|
||||
"/spine-lwjgl",
|
||||
"/spine-libgdx",
|
||||
"/spine-changelog"
|
||||
]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": [
|
||||
"Installation",
|
||||
"Download",
|
||||
"Assets",
|
||||
"Runtime Documentation",
|
||||
"FAQ"
|
||||
],
|
||||
"components": [
|
||||
"Main Components",
|
||||
"Utility Components",
|
||||
"SkeletonAnimation",
|
||||
"SkeletonGraphic",
|
||||
"SkeletonMecanim",
|
||||
"BoneFollower"
|
||||
],
|
||||
"animation": [
|
||||
"Events",
|
||||
"AnimationState",
|
||||
"callback",
|
||||
"Timeline",
|
||||
"Examples"
|
||||
],
|
||||
"rendering": [
|
||||
"Rendering",
|
||||
"shader",
|
||||
"material"
|
||||
],
|
||||
"advanced": [
|
||||
"On-Demand Loading",
|
||||
"UPM",
|
||||
"Packages"
|
||||
]
|
||||
},
|
||||
"rate_limit": 1.0
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"repo": "EsotericSoftware/spine-runtimes",
|
||||
"include_code": true,
|
||||
"language": "C#",
|
||||
"enable_codebase_analysis": true,
|
||||
"code_analysis_depth": "deep",
|
||||
"fetch_issues": true,
|
||||
"max_issues": 50,
|
||||
"fetch_changelog": true,
|
||||
"fetch_releases": true,
|
||||
"file_patterns": [
|
||||
"spine-unity/Assets/Spine/**/*.cs",
|
||||
"spine-unity/Assets/Spine Examples/**/*.cs",
|
||||
"spine-csharp/src/**/*.cs"
|
||||
]
|
||||
}
|
||||
],
|
||||
"analysis_features": {
|
||||
"pattern_detection": true,
|
||||
"test_extraction": true,
|
||||
"how_to_guides": true,
|
||||
"config_extraction": true,
|
||||
"architecture_overview": true,
|
||||
"api_reference": true,
|
||||
"dependency_graph": true
|
||||
},
|
||||
|
||||
"enhancement": {
|
||||
"enabled": true,
|
||||
"level": 3,
|
||||
"mode": "LOCAL"
|
||||
},
|
||||
|
||||
"chunking": {
|
||||
"enabled": true,
|
||||
"chunk_size": 1000,
|
||||
"chunk_overlap": 200
|
||||
},
|
||||
|
||||
"output_formats": [
|
||||
"claude",
|
||||
"markdown"
|
||||
],
|
||||
|
||||
"metadata": {
|
||||
"version": "4.2",
|
||||
"framework": "unity",
|
||||
"language": "csharp",
|
||||
"tags": ["unity", "spine", "2d-animation", "skeletal-animation", "csharp"],
|
||||
"documentation_url": "http://en.esotericsoftware.com/spine-unity",
|
||||
"repository_url": "https://github.com/EsotericSoftware/spine-runtimes"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "skill-seekers",
|
||||
"description": "Transform 17 source types (docs, GitHub, PDFs, videos, Jupyter, Confluence, Notion, Slack, and more) into AI-ready skills and RAG knowledge for 16+ LLM platforms.",
|
||||
"version": "3.3.0",
|
||||
"author": {
|
||||
"name": "Yusuf Karaaslan"
|
||||
},
|
||||
"homepage": "https://github.com/yusufkaraaslan/Skill_Seekers",
|
||||
"repository": "https://github.com/yusufkaraaslan/Skill_Seekers",
|
||||
"license": "MIT"
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"skill-seekers": {
|
||||
"command": "python",
|
||||
"args": ["-m", "skill_seekers.mcp.server_fastmcp"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
# Skill Seekers — Claude Code Plugin
|
||||
|
||||
Transform 18 source types into AI-ready skills and RAG knowledge, directly from Claude Code.
|
||||
|
||||
## Installation
|
||||
|
||||
### From the Official Plugin Directory
|
||||
|
||||
```
|
||||
/plugin install skill-seekers@claude-plugin-directory
|
||||
```
|
||||
|
||||
Or browse for it in `/plugin > Discover`.
|
||||
|
||||
### Local Installation (for development)
|
||||
|
||||
```bash
|
||||
claude --plugin-dir ./path/to/skill-seekers-plugin
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
|
||||
The plugin requires `skill-seekers` to be installed:
|
||||
|
||||
```bash
|
||||
pip install skill-seekers[mcp]
|
||||
```
|
||||
|
||||
## What's Included
|
||||
|
||||
### MCP Server (40 tools)
|
||||
|
||||
The plugin bundles the Skill Seekers MCP server providing tools for:
|
||||
- Scraping documentation, GitHub repos, PDFs, videos, and 13 other source types
|
||||
- Packaging skills for 21+ LLM platforms
|
||||
- Exporting to vector databases (Weaviate, Chroma, FAISS, Qdrant)
|
||||
- Managing configs, workflows, and sources
|
||||
|
||||
### Slash Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/skill-seekers:create-skill <source>` | Create a skill from any source (auto-detects type) |
|
||||
| `/skill-seekers:sync-config <config>` | Sync config URLs against live docs |
|
||||
| `/skill-seekers:install-skill <source>` | End-to-end: fetch, scrape, enhance, package, install |
|
||||
|
||||
### Agent Skill
|
||||
|
||||
The **skill-builder** skill is automatically available to Claude. It detects source types and uses the appropriate MCP tools to build skills autonomously.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```
|
||||
# Create a skill from a documentation site
|
||||
/skill-seekers:create-skill https://react.dev
|
||||
|
||||
# Create from a GitHub repo, targeting LangChain
|
||||
/skill-seekers:create-skill pallets/flask --target langchain
|
||||
|
||||
# Full install workflow with AI enhancement
|
||||
/skill-seekers:install-skill https://fastapi.tiangolo.com --enhance
|
||||
|
||||
# Sync an existing config
|
||||
/skill-seekers:sync-config react
|
||||
```
|
||||
|
||||
Or just ask Claude naturally:
|
||||
> "Create an AI skill from the React documentation"
|
||||
> "Scrape the Flask GitHub repo and package it for OpenAI"
|
||||
> "Export my skill to a Chroma vector database"
|
||||
|
||||
The skill-builder agent skill will automatically detect the intent and use the right tools.
|
||||
|
||||
## Remote MCP Alternative
|
||||
|
||||
By default, the plugin runs the MCP server locally via `python -m skill_seekers.mcp.server_fastmcp`. To use a remote server instead, edit `.mcp.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"skill-seekers": {
|
||||
"type": "http",
|
||||
"url": "https://your-hosted-server.com/mcp"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Supported Source Types
|
||||
|
||||
Documentation (web), GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML files, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence, Notion, Slack/Discord exports.
|
||||
|
||||
## License
|
||||
|
||||
MIT — https://github.com/yusufkaraaslan/Skill_Seekers
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
description: Create an AI skill from any source (URL, repo, PDF, video, notebook, etc.)
|
||||
---
|
||||
|
||||
# Create Skill
|
||||
|
||||
Create an AI-ready skill from a source. The source type is auto-detected.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/skill-seekers:create-skill <source> [--preset <level>] [--output <dir>]
|
||||
```
|
||||
|
||||
## Instructions
|
||||
|
||||
When the user provides a source via `$ARGUMENTS`, run the `skill-seekers create` command to generate a skill.
|
||||
|
||||
1. Parse the arguments: extract the source (first argument) and any flags.
|
||||
2. If no `--preset` is specified, default to `quick` for fast results.
|
||||
3. If no `--output` is specified, default to `./output`.
|
||||
4. Run the create command:
|
||||
```bash
|
||||
skill-seekers create "$SOURCE" --preset quick --output "$OUTPUT"
|
||||
```
|
||||
5. After completion, read the generated `SKILL.md` and summarize what was created.
|
||||
6. If the user wants to target a specific platform (e.g., Claude, OpenAI, LangChain), run the package command after:
|
||||
```bash
|
||||
skill-seekers package "$SKILL_DIR" --target "$PLATFORM"
|
||||
```
|
||||
|
||||
## Presets
|
||||
|
||||
- `-p quick` — 1-2 minutes, basic skill
|
||||
- `-p standard` — 5-10 minutes, good coverage
|
||||
- `-p comprehensive` — 20-60 minutes, full analysis
|
||||
|
||||
## Source Types (auto-detected)
|
||||
|
||||
- **URL** (https://...) — Documentation scraping
|
||||
- **owner/repo** or github.com URL — GitHub repo analysis
|
||||
- **file.pdf** — PDF extraction
|
||||
- **file.ipynb** — Jupyter notebook
|
||||
- **file.docx** — Word document
|
||||
- **file.epub** — EPUB book
|
||||
- **YouTube/Vimeo URL** — Video transcript
|
||||
- **./directory** — Local codebase analysis
|
||||
- **file.yaml** with OpenAPI — API spec
|
||||
- **file.pptx** — PowerPoint
|
||||
- **file.adoc** — AsciiDoc
|
||||
- **file.html** — HTML page
|
||||
- **file.rss** — RSS/Atom feed
|
||||
- **cmd.1** — Man page
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/skill-seekers:create-skill https://react.dev
|
||||
/skill-seekers:create-skill pallets/flask -p standard
|
||||
/skill-seekers:create-skill ./docs/api.pdf
|
||||
/skill-seekers:create-skill https://youtube.com/watch?v=abc123
|
||||
```
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: One-command skill creation and packaging for a target platform
|
||||
---
|
||||
|
||||
# Install Skill
|
||||
|
||||
End-to-end workflow: create a skill from any source, then package it for a target LLM platform.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/skill-seekers:install-skill <source> [--target <platform>] [--preset <level>]
|
||||
```
|
||||
|
||||
## Instructions
|
||||
|
||||
When the user provides a source via `$ARGUMENTS`:
|
||||
|
||||
1. Parse the arguments: extract source, `--target` (default: claude), `--preset` (default: quick).
|
||||
2. Run the create command:
|
||||
```bash
|
||||
skill-seekers create "$SOURCE" --preset "$PRESET" --output ./output
|
||||
```
|
||||
3. Find the generated skill directory (look for the directory containing SKILL.md in ./output/).
|
||||
4. Run the package command for the target platform:
|
||||
```bash
|
||||
skill-seekers package "$SKILL_DIR" --target "$TARGET"
|
||||
```
|
||||
5. Report what was created and where to find the packaged output.
|
||||
|
||||
## Target Platforms
|
||||
|
||||
`claude` (default), `openai`, `gemini`, `langchain`, `llama-index`, `haystack`, `markdown`, `chroma`, `weaviate`, `faiss`, `qdrant`, `pinecone`, `deepseek`, `kimi`, `qwen`, `fireworks`, `openrouter`, `together`, `minimax`, `opencode`, `ibm-bob`
|
||||
|
||||
> **IDE-specific installs** (`cursor`, `windsurf`, `continue`, `cline`) are not standalone `--target` values. Use `skill-seekers install-agent` or manually copy the output from `--target markdown` or `--target claude`.
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/skill-seekers:install-skill https://react.dev --target claude
|
||||
/skill-seekers:install-skill pallets/flask --target langchain -p standard
|
||||
/skill-seekers:install-skill ./docs/api.pdf --target openai
|
||||
```
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
description: Sync a scraping config's URLs against the live documentation site
|
||||
---
|
||||
|
||||
# Sync Config
|
||||
|
||||
Synchronize a Skill Seekers config file with the current state of a documentation site. Detects new pages, removed pages, and URL changes.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/skill-seekers:sync-config <config-path-or-name>
|
||||
```
|
||||
|
||||
## Instructions
|
||||
|
||||
When the user provides a config path or preset name via `$ARGUMENTS`:
|
||||
|
||||
1. If it's a preset name (e.g., `react`, `godot`), look for it in the `configs/` directory or fetch from the API.
|
||||
2. Run the sync command:
|
||||
```bash
|
||||
skill-seekers sync-config "$CONFIG"
|
||||
```
|
||||
3. Report what changed: new URLs found, removed URLs, and any conflicts.
|
||||
4. Ask the user if they want to update the config and re-scrape.
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/skill-seekers:sync-config configs/react.json
|
||||
/skill-seekers:sync-config react
|
||||
```
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
name: skill-builder
|
||||
description: Automatically detect source types and build AI skills using Skill Seekers. Use when the user wants to create skills from documentation, repos, PDFs, videos, or other knowledge sources.
|
||||
---
|
||||
|
||||
# Skill Builder
|
||||
|
||||
You have access to the Skill Seekers MCP server which provides 40 tools for converting knowledge sources into AI-ready skills.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when the user:
|
||||
- Wants to create an AI skill from a documentation site, GitHub repo, PDF, video, or other source
|
||||
- Needs to convert documentation into a format suitable for LLM consumption
|
||||
- Wants to update or sync existing skills with their source documentation
|
||||
- Needs to export skills to vector databases (Weaviate, Chroma, FAISS, Qdrant)
|
||||
- Asks about scraping, converting, or packaging documentation for AI
|
||||
|
||||
## Source Type Detection
|
||||
|
||||
Automatically detect the source type from user input:
|
||||
|
||||
| Input Pattern | Source Type | Tool to Use |
|
||||
|---------------|-------------|-------------|
|
||||
| `https://...` (not GitHub/YouTube) | Documentation | `scrape_docs` |
|
||||
| `owner/repo` or `github.com/...` | GitHub | `scrape_github` |
|
||||
| `*.pdf` | PDF | `scrape_pdf` |
|
||||
| YouTube/Vimeo URL or video file | Video | `scrape_video` |
|
||||
| Local directory path | Codebase | `scrape_codebase` |
|
||||
| `*.ipynb`, `*.html`, `*.yaml` (OpenAPI), `*.adoc`, `*.pptx`, `*.rss`, `*.1`-`.8` | Various | `scrape_generic` |
|
||||
| JSON config file | Unified | Use config with `scrape_docs` |
|
||||
|
||||
## Recommended Workflow
|
||||
|
||||
1. **Detect source type** from the user's input
|
||||
2. **Generate or fetch config** using `generate_config` or `fetch_config` if needed
|
||||
3. **Estimate scope** with `estimate_pages` for documentation sites
|
||||
4. **Scrape the source** using the appropriate scraping tool
|
||||
5. **Enhance** with `enhance_skill` if the user wants AI-powered improvements
|
||||
6. **Package** with `package_skill` for the target platform
|
||||
7. **Export to vector DB** if requested using `export_to_*` tools
|
||||
|
||||
## Available MCP Tools
|
||||
|
||||
### Config Management
|
||||
- `generate_config` — Generate a scraping config from a URL
|
||||
- `list_configs` — List available preset configs
|
||||
- `validate_config` — Validate a config file
|
||||
|
||||
### Scraping (use based on source type)
|
||||
- `scrape_docs` — Documentation sites
|
||||
- `scrape_github` — GitHub repositories
|
||||
- `scrape_pdf` — PDF files
|
||||
- `scrape_video` — Video transcripts
|
||||
- `scrape_codebase` — Local code analysis
|
||||
- `scrape_generic` — Jupyter, HTML, OpenAPI, AsciiDoc, PPTX, RSS, manpage, Confluence, Notion, chat
|
||||
|
||||
### Post-processing
|
||||
- `enhance_skill` — AI-powered skill enhancement
|
||||
- `package_skill` — Package for target platform
|
||||
- `upload_skill` — Upload to platform API
|
||||
- `install_skill` — End-to-end install workflow
|
||||
|
||||
### Advanced
|
||||
- `detect_patterns` — Design pattern detection in code
|
||||
- `extract_test_examples` — Extract usage examples from tests
|
||||
- `build_how_to_guides` — Generate how-to guides from tests
|
||||
- `split_config` — Split large configs into focused skills
|
||||
- `export_to_weaviate`, `export_to_chroma`, `export_to_faiss`, `export_to_qdrant` — Vector DB export
|
||||
@@ -0,0 +1,147 @@
|
||||
# Skill Seekers GitHub Action
|
||||
|
||||
Transform documentation, GitHub repos, PDFs, videos, and 13 other source types into AI-ready skills and RAG knowledge — directly in your CI/CD pipeline.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```yaml
|
||||
- uses: yusufkaraaslan/skill-seekers-action@v3
|
||||
with:
|
||||
source: 'https://react.dev'
|
||||
```
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Required | Default | Description |
|
||||
|-------|----------|---------|-------------|
|
||||
| `source` | Yes | — | Source URL, file path, or `owner/repo` |
|
||||
| `command` | No | `create` | Command: `create`, `scan`, `doctor`, `enhance`, `enhance-status`, `package`, `upload`, `install`, `install-agent`, `estimate`, `extract-test-examples`, `resume`, `quality`, `config`, `workflows`, `sync-config`, `stream`, `update`, `multilang` |
|
||||
| `target` | No | `claude` | Target platform: `claude`, `openai`, `gemini`, `langchain`, `llama-index`, `markdown` |
|
||||
| `config` | No | — | Path to JSON config file |
|
||||
| `output-dir` | No | `output` | Output directory |
|
||||
| `extra-args` | No | — | Additional CLI arguments |
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Description |
|
||||
|--------|-------------|
|
||||
| `skill-dir` | Path to the generated skill directory |
|
||||
| `skill-name` | Name of the generated skill |
|
||||
|
||||
## Examples
|
||||
|
||||
### Auto-update documentation skill weekly
|
||||
|
||||
```yaml
|
||||
name: Update AI Skills
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 6 * * 1' # Every Monday 6am UTC
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
update-skills:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: yusufkaraaslan/skill-seekers-action@v3
|
||||
with:
|
||||
source: 'https://react.dev'
|
||||
target: 'langchain'
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: react-skill
|
||||
path: output/
|
||||
```
|
||||
|
||||
### Generate skill from GitHub repo
|
||||
|
||||
```yaml
|
||||
- uses: yusufkaraaslan/skill-seekers-action@v3
|
||||
with:
|
||||
source: 'pallets/flask'
|
||||
command: 'create'
|
||||
target: 'claude'
|
||||
```
|
||||
|
||||
### Process PDF documentation
|
||||
|
||||
```yaml
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: yusufkaraaslan/skill-seekers-action@v3
|
||||
with:
|
||||
source: 'docs/api-reference.pdf'
|
||||
command: 'create'
|
||||
```
|
||||
|
||||
### Unified multi-source build with config
|
||||
|
||||
```yaml
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: yusufkaraaslan/skill-seekers-action@v3
|
||||
with:
|
||||
config: 'configs/my-project.json'
|
||||
command: 'create'
|
||||
target: 'openai'
|
||||
```
|
||||
|
||||
### Commit generated skill back to repo
|
||||
|
||||
```yaml
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: yusufkaraaslan/skill-seekers-action@v3
|
||||
id: generate
|
||||
with:
|
||||
source: 'https://fastapi.tiangolo.com'
|
||||
|
||||
- name: Commit skill
|
||||
run: |
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||
git add output/
|
||||
git diff --staged --quiet || git commit -m "Update AI skill: ${{ steps.generate.outputs.skill-name }}"
|
||||
git push
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Pass API keys as environment variables for AI-enhanced skills:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
## Supported Source Types
|
||||
|
||||
| Type | Example Source |
|
||||
|------|---------------|
|
||||
| Documentation (web) | `https://react.dev` |
|
||||
| GitHub repo | `pallets/flask` or `https://github.com/pallets/flask` |
|
||||
| PDF | `docs/manual.pdf` |
|
||||
| Video | `https://youtube.com/watch?v=...` |
|
||||
| Local codebase | `./src` |
|
||||
| Jupyter Notebook | `analysis.ipynb` |
|
||||
| OpenAPI/Swagger | `openapi.yaml` |
|
||||
| Word (.docx) | `docs/guide.docx` |
|
||||
| EPUB | `book.epub` |
|
||||
| PowerPoint | `slides.pptx` |
|
||||
| AsciiDoc | `docs/guide.adoc` |
|
||||
| HTML | `page.html` |
|
||||
| RSS/Atom | `feed.rss` |
|
||||
| Man pages | `tool.1` |
|
||||
| Confluence | Via config file |
|
||||
| Notion | Via config file |
|
||||
| Chat (Slack/Discord) | Via config file |
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,92 @@
|
||||
name: 'Skill Seekers - AI Knowledge Builder'
|
||||
description: 'Transform documentation, repos, PDFs, videos, and 13 other source types into AI skills and RAG knowledge'
|
||||
author: 'Yusuf Karaaslan'
|
||||
|
||||
branding:
|
||||
icon: 'book-open'
|
||||
color: 'blue'
|
||||
|
||||
inputs:
|
||||
source:
|
||||
description: 'Source URL, file path, or owner/repo for GitHub repos'
|
||||
required: true
|
||||
command:
|
||||
description: 'Command to run: create (auto-detect), scrape, github, pdf, video, analyze, unified'
|
||||
required: false
|
||||
default: 'create'
|
||||
target:
|
||||
description: 'Output target platform: claude, openai, gemini, langchain, llama-index, markdown, cursor, windsurf'
|
||||
required: false
|
||||
default: 'claude'
|
||||
config:
|
||||
description: 'Path to JSON config file (for unified/advanced scraping)'
|
||||
required: false
|
||||
output-dir:
|
||||
description: 'Output directory for generated skills'
|
||||
required: false
|
||||
default: 'output'
|
||||
extra-args:
|
||||
description: 'Additional CLI arguments to pass to skill-seekers'
|
||||
required: false
|
||||
default: ''
|
||||
|
||||
outputs:
|
||||
skill-dir:
|
||||
description: 'Path to the generated skill directory'
|
||||
value: ${{ steps.run.outputs.skill-dir }}
|
||||
skill-name:
|
||||
description: 'Name of the generated skill'
|
||||
value: ${{ steps.run.outputs.skill-name }}
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install Skill Seekers
|
||||
shell: bash
|
||||
run: pip install skill-seekers
|
||||
|
||||
- name: Run Skill Seekers
|
||||
id: run
|
||||
shell: bash
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ env.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ env.OPENAI_API_KEY }}
|
||||
GOOGLE_API_KEY: ${{ env.GOOGLE_API_KEY }}
|
||||
GITHUB_TOKEN: ${{ env.GITHUB_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
OUTPUT_DIR="${{ inputs.output-dir }}"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
CMD="${{ inputs.command }}"
|
||||
SOURCE="${{ inputs.source }}"
|
||||
TARGET="${{ inputs.target }}"
|
||||
CONFIG="${{ inputs.config }}"
|
||||
EXTRA="${{ inputs.extra-args }}"
|
||||
|
||||
# Build the command
|
||||
if [ "$CMD" = "create" ]; then
|
||||
skill-seekers create "$SOURCE" --target "$TARGET" --output "$OUTPUT_DIR" $EXTRA
|
||||
elif [ -n "$CONFIG" ]; then
|
||||
skill-seekers "$CMD" --config "$CONFIG" --target "$TARGET" --output "$OUTPUT_DIR" $EXTRA
|
||||
else
|
||||
skill-seekers "$CMD" "$SOURCE" --target "$TARGET" --output "$OUTPUT_DIR" $EXTRA
|
||||
fi
|
||||
|
||||
# Find the generated skill directory
|
||||
SKILL_DIR=$(find "$OUTPUT_DIR" -name "SKILL.md" -exec dirname {} \; | head -1)
|
||||
SKILL_NAME=$(basename "$SKILL_DIR" 2>/dev/null || echo "unknown")
|
||||
|
||||
echo "skill-dir=$SKILL_DIR" >> "$GITHUB_OUTPUT"
|
||||
echo "skill-name=$SKILL_NAME" >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "### Skill Generated" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "- **Name:** $SKILL_NAME" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "- **Directory:** $SKILL_DIR" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "- **Target:** $TARGET" >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,109 @@
|
||||
# Skill Seekers — Smithery MCP Registry
|
||||
|
||||
Publishing guide for the Skill Seekers MCP server on [Smithery](https://smithery.ai).
|
||||
|
||||
## Status
|
||||
|
||||
- **Namespace created:** `yusufkaraaslan`
|
||||
- **Server created:** `yusufkaraaslan/skill-seekers`
|
||||
- **Server page:** https://smithery.ai/servers/yusufkaraaslan/skill-seekers
|
||||
- **Release status:** Needs re-publish (initial release failed — Smithery couldn't scan GitHub URL as MCP endpoint)
|
||||
|
||||
## Publishing
|
||||
|
||||
Smithery requires a live, scannable MCP HTTP endpoint for URL-based publishing. Two options:
|
||||
|
||||
### Option A: Publish via Web UI (Recommended)
|
||||
|
||||
1. Go to https://smithery.ai/servers/yusufkaraaslan/skill-seekers/releases
|
||||
2. The server already exists — create a new release
|
||||
3. For the "Local" tab: follow the prompts to publish as a stdio server
|
||||
4. For the "URL" tab: provide a hosted HTTP endpoint URL
|
||||
|
||||
### Option B: Deploy HTTP endpoint first, then publish via CLI
|
||||
|
||||
1. Deploy the MCP server on Render/Railway/Fly.io:
|
||||
```bash
|
||||
# Using existing Dockerfile.mcp
|
||||
docker build -f Dockerfile.mcp -t skill-seekers-mcp .
|
||||
# Deploy to your hosting provider
|
||||
```
|
||||
2. Publish the live URL:
|
||||
```bash
|
||||
npx @smithery/cli@latest auth login
|
||||
npx @smithery/cli@latest mcp publish "https://your-deployed-url/mcp" \
|
||||
-n yusufkaraaslan/skill-seekers
|
||||
```
|
||||
|
||||
### CLI Authentication (already done)
|
||||
|
||||
```bash
|
||||
# Install via npx (no global install needed)
|
||||
npx @smithery/cli@latest auth login
|
||||
npx @smithery/cli@latest namespace show # Should show: yusufkaraaslan
|
||||
```
|
||||
|
||||
### After Publishing
|
||||
|
||||
Update the server page with metadata:
|
||||
|
||||
**Display name:** Skill Seekers — AI Skill & RAG Toolkit
|
||||
|
||||
**Description:**
|
||||
> Transform 18 source types into AI-ready skills and RAG knowledge. Ingest documentation sites, GitHub repos, PDFs, Jupyter notebooks, videos, Confluence, Notion, Slack/Discord exports, and more. Package for 21+ LLM platforms including Claude, GPT, Gemini, LangChain, LlamaIndex, and vector databases.
|
||||
|
||||
**Tags:** `ai`, `rag`, `documentation`, `skills`, `preprocessing`, `mcp`, `knowledge-base`, `vector-database`
|
||||
|
||||
## User Installation
|
||||
|
||||
Once published, users can add the server to their MCP client:
|
||||
|
||||
```bash
|
||||
# Via Smithery CLI (adds to Claude Desktop, Cursor, etc.)
|
||||
smithery mcp add yusufkaraaslan/skill-seekers --client claude
|
||||
|
||||
# Or configure manually — users need skill-seekers installed:
|
||||
pip install skill-seekers[mcp]
|
||||
```
|
||||
|
||||
### Manual MCP Configuration
|
||||
|
||||
For clients that use JSON config (Claude Desktop, Claude Code, Cursor):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"skill-seekers": {
|
||||
"command": "python",
|
||||
"args": ["-m", "skill_seekers.mcp.server_fastmcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Available Tools (40)
|
||||
|
||||
| Category | Tools | Description |
|
||||
|----------|-------|-------------|
|
||||
| Config | 3 | Generate, list, validate scraping configs |
|
||||
| Sync | 1 | Sync config URLs against live docs |
|
||||
| Scraping | 11 | Scrape docs, GitHub, PDF, video, codebase, generic (10 types) |
|
||||
| Packaging | 4 | Package, upload, enhance, install skills |
|
||||
| Splitting | 2 | Split large configs, generate routers |
|
||||
| Sources | 5 | Fetch, submit, manage config sources |
|
||||
| Vector DB | 4 | Export to Weaviate, Chroma, FAISS, Qdrant |
|
||||
| Workflows | 5 | List, get, create, update, delete workflows |
|
||||
| Quality | 3 | Quality checks, estimate scope, resume operations |
|
||||
| Agent Install | 2 | Install agent, extract test examples |
|
||||
|
||||
## Maintenance
|
||||
|
||||
- Update description/tags on major releases
|
||||
- No code changes needed — users always get the latest via `pip install`
|
||||
|
||||
## Notes
|
||||
|
||||
- Smithery CLI v4.7.0 removed the `--transport stdio` flag from the docs
|
||||
- The CLI `publish` command only supports URL-based (external) publishing
|
||||
- For local/stdio servers, use the web UI at smithery.ai/servers/new
|
||||
- The namespace and server entity are already created; only the release needs to succeed
|
||||
@@ -0,0 +1,111 @@
|
||||
# Skill Seekers Docker Compose
|
||||
# Complete deployment with MCP server and vector databases
|
||||
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# Main Skill Seekers CLI application
|
||||
skill-seekers:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: skill-seekers:latest
|
||||
container_name: skill-seekers
|
||||
environment:
|
||||
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
||||
- GOOGLE_API_KEY=${GOOGLE_API_KEY}
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- GITHUB_TOKEN=${GITHUB_TOKEN}
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- ./configs:/configs:ro
|
||||
- ./output:/output
|
||||
networks:
|
||||
- skill-seekers-net
|
||||
command: ["skill-seekers", "--help"]
|
||||
|
||||
# MCP Server (HTTP mode)
|
||||
mcp-server:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.mcp
|
||||
image: skill-seekers-mcp:latest
|
||||
container_name: skill-seekers-mcp
|
||||
ports:
|
||||
- "8765:8765"
|
||||
environment:
|
||||
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
||||
- GOOGLE_API_KEY=${GOOGLE_API_KEY}
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- GITHUB_TOKEN=${GITHUB_TOKEN}
|
||||
- MCP_TRANSPORT=http
|
||||
- MCP_PORT=8765
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- ./configs:/configs:ro
|
||||
- ./output:/output
|
||||
networks:
|
||||
- skill-seekers-net
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
# Weaviate Vector Database
|
||||
weaviate:
|
||||
image: semitechnologies/weaviate:latest
|
||||
container_name: weaviate
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
QUERY_DEFAULTS_LIMIT: 25
|
||||
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
|
||||
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
|
||||
DEFAULT_VECTORIZER_MODULE: 'none'
|
||||
ENABLE_MODULES: ''
|
||||
CLUSTER_HOSTNAME: 'node1'
|
||||
volumes:
|
||||
- weaviate-data:/var/lib/weaviate
|
||||
networks:
|
||||
- skill-seekers-net
|
||||
restart: unless-stopped
|
||||
|
||||
# Qdrant Vector Database
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
container_name: qdrant
|
||||
ports:
|
||||
- "6333:6333"
|
||||
- "6334:6334"
|
||||
volumes:
|
||||
- qdrant-data:/qdrant/storage
|
||||
networks:
|
||||
- skill-seekers-net
|
||||
restart: unless-stopped
|
||||
|
||||
# Chroma Vector Database
|
||||
chroma:
|
||||
image: ghcr.io/chroma-core/chroma:latest
|
||||
container_name: chroma
|
||||
ports:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
IS_PERSISTENT: 'TRUE'
|
||||
PERSIST_DIRECTORY: '/chroma/data'
|
||||
volumes:
|
||||
- chroma-data:/chroma/data
|
||||
networks:
|
||||
- skill-seekers-net
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
skill-seekers-net:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
weaviate-data:
|
||||
qdrant-data:
|
||||
chroma-data:
|
||||
@@ -0,0 +1,263 @@
|
||||
# Documentation Architecture
|
||||
|
||||
> **How Skill Seekers documentation is organized (v3.6.0 - 18 source types)**
|
||||
|
||||
---
|
||||
|
||||
## Philosophy
|
||||
|
||||
Our documentation follows these principles:
|
||||
|
||||
1. **Progressive Disclosure** - Start simple, add complexity as needed
|
||||
2. **Task-Oriented** - Organized by what users want to do
|
||||
3. **Single Source of Truth** - One authoritative reference per topic
|
||||
4. **Version Current** - Always reflect the latest release
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── README.md # Entry point - navigation hub
|
||||
├── ARCHITECTURE.md # This file
|
||||
│
|
||||
├── getting-started/ # New users (lowest cognitive load)
|
||||
│ ├── 01-installation.md
|
||||
│ ├── 02-quick-start.md
|
||||
│ ├── 03-your-first-skill.md
|
||||
│ └── 04-next-steps.md
|
||||
│
|
||||
├── user-guide/ # Common tasks (practical focus)
|
||||
│ ├── 01-core-concepts.md
|
||||
│ ├── 02-scraping.md
|
||||
│ ├── 03-enhancement.md
|
||||
│ ├── 04-packaging.md
|
||||
│ ├── 05-workflows.md
|
||||
│ └── 06-troubleshooting.md
|
||||
│
|
||||
├── reference/ # Technical details (comprehensive)
|
||||
│ ├── CLI_REFERENCE.md
|
||||
│ ├── MCP_REFERENCE.md
|
||||
│ ├── CONFIG_FORMAT.md
|
||||
│ └── ENVIRONMENT_VARIABLES.md
|
||||
│
|
||||
└── advanced/ # Power users (specialized)
|
||||
├── mcp-server.md
|
||||
├── mcp-tools.md
|
||||
├── custom-workflows.md
|
||||
└── multi-source.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Category Guidelines
|
||||
|
||||
### Getting Started
|
||||
|
||||
**Purpose:** Get new users to their first success quickly
|
||||
|
||||
**Characteristics:**
|
||||
- Minimal prerequisites
|
||||
- Step-by-step instructions
|
||||
- Copy-paste ready commands
|
||||
- Screenshots/output examples
|
||||
|
||||
**Files:**
|
||||
- `01-installation.md` - Install the tool
|
||||
- `02-quick-start.md` - 3 commands to first skill
|
||||
- `03-your-first-skill.md` - Complete walkthrough
|
||||
- `04-next-steps.md` - Where to go after first success
|
||||
|
||||
---
|
||||
|
||||
### User Guide
|
||||
|
||||
**Purpose:** Teach common tasks and concepts
|
||||
|
||||
**Characteristics:**
|
||||
- Task-oriented
|
||||
- Practical examples
|
||||
- Best practices
|
||||
- Common patterns
|
||||
|
||||
**Files:**
|
||||
- `01-core-concepts.md` - How it works
|
||||
- `02-scraping.md` - All 18 source types (docs, GitHub, PDF, video, Word, EPUB, Jupyter, HTML, OpenAPI, AsciiDoc, PPTX, RSS, man pages, Confluence, Notion, Slack/Discord, local codebase)
|
||||
- `03-enhancement.md` - AI enhancement
|
||||
- `04-packaging.md` - Platform export
|
||||
- `05-workflows.md` - Workflow presets
|
||||
- `06-troubleshooting.md` - Problem solving
|
||||
|
||||
---
|
||||
|
||||
### Reference
|
||||
|
||||
**Purpose:** Authoritative technical information
|
||||
|
||||
**Characteristics:**
|
||||
- Comprehensive
|
||||
- Precise
|
||||
- Organized for lookup
|
||||
- Always accurate
|
||||
|
||||
**Files:**
|
||||
- `CLI_REFERENCE.md` - All CLI commands (including 17 source-type subcommands)
|
||||
- `MCP_REFERENCE.md` - 40 MCP tools
|
||||
- `CONFIG_FORMAT.md` - JSON schema (covers all 18 source types)
|
||||
- `ENVIRONMENT_VARIABLES.md` - All env vars (including Confluence, Notion, Slack tokens)
|
||||
|
||||
---
|
||||
|
||||
### Advanced
|
||||
|
||||
**Purpose:** Specialized topics for power users
|
||||
|
||||
**Characteristics:**
|
||||
- Assumes basic knowledge
|
||||
- Deep dives
|
||||
- Complex scenarios
|
||||
- Integration topics
|
||||
|
||||
**Files:**
|
||||
- `mcp-server.md` - MCP server setup
|
||||
- `mcp-tools.md` - Advanced MCP usage
|
||||
- `custom-workflows.md` - Creating workflows
|
||||
- `multi-source.md` - Unified scraping
|
||||
|
||||
---
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### Files
|
||||
|
||||
- **getting-started:** `01-topic.md` (numbered for order)
|
||||
- **user-guide:** `01-topic.md` (numbered for order)
|
||||
- **reference:** `TOPIC_REFERENCE.md` (uppercase, descriptive)
|
||||
- **advanced:** `topic.md` (lowercase, specific)
|
||||
|
||||
### Headers
|
||||
|
||||
- H1: Title with version
|
||||
- H2: Major sections
|
||||
- H3: Subsections
|
||||
- H4: Details
|
||||
|
||||
Example:
|
||||
```markdown
|
||||
# Topic Guide
|
||||
|
||||
> **Skill Seekers v3.6.0**
|
||||
|
||||
## Major Section
|
||||
|
||||
### Subsection
|
||||
|
||||
#### Detail
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
Link to related docs using relative paths:
|
||||
|
||||
```markdown
|
||||
<!-- Within same directory -->
|
||||
See [Troubleshooting](06-troubleshooting.md)
|
||||
|
||||
<!-- Up one directory, then into reference -->
|
||||
See [CLI Reference](../reference/CLI_REFERENCE.md)
|
||||
|
||||
<!-- Up two directories (to root) -->
|
||||
See [Contributing](../../CONTRIBUTING.md)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Keeping Docs Current
|
||||
|
||||
1. **Update with code changes** - Docs must match implementation
|
||||
2. **Version in header** - Keep version current
|
||||
3. **Last updated date** - Track freshness
|
||||
4. **Deprecate old files** - Don't delete, redirect
|
||||
|
||||
### Review Checklist
|
||||
|
||||
Before committing docs:
|
||||
|
||||
- [ ] Commands actually work (tested)
|
||||
- [ ] No phantom commands documented
|
||||
- [ ] Links work
|
||||
- [ ] Version number correct
|
||||
- [ ] Date updated
|
||||
|
||||
---
|
||||
|
||||
## Adding New Documentation
|
||||
|
||||
### New User Guide
|
||||
|
||||
1. Add to `user-guide/` with next number
|
||||
2. Update `docs/README.md` navigation
|
||||
3. Add to table of contents
|
||||
4. Link from related guides
|
||||
|
||||
### New Reference
|
||||
|
||||
1. Add to `reference/` with `_REFERENCE` suffix
|
||||
2. Update `docs/README.md` navigation
|
||||
3. Link from user guides
|
||||
4. Add to troubleshooting if relevant
|
||||
|
||||
### New Advanced Topic
|
||||
|
||||
1. Add to `advanced/` with descriptive name
|
||||
2. Update `docs/README.md` navigation
|
||||
3. Link from appropriate user guide
|
||||
|
||||
---
|
||||
|
||||
## Deprecation Strategy
|
||||
|
||||
When content becomes outdated:
|
||||
|
||||
1. **Don't delete immediately** - Breaks external links
|
||||
2. **Add deprecation notice**:
|
||||
```markdown
|
||||
> ⚠️ **DEPRECATED**: This document is outdated.
|
||||
> See [New Guide](path/to/new.md) for current information.
|
||||
```
|
||||
3. **Move to archive** after 6 months:
|
||||
```
|
||||
docs/archive/legacy/
|
||||
```
|
||||
4. **Update navigation** to remove deprecated links
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
### Doc Changes
|
||||
|
||||
1. Edit relevant file
|
||||
2. Test all commands
|
||||
3. Update version/date
|
||||
4. Submit PR
|
||||
|
||||
### New Doc
|
||||
|
||||
1. Choose appropriate category
|
||||
2. Follow naming conventions
|
||||
3. Add to README.md
|
||||
4. Cross-link related docs
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Docs README](README.md) - Navigation hub
|
||||
- [Contributing Guide](../CONTRIBUTING.md) - How to contribute
|
||||
- [Repository README](../README.md) - Project overview
|
||||
@@ -0,0 +1,496 @@
|
||||
# Best Practices for High-Quality Skills
|
||||
|
||||
**Target Audience:** Anyone creating Claude skills | Already scraped documentation? Make it better!
|
||||
|
||||
**Time:** 5-10 minutes to review | Apply as you build
|
||||
|
||||
**Result:** Skills that Claude understands better and activates more reliably
|
||||
|
||||
---
|
||||
|
||||
## Quick Checklist
|
||||
|
||||
Before uploading a skill, check:
|
||||
|
||||
- [ ] SKILL.md has clear "When to Use" triggers
|
||||
- [ ] At least 5 code examples included
|
||||
- [ ] Prerequisites documented (if any)
|
||||
- [ ] Troubleshooting section present
|
||||
- [ ] Quality score 90+ (Grade A)
|
||||
|
||||
```bash
|
||||
# Check your skill quality
|
||||
skill-seekers quality output/myskill/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Structure Your SKILL.md Clearly
|
||||
|
||||
### Use Consistent Sections
|
||||
|
||||
Claude looks for specific sections to understand your skill:
|
||||
|
||||
```markdown
|
||||
# Skill Name
|
||||
|
||||
## Description
|
||||
Brief explanation of what this skill enables.
|
||||
|
||||
## When to Use This Skill
|
||||
- User asks about [specific topic]
|
||||
- User needs help with [specific task]
|
||||
- User mentions [keywords]
|
||||
|
||||
## Prerequisites
|
||||
What needs to be true before using this skill.
|
||||
|
||||
## Quick Reference
|
||||
Most common commands or patterns.
|
||||
|
||||
## Detailed Guide
|
||||
Step-by-step instructions with examples.
|
||||
|
||||
## Troubleshooting
|
||||
Common issues and solutions.
|
||||
```
|
||||
|
||||
### Why This Matters
|
||||
|
||||
Claude uses the "When to Use" section to decide if your skill matches the user's question. Vague triggers = skill doesn't activate.
|
||||
|
||||
**Bad Example:**
|
||||
```markdown
|
||||
## When to Use This Skill
|
||||
Use this skill for API-related questions.
|
||||
```
|
||||
|
||||
**Good Example:**
|
||||
```markdown
|
||||
## When to Use This Skill
|
||||
- User asks about Steam Inventory API methods
|
||||
- User needs to implement item drops in a Steam game
|
||||
- User wants to grant promotional items to players
|
||||
- User mentions: SteamInventory, GetAllItems, AddPromoItems
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Include Real Code Examples
|
||||
|
||||
Skills with 5+ code examples work significantly better. Claude learns patterns from examples.
|
||||
|
||||
### What Works
|
||||
|
||||
**Include a variety:**
|
||||
- Basic usage (getting started)
|
||||
- Common patterns (day-to-day use)
|
||||
- Advanced usage (edge cases)
|
||||
- Error handling (when things go wrong)
|
||||
|
||||
**Example from a good SKILL.md:**
|
||||
```markdown
|
||||
## Quick Reference
|
||||
|
||||
### Get All Items
|
||||
```cpp
|
||||
SteamInventoryResult_t resultHandle;
|
||||
bool success = SteamInventory()->GetAllItems(&resultHandle);
|
||||
```
|
||||
|
||||
### Grant Promotional Items
|
||||
```cpp
|
||||
void CInventory::GrantPromoItems()
|
||||
{
|
||||
SteamItemDef_t newItems[2];
|
||||
newItems[0] = 110;
|
||||
newItems[1] = 111;
|
||||
SteamInventory()->AddPromoItems(&s_GenerateRequestResult, newItems, 2);
|
||||
}
|
||||
```
|
||||
|
||||
### Handle Async Results
|
||||
```cpp
|
||||
void OnSteamInventoryResult(SteamInventoryResultReady_t *pResult)
|
||||
{
|
||||
if (pResult->m_result == k_EResultOK) {
|
||||
// Process items
|
||||
}
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
### What to Avoid
|
||||
|
||||
**Generic placeholder text:**
|
||||
```markdown
|
||||
## Quick Reference
|
||||
|
||||
*Quick reference patterns will be added as you use the skill.*
|
||||
```
|
||||
|
||||
**Code without context:**
|
||||
```markdown
|
||||
`GetAllItems()`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Document Prerequisites
|
||||
|
||||
Claude can check conditions before proceeding. This prevents errors mid-execution.
|
||||
|
||||
### Good Pattern
|
||||
|
||||
```markdown
|
||||
## Before You Start
|
||||
|
||||
Make sure you have:
|
||||
- [ ] Python 3.10+ installed
|
||||
- [ ] API key set in environment (`export API_KEY=...`)
|
||||
- [ ] Network access to api.example.com
|
||||
|
||||
### Quick Check
|
||||
```bash
|
||||
python3 --version # Should show 3.10+
|
||||
echo $API_KEY # Should not be empty
|
||||
curl api.example.com/health # Should return 200
|
||||
```
|
||||
```
|
||||
|
||||
### Why It Matters
|
||||
|
||||
Without prerequisites, Claude might:
|
||||
1. Start a complex workflow
|
||||
2. Fail halfway through
|
||||
3. Leave the user with a broken state
|
||||
|
||||
With prerequisites, Claude can:
|
||||
1. Check conditions first
|
||||
2. Report what's missing
|
||||
3. Guide the user to fix issues before starting
|
||||
|
||||
---
|
||||
|
||||
## 4. Add Troubleshooting Sections
|
||||
|
||||
Real-world usage hits errors. Document the common ones.
|
||||
|
||||
### Template
|
||||
|
||||
```markdown
|
||||
## Troubleshooting
|
||||
|
||||
### "Connection refused" error
|
||||
**Cause:** Service not running or firewall blocking
|
||||
|
||||
**Solution:**
|
||||
1. Check if the service is running: `systemctl status myservice`
|
||||
2. Verify firewall settings: `sudo ufw status`
|
||||
3. Test connectivity: `curl -v https://api.example.com`
|
||||
|
||||
### "Permission denied" error
|
||||
**Cause:** Insufficient file permissions
|
||||
|
||||
**Solution:**
|
||||
1. Check file permissions: `ls -la /path/to/file`
|
||||
2. Fix permissions: `chmod 644 /path/to/file`
|
||||
3. Verify ownership: `chown user:group /path/to/file`
|
||||
|
||||
### "Rate limited" error
|
||||
**Cause:** Too many API requests
|
||||
|
||||
**Solution:**
|
||||
1. Wait 60 seconds before retrying
|
||||
2. Implement exponential backoff in your code
|
||||
3. Consider caching responses
|
||||
```
|
||||
|
||||
### What to Include
|
||||
|
||||
- Error message (exact text users see)
|
||||
- Cause (why it happens)
|
||||
- Solution (step-by-step fix)
|
||||
- Prevention (how to avoid it)
|
||||
|
||||
---
|
||||
|
||||
## 5. Organize Reference Files
|
||||
|
||||
The `references/` directory should be easy to navigate.
|
||||
|
||||
### Good Structure
|
||||
|
||||
```
|
||||
output/myskill/
|
||||
├── SKILL.md # Main entry point
|
||||
└── references/
|
||||
├── index.md # Category overview
|
||||
├── getting_started.md # Installation, setup
|
||||
├── api_reference.md # API methods, classes
|
||||
├── guides.md # How-to tutorials
|
||||
└── advanced.md # Complex scenarios
|
||||
```
|
||||
|
||||
### Category Guidelines
|
||||
|
||||
| Category | Contains | Keywords |
|
||||
|----------|----------|----------|
|
||||
| getting_started | Installation, setup, quickstart | intro, install, setup, quickstart |
|
||||
| api_reference | Methods, classes, parameters | api, method, function, class, reference |
|
||||
| guides | Step-by-step tutorials | guide, tutorial, how-to, example |
|
||||
| concepts | Architecture, design patterns | concept, overview, architecture |
|
||||
| advanced | Complex scenarios, internals | advanced, internal, extend |
|
||||
|
||||
### Navigation Table in SKILL.md
|
||||
|
||||
```markdown
|
||||
## Navigation
|
||||
|
||||
| Topic | File | Description |
|
||||
|-------|------|-------------|
|
||||
| Getting Started | references/getting_started.md | Installation and setup |
|
||||
| API Reference | references/api_reference.md | Complete API documentation |
|
||||
| Guides | references/guides.md | Step-by-step tutorials |
|
||||
| Advanced | references/advanced.md | Complex scenarios |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Run Quality Checks
|
||||
|
||||
Always check quality before uploading:
|
||||
|
||||
```bash
|
||||
# Check quality score
|
||||
skill-seekers quality output/myskill/
|
||||
|
||||
# Expected output:
|
||||
# ✅ Grade: A (Score: 95)
|
||||
# ✅ No errors
|
||||
# ⚠️ 1 warning: Consider adding more code examples
|
||||
```
|
||||
|
||||
### Quality Targets
|
||||
|
||||
| Grade | Score | Status |
|
||||
|-------|-------|--------|
|
||||
| A | 90-100 | Ready to upload |
|
||||
| B | 80-89 | Good, minor improvements possible |
|
||||
| C | 70-79 | Review warnings before uploading |
|
||||
| D | 60-69 | Needs work |
|
||||
| F | < 60 | Significant issues |
|
||||
|
||||
### Common Issues
|
||||
|
||||
**"Missing SKILL.md"**
|
||||
- Run the scraper first
|
||||
- Or create manually
|
||||
|
||||
**"No code examples found"**
|
||||
- Add code blocks to SKILL.md
|
||||
- Run enhancement: `skill-seekers enhance output/myskill/`
|
||||
|
||||
**"Generic description"**
|
||||
- Rewrite "When to Use" section
|
||||
- Add specific keywords and use cases
|
||||
|
||||
---
|
||||
|
||||
## 7. Test Your Skill
|
||||
|
||||
Before uploading, test with Claude:
|
||||
|
||||
### Manual Testing
|
||||
|
||||
1. Upload the skill to Claude
|
||||
2. Ask a question your skill should answer
|
||||
3. Check if Claude activates the skill
|
||||
4. Verify the response uses skill content
|
||||
|
||||
### Test Questions
|
||||
|
||||
For a Steam Inventory skill:
|
||||
```
|
||||
"How do I get all items in a player's Steam inventory?"
|
||||
"What's the API call for granting promotional items?"
|
||||
"Show me how to handle async inventory results"
|
||||
```
|
||||
|
||||
### What to Look For
|
||||
|
||||
**Good activation:**
|
||||
- Claude references your skill
|
||||
- Response includes examples from your SKILL.md
|
||||
- Specific, accurate information
|
||||
|
||||
**Poor activation:**
|
||||
- Claude gives generic answer
|
||||
- No skill reference
|
||||
- Information doesn't match your docs
|
||||
|
||||
---
|
||||
|
||||
## Real-World Example
|
||||
|
||||
### Before Improvement
|
||||
|
||||
```markdown
|
||||
# React Skill
|
||||
|
||||
## Description
|
||||
React documentation.
|
||||
|
||||
## When to Use
|
||||
For React questions.
|
||||
|
||||
## Quick Reference
|
||||
See references.
|
||||
```
|
||||
|
||||
**Quality Score:** 45 (Grade F)
|
||||
|
||||
### After Improvement
|
||||
|
||||
```markdown
|
||||
# React Skill
|
||||
|
||||
## Description
|
||||
Complete React 18+ documentation including hooks, components, and best practices.
|
||||
|
||||
## When to Use This Skill
|
||||
- User asks about React hooks (useState, useEffect, useContext)
|
||||
- User needs help with React component lifecycle
|
||||
- User wants to implement React patterns (render props, HOCs, custom hooks)
|
||||
- User mentions: React, JSX, virtual DOM, fiber, concurrent mode
|
||||
|
||||
## Prerequisites
|
||||
- Node.js 16+ for development
|
||||
- Basic JavaScript/ES6 knowledge
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### useState Hook
|
||||
```jsx
|
||||
const [count, setCount] = useState(0);
|
||||
```
|
||||
|
||||
### useEffect Hook
|
||||
```jsx
|
||||
useEffect(() => {
|
||||
document.title = `Count: ${count}`;
|
||||
}, [count]);
|
||||
```
|
||||
|
||||
### Custom Hook
|
||||
```jsx
|
||||
function useWindowSize() {
|
||||
const [size, setSize] = useState({ width: 0, height: 0 });
|
||||
useEffect(() => {
|
||||
const handleResize = () => {
|
||||
setSize({ width: window.innerWidth, height: window.innerHeight });
|
||||
};
|
||||
window.addEventListener('resize', handleResize);
|
||||
handleResize();
|
||||
return () => window.removeEventListener('resize', handleResize);
|
||||
}, []);
|
||||
return size;
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Invalid hook call"
|
||||
**Cause:** Hook called outside component or conditionally
|
||||
|
||||
**Solution:**
|
||||
1. Only call hooks at top level of function components
|
||||
2. Don't call hooks inside loops or conditions
|
||||
3. Check for multiple React copies: `npm ls react`
|
||||
```
|
||||
|
||||
**Quality Score:** 94 (Grade A)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Practice | Why It Matters | Quick Check |
|
||||
|----------|---------------|-------------|
|
||||
| Clear structure | Claude knows where to look | Has all standard sections? |
|
||||
| Code examples (5+) | Claude learns patterns | Count code blocks |
|
||||
| Prerequisites | Prevents mid-task failures | Prerequisites section exists? |
|
||||
| Troubleshooting | Handles real-world errors | Common errors documented? |
|
||||
| Organized references | Easy navigation | Categories make sense? |
|
||||
| Quality check | Catches issues early | Score 90+? |
|
||||
| Testing | Confirms it works | Claude activates skill? |
|
||||
|
||||
**Final command before upload:**
|
||||
```bash
|
||||
skill-seekers quality output/myskill/
|
||||
```
|
||||
|
||||
That's it! Follow these practices and your skills will work better with Claude.
|
||||
|
||||
---
|
||||
|
||||
## 8. Tips for Specific Source Types
|
||||
|
||||
Skill Seekers supports **18 source types**. Here are tips for getting the best results from each category:
|
||||
|
||||
### Documentation (Web)
|
||||
- Always test CSS selectors before large scrapes: `skill-seekers create --max-pages 3 --verbose`
|
||||
- Use `--async` for large sites (2-3x faster)
|
||||
|
||||
### GitHub Repos
|
||||
- Use `--analysis-depth c3x` for deep analysis (patterns, tests, architecture)
|
||||
- Set `GITHUB_TOKEN` to avoid rate limits
|
||||
|
||||
### PDFs & Office Documents (PDF, Word, EPUB, PPTX)
|
||||
- Use `--enable-ocr` for scanned PDFs
|
||||
- For Word/PPTX, embedded images are extracted automatically; add `--extract-images` for PDFs
|
||||
- EPUB works best with DRM-free files
|
||||
|
||||
### Video
|
||||
- Run `skill-seekers create --setup` first to install GPU-optimized dependencies
|
||||
- YouTube and Vimeo URLs are auto-detected; local video files also work
|
||||
|
||||
### Jupyter Notebooks
|
||||
- Ensure notebooks are saved (unsaved cell outputs won't be captured)
|
||||
- Both code cells and markdown cells are extracted
|
||||
|
||||
### OpenAPI/Swagger Specs
|
||||
- Both YAML and JSON specs are supported (OpenAPI 3.x and Swagger 2.0)
|
||||
- Endpoints, schemas, and examples are parsed into structured API reference
|
||||
|
||||
### AsciiDoc & Man Pages
|
||||
- AsciiDoc requires `asciidoctor` (install via your package manager or gem)
|
||||
- Man pages in sections `.1` through `.8` are supported
|
||||
|
||||
### RSS/Atom Feeds
|
||||
- Useful for converting blog posts and changelogs into skills
|
||||
- Set `--max-items` to limit how many entries are extracted
|
||||
|
||||
### Confluence & Notion
|
||||
- API mode requires authentication tokens (see FAQ for setup)
|
||||
- Export directory mode works offline with HTML/Markdown exports
|
||||
|
||||
### Slack & Discord
|
||||
- Use official export tools (Slack Workspace Export, DiscordChatExporter)
|
||||
- Specify `--platform slack` or `--platform discord` explicitly
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Enhancement Guide](features/ENHANCEMENT.md) - AI-powered SKILL.md improvement
|
||||
- [Upload Guide](guides/UPLOAD_GUIDE.md) - How to upload skills to Claude
|
||||
- [CLI Reference](reference/CLI_REFERENCE.md) - Complete command reference
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
This guide was contributed by the [AI Writing Guide](https://github.com/jmagly/ai-writing-guide) project, which uses Skill Seekers for documentation-to-skill conversion. Best practices here are informed by research on production-grade agentic workflows.
|
||||
|
||||
Found an issue or want to improve this guide? PRs welcome!
|
||||
@@ -0,0 +1,704 @@
|
||||
# Skill Seekers — Bug Audit
|
||||
|
||||
> **Addendum (2026-06-11):** Follow-up review during the Grand Unification work
|
||||
> (see [UNIFICATION_PLAN.md](UNIFICATION_PLAN.md)) found that two fixes claimed
|
||||
> below were no-ops as shipped and have since been fixed properly:
|
||||
> 1. **MCP-03 (`dry_run` for unified configs)** — the injected flag never
|
||||
> reached `UnifiedScraper`; it is now genuinely honored (constructor param,
|
||||
> preview-and-return, no directory creation) for both CLI and MCP paths.
|
||||
> 2. **ENH timeout fallback** — the raw-config fallbacks for enhancement
|
||||
> settings (incl. timeout/"unlimited") were dead code because
|
||||
> `ExecutionContext.get()` never raises; the gate now uses
|
||||
> `ExecutionContext.is_initialized()`.
|
||||
>
|
||||
> The findings below are preserved unchanged as a historical record.
|
||||
|
||||
**Date:** 2026-06-10
|
||||
**Scope:** Full audit of `src/skill_seekers/` (~80K LOC): scan/config pipeline, CLI core & dispatch, platform adaptors, MCP server & tools, codebase-analysis engines (AST/pattern/dependency/guide/router), document & media & remote scrapers, enhancement + unified builder, argument/parser system, and infra (embedding, sync, storage, benchmark, presets).
|
||||
|
||||
**Method:** Two-pass. Pass 1 — 10 parallel review agents, one per subsystem, surfaced candidate defects. Pass 2 — a second set of agents (plus manual spot-checks) re-read the actual code behind **every** finding, checked reachability/trigger, marked each `Confirmed` / `Corrected` / `False-Positive`, deepened it, and hunted for additional related bugs. Only `Confirmed` and `Corrected` findings appear in the body; dismissed items are recorded in Appendix A for transparency.
|
||||
|
||||
**Severity rubric**
|
||||
- **Critical** — silent data loss/corruption of the primary output, or a core feature completely (and silently) non-functional.
|
||||
- **High** — wrong result / broken feature under common conditions, or a crash on realistic input.
|
||||
- **Medium** — wrong behaviour under specific conditions, misleading output, or latent correctness hazard with a plausible trigger.
|
||||
- **Low** — cosmetic, rare trigger, dead code, type-annotation, or defensive gap.
|
||||
|
||||
**No security/RCE findings.** Subprocess calls use argv arrays (no `shell=True`); path-traversal guards in `config_publisher`, `marketplace_publisher`, and `workflow_tools` were verified present and correct.
|
||||
|
||||
---
|
||||
|
||||
## Summary counts
|
||||
|
||||
| Severity | Count |
|
||||
|----------|-------|
|
||||
| Critical | 5 |
|
||||
| High | 18 |
|
||||
| Medium | 45 |
|
||||
| Low | 67 |
|
||||
| **Total confirmed/corrected** | **135** |
|
||||
| of which newly found in the verification pass (★) | 33 |
|
||||
| Dismissed (false-positive / no-impact) | 7 (Appendix A) |
|
||||
|
||||
IDs are subsystem-prefixed: `SCAN` (scan+config), `CLI` (core/dispatch), `ADP` (adaptors), `MCP`, `CBA` (codebase-analysis: AST/pattern/dependency), `CBB` (codebase-analysis: example/guide/router), `DOC` (document scrapers), `MED` (media/structured/remote scrapers), `ENH` (enhancement + unified builder), `INF` (args/parsers + infra). `★` marks bugs newly discovered during the verification pass.
|
||||
|
||||
---
|
||||
|
||||
## Resolution status (this PR)
|
||||
|
||||
Worked through the audit Critical → Low. Every Critical and High finding is fixed, and the entire Medium tier is fixed except one deferred item. Each fix carries a regression test where behavioral; the rest are covered by existing suites. Lint + format are clean throughout.
|
||||
|
||||
| Tier | Total | Fixed | Remaining |
|
||||
|------|-------|-------|-----------|
|
||||
| Critical | 5 | 5 | 0 |
|
||||
| High | 18 | 18 | 0 |
|
||||
| Medium | 45* | 44 | 1 (DOC-04) |
|
||||
| Low | 67* | 0 | 67 (not in scope for this PR) |
|
||||
|
||||
\* Medium/Low counts include sub-findings; the body tables enumerate each fixed ID.
|
||||
|
||||
**Delivered as 8 commits:**
|
||||
1. Critical + High behavioral bugs (each with a regression test).
|
||||
2. `--output` honored for every source type + config-file precedence (CLI-01/02, CFG-01/02).
|
||||
3. `ExecutionContext` made the real single source of truth (RT-01..11): ~12 inert flags wired, 4 bypasses fixed, dead `rag` section removed, `--languages`/`--dry-run` wired.
|
||||
4. Medium batch 1 — codebase-analysis, enhancement, infra, scan/config, adaptors.
|
||||
5. Medium batch 2 — guides + document/remote scrapers.
|
||||
6. DOC-07/MED-04 — SKILL.md nav links fixed across all 7 scrapers.
|
||||
7. ADP-01 — `--streaming` wired into the 8 RAG/vector adaptors (was dead for every target).
|
||||
8. CLI-04 — all 15 commands migrated off `_reconstruct_argv` to namespace dispatch (also fixed a pre-existing live bug: `skill-seekers workflows …` was broken).
|
||||
|
||||
**Remaining / out of scope:**
|
||||
- **DOC-04** (Medium, deferred): `DEFAULT_MAX_PAGES = -1` means a config without `max_pages` crawls unbounded. The fix (a finite default) is a **user-visible behavior change** for existing configs, so it's left for a maintainer product decision rather than changed unilaterally.
|
||||
- **Low tier (67):** cosmetic / rare-trigger / dead-code / type-annotation items — catalogued in the body, not addressed in this PR.
|
||||
- **`tests/test_bootstrap_skill*.py`** were hardened (out of band) to guard a fork-bomb: the bootstrap/`create` subprocess tests spawn real `skill-seekers create` runs (and LLM enhance agents). Because of this, the *full* `pytest tests/` is unsafe to run repeatedly; verification used targeted per-module suites + `-m "not slow and not integration"`.
|
||||
|
||||
---
|
||||
|
||||
## Independent re-review of this PR — 2026-06-10 (second pass)
|
||||
|
||||
After the fixes above, every commit in this PR was re-reviewed independently (one reviewer per commit) to verify the fixes are correct, complete, and don't introduce regressions — i.e. not taking the audit's own self-grade at face value. The reviewers confirmed the large majority of fixes are correct, and surfaced the following issues, **all now resolved in this PR** (each with a regression test unless noted):
|
||||
|
||||
### Regressions this PR had introduced (now fixed)
|
||||
|
||||
| ID | Sev | Fix |
|
||||
|----|-----|-----|
|
||||
| **ENH-03-bis** | Blocker | The ENH-03 retry path was mis-indented: when the exit-75 retry failed, `new_mtime` was unbound → `UnboundLocalError`. Re-nested under the returncode/exists guards. Tests: `test_enhance_skill_local.py::TestHeadlessSuccessGate::{test_failed_retry_returns_false_without_crashing,test_successful_retry_counts_as_success}`. |
|
||||
| **R1 (CLI-04-bis)** | High | `skill-seekers estimate <cfg>` crawled **unlimited** instead of capping at 1000 — the unified `estimate_parser`'s `--max-discovery` had no default (`None`), which `estimate_pages()` treats as unlimited. Added `default=DEFAULT_MAX_DISCOVERY`. Test: `test_estimate_pages.py::TestEstimateParserDefault`. |
|
||||
| **R2 (CBA-14-bis)** | Medium | The JS method regex required `)` immediately before `{`, silently dropping **TypeScript** class methods with return-type annotations (`greet(): string {`). Made the return type optional in the pattern. Test: `test_code_analyzer.py::...::test_typescript_class_methods_with_return_types_not_dropped`. |
|
||||
| **R3 (CLI-02-bis)** | Medium | `github_scraper._save_data()` did `os.makedirs("output")` but wrote to a `--output`-derived `data_file`, raising `FileNotFoundError` for `create owner/repo --output <nested/path>`. Now creates `dirname(data_file)`. Test: `test_github_scraper.py::...::test_save_data_creates_nested_output_parent`. |
|
||||
|
||||
### Audit fixes that were incomplete / over-claimed (now corrected)
|
||||
|
||||
| ID | Sev | Resolution |
|
||||
|----|-----|-----------|
|
||||
| **I1 (MCP-03)** | High | `skip_scrape` was a no-op (set an attribute no converter read). Now honored at the `SkillConverter.run()` chokepoint for single-source paths and by `UnifiedScraper.run()` for multi-source configs, which reloads each configured source from `.skillseeker-cache`/cached extraction files before conflict detection and build. Tests: `test_skill_converter.py::TestSkipScrape`, `test_unified_scraper_orchestration.py::TestUnifiedSkipScrape`, `test_unified_mcp_integration.py::test_mcp_scrape_docs_unified_skip_scrape_sets_attribute_without_warning`. |
|
||||
| **I2 (RT-06)** | Medium | Config-file **inline** `stages` never ran — `_build_inline_engine` re-read `args.enhance_stage` (None for config-only) instead of the resolved list. Now takes the resolved `inline_stages`. Test: `test_workflow_runner.py::...::test_inline_stages_from_config_execute`. |
|
||||
| **I3 (ENH-06)** | Low | The OpenAI branch of `_call_api` still hardcoded `timeout=120`; now forwards the caller's `request_timeout`. Test: `test_agent_client.py::...::test_openai_forwards_caller_timeout`. |
|
||||
| **I4 (RT-04)** | Low | RT-04 claimed `--timeout` is wired on `create`, but the flag isn't registered there, so the mapping is inert on that path (it only fires for config dicts carrying `timeout`). **Claim corrected here** — no CLI flag added (avoids new surface area for an over-claim). |
|
||||
|
||||
### Low-tier / pre-existing issues fixed opportunistically
|
||||
|
||||
| ID | Area | Fix |
|
||||
|----|------|-----|
|
||||
| **CBA-09** | Kotlin | `is_suspend` used a substring test, mis-flagging a function *named* `suspendCoroutine`. Now word-boundary (`\bsuspend\b`). Test in `test_kotlin_support.py`. |
|
||||
| **P1** | chat | Discord 429 retry was unbounded (hang risk) — the Slack path in the same commit was bounded; Discord now caps consecutive 429s at 3 and stops with partial data. |
|
||||
| **P2** | doc | `_normalize_url` stripped `?ref=`, which is content-bearing on some sites (GitHub `?ref=branch`, SPA routers) → could collapse distinct pages. `ref` removed from the strip set (`ref_src` kept). Tests in `test_scraper_features.py::TestNormalizeUrl`. |
|
||||
| **P3** | doc | `--dry-run` supplied via config still created empty output dirs (the ctor used the param, not `self.dry_run`). Now uses `self.dry_run`/`self.resume`. |
|
||||
| **P5** | adaptors | `claude.py` enhancement did rename-then-write, leaving only `SKILL.md.backup` if the write failed. Now writes to a temp file, copies the backup, then `os.replace()` (atomic). |
|
||||
| **P6** | mcp | `submit_config` dedup used GitHub's fuzzy title search (`[CONFIG] react` matched `react-native`). Added an exact-title guard. |
|
||||
| **P8** | scrapers | asciidoc/jupyter `_generate_index` still had inline filename logic (drift risk vs the nav helper). Both index generators now route through `_ref_filename`, so index links and reference files share one source of truth. |
|
||||
| **P9** | config | `config status` omitted the Moonshot key row; added it. |
|
||||
|
||||
### Deferred (genuine larger work — not safe to do inline; tracked here)
|
||||
|
||||
- **Streaming adaptor format (ADP-01 follow-up):** all 8 RAG/vector adaptors emit a *generic* streamed format rather than each platform's native shape (only the unregistered example classes override the converter). `--streaming` works; per-platform fidelity is a follow-up.
|
||||
- **Unified multi-source `skip_scrape` (I1 remainder):** fixed in #405 by rebuilding `scraped_data` from cached per-source files before the unified build, and by forwarding `--skip-scrape` through the config-source `create` path.
|
||||
- **Kotlin brace-depth / `is_suspend` corpus robustness (CBA-08, CBA-11):** the brace-depth nesting check is fooled by braces inside strings/comments; a robust fix needs a string/comment-aware scan (shared with the still-open CBA-11).
|
||||
### Low-tier disposition (2026-06-11)
|
||||
|
||||
After fixing the ~12 impactful Low items above, the remaining Low tier was triaged
|
||||
and batch-fixed. Issues #405–#408 track the 4 larger deferred items.
|
||||
|
||||
**Fixed in this PR (data-loss/crash + minor correctness):**
|
||||
DOC-11, DOC-13, DOC-14, DOC-15, DOC-17, MED-06, MED-07, MED-08, MED-11, MED-12,
|
||||
MED-14, MED-17, MED-18, ENH-13, ENH-14, ENH-15, ENH-16, ENH-17, ADP-03, ADP-04,
|
||||
ADP-05, ADP-06, MCP-11, MCP-15, MCP-16, MCP-17, SCAN-05, SCAN-06, SCAN-07, CLI-08,
|
||||
CLI-09, INF-06, CBA-12, CBA-16, CBB-09, CBB-10, CBB-11, CBB-14, CBB-15, CBB-16
|
||||
(plus the re-review R/I/P items). MED-16 was already fixed in batch 2.
|
||||
|
||||
**Intentionally NOT changed (with reason):**
|
||||
- **Product decisions (left as-is, like DOC-04):** MCP-10 (sync caps at 500 — a
|
||||
sensible safety bound), CLI-07 (a failed *optional* enhancement returning a
|
||||
non-zero exit from `create` is a behavior change — needs a maintainer call).
|
||||
- **Cosmetic / doc-only (dropped):** ADP-07 (docstring), CLI-05 (a misleading
|
||||
success-log path), MED-13 (type-annotation lie), MCP-08 (param typing), MED-09,
|
||||
MED-10 (already corrected/dead per Appendix).
|
||||
- **Still deferred (genuinely larger work):** MED-15 (OpenAPI external/multi-file
|
||||
`$ref` bundling — needs a pre-bundle pass; the MED-07 *cycle* guard is fixed),
|
||||
MCP-09 (a truly structured MCP error channel — the text already carries ✅/❌
|
||||
markers).
|
||||
|
||||
- The original **DOC-04** decision stands (issue #408): keep `-1` (unbounded).
|
||||
|
||||
---
|
||||
|
||||
## Fix log — 2026-06-10
|
||||
|
||||
All 5 Criticals + all 18 High findings fixed with regression tests (lint/format clean; affected suites green). A few mechanical/wide fixes (CLI-01, CLI-02, CBB-02/03/12, MCP-02..05) are covered by the broad suite rather than dedicated new tests.
|
||||
|
||||
### Critical
|
||||
|
||||
| ID | Status | Fix summary |
|
||||
|----|--------|-------------|
|
||||
| ENH-01 | ✅ Fixed | `enhance_skill.py`, `agent_client.py`, `adaptors/claude.py` refuse to save/return a `max_tokens`-truncated (or empty) response; unified-scraper path covered via `agent_client`. |
|
||||
| SCAN-01 | ✅ Fixed | Schema hint `code_analysis_depth` `"standard"` → `"deep"`; pinned to the validator by a regression test. |
|
||||
| CBA-13 | ✅ Fixed | GDScript class dict key `"bases"` → `"base_classes"` (+ `docstring`, `_offset_to_line`). |
|
||||
| CBB-01 | ✅ Fixed | `guide_enhancer._enhance_via_local` now calls the real `_call_ai`. |
|
||||
| MCP-01 | ✅ Fixed | `config_publisher` clone/pull/push authenticate via an explicit token URL (origin stays tokenless). |
|
||||
| ★MCP-12 | ✅ Fixed | `get_source` `KeyError` translated to a helpful `ValueError`. |
|
||||
| ★MCP-13 | ✅ Fixed | cached-repo re-pull uses the token URL (was tokenless origin). |
|
||||
| ★MCP-14 | ✅ Fixed | feature branch restored in a `finally` so the cache can't be stranded. |
|
||||
| MCP-07 | ✅ Fixed | commit `action` uses pre-copy existence (`add` vs `update`). |
|
||||
| ADP-06 | ✅ Fixed | `adaptors/claude.py` validates non-empty before renaming the original (folded into the ENH-01 fix). |
|
||||
|
||||
### High
|
||||
|
||||
| ID | Status | Fix summary |
|
||||
|----|--------|-------------|
|
||||
| SCAN-02 | ✅ Fixed | Guard the non-numeric `confidence` parse so one bad AI entry can't crash the scan. |
|
||||
| CLI-01 | ✅ Fixed | `_build_config` reads the negative dests (`not no_issues`/`no_changelog`/`no_releases`); `--no-*` opt-outs now work. |
|
||||
| CLI-02 | ✅ Fixed | `--output` now honored across the whole single-source create path: base `SkillConverter` + all 16 overriding converters read `config["output_dir"]` (incl. 5 type-annotated `skill_dir: str = …` that a first pass missed), and `data_file`/`data_dir`/`checkpoint` are derived from `skill_dir` so intermediate artifacts follow `--output` too (behavior-preserving fallback to `output/<name>`). `_build_config` passes `output_dir` for every branch. The unified/`config` multi-source path also honors `--output`: `unified_scraper`/`unified_skill_builder` derive the final skill from `config["output_dir"]`, and the docs/github sub-converters now write **straight into the `.skillseeker-cache/` cache** (their sub-config gets `output_dir`), removing the `output/{subname}` hardcodes and the move-to-cache dance. (`pdf_extractor_poc.py` keeps an `output/` *default* but already accepts `image_dir` as a constructor arg — standalone PoC, not the config flow.) |
|
||||
| CBA-01 | ✅ Fixed | Singleton precedence: `(has_instance_method or has_init_control) and confidence >= 0.5`. |
|
||||
| CBA-02 | ✅ Fixed | `total_emissions` iterates `.values()` (was `.items()` → always 2×#signals). |
|
||||
| CBA-03 | ✅ Fixed | JS/TS class body delimited by the matching brace (brace-count), not the first `}`. |
|
||||
| CBA-04 | ✅ Fixed | New `_base_root`/`_matches_base` normalize generic/qualified bases so subclass detection matches. |
|
||||
| CBB-02 | ✅ Fixed | Removed the positional verification overwrite that misattributed checks to steps. |
|
||||
| CBB-03 | ✅ Fixed | `_generate_examples_from_github` works on a copy + index set instead of mutating shared insights. |
|
||||
| CBB-12 | ✅ Fixed | `dependencies=list(imports)` — per-example copy (no shared mutable list). |
|
||||
| ENH-02 | ✅ Fixed | `SKILL_SEEKER_PROVIDER` override so a Moonshot/Kimi `sk-` key isn't misrouted to OpenAI. |
|
||||
| ENH-03 | ✅ Fixed | Headless success gates on mtime advance, not file growth (matches background/daemon). |
|
||||
| ENH-04 | ✅ Fixed | Kimi parser uses `re.DOTALL` + record-boundary anchor (multi-line / apostrophe safe). |
|
||||
| ENH-05 | ✅ Fixed | Only read the response file when `output_file` was requested; dropped the stray-`.json` glob. |
|
||||
| MCP-02 | ✅ Fixed | Package-path regex matches `Output:` / `Package created:` (the strings actually printed). |
|
||||
| MCP-03 | ✅ Fixed | `scrape_docs` honors `skip_scrape` and applies `dry_run` for unified configs too. |
|
||||
| MCP-04 | ✅ Fixed | `submit_config` searches for an existing open issue before creating (idempotent). |
|
||||
| MCP-05 | ✅ Fixed | `install_skill` detects failure by the specific `_run_converter` markers, not any `❌`. |
|
||||
|
||||
Regression tests added — Critical: `test_agent_client.py::TestCallApiTruncation`, `test_scan_command.py::TestGenerateSchemaHintDepth`, `test_code_analyzer.py::TestGDScriptParsing`, `test_guide_enhancer.py::TestEnhanceViaLocalRegression`, `test_config_publisher.py` (re-pull + KeyError + add/update). High: `test_agent_client.py` (`TestProviderOverride`, `TestKimiOutputParsing`, `TestCallLocalStrayJson`), `test_scan_command.py` (non-numeric confidence), `test_enhance_skill_local.py::TestHeadlessSuccessGate`, `test_pattern_recognizer.py::TestBaseClassMatching`, `test_code_analyzer.py` (JS multi-method), `test_signal_flow_analyzer.py`. (CBB-02/03/12, CLI-01, MCP-02..05 are covered by the broad suite rather than dedicated new tests.)
|
||||
|
||||
### Config-override follow-up (found while auditing "what else is overridden by hardcoded values")
|
||||
|
||||
| ID | Status | Fix summary |
|
||||
|----|--------|-------------|
|
||||
| CFG-01 | ✅ Fixed | **Web `--config` `selectors`/`url_patterns` silently ignored.** `_merge_json_config` only fills missing keys, but `_build_config` pre-set the hardcoded `selectors`/`url_patterns` before merging → the user's config-file values were shadowed. Now applied via `setdefault` **after** the merge, so the file wins. (Verified `doc_scraper` actually reads `config["selectors"]`/`url_patterns`.) |
|
||||
| CFG-02 | ✅ Fixed | **Web config-file `workers`/`async_mode`/`browser_wait_until`/`browser_extra_wait` dropped.** `ExecutionContext._load_config_file` copied only `max_pages`/`rate_limit`/`browser`; now copies all scraping-tuning keys. Test: `test_execution_context.py::...test_simple_web_config_format`. |
|
||||
| CFG-03 | ℹ️ Noted | `--config` is only merged for `web`/`github` source types; for `pdf`/`video`/etc. the documented path is the unified `sources` config (which reads per-source keys correctly). Low impact; left as a design note. |
|
||||
|
||||
### Runtime-config (ExecutionContext) audit — is every field wired through args→context→consumer, with no bypasses?
|
||||
|
||||
The singleton claims "all components read from this context instead of parsing their own argv." Audit found it was **not** the single source of truth: ~12 registered flags never reached the context, several consumers bypassed it, and two sections were dead. All fixed (regression tests in `test_execution_context.py`).
|
||||
|
||||
| ID | Status | Fix summary |
|
||||
|----|--------|-------------|
|
||||
| RT-01 | ✅ Fixed | **5 analysis `--skip-*` flags inert.** `--skip-config-patterns`/`--skip-api-reference`/`--skip-dependency-graph`/`--skip-docs`/`--no-comments` were never mapped in `_args_to_data`, yet `create_command` reads those exact `ctx.analysis.*` fields → the skipped steps ran anyway. Now mapped. |
|
||||
| RT-02 | ✅ Fixed | **`--skip-config` orphan dest** now aliased to `skip_config_patterns`. |
|
||||
| RT-03 | ✅ Fixed | **`--dry-run` dead on `create`.** `ctx.output.dry_run` was set but never put in the converter config; `_build_config` now passes `"dry_run"`, and `doc_scraper` reads `config["dry_run"]` (was only honoring the unused ctor param). |
|
||||
| RT-04 | ✅ Fixed | **`--timeout` not wired on `create`** → mapped to `ctx.enhancement.timeout` in `_args_to_data`. |
|
||||
| RT-05 | ✅ Fixed | **`get_agent_client()` dropped `api_key`** → a CLI `--api-key` was lost and AgentClient env-detected only. Now forwarded. |
|
||||
| RT-06 | ✅ Fixed | **Config-file `workflows`/`stages`/`workflow_vars` never ran** — `run_workflows`/`collect_workflow_vars` read only argv. Now fall back to `ctx.enhancement.*` so config-declared workflows execute. |
|
||||
| RT-07 | ✅ Fixed | **Confluence/Notion `max_pages` bypass** — read `getattr(self.args, …)` (dropping config-file `max_pages`); now read `ctx.scraping.max_pages`. |
|
||||
| RT-08 | ✅ Fixed | **`unified_scraper` dropped `ctx.enhancement.timeout` and `api_key`** (re-read raw config / built `AgentClient(mode="api")`); now use the context. |
|
||||
| RT-09 | ✅ Fixed | **`scraping.languages` was a dead field with a name collision** (default `["en"]`, read nowhere; `--languages` is a *code*-filter the local converter read from argv, which would have broken if fed `["en"]`). Repurposed the field as the code-language filter (`default=None`), wired `--languages` into it, and `create_command` now reads `ctx.scraping.languages`. |
|
||||
| RT-10 | ✅ Fixed (removed) | **`rag.*` section was write-only/dead.** Chunking lives in the separate `package` command, which parses its own (richer) args and runs as its own process — the create/scrape context never chunks and `ctx.rag` was read nowhere. Removed `RAGSettings` from the context. As cleanup, `common.py` now sources `DEFAULT_CHUNK_TOKENS`/`_OVERLAP` from `defaults.json` (was hardcoded `512`/`50`), so `defaults.json`'s `rag` block is the single source of truth for package chunking. *(Deviation: the earlier plan said "wire rag via package"; on inspection that's unsound — `create` and `package` are separate processes, so a create-time context can't reach package. Removal is the honest fix.)* |
|
||||
| RT-11 | ✅ Kept (not a bug) | **`source.*`** is read nowhere in production (consumers use `create_command.self.source_info`), but it is **tested public API** and legitimately belongs in an execution context. Retained — removing tested API to chase a cosmetic dual-representation would be over-reach. |
|
||||
|
||||
### Medium tier — batch 1 (codebase-analysis + enhancement + infra + scan/config + adaptors)
|
||||
|
||||
All ✅ Fixed; verified by existing suites + new regression tests where behavioral.
|
||||
|
||||
| ID | Fix summary |
|
||||
|----|-------------|
|
||||
| CBA-05 / CBA-06 | Parenthesized the `A and B or C` precedence so the `"protocol"`/`"event"` clause no longer boosts *every* pattern type. |
|
||||
| CBA-07 | `_resolve_import` converts dotted module names → slash paths + suffix-matches `file_nodes`, so the dependency graph has edges (cycle detection was empty for all dotted-import languages). |
|
||||
| CBA-08 | Kotlin top-level-fn detection uses brace-depth, not the brittle `indent > 4`. |
|
||||
| CBA-09 | Kotlin `is_suspend` reads `match.group(0)` (the matched modifiers), not a fixed look-behind window. |
|
||||
| CBA-10 | `gd_resource` regex ends with `\s*[\]\s]` so `script_class` is captured on compact headers. |
|
||||
| CBA-14 | JS method extractor requires a trailing `{` (declarations only) + a fuller keyword blocklist, so call-sites like `setTimeout(...)` aren't counted as methods (unmasked by the CBA-03 brace fix). |
|
||||
| CBA-15 | GDScript signal doc comment = nearest preceding non-blank line (was always `None` for col-0/indented signals). |
|
||||
| ENH-06 | `_call_api` threads the caller's `timeout` (was hardcoded 120s, killing large prompts). |
|
||||
| ENH-07 | `scrape_all_sources` reports the count of items scraped (was the bucket count, always ~17) and returns it. |
|
||||
| ENH-08 | `run()` aborts with non-zero when every source failed (was building an empty skill + exit 0). |
|
||||
| ENH-09 | "Official Documentation" URL filters `type == "documentation"` (was `sources[0]`). |
|
||||
| ENH-10 | `DEBUG:` synthesis logs downgraded to `logger.debug`. |
|
||||
| ENH-11 | Gemini API honors `max_output_tokens`/`timeout` and rejects a truncated reply. |
|
||||
| ENH-12 | Moonshot/Kimi added to the auto-detect priority in `enhance_command._pick_mode` + `enhance_skill_local._detect_api_target` (Moonshot-only users were dropped to LOCAL). |
|
||||
| SCAN-03 | `get_api_key`/`set_api_key` include `moonshot` → `MOONSHOT_API_KEY`. |
|
||||
| SCAN-04 | `scan --dry-run` resolves with `auto_fetch=False` (no network, no stray `./configs/` write). |
|
||||
| MCP-06 | `list_configs` falls back to `sources[].base_url` for unified configs. |
|
||||
| ADP-02 | `chroma`/`pinecone` split `except ImportError` from `except Exception` (don't misreport a broken-but-installed package as "not installed"). |
|
||||
| CLI-03 | `CreateCommand.execute` calls `ExecutionContext.reset()` first so a 2nd in-process create rebuilds the context. |
|
||||
| INF-01 | Embedding cache key includes `normalize` (`{model}:{int(normalize)}:{text}`); threaded through `server.py`. |
|
||||
| INF-02 | `cleanup_old` strips the full `_YYYYMMDD_HHMMSS` timestamp via regex (retention was per-name-per-day). |
|
||||
| INF-03 | `memory()` samples RSS on a background thread to capture the true peak. |
|
||||
| INF-04 | `compare()` guards division by zero on instant ops. |
|
||||
| INF-05 | `check_header_changes` returns `True` when the server provides no `Last-Modified`/`ETag` validators. |
|
||||
|
||||
### Medium tier — batch 2 (guides + document scrapers + remote scrapers)
|
||||
|
||||
All ✅ Fixed; verified by the guide/scraper suites + new regression tests where behavioral.
|
||||
|
||||
| ID | Fix summary |
|
||||
|----|-------------|
|
||||
| CBB-04 | `_extract_steps_python` descends one level into the test-function wrapper (else module body) instead of `ast.walk`, which flattened nested control flow out of context. |
|
||||
| CBB-05 | `ai_enhancer` batch analysis maps back by an echoed `index` (was positional → a dropped/reordered entry shifted every later example's analysis). |
|
||||
| CBB-06 | `_is_test_class` matches a base that *is* a `TestCase` (bare/`*TestCase`), not the `"Test" in base.id` substring that matched `LatestConfig`/`TestableMixin`. |
|
||||
| CBB-07 | Index TOC sorts by a difficulty rank map (was alphabetical → advanced/beginner/intermediate). |
|
||||
| CBB-08 | c3x failure returns `analysis_type="c3x_failed"` (distinct from empty) and the temp dir is removed in `finally` (was leaked every run). |
|
||||
| CBB-13 | `guide_enhancer` step-enhancements map by the model's explicit `step_index`; positional fallback only when none are indexed (extracted to `_parse_step_enhancements`). |
|
||||
| DOC-02 | `smart_categorize` assigns to the highest-scoring category (was the first over threshold → config-order dependent). |
|
||||
| DOC-03 | Link extraction strips tracking params (`utm_*`/`fbclid`/…) + sorts the query before dedup, so tracking variants don't re-crawl the same page (preserves `?lang=`). |
|
||||
| DOC-05 | PDF cross-page code merge requires a real continuation token / unbalanced bracket / block-opener colon (was `any([...])` ≈ always true) and keeps both pages' `code_blocks_count` consistent. |
|
||||
| DOC-06 | Async `--dry-run` caps the preview at 20 even for unlimited configs (matches sync; was previewing the whole site). |
|
||||
| MED-01 | `github_fetcher` follows `Link: rel="next"` pagination (was a single ≤100 page). Regression test added. |
|
||||
| MED-02 | Slack: per-channel try/except so one channel can't abort all; `conversations_history` retries on 429 with `Retry-After`. |
|
||||
| MED-03 | Discord: `ClientTimeout(30)`, 429 `Retry-After` retry, and a guarded `before` cursor (`.get("id")`). |
|
||||
|
||||
### Medium tier — batch 3 (systemic nav-link fix)
|
||||
|
||||
| ID | Status | Fix summary |
|
||||
|----|--------|-------------|
|
||||
| DOC-07 + MED-04 | ✅ Fixed | **Broken SKILL.md nav links across all 7 scrapers** (pdf/html/word/epub/jupyter/asciidoc/pptx). Each scraper now routes its SKILL.md nav, `index.md`, and reference-file writer through ONE filename helper (`_reference_filename`, or the pre-existing `_ref_filename` for asciidoc/jupyter), so nav links match the actual range/basename filenames instead of pointing at nonexistent `sanitize(title).md`. Regression test: `test_pdf_scraper.py::...test_reference_filename_matches_nav_and_index`. |
|
||||
| ADP-01 | ✅ Fixed | **`--streaming` was dead for every target.** `StreamingAdaptorMixin` was never inherited, so `hasattr(adaptor, "package_streaming")` was always False. Fixed the mixin's fragile `sys.path` import (now `from skill_seekers.cli.streaming_ingest import …`) and made all 8 RAG/vector adaptors (langchain, llama-index, chroma, haystack, weaviate, qdrant, faiss, pinecone) inherit it. `--streaming` now produces a real streamed package; non-streaming targets (claude/markdown) still fall back with the announced message. Regression + end-to-end test in `test_adaptors/test_langchain_adaptor.py`. |
|
||||
|
||||
| CLI-04 | ✅ Fixed | **`_reconstruct_argv` argv round-trip removed; all 15 legacy commands migrated to namespace dispatch.** Each command's `main(args=None)` now accepts the central parsed namespace (re-parses only when invoked standalone via the `skill-seekers-<cmd>` entry points). The dispatcher calls `module.main(args=args)` directly and `_reconstruct_argv` is deleted. Fixed the drift the round-trip had been masking: aligned 4 central positional dests to the modules (`skill_directory`→`skill_dir`, `input_file`→`input` via `metavar`), added `--welcome` to the config parser, and backfill module-parser defaults for options the central parser doesn't expose. **This also fixed a pre-existing live bug: `skill-seekers workflows …` was broken** (`unrecognized arguments: --workflows-action`) and now works. Verified: 345 command-suite tests + all 16 commands dispatch cleanly. |
|
||||
|
||||
Deferred (Medium):
|
||||
- **DOC-04** — `DEFAULT_MAX_PAGES = -1` (unbounded default crawl). Changing the default to finite is a user-visible behavior change; left for an explicit decision (not selected for fixing).
|
||||
|
||||
> **Test-harness note:** the *full* `pytest tests/` run is unsafe to execute repeatedly — `@pytest.mark.slow` tests like `test_bootstrap_skill::test_bootstrap_script_runs` and the `test_create_integration_basic` subprocess tests invoke the real `create` pipeline, which spawns live LLM agents (a fork-bomb). Verify with targeted per-module test files and `-m "not slow and not integration"`.
|
||||
|
||||
---
|
||||
|
||||
## Triage index — Critical & High
|
||||
|
||||
| ID | Sev | Title | File |
|
||||
|----|-----|-------|------|
|
||||
| ENH-01 | Critical | Truncated SKILL.md silently overwrites the original | `enhance_skill.py` / `agent_client.py` / `adaptors/claude.py` |
|
||||
| SCAN-01 | Critical | Scan AI github-config generation always rejected by validator | `scan_command.py:426` |
|
||||
| CBB-01 | Critical | LOCAL-mode guide enhancement calls a non-existent method (silent no-op) | `guide_enhancer.py:307` |
|
||||
| MCP-01 | Critical | `push_config` strips its own token → broken for private repos | `mcp/config_publisher.py:170,210` |
|
||||
| CBA-13 ★ | Critical | GDScript class dicts use `"bases"`; consumer reads `"base_classes"` | `code_analyzer.py:1986` |
|
||||
| SCAN-02 | High | Non-numeric AI `confidence` crashes the whole scan | `scan_command.py:394` |
|
||||
| CLI-01 | High | `--no-issues/--no-changelog/--no-releases` are dead on `create` | `create_command.py:270` |
|
||||
| CLI-02 | High | `--output` ignored for all non-local sources; enhancement targets wrong dir | `create_command.py:215` |
|
||||
| CBA-01 | High | Singleton precedence bug bypasses the confidence gate | `pattern_recognizer.py:453` |
|
||||
| CBA-02 | High | `total_emissions` iterates `.items()` → always 2×#signals | `signal_flow_analyzer.py:172` |
|
||||
| CBA-03 | High | JS/TS class body truncated at first `}` → later methods dropped | `code_analyzer.py:301` |
|
||||
| CBA-04 | High | Subclass detection misses generic/qualified bases | `pattern_recognizer.py:819` |
|
||||
| CBB-02 | High | Verification points overwritten/misattributed across steps | `how_to_guide_builder.py:185` |
|
||||
| MCP-02 | High | `install_skill` package-path regex never matches real output | `tools/packaging_tools.py:602` |
|
||||
| MCP-03 | High | `scrape_docs` drops `skip_scrape` and `dry_run` (unified) | `tools/scraping_tools.py:144` |
|
||||
| MCP-04 | High | `submit_config` has no idempotency guard → duplicate issues | `tools/source_tools.py:570` |
|
||||
| MCP-05 | High | `install_skill` infers failure from `"❌"` substring | `tools/packaging_tools.py:523` |
|
||||
| ENH-02 | High | Direct Moonshot key misrouted to OpenAI | `agent_client.py:202` |
|
||||
| ENH-03 | High | Headless "success" requires the file to *grow* | `enhance_skill_local.py:900` |
|
||||
| ENH-04 | High | Kimi stdout parser drops all multi-line responses | `agent_client.py:467` |
|
||||
| ENH-05 | High | Stray `.json` in agent cwd returned as the LLM response | `agent_client.py:419` |
|
||||
| CBB-03 | High | `common_problems.remove()` mutates shared insights | `generate_router.py:421` |
|
||||
| CBB-12 ★ | High | `dependencies=imports` shares one mutable list across examples | `test_example_extractor.py:394` |
|
||||
|
||||
---
|
||||
|
||||
# Critical
|
||||
|
||||
## ENH-01 — Truncated SKILL.md silently overwrites the original `Confirmed · High`
|
||||
**Location:** `cli/enhance_skill.py:75-94`; `cli/agent_client.py:286-301`; reachable save path `cli/adaptors/claude.py:384-405`; unified path `cli/unified_scraper.py:2079`.
|
||||
**Trigger:** Enhancement output exceeds `max_tokens` (4096 in `enhance_skill.py`/`agent_client.py`; 8192 in the unified path), or the model otherwise stops with `stop_reason="max_tokens"`.
|
||||
**Mechanism:** All call sites read `message.content[0].text` / iterate for `block.text` with **no** check of `message.stop_reason` (Anthropic) or `choices[0].finish_reason` (OpenAI). The reachable save path renames the original to `SKILL.md.backup` **first** (claude.py:400), then writes the new content (claude.py:404). So a truncated body overwrites the live file; the only intact copy is the backup.
|
||||
**Impact:** Any non-trivial skill gets silently chopped to ~4k output tokens (broken frontmatter/fences) and saved as "complete." Silent data loss of the primary deliverable; the run reports success.
|
||||
```python
|
||||
message = client.messages.create(model=..., max_tokens=4096, ...)
|
||||
enhanced_content = message.content[0].text # no stop_reason check
|
||||
if skill_md_path.exists():
|
||||
skill_md_path.rename(skill_md_path.with_suffix(".md.backup")) # original moved away first
|
||||
skill_md_path.write_text(enhanced_content, ...) # truncated body wins
|
||||
```
|
||||
**Fix:** Check the stop reason before touching the original; abort on truncation. Also write to a temp file and `os.replace` only after a successful, complete write (so failure can never leave the dir without a `SKILL.md`).
|
||||
```python
|
||||
if getattr(message, "stop_reason", None) == "max_tokens":
|
||||
print("❌ Response truncated (max_tokens); leaving original SKILL.md intact.")
|
||||
return False
|
||||
```
|
||||
|
||||
## SCAN-01 — Scan AI github-config generation always rejected by the validator `Confirmed · High`
|
||||
**Location:** `cli/scan_command.py:426` (`_GENERATE_SCHEMA_HINT`); validated at `scan_command.py:555` via `UniSkillConfigValidator(data).validate()`; rejection at `cli/config_validator.py:266-272` against `VALID_DEPTH_LEVELS` (`config_validator.py:71`).
|
||||
**Trigger:** Any unmapped detection that falls through to `generate_config_with_ai`. The hint sits inside the `"type": "github"` source block, and `"standard"` is a plain literal, so a prompt-compliant model copies it verbatim.
|
||||
**Mechanism:** The schema hint tells the AI to emit `"code_analysis_depth": "standard"`, but the validator's github-source branch only accepts `{surface, deep, full}`. `generate_config_with_ai` validates AI output and rejects on raise; the retry reuses the same hint, so attempt 2 reproduces the same invalid value. After `max_attempts` the detection lands in `result.failed`.
|
||||
**Impact:** The headline `scan` feature — "generate a config for an unmapped framework" — systematically produces nothing for any library with a github source. Two wasted AI calls per detection, zero output.
|
||||
```python
|
||||
"code_analysis_depth": "standard", # scan_command.py:426 — not a valid level
|
||||
# config_validator.py:71
|
||||
VALID_DEPTH_LEVELS = {"surface", "deep", "full"}
|
||||
```
|
||||
**Fix:** `"code_analysis_depth": "deep",` (or `"surface"`).
|
||||
|
||||
## CBB-01 — LOCAL-mode guide enhancement calls a non-existent method `Confirmed · High`
|
||||
**Location:** `cli/guide_enhancer.py:307`.
|
||||
**Trigger:** `build-how-to-guides` runs with AI enhancement when no API key is set (`auto` → `AgentClient.mode == "local"`) or `ai_mode="local"`.
|
||||
**Mechanism:** `_enhance_via_local` calls `self._call_claude_local(prompt)`, which is defined nowhere (only `_call_ai` at :263 exists; every other call site uses it). The `AttributeError` is swallowed by the broad `except Exception` in `enhance_guide` (:106-109), which logs and returns the original `guide_data`.
|
||||
**Impact:** Every LOCAL-mode (no-API-key) guide build silently returns **unenhanced** guides while logging a generic "AI enhancement failed." Local-mode guide enhancement has never worked.
|
||||
```python
|
||||
def _enhance_via_local(self, guide_data: dict) -> dict:
|
||||
prompt = self._create_enhancement_prompt(guide_data)
|
||||
response = self._call_claude_local(prompt) # NameError: no such method
|
||||
```
|
||||
**Fix:** `response = self._call_ai(prompt)`.
|
||||
|
||||
## MCP-01 — `push_config` strips its own auth token → broken for private repos `Confirmed · High`
|
||||
**Location:** `mcp/config_publisher.py:170,210` (compounded by ★MCP-12 / ★MCP-13).
|
||||
**Trigger:** Any `push_config` to a private/authenticated source repo.
|
||||
**Mechanism:** The clone uses a token-injected URL, but line 170 immediately runs `origin.set_url(git_url)` (tokenless) "to scrub the token," and the push at line 210 goes through that tokenless `origin`. `marketplace_publisher.py:160` does it correctly — it pushes to a freshly token-injected URL.
|
||||
**Impact:** The push fails with an auth error *after* the commit, leaving the cache repo with an orphan commit. `push_config` to the primary (private) config source is effectively non-functional. ★MCP-12 and ★MCP-13 (below) make the second run fail at pull and leave the cache on a stranded branch — together they fully break the feature and corrupt the cache.
|
||||
```python
|
||||
repo_obj.remotes.origin.set_url(git_url) # :170 scrubs token from origin
|
||||
...
|
||||
repo.remotes.origin.push(target_branch) # :210 pushes via tokenless origin
|
||||
```
|
||||
**Fix:** `repo.git.push(self.git_repo.inject_token(git_url, token), target_branch)`.
|
||||
|
||||
## CBA-13 ★ — GDScript class dicts use key `"bases"`; consumer reads `"base_classes"` `Confirmed · High`
|
||||
**Location:** `cli/code_analyzer.py:1986` vs `cli/pattern_recognizer.py:367`.
|
||||
**Trigger:** Any GDScript file with `class_name Foo extends Bar` run through `PatternRecognizer`.
|
||||
**Mechanism:** `_analyze_gdscript` emits class dicts with the key `"bases"` — the only one of 11 analyzers to do so (the other 10 use `"base_classes"`). `_convert_to_signatures` reads `cls.get("base_classes", [])`, so for GDScript the base list is always `[]`.
|
||||
**Impact:** All inheritance-based pattern detection (Strategy/Template-Method `subclasses`, Observer/Chain families — every `name in cls.base_classes` check) is silently dead for GDScript, a first-class target (C3.10 signal-flow pipeline).
|
||||
```python
|
||||
classes.append({
|
||||
"name": class_name,
|
||||
"bases": [extends] if extends else [], # wrong key
|
||||
...
|
||||
})
|
||||
```
|
||||
**Fix:** rename to `"base_classes"`.
|
||||
|
||||
---
|
||||
|
||||
# High
|
||||
|
||||
## SCAN-02 — Non-numeric AI `confidence` crashes the whole scan `Confirmed · High`
|
||||
**Location:** `cli/scan_command.py:394`; propagates through `run_scan` → `_amain` (only catches `KeyboardInterrupt`/`RuntimeError`).
|
||||
**Trigger:** AI returns `"confidence": "high"` / a list / explicit `null` (LLMs do this despite the "0-1" instruction). Missing key is safe (defaults to `0.0`).
|
||||
**Mechanism:** `float(entry.get("confidence", 0.0))` is inside the per-entry loop, which is **not** wrapped in try/except (only `client.call` is). `float("high")` raises `ValueError`; it unwinds out of `asyncio.run`.
|
||||
**Impact:** One malformed entry aborts the entire command with a traceback, discarding all valid detections.
|
||||
```python
|
||||
confidence = float(entry.get("confidence", 0.0)) # unguarded
|
||||
```
|
||||
**Fix:** wrap in `try/except (TypeError, ValueError): continue`.
|
||||
|
||||
## CLI-01 — `--no-issues/--no-changelog/--no-releases` are dead on `create` `Confirmed · High`
|
||||
**Location:** `cli/create_command.py:270-283`; dests in `arguments/create.py:298-318` & `arguments/github.py:49-69`; consumer `github_scraper.py:257-266`.
|
||||
**Trigger:** `skill-seekers create owner/repo --no-issues` (or `--no-changelog`/`--no-releases`).
|
||||
**Mechanism:** The parser defines the **negative** dests `no_issues/no_changelog/no_releases`, but `_build_config` reads the never-defined positive keys `include_issues/include_changelog/include_releases` via `getattr(..., True)`, so they're always `True`. No `no_*→include_*` translation exists anywhere. `include_code` has no flag at all.
|
||||
**Impact:** `--no-issues` still fetches every issue (potential rate-limit exhaustion); the three opt-out flags are no-ops on the only entry point that exists.
|
||||
```python
|
||||
"include_issues": getattr(self.args, "include_issues", True), # attr never exists
|
||||
```
|
||||
**Fix:** `"include_issues": not getattr(self.args, "no_issues", False)` (and the others).
|
||||
|
||||
## CLI-02 — `--output` ignored for all non-local sources; enhancement targets the wrong dir `Confirmed · High`
|
||||
**Location:** `cli/create_command.py:215-430` (`_build_config`) & `:450-457` (`_run_enhancement`); scrapers `doc_scraper.py:209`, `notion_scraper.py:99`, `pdf_scraper.py:79`, `unified_scraper.py:120`.
|
||||
**Trigger:** `create <web/github/pdf/... source> --output /tmp/x` (anything but `output/{name}`), especially with `--enhance-level > 0`.
|
||||
**Mechanism:** `_build_config` only forwards `output_dir` for the `local` branch; the other scrapers hardcode `self.skill_dir = f"output/{name}"`. But `_run_enhancement` computes `skill_dir = ctx.output.output_dir`, which *does* honor `--output`.
|
||||
**Impact:** Output always lands in `output/{name}` (flag silently ignored); enhancement then runs against the empty `--output` dir and fails — and since `_run_enhancement` swallows exceptions (★CLI-06), the run still exits 0 with an un-enhanced skill in the wrong place.
|
||||
**Fix:** have every converter read `config.get("output_dir") or f"output/{self.name}"`, and set `config["output_dir"]` in every `_build_config` branch.
|
||||
|
||||
## CBA-01 — Singleton precedence bug bypasses the confidence gate `Confirmed · High`
|
||||
**Location:** `cli/pattern_recognizer.py:453`.
|
||||
**Mechanism:** `if has_instance_method or has_init_control and confidence >= 0.5:` parses as `has_instance_method or (...)`. Any class with a method named `instance`/`getInstance` is flagged Singleton at confidence 0.4, bypassing the 0.5 gate.
|
||||
**Impact:** False-positive Singletons for any class exposing an `instance`-style accessor (e.g. ORM models).
|
||||
**Fix:** `if (has_instance_method or has_init_control) and confidence >= 0.5:`.
|
||||
|
||||
## CBA-02 — `total_emissions` iterates `.items()` → always 2× signal count `Confirmed · High`
|
||||
**Location:** `cli/signal_flow_analyzer.py:172` (line 171 correctly uses `.values()`).
|
||||
**Mechanism:** `sum(len(emits) for emits in self.signal_emissions.items())` — each `emits` is a `(key, list)` 2-tuple, so `len` is always 2. Result = `2 × (#emitting signals)`, decoupled from real emission count.
|
||||
**Impact:** "Total Emissions" in every Godot signal-flow report (`signal_flow.json`, `signal_reference.md`) is meaningless.
|
||||
**Fix:** `.values()`.
|
||||
|
||||
## CBA-03 — JS/TS class body truncated at first `}` `Confirmed · High`
|
||||
**Location:** `cli/code_analyzer.py:301`.
|
||||
**Mechanism:** `class_block_end = content.find("}", class_block_start)` finds the **first** brace, not the matching one (C#/Java/Kotlin brace-count; JS doesn't). For any real JS class, the body ends at the first method's closing brace.
|
||||
**Impact:** All methods after the first are dropped from JS/TS classes, breaking method-count pattern heuristics (Builder/Observer/Strategy). (Fixing this surfaces ★CBA-14.)
|
||||
**Fix:** use the brace-counting loop already used in `_analyze_csharp`/`_analyze_java`.
|
||||
|
||||
## CBA-04 — Subclass detection misses generic/qualified bases `Confirmed · High`
|
||||
**Location:** `cli/pattern_recognizer.py:819` (also 805/926/1297/1324/1476).
|
||||
**Mechanism:** `class_sig.name in cls.base_classes` is exact-match, but C# bases are stored verbatim (`BaseStrategy<Foo>`), so `"BaseStrategy" in ["BaseStrategy<Foo>"]` is False. (Java strips generics, so Java works; C#/qualified `Namespace.Base` fail.)
|
||||
**Impact:** Strategy/Template-Method built on C# generic bases are missed entirely; sibling/family evidence lost.
|
||||
**Fix:** normalize bases before compare: `b.split("<",1)[0].split(".")[-1].strip()`.
|
||||
|
||||
## CBB-02 — Verification points overwritten / misattributed across steps `Confirmed · High`
|
||||
**Location:** `cli/how_to_guide_builder.py:185-187`.
|
||||
**Mechanism:** `_extract_steps_python` (:222-226) already pairs each step with its following assertion. Then `analyze_workflow` does `step.verification = verifications[i]` positionally, but `steps` (non-assert statements) and `verifications` (assert lines) are different filtered lists with divergent lengths/order.
|
||||
**Impact:** Guides show verification code under the wrong step (e.g. step 1 shows a check that validates step 4's result), and the correct pairing is thrown away.
|
||||
**Fix:** remove the positional overwrite loop; rely on the per-step pairing already done.
|
||||
|
||||
## CBB-03 — `common_problems.remove()` mutates shared insights `Confirmed · High`
|
||||
**Location:** `cli/generate_router.py:421,436` (readers at :865, :1022).
|
||||
**Mechanism:** `common_problems = self.github_issues.get("common_problems", [])` is the shared list (no copy); `.remove(issue)` deletes from it in place for up to 3 skills, **before** the "Common Issues" section and the issues-reference file read it.
|
||||
**Impact:** Up to 3 of the most label-relevant issues vanish from the summary and `github_issues.md`; output is order-dependent and non-idempotent.
|
||||
**Fix:** iterate `list(...)` copy and track used issues by id; never `.remove()` the shared list.
|
||||
|
||||
## CBB-12 ★ — `dependencies=imports` shares one mutable list across all examples `Corrected · High→Medium (latent)`
|
||||
**Location:** `cli/test_example_extractor.py:394,446,490,532` (list built once at :191).
|
||||
**Mechanism:** The same `imports` list object is assigned as `dependencies` to every `TestExample` from a file, and `how_to_guide_builder._detect_prerequisites` aliases it again into `metadata["required_imports"]`. Aliasing is real, **but** all current consumers only read the list — none mutate it — so there is no observable corruption today.
|
||||
**Impact:** Latent: any future `.append`/`.sort` on one example's `dependencies` would silently affect all siblings (and the guide prerequisites). Reclassified **Medium** (latent) after verification.
|
||||
**Fix:** `dependencies=list(imports)` at each call site.
|
||||
|
||||
## MCP-02 — `install_skill` package-path regex never matches real output `Confirmed · High`
|
||||
**Location:** `mcp/tools/packaging_tools.py:602` vs `cli/package_skill.py:177`.
|
||||
**Mechanism:** The regex matches `"saved to:"`, but `package_skill.py` prints `"Output: <path>"` and `"✅ Package created: <path>"`. The match always fails, so `zip_path` falls back to a constructed name (`{name}.zip`) that differs from real adaptor names (`{dir}-gemini.tar.gz`, etc.).
|
||||
**Impact:** The upload phase receives a guessed/non-existent path while the workflow reports success.
|
||||
**Fix:** match what is actually printed: `r"(?im)^\s*(?:Output|✅ Package created):\s*(.+\.(?:zip|tar\.gz))\s*$"` (better: return the path structurally).
|
||||
|
||||
## MCP-03 — `scrape_docs` drops `skip_scrape` entirely and `dry_run` for unified configs `Confirmed · High`
|
||||
**Location:** `mcp/tools/scraping_tools.py:144-216` (wrapper `server_fastmcp.py:353-381`).
|
||||
**Mechanism:** The FastMCP wrapper declares `skip_scrape`, `dry_run`, `enhance_local` and forwards them, but the impl never reads `skip_scrape`/`enhance_local`, and only sets `converter.dry_run` in the non-unified branch. `server_legacy.py` honored all three.
|
||||
**Impact:** `skip_scrape=True` re-scrapes from the network (20-45 min wasted, ignores cache); `dry_run=True` performs a full scrape for unified configs. Real server_fastmcp ↔ server_legacy drift.
|
||||
**Fix:** read `skip_scrape`, set `converter.dry_run = dry_run` and `converter.skip_scrape = skip_scrape` in both branches.
|
||||
|
||||
## MCP-04 — `submit_config` has no idempotency guard `Confirmed · High`
|
||||
**Location:** `mcp/tools/source_tools.py:529-574`.
|
||||
**Mechanism:** Calls `repo.create_issue(title=f"[CONFIG] {name}", ...)` unconditionally; unlike scan's `maybe_publish` (`_find_existing_issue`), it never searches for an existing open issue.
|
||||
**Impact:** Re-runs create duplicate `[CONFIG] {name}` issues, polluting the review queue.
|
||||
**Fix:** query `is:issue is:open in:title "{name}"` before creating; return the existing URL if found.
|
||||
|
||||
## MCP-05 — `install_skill` infers scrape failure from `"❌"` substring `Confirmed · Medium-High`
|
||||
**Location:** `mcp/tools/packaging_tools.py:523`.
|
||||
**Mechanism:** `if "❌" in scrape_output:` discards the structured returncode that `_run_converter` already provides.
|
||||
**Impact:** A single skipped page logging `❌` aborts the workflow; a real failure without that emoji proceeds to package an empty skill and reports success. (★MCP-15 is the identical issue for the packaging phase, which has no check at all.)
|
||||
**Fix:** return a status flag from `scrape_docs_tool`/`_run_converter` and branch on it.
|
||||
|
||||
## ENH-02 — Direct Moonshot/Kimi key misrouted to OpenAI `Confirmed · High`
|
||||
**Location:** `cli/agent_client.py:202-217`.
|
||||
**Mechanism:** `_detect_provider_from_key` returns `"moonshot"` for an `sk-` key only if `MOONSHOT_API_KEY` env equals it; otherwise any `sk-` key → `"openai"`. Moonshot keys are `sk-`-prefixed, so a key passed via `--api-key` without the env var builds an `OpenAI` client with `gpt-4o`.
|
||||
**Impact:** Direct Moonshot-key usage fails auth (hits `api.openai.com`) or routes to the wrong provider, surfacing as a misleading OpenAI error.
|
||||
**Fix:** accept an explicit provider hint (e.g. `SKILL_SEEKER_PROVIDER`) before defaulting to OpenAI.
|
||||
|
||||
## ENH-03 — Headless "success" requires the file to *grow* `Confirmed · High`
|
||||
**Location:** `cli/enhance_skill_local.py:900` (headless), `:1087` (background), `:1213` (daemon).
|
||||
**Mechanism:** Headless gates success on `new_mtime > initial_mtime and new_size > initial_size`; background/daemon gate only on `returncode == 0`. The three modes disagree.
|
||||
**Impact:** A legitimate condensing/same-size rewrite is reported as failure in the default headless mode ("SKILL.md was not updated"), so CI/automation treats valid runs as errors.
|
||||
**Fix:** gate on `new_mtime > initial_mtime` (file changed), not strictly larger size; share one predicate across modes.
|
||||
|
||||
## ENH-04 — Kimi stdout parser drops all multi-line responses `Confirmed · High`
|
||||
**Location:** `cli/agent_client.py:467`.
|
||||
**Mechanism:** `re.findall(r"TextPart\(type='text', text='(.+?)'\)", raw_output)` has no `re.DOTALL`, so `.` doesn't match newlines; every multi-line SKILL.md matches nothing and the function returns the raw `TurnBegin(...)/StepBegin(...)` debug dump. The non-greedy `'` also splits on embedded apostrophes.
|
||||
**Impact:** With `--agent kimi`, SKILL.md is replaced by Kimi's raw debug log (or fragmented at apostrophes).
|
||||
**Fix:** add `re.DOTALL` and anchor the terminator to the next record boundary.
|
||||
|
||||
## ENH-05 — Stray `.json` in agent cwd returned as the LLM response `Confirmed · High (conditional)`
|
||||
**Location:** `cli/agent_client.py:385-393, 419-435` *(downgraded from Critical — conditional trigger)*.
|
||||
**Mechanism:** When `output_file` is None, the prompt is `prompt.md` and no "write response" instruction is added, yet the code returns `temp/response.json` if present, else **any** `*.json` in the temp dir. The guard `!= "prompt.json"` is dead (the file is `prompt.md`). The subprocess cwd is `temp_path`, so any incidental `.json` the agent writes becomes the "response," shadowing stdout.
|
||||
**Impact:** Non-deterministic: enhancement can return an unrelated JSON file's contents instead of the model's actual output. Conditional on the agent writing a stray `.json`, hence High not Critical.
|
||||
**Fix:** only consult the response file when `output_file` was explicitly requested (and match exactly that file); otherwise return stdout.
|
||||
|
||||
---
|
||||
|
||||
# Medium
|
||||
|
||||
## Scan + config
|
||||
- **SCAN-03 — `get_api_key("moonshot")` env fallback dead.** `config_manager.py:314` — `env_map` omits `moonshot`, so a user who exports `MOONSHOT_API_KEY` (no config-file key) gets `None`, while `config_command.py:329` shows "(from environment)". `set_api_key`'s error text also omits moonshot. *Fix:* add `"moonshot": "MOONSHOT_API_KEY"`. `Confirmed`
|
||||
- **★SCAN-04 — `--dry-run` writes `./configs/` and hits the network.** `scan_command.py:1162` calls `resolve_config_path(lookup, auto_fetch=allow_network)` (default True) in the dry-run branch; on an API hit `fetch_config_from_api` performs HTTP GETs and writes `./configs/<name>.json`, contradicting the "DRY RUN — no files written" promise (`:1287`). *Fix:* pass `auto_fetch=False` in dry-run. `Confirmed`
|
||||
|
||||
## CLI core
|
||||
- **CLI-03 — `ExecutionContext.reset()` never called in production.** `execution_context.py:207-228` — the singleton guard returns the existing instance; `reset()` exists but only tests call it. A 2nd in-process `create` reuses the 1st's name/output/settings. Latent today (MCP runs scrapes in subprocesses; CLI is one-shot). *Fix:* call `reset()` at the top of `CreateCommand.execute()`. `Confirmed`
|
||||
- **★CLI-04 — Legacy `_reconstruct_argv` round-trips the full namespace.** `main.py:179-186` re-serializes every namespace attribute to argv for ~14 legacy commands; attributes unknown to a sub-parser are emitted as `--unknown-flag value`, which the sub-parser can reject. Brittle; the documented migration to `COMMAND_CLASSES` is the fix. `Confirmed`
|
||||
|
||||
## Adaptors
|
||||
- **ADP-01 — `--streaming` is dead for every adaptor.** `streaming_adaptor.py:21` + `package_skill.py:149` — `StreamingAdaptorMixin` is never imported or inherited by any registered adaptor, so `hasattr(adaptor,"package_streaming")` is always False. *Corrected:* the fallback **announces** "Streaming not supported… using standard packaging" (not silent), but the feature is fully dead. *Fix:* have RAG adaptors inherit the mixin or register the streaming variants. `Corrected`
|
||||
- **ADP-02 — `except (ImportError, Exception)` masks all import errors.** `chroma.py:236`, `pinecone_adaptor.py:328` — `Exception` subsumes `ImportError`, so a broken-but-installed package (e.g. chromadb/py3.14 pydantic-v1) is misreported as "not installed." *Fix:* split the handlers. `Confirmed`
|
||||
|
||||
## MCP
|
||||
- **MCP-06 — `list_configs` reads top-level `base_url`; unified configs store it under `sources[0]`.** `tools/config_tools.py:136` — every config `generate_config` produces lists a blank URL. *Fix:* fall back to `sources[].base_url`. `Confirmed`
|
||||
- **★MCP-12 — `config_publisher` `get_source` raises `KeyError`; the `if not source:` branch is dead.** `config_publisher.py:140-143` — an unknown source name raises a raw `KeyError` (quoted message) instead of the curated "Available sources" error. *Fix:* wrap in `try/except KeyError`. `Confirmed`
|
||||
- **★MCP-13 — `config_publisher` tokenless re-pull on the 2nd run.** `config_publisher.py:161-172` — on a cached repo it pulls via `origin` (scrubbed tokenless by the prior run's :170) before re-injecting the token, so a 2nd push to a private repo fails at pull. Compounds MCP-01. *Fix:* `origin.set_url(clone_url)` before pulling. `Confirmed`
|
||||
|
||||
## Codebase analysis (AST / pattern / dependency)
|
||||
- **CBA-05 — "protocol" boosts ANY pattern.** `pattern_recognizer.py:1538-1542` — `type=="Strategy" and "duck typing" in s or "protocol" in s` precedence → the `"protocol"` clause fires for any Python pattern. *Fix:* parenthesize the OR group. `Confirmed`
|
||||
- **CBA-06 — "event" boosts ANY pattern (+ false "EventEmitter detected").** `pattern_recognizer.py:1559-1563` — same precedence flaw for JS/TS. `Confirmed`
|
||||
- **CBA-07 — `_resolve_import` never maps dotted module → file path.** `dependency_analyzer.py:784-814` — `file_nodes` keys are file paths (`src/pkg/mod.py`) while `imported_module` is `pkg.mod`; no dot→slash conversion. The dependency graph is essentially edgeless and cycle detection returns empty for **all dotted-import languages** (Python/Java/C#/Kotlin/Go/Rust/Ruby/PHP). *Corrected:* Godot resource graphs (slash paths) **do** resolve. *Fix:* convert dots to slashes + suffix-match against `file_nodes`. `Confirmed`
|
||||
- **CBA-08 — Kotlin top-level-fn `indent > 4` heuristic.** `code_analyzer.py:1379-1383` — tab/2-space-indented methods are misclassified as top-level, and indented top-level fns are dropped. *Fix:* track brace depth instead of indentation. `Confirmed`
|
||||
- **CBA-09 — Kotlin `is_suspend` look-behind reads before the modifier.** `code_analyzer.py:1385` — the 50-char window precedes the match (which already includes `suspend`), so real `suspend fun` reads False and a neighboring `suspend` reads True. *Fix:* `"suspend" in match.group(0)`. `Confirmed`
|
||||
- **CBA-10 — `gd_resource` regex drops `script_class` on compact headers.** `code_analyzer.py:1875` — trailing `\s+` can't match `]`, so `[gd_resource type="X" script_class="Y"]` yields `script_class=None`. *Fix:* end with `\s*[\]\s]`. `Confirmed`
|
||||
- **★CBA-14 — JS method extractor matches call-sites and keywords.** `code_analyzer.py:391-419` — `(\w+)\s*\(...\)` matches any `ident(args)`; only 4 keywords are blocklisted, so `this.helper(b)`, `setTimeout(...)` become "methods." Masked today by CBA-03 (truncation); surfaces once CBA-03 is fixed. *Fix:* anchor to `\s*\{` declarations and exclude the full JS keyword set. `Confirmed`
|
||||
- **★CBA-15 — GDScript signal doc comment read from the wrong line.** `code_analyzer.py:2043-2047` — for a signal at column 0, `content[:start].split("\n")[-1]` is the empty pre-`signal` text; the doc comment is `lines[-2]`. So `## doc \n signal x` always yields `documentation=None`. *Fix:* use `lines[-2]`. `Confirmed`
|
||||
|
||||
## Codebase analysis (example / guide / router)
|
||||
- **CBB-04 — `ast.walk` flattens nested statements.** `how_to_guide_builder.py:204-206` — statements inside `if`/`for`/`with`/`try` are hoisted into the flat step list, losing control-flow context. *Fix:* iterate top-level body only. `Confirmed`
|
||||
- **CBB-05 — AI `ai_analysis` aligned by position, not `example_id`.** Real site `ai_enhancer.py:360-362` (the extractor re-merge at `test_example_extractor.py:1088` is actually order-safe). If the model drops/reorders an entry, every later example gets another's analysis (and `tutorial_group`, which drives guide grouping). *Fix:* key by `example_id`. `Confirmed`
|
||||
- **CBB-06 — `_is_test_class` `"Test" in base.id` over-matches.** `test_example_extractor.py:218-225` — any base whose name *contains* "Test" (`LatestConfig`, `TestableMixin`) flags the class as a test. *Fix:* `endswith("TestCase")` / explicit bases. `Confirmed`
|
||||
- **CBB-07 — Index TOC sorts complexity alphabetically.** `how_to_guide_builder.py:789` — `sorted(..., key=g.complexity_level)` → `advanced, beginner, intermediate`. *Fix:* explicit rank map. `Confirmed`
|
||||
- **CBB-08 — c3x failures look like empty results; temp dir leaks.** `unified_codebase_analyzer.py:288-305` — broad `except` returns `analysis_type:"c3x"` with empty arrays + an easily-ignored `error` key; `tempfile.mkdtemp` (:255) is never cleaned. *Fix:* mark a distinct failure type and `shutil.rmtree` in `finally`. `Confirmed`
|
||||
- **★CBB-13 — Guide step-description index trust.** `guide_enhancer.py:552-560` + `how_to_guide_builder.py:1050-1055` — step explanations routed by AI-supplied `step_index` with an enumerate fallback; a dropped entry shifts every subsequent explanation onto the wrong step. *Fix:* require an explicit in-range `step_index`; skip otherwise. `Confirmed`
|
||||
|
||||
## Document scrapers
|
||||
- **DOC-02 — `smart_categorize` assigns to first category over threshold, not max.** `doc_scraper.py:1837-1840` — `break`s on the first category reaching score 2 (config-order dependent); siblings use `max(scores,...)`. *Corrected severity High→Medium.* *Fix:* collect all scores, assign to `max`. `Confirmed`
|
||||
- **DOC-03 — Link extraction keeps query strings.** `doc_scraper.py:432-437` — only `#fragment` is stripped, so `?lang=`/`?utm=` variants are distinct pages (crawler-trap risk in unlimited mode). *Fix:* drop tracking params / normalize query before dedup. `Confirmed`
|
||||
- **DOC-04 — `DEFAULT_MAX_PAGES = -1` ⇒ unbounded default crawl.** `doc_scraper.py:1443-1450`; `defaults.json:7` — a config without `max_pages` enters unlimited mode. *Fix:* finite default (e.g. 1000); require explicit `-1` for unlimited. `Confirmed (by-design footgun)`
|
||||
- **DOC-05 — PDF cross-page code merge heuristic is ~always true; count drift.** `pdf_extractor_poc.py:606-623` — `any([not endswith("}"), not endswith(";"), ...])` merges unrelated adjacent blocks; `code_blocks_count` decremented on next page but not incremented on current. *Fix:* require a real continuation token; recompute counts from `len(code_samples)`. `Confirmed`
|
||||
- **DOC-06 — Async unlimited dry-run doesn't cap at 20.** `doc_scraper.py:1613-1620` — sets `preview_limit=inf` before checking `dry_run` (sync caps at 20 unconditionally), so async `--dry-run` on a default config previews the entire site. *Fix:* set `preview_limit=20` when `dry_run` first. `Confirmed`
|
||||
- **DOC-07 — Broken SKILL.md nav links (pdf/html/word/epub).** `pdf_scraper.py:530`, `html_scraper.py:1562`, `word_scraper.py:631`, `epub_scraper.py:800` — nav links to `references/{sanitize(title)}.md` but files are written with range/basename names (`_pX-pY`, `_sX-sY`, raw basename). Dead links for any multi-section/multi-category source. (Same root cause as MED-04 — see systemic pattern S2.) *Fix:* derive nav names from the same filename helper that writes the files. `Confirmed`
|
||||
|
||||
## Media / structured / remote scrapers
|
||||
- **MED-01 — GitHub issues capped at 100 (no Link pagination).** `github_fetcher.py:299-345` — single page, `per_page=min(max,100)`, no `rel="next"`; PR filtering reduces further. *Corrected severity High→Medium.* *Fix:* loop on the `next` link. `Confirmed`
|
||||
- **MED-02 — Slack 429 aborts all channels.** `chat_scraper.py:648-722` — `conversations_history` has no 429/retry, and the per-channel loop is in one try whose `except SlackApiError` raises a fatal `RuntimeError`, losing already-fetched channels. *Fix:* per-channel try/except + `Retry-After` backoff. `Confirmed`
|
||||
- **MED-03 — Discord session has no timeout; 429 silently breaks.** `chat_scraper.py:827-878` — no `ClientTimeout`; a 429 is treated as a generic non-200 → silent `break`. *Fix:* add timeout + honor `Retry-After`. `Confirmed`
|
||||
- **MED-04 — Broken SKILL.md nav links (jupyter/asciidoc/pptx).** `jupyter_scraper.py:966`, `asciidoc_scraper.py:841`, `pptx_scraper.py:1467` — same nav vs file naming mismatch as DOC-07. *Verified:* Confluence/RSS are consistent (not affected). *Fix:* shared filename helper. `Confirmed`
|
||||
|
||||
## Enhancement + unified builder
|
||||
- **ENH-06 — `_call_api` ignores the caller's `timeout`.** `agent_client.py:295-309` — `call()` computes a timeout (default 45 min) but `_call_api` hardcodes `timeout=120`. Large prompts fail at 2 min with a misleading "connection error." *Fix:* thread `timeout` through. `Confirmed`
|
||||
- **ENH-07 — `scrape_all_sources` reports the bucket count, not sources scraped.** `unified_scraper.py:256` — `len(self.scraped_data)` is always ~17. *Fix:* `sum(len(v) for v in ...)`. `Confirmed`
|
||||
- **ENH-08 — All-sources-fail still builds and returns 0.** `unified_scraper.py:252-254` — per-source exceptions swallowed; `run()` proceeds to `build_skill` on empty data and exits 0. *Fix:* count scraped sources; return non-zero on zero. `Confirmed`
|
||||
- **ENH-09 — "Official Documentation" reads `sources[0]` regardless of type.** `unified_skill_builder.py:280` — prints `N/A` (or a non-docs URL) if github is listed first. *Fix:* filter `type == "documentation"`. `Confirmed`
|
||||
- **ENH-10 — `DEBUG:` log lines left in production synthesis.** `unified_skill_builder.py:327-332` — `logger.info("DEBUG: ...")` and `logger.warning("DEBUG: ... NOT FOUND!")` fire on every docs+github synthesis. *Fix:* `logger.debug`, drop the alarm wording. `Confirmed`
|
||||
- **★ENH-11 — Gemini API mode ignores `timeout` and `max_tokens`.** `agent_client.py:312-315` — `generate_content(prompt)` with no `generation_config`/timeout; output capped at the model default, request unbounded. With ENH-01 the truncation is doubly invisible. *Fix:* pass `max_output_tokens`/`request_options` and check `finish_reason`. `Confirmed`
|
||||
- **★ENH-12 — Provider-priority lists are inconsistent; Moonshot dropped in `_pick_mode`.** `agent_client.py:84-90` honors `MOONSHOT_API_KEY`, but `enhance_command._pick_mode` and `enhance_skill_local._detect_api_target` only check claude/gemini/openai. A Moonshot-only user is silently dropped to LOCAL mode by the `enhance` command. *Fix:* add a moonshot/kimi branch; share one priority order. `Confirmed`
|
||||
|
||||
## Args / infra
|
||||
- **INF-01 — Embedding cache key omits `normalize`.** `embedding/generator.py:414-418`; `server.py:138,182,209` — cache keyed `(model,text)` only; a `normalize=False` request after a cached `normalize=True` returns the wrong (normalized) vector. *Fix:* include `int(normalize)` in the key. `Confirmed`
|
||||
- **INF-02 — `cleanup_old` folds the date into the group key.** `benchmark/runner.py:293-295` — filename is `{name}_{%Y%m%d_%H%M%S}`; `"_".join(parts[:-1])` leaves `{name}_{date}`, so retention is per-name-per-day → old files never pruned. *Fix:* `parts[:-2]`. `Confirmed`
|
||||
- **INF-03 — `peak_mb` never samples during the op.** `benchmark/framework.py:184-192` — only `max(before, after)`; transient spikes invisible, so leak/memory recommendations under-report. *Fix:* sample RSS on a background thread during `yield`. `Confirmed`
|
||||
- **INF-04 — `compare()` divides by zero on instant ops.** `benchmark/runner.py:160,195` — no guard on `current.duration == 0`. *Fix:* guard with `float("inf")`. `Confirmed`
|
||||
- **INF-05 — Header change-detection misses validator-less servers.** `sync/detector.py:270-277` — returns `False` (unchanged) when neither `Last-Modified` nor `ETag` is present, so `batch_check_headers` silently drops genuinely-changed pages (the `except` branch returns `True`, inconsistently). *Fix:* return `True` when no validators are available. `Confirmed`
|
||||
|
||||
---
|
||||
|
||||
# Low
|
||||
|
||||
## Scan + config
|
||||
- **SCAN-05 — API-fetched configs written to `./configs/` regardless of `--out`.** `config_fetcher.py:196` — hardcoded `destination="configs"` pollutes CWD (scan re-writes to out_dir anyway). *Fix:* thread the out dir / temp dir through. `Confirmed`
|
||||
- **SCAN-06 — Module-level `logging.basicConfig` in a library.** `config_validator.py:33` — can win over scan's `--verbose` depending on import order. *Fix:* remove from the library module. `Confirmed`
|
||||
- **SCAN-07 — Duplicate-slug detections double-count.** `scan_command.py:1149-1199` — the diff dedups by slug, but the main loop double-writes/double-counts duplicate-slug detections. *Fix:* dedup detections by `_config_filename_for`. `Confirmed`
|
||||
|
||||
## CLI core
|
||||
- **CLI-05 — `run()` success log prints `output/{name}` for local builds.** `skill_converter.py:46` + `codebase_scraper.py:2154` — artifacts go to `output_dir` (honors `--output`) but the base log shows `skill_dir`. *Fix:* set `self.skill_dir = str(self.output_dir)`. `Confirmed`
|
||||
- **CLI-06 — confluence/notion `max_pages` dead `DEFAULTS` fallback.** `create_command.py:404,415` — `getattr(..., DEFAULTS[...])` fallback is unreachable (`max_pages` always present as `None`); harmless because `None` is treated as unlimited like `-1`. *Corrected (no behavioral impact).* `Corrected`
|
||||
- **★CLI-07 — `_run_enhancement`/`_run_workflows` swallow all exceptions; exit 0.** `create_command.py:504-516` — enhancement/workflow failure never affects the exit code (CI can't detect it). Compounds CLI-02. *Fix:* track success, return non-zero or emit a clear failure summary. `Confirmed`
|
||||
- **★CLI-08 — `merge_sources(mode="ai-enhanced")` not recognized.** `merge_sources.py:766-769` — only `"claude-enhanced"` is special-cased, so the canonical `"ai-enhanced"` falls through to rule-based merging for direct callers. *Fix:* accept both spellings. `Confirmed`
|
||||
- **★CLI-09 — Module-level `logging.basicConfig` in `merge_sources.py:36`.** Same library-logging hazard as SCAN-06 (imported transitively via `generate_router`). *Fix:* remove. `Confirmed`
|
||||
|
||||
## Adaptors
|
||||
- **ADP-03 — Weaviate adaptor uses v3-only API; pin allows v4.** `weaviate.py:341-379`; pin `weaviate-client>=3.25.0` — `weaviate.Client`, `schema.create_class`, `client.batch` all raise `AttributeError` on v4. *Fix:* cap `<4` or port to the v4 API. `Confirmed`
|
||||
- **ADP-04 — OpenAI upload uses `client.beta.vector_stores`.** `openai.py:260-280`; pin `openai>=1.0.0` — promoted to `client.vector_stores` in newer SDKs; deprecated path may raise. *Fix:* prefer top-level with fallback; pin a known-good range. `Confirmed (Conf: Low)`
|
||||
- **ADP-05 — Unquoted YAML name/description in Claude frontmatter.** `claude.py:76-80` — colon-space / leading-quote in `description` produces invalid YAML (OpenCode/IBM-Bob quote; Claude doesn't). *Fix:* quote/escape the values. `Confirmed`
|
||||
- **ADP-06 — Enhance renames SKILL.md to `.backup` before writing; empty response loses it.** `openai_compatible.py:349-357` (+ openai/gemini/claude) — a successful-but-empty/None response leaves the dir with no `SKILL.md`. *Fix:* validate non-empty before renaming; write-temp-then-swap. `Confirmed`
|
||||
- **ADP-07 — LlamaIndex uses `format="hex"` while base docstring claims uuid5.** `llama_index.py:43` vs `base.py:481` — doc/behavior mismatch only (hex ids are valid). *Fix:* correct the docstring. `Confirmed`
|
||||
- *Verified correct:* the thin OpenAI-compatible subclasses (kimi/deepseek/qwen/openrouter/together/fireworks/minimax) have correct, distinct model ids / endpoints / env vars / registry keys; vector-DB dimensions/metrics are consistent.
|
||||
|
||||
## MCP
|
||||
- **MCP-07 — `config_publisher` commit `action` term is meaningless.** `config_publisher.py:204` — `target_file.exists()` is always True post-copy. *Fix:* capture existence before copy, or use `"update" if force else "add"`. `Confirmed`
|
||||
- **MCP-08 — Three tool params typed `str` but default `None`.** `server_fastmcp.py:1342,1403,1404` — should be `str | None`; schema advertises non-nullable. `Confirmed`
|
||||
- **MCP-09 — `enhance_skill` returns identical shape on success/failure.** `tools/packaging_tools.py:329-344` — callers can't distinguish without substring scanning. *Fix:* structured error signal. `Confirmed`
|
||||
- **MCP-10 — `sync_config` wrapper hardcodes `max_pages=500`.** `server_fastmcp.py:279` vs `tools/sync_config_tools.py:40` — the impl's `DEFAULTS` (-1/unlimited) default is dead; MCP path silently caps at 500. *Fix:* default to `None`, forward only if set. `Confirmed`
|
||||
- **MCP-11 — Unlimited temp path is non-unique and uses `str.replace`.** `tools/scraping_tools.py:169` — `config_path.replace(".json", "_unlimited_temp.json")` collides on concurrency and rewrites any `.json` substring. *Fix:* `tempfile.mkstemp`. `Confirmed`
|
||||
- **★MCP-14 — `config_publisher` left on the feature branch after a mid-flow exception.** `config_publisher.py:198-219` — no try/finally to restore `branch`; a push failure strands the cache on `config/<name>`. *Fix:* restore branch in `finally`. `Confirmed`
|
||||
- **★MCP-15 — `package_skill_tool` returncode swallowed in `install_skill`.** `tools/packaging_tools.py:85-143` — no success check after packaging; a failed package still yields a fabricated `zip_path`. *Fix:* structured success signal; abort on failure. `Confirmed`
|
||||
- **★MCP-16 — `generate_config` silently overwrites an existing config.** `tools/config_tools.py:80-84` — no `exists()`/`force` guard; clobbers hand-edited configs. *Fix:* add a `force` arg and guard. `Confirmed`
|
||||
- **★MCP-17 — `_run_converter` attaches a handler to the shared `skill_seekers` logger.** `tools/scraping_tools.py:41-70` — concurrent converter tools cross-contaminate captured logs. *Fix:* per-call scoped capture. `Confirmed`
|
||||
|
||||
## Codebase analysis
|
||||
- **CBA-11 — Comment extractors match `//` / `/* */` inside string literals.** `code_analyzer.py:581,588` — URLs in strings yield phantom comments. *Fix:* strip string literals first. `Confirmed`
|
||||
- **CBA-12 — Single-letter CSS classes `c`/`r` matched as C/R.** `language_detector.py:688,708` — bare-name branch returns confidence 1.0 for ambiguous tokens. *Fix:* require a prefix for `c`/`r`. `Confirmed`
|
||||
- **★CBA-16 — Go grouped-import line numbers off by one.** `dependency_analyzer.py:495` — uses `match.start()` (at `import`) instead of `match.start(1)` (after `import (\n`). *Fix:* `block_start = match.start(1)`. `Confirmed`
|
||||
|
||||
## Codebase analysis (guides)
|
||||
- **CBB-09 — Duplicate `LANGUAGE_ALIASES`; `c++→cpp` but no `cpp` in `PATTERNS`.** `test_example_extractor.py:731-743` — C++ test files silently yield nothing. *Fix:* remove the dup; add `cpp` patterns or drop the alias. `Confirmed`
|
||||
- **CBB-10 — Space-form `ENV KEY VALUE` dropped; `_infer_purpose` substring-matches.** `config_extractor.py:601-611,350` — legacy ENV skipped; `"db" in "dbeaver"`. *Fix:* handle space form; token-boundary match. `Confirmed`
|
||||
- **CBB-11 — Step code fences hardcoded ```python```.** `how_to_guide_builder.py:650,663,679` — wrong highlighting for non-Python guides despite `guide.language`. *Fix:* `f"```{guide.language}"`. `Confirmed`
|
||||
- **★CBB-14 — Basic-mode maps best_practices onto step explanations by index.** `how_to_guide_builder.py:1098-1101` — unrelated best-practice strings attached to steps positionally. *Fix:* render as a separate list. `Confirmed`
|
||||
- **★CBB-15 — `ast.unparse(node.body)` on a statement list for `setUp`.** `test_example_extractor.py:266-271` — relies on undocumented unparse-of-list behavior. *Fix:* `"\n".join(ast.unparse(s) for s in node.body)`. `Confirmed`
|
||||
- **★CBB-16 — Regex `group(1)` assumption → `Test: None` for Kotlin/GDScript/C#.** `test_example_extractor.py:762-769` — alternation patterns put the name in group 2/3; `group(1)` is None, and the body-slice re-search can land inside the current body. *Fix:* `next((g for g in match.groups() if g), match.group(0))`. `Confirmed`
|
||||
- **★CBB-17 — Redundant triple-fetch of `ai_analysis` with divergent null handling.** `how_to_guide_builder.py:957-961, 995, 1004-1012` — maintenance hazard. *Fix:* fetch once, reuse. `Confirmed`
|
||||
|
||||
## Document scrapers
|
||||
- **DOC-08 — llms.txt sections with identical titles overwrite each other.** `doc_scraper.py:867-875` — same title → same URL → same `{safe_title}_{url_hash}` filename. *Corrected:* narrow (identical-title sections only), not the general case. *Fix:* add a section index/counter. `Corrected`
|
||||
- **DOC-09 — Parallel/async submit guard off-by-one (`<=` vs `<`).** `doc_scraper.py:1523,1647` vs `1457` — parallel mode over-scrapes `max_pages` by ≥1. *Fix:* use `<`. `Confirmed`
|
||||
- **DOC-10 — `is_valid_url` rejects the exact base page.** `doc_scraper.py:301-303` — trailing-slash `base_dir` drops the slashless base URL. *Corrected:* negligible — the start page is enqueued directly via `start_urls`. *Fix:* also accept `url == base_url`. `Corrected (negligible)`
|
||||
- **DOC-11 — EPUB section images written as 0-byte files with broken links.** `epub_scraper.py:984-994` + `598-613` — `data=b""` passes the `isinstance(bytes)` check → 0-byte PNG + `![Image]` link. *Fix:* `if isinstance(...) and len(data) > 0`. `Confirmed`
|
||||
- **DOC-12 — `_extract_html_as_markdown` returns `links=[]` → BFS dead-ends.** `doc_scraper.py:509-510,667-711` — a `.md` URL serving HTML contributes no further links. *Fix:* extract same-prefix links. `Confirmed`
|
||||
- **DOC-13 — `_extract_tables` drops body rows equal to the header.** `html_scraper.py:868-872` (+ shared helper, confluence) — `cells != headers` drops legitimate data rows. *Fix:* skip the first row only when it came from `<thead>`. `Confirmed`
|
||||
- **DOC-14 — Non-atomic checkpoint/page writes corrupt on interrupt.** `doc_scraper.py:334, 874` (the earlier-cited :771 is unrelated) — direct `open(...,'w')`; a second Ctrl-C truncates the checkpoint, and `load_checkpoint` silently starts fresh, losing progress. *Fix:* temp-file + `os.replace`. `Confirmed`
|
||||
- **DOC-15 — `int(img.get("width") or 0)` crashes on `"100%"`/`"50px"`.** `html_scraper.py:911-912` (crash), `word_scraper.py:826` (silently drops image), `epub_scraper.py:991` (crash). *Fix:* parse leading digits defensively. `Confirmed`
|
||||
- **★DOC-16 — Async dry-run discovered links bypass the cap.** `doc_scraper.py:1647-1664` — combined with DOC-06, async `--dry-run` issues real GETs across the whole site. *Fix:* the DOC-06 fix bounds this. `Confirmed`
|
||||
- **★DOC-17 — pdf legacy image branch writes raw `img["data"]` with no guard.** `pdf_scraper.py:341-356` — `b""` → 0-byte file; missing/non-bytes → `KeyError`/`TypeError` aborts the reference-file write. *Fix:* `isinstance`/non-empty guard + `.get`. `Confirmed`
|
||||
- **★DOC-18 — `smart_categorize` output is config-order-dependent.** `doc_scraper.py:1809-1846` — corollary of DOC-02; identical configs differing only in key order produce different groupings. *Fix:* same as DOC-02 (pick max, stable tiebreak). `Confirmed`
|
||||
|
||||
## Media / structured / remote scrapers
|
||||
- **MED-05 — `VideoScraperResult.to_dict()` drops `config`.** `video_models.py:827-848` — `--from-json` reload loses clip range/languages/whisper model. *Corrected:* latent — no current reader of `result.config`. *Fix:* serialize/restore `config`. `Corrected (latent)`
|
||||
- **MED-06 — `.man` files dropped when `--sections` is set.** `man_scraper.py:317-353` — section-less `.man` pages have `section_num=None`, never in the list. *Fix:* only filter when `section_num is not None`. `Confirmed`
|
||||
- **MED-07 — OpenAPI `$ref` has no cycle set.** `openapi_scraper.py:775-855` — self-referential schemas expand to the depth-10 cap (bounded — no hang). *Fix:* track visited ref names; emit a stub on cycle. `Confirmed`
|
||||
- **MED-08 — RSS link-following has no global time budget.** `rss_scraper.py:191-206,484-489` — serial, 15s/req + 1s sleep; ~13 min on a 50-entry feed with slow hosts. *Fix:* time budget or concurrency. `Confirmed`
|
||||
- **MED-09 — Confluence pagination compares against constant `limit`.** `confluence_scraper.py:455-528` — `if len(batch) < limit` uses 50, not the requested `min(...)`. *Corrected:* sloppy but not a reachable premature-stop (outer `while` already bounds it). *Fix:* compare against the requested size. `Corrected`
|
||||
- **MED-10 — OCR keyframe temp JPEGs never deleted.** `video_visual.py:702-715` — `delete=False`, never unlinked. *Corrected:* `extract_keyframes` is dead code (no production caller); the active pipeline writes intended output frames and cleans stale ones. *Fix:* delete the dead function or have it clean up. `Corrected (dead code)`
|
||||
- **MED-11 — Confluence `created` only set for version-1 pages.** `confluence_scraper.py:494-498` — empty for all edited pages. *Fix:* use `history.createdDate`. `Confirmed`
|
||||
- **MED-12 — `get_releases()` / file-tree walk uncapped.** `github_scraper.py:686-708, 939-968` — unlike issues' `islice`, can exhaust the rate-limit quota on large repos. *Fix:* `islice` / depth cap. `Confirmed`
|
||||
- **MED-13 — `max_pages: int` assigned `float("inf")`.** `notion_scraper.py:96-98`, `confluence_scraper.py:241-243` — type-annotation lie; works only because comparisons accept inf. *Fix:* `sys.maxsize` or annotate `float`. `Confirmed`
|
||||
- **MED-14 — Video AI ref cleaning accepts any response >50% of input.** `video_scraper.py:299-312` — no `stop_reason` check; a truncated (>50%) response overwrites the reference file. *Fix:* require `stop_reason == "end_turn"`. `Confirmed`
|
||||
- **★MED-15 — OpenAPI external/relative `$ref` left as an unresolved stub.** `openapi_scraper.py:916-943` — only `#/`-refs resolve; multi-file bundles emit stubs with a debug-only log. *Fix:* pre-bundle, or warn + mark `_ref_unresolved`. `Confirmed`
|
||||
- **★MED-16 — Discord `before = batch[-1]["id"]` `KeyError` on malformed message.** `chat_scraper.py:875` — aborts the whole fetch. *Fix:* `.get("id")` + break. `Confirmed`
|
||||
- **★MED-17 — Slack `conversations_list` is single-page.** `chat_scraper.py:654-662` — no `next_cursor`; >200 channels truncated. *Fix:* paginate via `response_metadata.next_cursor`. `Confirmed`
|
||||
- **★MED-18 — GitHub "all" mode `//2` quota split under-fetches.** `github_fetcher.py:289-297` — rigid open/closed split never reallocates unused quota; magnifies MED-01. *Fix:* fetch open then request `max - len(open)` closed. `Confirmed`
|
||||
|
||||
## Enhancement + builder
|
||||
- **ENH-13 — Unknown provider → `_init_api_client` returns None, mode stays "api".** `agent_client.py:219-254` — `_call_api` then silently returns None (no fallback to LOCAL). *Fix:* `else:` raise or set `self.mode = "local"`. `Confirmed`
|
||||
- **ENH-14 — `quality_metrics --threshold` parsed but never used.** `quality_metrics.py:542` — `main()` always returns 0, so quality gating is a no-op (also a 0-10 vs percentage scale mismatch). *Fix:* compare the score and return non-zero. `Confirmed`
|
||||
- **ENH-15 — `prompt_file.write_text` missing `encoding="utf-8"`.** `agent_client.py:393` — `UnicodeEncodeError` on non-UTF-8 locales for emoji/CJK prompts (every read uses utf-8). *Fix:* add `encoding="utf-8"`. `Confirmed`
|
||||
- **★ENH-16 — `_call_api` error classification by name substring.** `agent_client.py:317-353` — `"auth"/"rate" in type-name` misfires; a 429 whose class lacks "rate" gets a generic message. *Fix:* branch on HTTP status / typed SDK exceptions. `Confirmed (diagnostic)`
|
||||
- **★ENH-17 — PDF content dropped when base SKILL.md lacks a Reference heading.** `unified_skill_builder.py:619-623, 659-663` — `insertion_index == -1` → assembled PDF lines silently discarded (no append-at-end fallback, unlike the docs+github+pdf path). *Fix:* append at end when no heading found. `Confirmed`
|
||||
|
||||
## Args / infra
|
||||
- **INF-06 — `--skip-config` is an orphan flag.** `arguments/create.py:427-433` — dest `skip_config` read nowhere (the real flag is `--skip-config-patterns`). *Fix:* remove or alias. `Confirmed`
|
||||
- **INF-07 — Azure server-side copy uses an unauthenticated source URL.** `storage/azure_storage.py:223` — `start_copy_from_url(source_blob.url)` (no SAS) → 403 on private containers (S3/GCS are fine). *Fix:* append a short-lived read SAS. `Confirmed`
|
||||
- **INF-08 — `presets/` package is dead code.** `presets/manager.py`, `presets/analyze_presets.py` — imported by no command (analyze parser not registered). Latent: `resolve_enhance_level` assumes `enhance_level` defaults to `None`, but the analyze parser forces `0`, so `--enhance` would be dead if revived. *Fix:* delete, or set the default back to `None`. `Confirmed`
|
||||
- **★INF-09 — `create --no-preserve-code-blocks` / `--no-preserve-paragraphs` inert.** `arguments/create.py:810-822` — registered for `create` but read nowhere in `_args_to_data`/`_build_config` (only the `package` command consumes them). *Fix:* wire into the RAG config or drop from `create`. `Confirmed`
|
||||
- **★INF-10 — `create --preset/-p` consumed nowhere.** `arguments/create.py:75-82` — the only reader lives in the dead `presets` package; `--preset comprehensive` is parsed and discarded. *Fix:* wire to analysis depth/features or remove. `Confirmed`
|
||||
|
||||
---
|
||||
|
||||
# Appendix A — Investigated and dismissed / corrected severity
|
||||
|
||||
These were checked against the source and are **not** defects (or were materially downgraded). Recorded for coverage.
|
||||
|
||||
- **C6 → DOC-01 (pdf `next(iter(...))` StopIteration).** *Downgraded Critical → Low (not in body).* The `elif self.categories:` guard means a plain dict is non-empty there, so `next(iter(values()))` cannot raise; it is only a defensive inconsistency vs the siblings (which pass `, None`). Low-priority robustness fix: add the `, None` default at `pdf_scraper.py:192`.
|
||||
- **CLI `_validate_arguments` false "not applicable" warnings — FALSE POSITIVE.** `create_command.py:111-136`. Verified every argument dict key equals its argparse dest, so the compatible-set membership check aligns; warnings only fire for genuinely-incompatible args.
|
||||
- **CBA GDScript `class_name` line via `.count("\n")` — FALSE POSITIVE (correctness).** `code_analyzer.py:1988`. Produces the identical line number as `_offset_to_line`; only a style/perf nit.
|
||||
- **CBB JS/TS config extraction duplicate settings — FALSE POSITIVE.** `config_extractor.py:584-586`. The three regexes are mutually exclusive per declaration; tested, no duplicate emissions. (A separate `version = 1.5` → `1` truncation defect exists but was not the claim.)
|
||||
- **DOC PDF chapter-boundary chunk inverts start/end — FALSE POSITIVE.** `pdf_extractor_poc.py:660-669`. The non-empty-`current_chunk` guard guarantees `end_page ≥ start_page`.
|
||||
- **ENH `enhance --model` dropped — CORRECTED (dead path).** `enhance_skill_local.py:1312-1340` does drop `--model`, but it's an unreachable legacy `main()`; the shipped `enhance` command routes through `enhance_command.py`, which forwards `--model` correctly to `enhance_skill.py` (which accepts it).
|
||||
- **Scan archive/diff ordering & `set_api_key` display name — FALSE POSITIVE.** `scan_command.py:1137-1143` ordering is correct by design; `config_manager.py` `capitalize()` is display-only (the real defect is the env_map omission, SCAN-03).
|
||||
|
||||
---
|
||||
|
||||
# Appendix B — Systemic patterns
|
||||
|
||||
Cross-cutting root causes worth fixing once rather than per-site:
|
||||
|
||||
- **S1 — Truncated/empty LLM output accepted as complete.** No `stop_reason`/`finish_reason` check: ENH-01 (`enhance_skill`/`agent_client`), ENH-11 (Gemini), MED-14 (video), ADP-06 (rename-before-write). Add a single "complete and non-empty?" gate before any overwrite, and always write-temp-then-swap.
|
||||
- **S2 — SKILL.md nav links derived differently from reference filenames.** DOC-07 (pdf/html/word/epub) + MED-04 (jupyter/asciidoc/pptx): 7 scrapers link to `sanitize(title).md` while files use range/basename names. Extract one `_ref_filename()` helper per scraper and use it for both writing and nav. (★MED-N5: PPTX duplicates the naming logic in 3 places.)
|
||||
- **S3 — Success/failure inferred from emoji/text substrings.** MCP-05 (`"❌"`), MCP-09/★MCP-15 (enhance/package), ENH-04 (Kimi regex). Return structured status/returncodes; never scan stdout for `❌`.
|
||||
- **S4 — `A or B and C` precedence bugs.** CBA-01, CBA-05, CBA-06, CBB-06. Audit every multi-clause boolean for missing parentheses.
|
||||
- **S5 — Shared-mutable-list aliasing.** CBB-03 (`common_problems.remove`), CBB-12 (`dependencies=imports`). Copy lists at construction (`list(...)`).
|
||||
- **S6 — Pagination / `max_pages` inconsistency.** MED-01/★MED-18 (github), MED-02/★MED-17 (slack), MED-03/★MED-16 (discord), MED-09 (confluence), DOC-04/DOC-09 (doc_scraper), MED-13 (`inf`-as-`int`). Standardize a cursor-following helper with explicit caps.
|
||||
- **S7 — Negative→positive flag translation gaps & inert flags.** CLI-01 (`--no-*`), INF-06/INF-09/INF-10 (`--skip-config`, `--no-preserve-*`, `--preset`). Add a registration test asserting every declared arg's dest is read somewhere.
|
||||
- **S8 — Library modules calling `logging.basicConfig`.** SCAN-06, ★CLI-09. Only the entry point should configure logging.
|
||||
- **S9 — Provider/model/timeout plumbing not threaded to the SDK call.** ENH-02, ENH-06, ENH-11, ENH-12. Centralize provider detection and request-option passing in one place.
|
||||
|
||||
---
|
||||
|
||||
# Suggested fix order
|
||||
|
||||
1. **S1 + ENH-01** (silent data loss of the primary artifact) — highest blast radius, low risk.
|
||||
2. **SCAN-01, CBB-01, MCP-01(+★MCP-12/13/14), CBA-13** (core features silently broken) — small, surgical fixes.
|
||||
3. **CLI-01, CLI-02(+★CLI-07), SCAN-02** (user-visible flag/crash bugs on the main `create`/`scan` paths).
|
||||
4. **S3 + S6** (MCP install/upload reliability; scraper pagination correctness).
|
||||
5. **CBA-01..04 + ★CBA-13..16** (pattern/dependency-analysis correctness — many share the precedence/regex root causes in S4).
|
||||
6. Medium/Low by subsystem as code is touched.
|
||||
@@ -0,0 +1,762 @@
|
||||
# Docker Deployment Guide
|
||||
|
||||
Complete guide for deploying Skill Seekers using Docker.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Quick Start](#quick-start)
|
||||
- [Building Images](#building-images)
|
||||
- [Running Containers](#running-containers)
|
||||
- [Docker Compose](#docker-compose)
|
||||
- [Configuration](#configuration)
|
||||
- [Data Persistence](#data-persistence)
|
||||
- [Networking](#networking)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Single Container Deployment
|
||||
|
||||
```bash
|
||||
# Pull pre-built image (when available)
|
||||
docker pull skillseekers/skillseekers:latest
|
||||
|
||||
# Or build locally
|
||||
docker build -t skillseekers:latest .
|
||||
|
||||
# Run MCP server
|
||||
docker run -d \
|
||||
--name skillseekers-mcp \
|
||||
-p 8765:8765 \
|
||||
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
|
||||
-e GITHUB_TOKEN=$GITHUB_TOKEN \
|
||||
-v skillseekers-data:/app/data \
|
||||
--restart unless-stopped \
|
||||
skillseekers:latest
|
||||
```
|
||||
|
||||
### Multi-Service Deployment
|
||||
|
||||
```bash
|
||||
# Start all services
|
||||
docker-compose up -d
|
||||
|
||||
# Check status
|
||||
docker-compose ps
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
## Building Images
|
||||
|
||||
### 1. Production Image
|
||||
|
||||
The Dockerfile uses multi-stage builds for optimization:
|
||||
|
||||
```dockerfile
|
||||
# Build stage
|
||||
FROM python:3.12-slim as builder
|
||||
WORKDIR /build
|
||||
COPY requirements.txt .
|
||||
RUN pip install --user --no-cache-dir -r requirements.txt
|
||||
|
||||
# Runtime stage
|
||||
FROM python:3.12-slim
|
||||
WORKDIR /app
|
||||
COPY --from=builder /root/.local /root/.local
|
||||
COPY . .
|
||||
ENV PATH=/root/.local/bin:$PATH
|
||||
CMD ["python", "-m", "skill_seekers.mcp.server_fastmcp"]
|
||||
```
|
||||
|
||||
**Build the image:**
|
||||
|
||||
```bash
|
||||
# Standard build
|
||||
docker build -t skillseekers:latest .
|
||||
|
||||
# Build with specific features
|
||||
docker build \
|
||||
--build-arg INSTALL_EXTRAS="all-llms,embedding" \
|
||||
-t skillseekers:full \
|
||||
.
|
||||
|
||||
# Build with cache
|
||||
docker build \
|
||||
--cache-from skillseekers:latest \
|
||||
-t skillseekers:v3.6.0 \
|
||||
.
|
||||
```
|
||||
|
||||
### 2. Development Image
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile.dev
|
||||
FROM python:3.12
|
||||
WORKDIR /app
|
||||
RUN pip install -e ".[dev]"
|
||||
COPY . .
|
||||
CMD ["python", "-m", "skill_seekers.mcp.server_fastmcp", "--reload"]
|
||||
```
|
||||
|
||||
**Build and run:**
|
||||
|
||||
```bash
|
||||
docker build -f Dockerfile.dev -t skillseekers:dev .
|
||||
|
||||
docker run -it \
|
||||
--name skillseekers-dev \
|
||||
-p 8765:8765 \
|
||||
-v $(pwd):/app \
|
||||
skillseekers:dev
|
||||
```
|
||||
|
||||
### 3. Image Optimization
|
||||
|
||||
**Reduce image size:**
|
||||
|
||||
```bash
|
||||
# Multi-stage build
|
||||
FROM python:3.12-slim as builder
|
||||
...
|
||||
FROM python:3.12-alpine # Smaller base
|
||||
|
||||
# Remove build dependencies
|
||||
RUN pip install --no-cache-dir ... && \
|
||||
rm -rf /root/.cache
|
||||
|
||||
# Use .dockerignore
|
||||
echo ".git" >> .dockerignore
|
||||
echo "tests/" >> .dockerignore
|
||||
echo "*.pyc" >> .dockerignore
|
||||
```
|
||||
|
||||
**Layer caching:**
|
||||
|
||||
```dockerfile
|
||||
# Copy requirements first (changes less frequently)
|
||||
COPY requirements.txt .
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
# Copy code later (changes more frequently)
|
||||
COPY . .
|
||||
```
|
||||
|
||||
## Running Containers
|
||||
|
||||
### 1. MCP Server
|
||||
|
||||
```bash
|
||||
# HTTP transport (recommended for production)
|
||||
docker run -d \
|
||||
--name skillseekers-mcp \
|
||||
-p 8765:8765 \
|
||||
-e MCP_TRANSPORT=http \
|
||||
-e MCP_PORT=8765 \
|
||||
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
|
||||
-v skillseekers-data:/app/data \
|
||||
--restart unless-stopped \
|
||||
skillseekers:latest
|
||||
|
||||
# stdio transport (for local tools)
|
||||
docker run -it \
|
||||
--name skillseekers-stdio \
|
||||
-e MCP_TRANSPORT=stdio \
|
||||
skillseekers:latest
|
||||
```
|
||||
|
||||
### 2. Embedding Server
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name skillseekers-embed \
|
||||
-p 8000:8000 \
|
||||
-e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
-e VOYAGE_API_KEY=$VOYAGE_API_KEY \
|
||||
-v skillseekers-cache:/app/cache \
|
||||
--restart unless-stopped \
|
||||
skillseekers:latest \
|
||||
python -m skill_seekers.embedding.server --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### 3. Sync Monitor
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name skillseekers-sync \
|
||||
-e SYNC_WEBHOOK_URL=$SYNC_WEBHOOK_URL \
|
||||
-v skillseekers-configs:/app/configs \
|
||||
--restart unless-stopped \
|
||||
skillseekers:latest \
|
||||
skill-seekers-sync start --config configs/react.json
|
||||
```
|
||||
|
||||
### 4. Interactive Commands
|
||||
|
||||
```bash
|
||||
# Run scraping
|
||||
docker run --rm \
|
||||
-e GITHUB_TOKEN=$GITHUB_TOKEN \
|
||||
-v $(pwd)/output:/app/output \
|
||||
skillseekers:latest \
|
||||
skill-seekers create --config configs/react.json
|
||||
|
||||
# Generate skill
|
||||
docker run --rm \
|
||||
-v $(pwd)/output:/app/output \
|
||||
skillseekers:latest \
|
||||
skill-seekers package output/react/
|
||||
|
||||
# Interactive shell
|
||||
docker run --rm -it \
|
||||
skillseekers:latest \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
## Docker Compose
|
||||
|
||||
### 1. Basic Setup
|
||||
|
||||
**docker-compose.yml:**
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
mcp-server:
|
||||
image: skillseekers:latest
|
||||
container_name: skillseekers-mcp
|
||||
ports:
|
||||
- "8765:8765"
|
||||
environment:
|
||||
- MCP_TRANSPORT=http
|
||||
- MCP_PORT=8765
|
||||
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
||||
- GITHUB_TOKEN=${GITHUB_TOKEN}
|
||||
- LOG_LEVEL=INFO
|
||||
volumes:
|
||||
- skillseekers-data:/app/data
|
||||
- skillseekers-logs:/app/logs
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
embedding-server:
|
||||
image: skillseekers:latest
|
||||
container_name: skillseekers-embed
|
||||
ports:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- VOYAGE_API_KEY=${VOYAGE_API_KEY}
|
||||
volumes:
|
||||
- skillseekers-cache:/app/cache
|
||||
command: ["python", "-m", "skill_seekers.embedding.server", "--host", "0.0.0.0"]
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: skillseekers-nginx
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./certs:/etc/nginx/certs:ro
|
||||
depends_on:
|
||||
- mcp-server
|
||||
- embedding-server
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
skillseekers-data:
|
||||
skillseekers-logs:
|
||||
skillseekers-cache:
|
||||
```
|
||||
|
||||
### 2. With Monitoring Stack
|
||||
|
||||
**docker-compose.monitoring.yml:**
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# ... (previous services)
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:latest
|
||||
container_name: skillseekers-prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- prometheus-data:/prometheus
|
||||
command:
|
||||
- '--config.file=/etc/prometheus/prometheus.yml'
|
||||
- '--storage.tsdb.path=/prometheus'
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: skillseekers-grafana
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
|
||||
volumes:
|
||||
- grafana-data:/var/lib/grafana
|
||||
- ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
|
||||
restart: unless-stopped
|
||||
|
||||
loki:
|
||||
image: grafana/loki:latest
|
||||
container_name: skillseekers-loki
|
||||
ports:
|
||||
- "3100:3100"
|
||||
volumes:
|
||||
- loki-data:/loki
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
prometheus-data:
|
||||
grafana-data:
|
||||
loki-data:
|
||||
```
|
||||
|
||||
### 3. Commands
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
docker-compose up -d
|
||||
|
||||
# Start with monitoring
|
||||
docker-compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d
|
||||
|
||||
# Check status
|
||||
docker-compose ps
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f mcp-server
|
||||
|
||||
# Scale services
|
||||
docker-compose up -d --scale mcp-server=3
|
||||
|
||||
# Stop services
|
||||
docker-compose down
|
||||
|
||||
# Stop and remove volumes
|
||||
docker-compose down -v
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. Environment Variables
|
||||
|
||||
**Using .env file:**
|
||||
|
||||
```bash
|
||||
# .env
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
GITHUB_TOKEN=ghp_...
|
||||
OPENAI_API_KEY=sk-...
|
||||
VOYAGE_API_KEY=...
|
||||
LOG_LEVEL=INFO
|
||||
MCP_PORT=8765
|
||||
```
|
||||
|
||||
**Load in docker-compose:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mcp-server:
|
||||
env_file:
|
||||
- .env
|
||||
```
|
||||
|
||||
### 2. Config Files
|
||||
|
||||
**Mount configuration:**
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
-v $(pwd)/configs:/app/configs:ro \
|
||||
skillseekers:latest
|
||||
```
|
||||
|
||||
**docker-compose.yml:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mcp-server:
|
||||
volumes:
|
||||
- ./configs:/app/configs:ro
|
||||
```
|
||||
|
||||
### 3. Secrets Management
|
||||
|
||||
**Docker Secrets (Swarm mode):**
|
||||
|
||||
```bash
|
||||
# Create secrets
|
||||
echo $ANTHROPIC_API_KEY | docker secret create anthropic_key -
|
||||
echo $GITHUB_TOKEN | docker secret create github_token -
|
||||
|
||||
# Use in service
|
||||
docker service create \
|
||||
--name skillseekers-mcp \
|
||||
--secret anthropic_key \
|
||||
--secret github_token \
|
||||
skillseekers:latest
|
||||
```
|
||||
|
||||
**docker-compose.yml (Swarm):**
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
secrets:
|
||||
anthropic_key:
|
||||
external: true
|
||||
github_token:
|
||||
external: true
|
||||
|
||||
services:
|
||||
mcp-server:
|
||||
secrets:
|
||||
- anthropic_key
|
||||
- github_token
|
||||
environment:
|
||||
- ANTHROPIC_API_KEY_FILE=/run/secrets/anthropic_key
|
||||
```
|
||||
|
||||
## Data Persistence
|
||||
|
||||
### 1. Named Volumes
|
||||
|
||||
```bash
|
||||
# Create volume
|
||||
docker volume create skillseekers-data
|
||||
|
||||
# Use in container
|
||||
docker run -v skillseekers-data:/app/data skillseekers:latest
|
||||
|
||||
# Backup volume
|
||||
docker run --rm \
|
||||
-v skillseekers-data:/data \
|
||||
-v $(pwd):/backup \
|
||||
alpine \
|
||||
tar czf /backup/backup.tar.gz /data
|
||||
|
||||
# Restore volume
|
||||
docker run --rm \
|
||||
-v skillseekers-data:/data \
|
||||
-v $(pwd):/backup \
|
||||
alpine \
|
||||
sh -c "cd /data && tar xzf /backup/backup.tar.gz --strip 1"
|
||||
```
|
||||
|
||||
### 2. Bind Mounts
|
||||
|
||||
```bash
|
||||
# Mount host directory
|
||||
docker run -v /opt/skillseekers/output:/app/output skillseekers:latest
|
||||
|
||||
# Read-only mount
|
||||
docker run -v $(pwd)/configs:/app/configs:ro skillseekers:latest
|
||||
```
|
||||
|
||||
### 3. Data Migration
|
||||
|
||||
```bash
|
||||
# Export from container
|
||||
docker cp skillseekers-mcp:/app/data ./data-backup
|
||||
|
||||
# Import to new container
|
||||
docker cp ./data-backup new-container:/app/data
|
||||
```
|
||||
|
||||
## Networking
|
||||
|
||||
### 1. Bridge Network (Default)
|
||||
|
||||
```bash
|
||||
# Containers can communicate by name
|
||||
docker network create skillseekers-net
|
||||
|
||||
docker run --network skillseekers-net skillseekers:latest
|
||||
```
|
||||
|
||||
### 2. Host Network
|
||||
|
||||
```bash
|
||||
# Use host network stack
|
||||
docker run --network host skillseekers:latest
|
||||
```
|
||||
|
||||
### 3. Custom Network
|
||||
|
||||
**docker-compose.yml:**
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
frontend:
|
||||
driver: bridge
|
||||
backend:
|
||||
driver: bridge
|
||||
internal: true # No external access
|
||||
|
||||
services:
|
||||
nginx:
|
||||
networks:
|
||||
- frontend
|
||||
|
||||
mcp-server:
|
||||
networks:
|
||||
- frontend
|
||||
- backend
|
||||
|
||||
database:
|
||||
networks:
|
||||
- backend
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### 1. Health Checks
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mcp-server:
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
### 2. Resource Limits
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mcp-server:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 4G
|
||||
reservations:
|
||||
cpus: '1.0'
|
||||
memory: 2G
|
||||
```
|
||||
|
||||
### 3. Logging
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mcp-server:
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels: "service=mcp"
|
||||
|
||||
# Or use syslog
|
||||
logging:
|
||||
driver: "syslog"
|
||||
options:
|
||||
syslog-address: "udp://192.168.1.100:514"
|
||||
```
|
||||
|
||||
### 4. Metrics
|
||||
|
||||
```bash
|
||||
# Docker stats
|
||||
docker stats skillseekers-mcp
|
||||
|
||||
# cAdvisor for metrics
|
||||
docker run -d \
|
||||
--name cadvisor \
|
||||
-p 8080:8080 \
|
||||
-v /:/rootfs:ro \
|
||||
-v /var/run:/var/run:ro \
|
||||
-v /sys:/sys:ro \
|
||||
-v /var/lib/docker:/var/lib/docker:ro \
|
||||
gcr.io/cadvisor/cadvisor:latest
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Container Won't Start
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
docker logs skillseekers-mcp
|
||||
|
||||
# Inspect container
|
||||
docker inspect skillseekers-mcp
|
||||
|
||||
# Run with interactive shell
|
||||
docker run -it --entrypoint /bin/bash skillseekers:latest
|
||||
```
|
||||
|
||||
#### 2. Port Already in Use
|
||||
|
||||
```bash
|
||||
# Find process using port
|
||||
sudo lsof -i :8765
|
||||
|
||||
# Kill process
|
||||
kill -9 <PID>
|
||||
|
||||
# Or use different port
|
||||
docker run -p 8766:8765 skillseekers:latest
|
||||
```
|
||||
|
||||
#### 3. Volume Permission Issues
|
||||
|
||||
```bash
|
||||
# Run as specific user
|
||||
docker run --user $(id -u):$(id -g) skillseekers:latest
|
||||
|
||||
# Fix permissions
|
||||
docker run --rm \
|
||||
-v skillseekers-data:/data \
|
||||
alpine chown -R 1000:1000 /data
|
||||
```
|
||||
|
||||
#### 4. Network Connectivity
|
||||
|
||||
```bash
|
||||
# Test connectivity
|
||||
docker exec skillseekers-mcp ping google.com
|
||||
|
||||
# Check DNS
|
||||
docker exec skillseekers-mcp cat /etc/resolv.conf
|
||||
|
||||
# Use custom DNS
|
||||
docker run --dns 8.8.8.8 skillseekers:latest
|
||||
```
|
||||
|
||||
#### 5. High Memory Usage
|
||||
|
||||
```bash
|
||||
# Set memory limit
|
||||
docker run --memory=4g skillseekers:latest
|
||||
|
||||
# Check memory usage
|
||||
docker stats skillseekers-mcp
|
||||
|
||||
# Enable memory swappiness
|
||||
docker run --memory=4g --memory-swap=8g skillseekers:latest
|
||||
```
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# Enter running container
|
||||
docker exec -it skillseekers-mcp /bin/bash
|
||||
|
||||
# View environment variables
|
||||
docker exec skillseekers-mcp env
|
||||
|
||||
# Check processes
|
||||
docker exec skillseekers-mcp ps aux
|
||||
|
||||
# View logs in real-time
|
||||
docker logs -f --tail 100 skillseekers-mcp
|
||||
|
||||
# Inspect container details
|
||||
docker inspect skillseekers-mcp | jq '.[]'
|
||||
|
||||
# Export container filesystem
|
||||
docker export skillseekers-mcp > container.tar
|
||||
```
|
||||
|
||||
## Production Best Practices
|
||||
|
||||
### 1. Image Management
|
||||
|
||||
```bash
|
||||
# Tag images with versions
|
||||
docker build -t skillseekers:2.9.0 .
|
||||
docker tag skillseekers:2.9.0 skillseekers:latest
|
||||
|
||||
# Use private registry
|
||||
docker tag skillseekers:latest registry.example.com/skillseekers:latest
|
||||
docker push registry.example.com/skillseekers:latest
|
||||
|
||||
# Scan for vulnerabilities
|
||||
docker scan skillseekers:latest
|
||||
```
|
||||
|
||||
### 2. Security
|
||||
|
||||
```bash
|
||||
# Run as non-root user
|
||||
RUN useradd -m -s /bin/bash skillseekers
|
||||
USER skillseekers
|
||||
|
||||
# Read-only root filesystem
|
||||
docker run --read-only --tmpfs /tmp skillseekers:latest
|
||||
|
||||
# Drop capabilities
|
||||
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE skillseekers:latest
|
||||
|
||||
# Use security scanning
|
||||
trivy image skillseekers:latest
|
||||
```
|
||||
|
||||
### 3. Resource Management
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mcp-server:
|
||||
# CPU limits
|
||||
cpus: 2.0
|
||||
cpu_shares: 1024
|
||||
|
||||
# Memory limits
|
||||
mem_limit: 4g
|
||||
memswap_limit: 8g
|
||||
mem_reservation: 2g
|
||||
|
||||
# Process limits
|
||||
pids_limit: 200
|
||||
```
|
||||
|
||||
### 4. Backup & Recovery
|
||||
|
||||
```bash
|
||||
# Backup script
|
||||
#!/bin/bash
|
||||
docker-compose down
|
||||
tar czf backup-$(date +%Y%m%d).tar.gz volumes/
|
||||
docker-compose up -d
|
||||
|
||||
# Automated backups
|
||||
0 2 * * * /opt/skillseekers/backup.sh
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- See [KUBERNETES_DEPLOYMENT.md](./KUBERNETES_DEPLOYMENT.md) for Kubernetes deployment
|
||||
- Review [PRODUCTION_DEPLOYMENT.md](./PRODUCTION_DEPLOYMENT.md) for general production guidelines
|
||||
- Check [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common issues
|
||||
|
||||
---
|
||||
|
||||
**Need help?** Open an issue on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers/issues).
|
||||
@@ -0,0 +1,863 @@
|
||||
# Frequently Asked Questions (FAQ)
|
||||
|
||||
**Version:** 3.6.0
|
||||
**Last Updated:** 2026-03-15
|
||||
|
||||
---
|
||||
|
||||
## General Questions
|
||||
|
||||
### What is Skill Seekers?
|
||||
|
||||
Skill Seekers is a Python tool that converts 18 source types — documentation websites, GitHub repos, PDFs, videos, Word docs, EPUB books, Jupyter notebooks, local HTML files, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence wikis, Notion pages, Slack/Discord exports, and local codebases — into AI-ready formats for 21+ platforms: LLM platforms (Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown), RAG frameworks (LangChain, LlamaIndex, Haystack), vector databases (ChromaDB, FAISS, Weaviate, Qdrant, Pinecone), and AI coding assistants (Cursor, Windsurf, Cline, Continue.dev, Roo, Aider, Bolt, Kilo, Kimi Code).
|
||||
|
||||
**Use Cases:**
|
||||
- Create custom documentation skills for your favorite frameworks
|
||||
- Analyze GitHub repositories and extract code patterns
|
||||
- Convert PDF manuals into searchable AI skills
|
||||
- Import knowledge from Confluence, Notion, or Slack/Discord
|
||||
- Extract content from videos (YouTube, Vimeo, local files)
|
||||
- Convert Jupyter notebooks, EPUB books, or PowerPoint slides into skills
|
||||
- Parse OpenAPI/Swagger specs into API reference skills
|
||||
- Combine multiple sources (docs + code + PDFs + more) into unified skills
|
||||
|
||||
### Which platforms are supported?
|
||||
|
||||
**Supported Platforms (30+):**
|
||||
|
||||
*LLM Platforms (12):*
|
||||
1. **Claude AI** - ZIP format with YAML frontmatter
|
||||
2. **Google Gemini** - tar.gz format for Grounded Generation
|
||||
3. **OpenAI ChatGPT** - ZIP format for Vector Stores
|
||||
4. **MiniMax** - ZIP format
|
||||
5. **OpenCode** - ZIP format
|
||||
6. **Kimi** - ZIP format
|
||||
7. **DeepSeek** - ZIP format
|
||||
8. **Qwen** - ZIP format
|
||||
9. **OpenRouter** - ZIP format for multi-model routing
|
||||
10. **Together AI** - ZIP format for open-source models
|
||||
11. **Fireworks AI** - ZIP format for fast inference
|
||||
12. **Generic Markdown** - ZIP format with markdown files
|
||||
|
||||
*RAG Frameworks:*
|
||||
13. **LangChain** - Document objects for QA chains and agents
|
||||
14. **LlamaIndex** - TextNodes for query engines
|
||||
15. **Haystack** - Document objects for enterprise RAG
|
||||
|
||||
*Vector Databases:*
|
||||
16. **ChromaDB** - Direct collection upload
|
||||
17. **FAISS** - Index files for local similarity search
|
||||
18. **Weaviate** - Vector objects with schema creation
|
||||
19. **Qdrant** - Points with payload indexing
|
||||
20. **Pinecone** - Ready-to-upsert format
|
||||
|
||||
*AI Coding Assistants (9):*
|
||||
21. **Cursor** - .cursorrules persistent context
|
||||
22. **Windsurf** - .windsurfrules AI coding rules
|
||||
23. **Cline** - .clinerules + MCP integration
|
||||
24. **Continue.dev** - HTTP context server (all IDEs)
|
||||
25. **Roo** - .roorules AI coding rules
|
||||
26. **Aider** - Terminal AI coding assistant
|
||||
27. **Bolt** - Web IDE AI context
|
||||
28. **Kilo** - IDE AI context
|
||||
29. **Kimi Code** - IDE AI context
|
||||
|
||||
Each platform has a dedicated adaptor for optimal formatting and upload.
|
||||
|
||||
### Is it free to use?
|
||||
|
||||
**Tool:** Yes, Skill Seekers is 100% free and open-source (MIT license).
|
||||
|
||||
**API Costs:**
|
||||
- **Scraping:** Free (just bandwidth)
|
||||
- **AI Enhancement (API mode):** ~$0.15-0.30 per skill (Claude API)
|
||||
- **AI Enhancement (LOCAL mode):** Free! (uses your Claude Code Max plan)
|
||||
- **Upload:** Free (platform storage limits apply)
|
||||
|
||||
**Recommendation:** Use LOCAL mode for free AI enhancement or skip enhancement entirely.
|
||||
|
||||
### How do I set up video extraction?
|
||||
|
||||
**Quick setup:**
|
||||
```bash
|
||||
# 1. Install video support
|
||||
pip install skill-seekers[video-full]
|
||||
|
||||
# 2. Auto-detect GPU and install visual deps
|
||||
skill-seekers create --setup
|
||||
```
|
||||
|
||||
The `--setup` command auto-detects your GPU vendor (NVIDIA CUDA, AMD ROCm, or CPU-only) and installs the correct PyTorch variant along with easyocr and other visual extraction dependencies. This avoids the ~2GB NVIDIA CUDA download that would happen if easyocr were installed via pip on non-NVIDIA systems.
|
||||
|
||||
**What it detects:**
|
||||
- **NVIDIA:** Uses `nvidia-smi` to find CUDA version → installs matching `cu124`/`cu121`/`cu118` PyTorch
|
||||
- **AMD:** Uses `rocminfo` to find ROCm version → installs matching ROCm PyTorch
|
||||
- **CPU-only:** Installs lightweight CPU-only PyTorch
|
||||
|
||||
### What source types are supported?
|
||||
|
||||
Skill Seekers supports **18 source types**:
|
||||
|
||||
| # | Source Type | CLI Command | Auto-Detection |
|
||||
|---|------------|-------------|----------------|
|
||||
| 1 | Documentation (web) | `scrape` / `create <url>` | HTTP/HTTPS URLs |
|
||||
| 2 | GitHub repo | `github` / `create owner/repo` | `owner/repo` or github.com URLs |
|
||||
| 3 | PDF | `pdf` / `create file.pdf` | `.pdf` extension |
|
||||
| 4 | Word (.docx) | `word` / `create file.docx` | `.docx` extension |
|
||||
| 5 | EPUB | `epub` / `create file.epub` | `.epub` extension |
|
||||
| 6 | Video | `video` / `create <url/file>` | YouTube/Vimeo URLs, video extensions |
|
||||
| 7 | Local codebase | `analyze` / `create ./path` | Directory paths |
|
||||
| 8 | Jupyter Notebook | `jupyter` / `create file.ipynb` | `.ipynb` extension |
|
||||
| 9 | Local HTML | `html` / `create file.html` | `.html`/`.htm` extensions |
|
||||
| 10 | OpenAPI/Swagger | `openapi` / `create spec.yaml` | `.yaml`/`.yml` with OpenAPI content |
|
||||
| 11 | AsciiDoc | `asciidoc` / `create file.adoc` | `.adoc`/`.asciidoc` extensions |
|
||||
| 12 | PowerPoint | `pptx` / `create file.pptx` | `.pptx` extension |
|
||||
| 13 | RSS/Atom | `rss` / `create feed.rss` | `.rss`/`.atom` extensions |
|
||||
| 14 | Man pages | `manpage` / `create cmd.1` | `.1`-`.8`/`.man` extensions |
|
||||
| 15 | Confluence | `confluence` | API or export directory |
|
||||
| 16 | Notion | `notion` | API or export directory |
|
||||
| 17 | Slack/Discord | `chat` | Export directory or API |
|
||||
|
||||
The `create` command auto-detects the source type from your input, so you often don't need to specify a subcommand.
|
||||
|
||||
### How long does it take to create a skill?
|
||||
|
||||
**Typical Times:**
|
||||
- Documentation scraping: 5-45 minutes (depends on size)
|
||||
- GitHub analysis: 1-5 minutes (basic) or 20-60 minutes (C3.x deep analysis)
|
||||
- PDF extraction: 30 seconds - 5 minutes
|
||||
- Video extraction: 2-10 minutes (depends on length and visual analysis)
|
||||
- Word/EPUB/PPTX: 10-60 seconds
|
||||
- Jupyter notebook: 10-30 seconds
|
||||
- OpenAPI spec: 5-15 seconds
|
||||
- Confluence/Notion import: 1-5 minutes (depends on space size)
|
||||
- AI enhancement: 30-60 seconds (LOCAL or API mode)
|
||||
- Total workflow: 10-60 minutes
|
||||
|
||||
**Speed Tips:**
|
||||
- Use `--async` for 2-3x faster scraping
|
||||
- Use `--skip-scrape` to rebuild without re-scraping
|
||||
- Skip AI enhancement for faster workflow
|
||||
|
||||
---
|
||||
|
||||
## Installation & Setup
|
||||
|
||||
### How do I install Skill Seekers?
|
||||
|
||||
```bash
|
||||
# Basic installation
|
||||
pip install skill-seekers
|
||||
|
||||
# With all platform support
|
||||
pip install skill-seekers[all-llms]
|
||||
|
||||
# Development installation
|
||||
git clone https://github.com/yusufkaraaslan/Skill_Seekers.git
|
||||
cd Skill_Seekers
|
||||
pip install -e ".[all-llms,dev]"
|
||||
```
|
||||
|
||||
### What Python version do I need?
|
||||
|
||||
**Required:** Python 3.10 or higher
|
||||
**Tested on:** Python 3.10, 3.11, 3.12, 3.13
|
||||
**OS Support:** Linux, macOS, Windows (WSL recommended)
|
||||
|
||||
**Check your version:**
|
||||
```bash
|
||||
python --version # Should be 3.10+
|
||||
```
|
||||
|
||||
### Why do I get "No module named 'skill_seekers'" error?
|
||||
|
||||
**Common Causes:**
|
||||
1. Package not installed
|
||||
2. Wrong Python environment
|
||||
|
||||
**Solutions:**
|
||||
```bash
|
||||
# Install package
|
||||
pip install skill-seekers
|
||||
|
||||
# Or for development
|
||||
pip install -e .
|
||||
|
||||
# Verify installation
|
||||
skill-seekers --version
|
||||
```
|
||||
|
||||
### How do I set up API keys?
|
||||
|
||||
```bash
|
||||
# Claude AI (for enhancement and upload)
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# Google Gemini (for upload)
|
||||
export GOOGLE_API_KEY=AIza...
|
||||
|
||||
# OpenAI ChatGPT (for upload)
|
||||
export OPENAI_API_KEY=sk-...
|
||||
|
||||
# GitHub (for higher rate limits)
|
||||
export GITHUB_TOKEN=ghp_...
|
||||
|
||||
# Make permanent (add to ~/.bashrc or ~/.zshrc)
|
||||
echo 'export ANTHROPIC_API_KEY=sk-ant-...' >> ~/.bashrc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage Questions
|
||||
|
||||
### How do I scrape documentation?
|
||||
|
||||
**Using preset config:**
|
||||
```bash
|
||||
skill-seekers create --config react
|
||||
```
|
||||
|
||||
**Using custom URL:**
|
||||
```bash
|
||||
skill-seekers create https://docs.example.com --name my-framework
|
||||
```
|
||||
|
||||
**From custom config file:**
|
||||
```bash
|
||||
skill-seekers create --config configs/my-framework.json
|
||||
```
|
||||
|
||||
### How do I create skills for an entire project at once?
|
||||
|
||||
Use `skill-seekers scan` — an AI agent inspects the project's manifests, README, Dockerfile/CI, and source imports, then emits one config per detected framework plus a `<project>-codebase.json` for your own code:
|
||||
|
||||
```bash
|
||||
# AI-detect frameworks and emit configs
|
||||
skill-seekers scan ./my-react-app --out ./configs/scanned/
|
||||
# → react.json, typescript.json, vite.json, my-react-app-codebase.json
|
||||
|
||||
# Then build any of them
|
||||
skill-seekers create ./configs/scanned/react.json
|
||||
```
|
||||
|
||||
Re-running `scan` against the same output directory reports diffs (added / version-bumped / removed) so you can keep skills in sync with your real dependencies. See [CLI Reference: scan](reference/CLI_REFERENCE.md#scan) for all flags.
|
||||
|
||||
### Can I analyze GitHub repositories?
|
||||
|
||||
Yes! Skill Seekers has powerful GitHub analysis:
|
||||
|
||||
```bash
|
||||
# Basic analysis (fast)
|
||||
skill-seekers create https://github.com/facebook/react
|
||||
|
||||
# Deep C3.x analysis (includes patterns, tests, guides)
|
||||
skill-seekers create https://github.com/vercel/next.js --depth full
|
||||
```
|
||||
|
||||
**C3.x Features:**
|
||||
- Design pattern detection (10 GoF patterns)
|
||||
- Test example extraction
|
||||
- How-to guide generation
|
||||
- Configuration pattern extraction
|
||||
- Architectural overview
|
||||
- API reference generation
|
||||
|
||||
### Can I extract content from PDFs?
|
||||
|
||||
Yes! PDF extraction with OCR support:
|
||||
|
||||
```bash
|
||||
# Basic PDF extraction
|
||||
skill-seekers create --pdf manual.pdf --name product-manual
|
||||
|
||||
# With OCR (for scanned PDFs)
|
||||
skill-seekers create --pdf scanned.pdf --ocr
|
||||
|
||||
# Images and tables are extracted automatically
|
||||
skill-seekers create --pdf document.pdf
|
||||
```
|
||||
|
||||
### How do I scrape a Jupyter Notebook?
|
||||
|
||||
```bash
|
||||
# Extract cells, outputs, and markdown from a notebook
|
||||
skill-seekers create analysis.ipynb --name data-analysis
|
||||
|
||||
# Or use auto-detection
|
||||
skill-seekers create analysis.ipynb
|
||||
```
|
||||
|
||||
Jupyter extraction preserves code cells, markdown cells, and cell outputs. It works with `.ipynb` files from JupyterLab, Google Colab, and other notebook environments.
|
||||
|
||||
### How do I import from Confluence or Notion?
|
||||
|
||||
**Confluence:**
|
||||
```bash
|
||||
# From Confluence Cloud API
|
||||
export CONFLUENCE_URL=https://yourorg.atlassian.net
|
||||
export CONFLUENCE_TOKEN=your-api-token
|
||||
export CONFLUENCE_EMAIL=your-email@example.com
|
||||
skill-seekers create --space-key MYSPACE --name my-wiki
|
||||
|
||||
# From a Confluence HTML/XML export directory
|
||||
skill-seekers create --conf-export-path ./confluence-export --name my-wiki
|
||||
```
|
||||
|
||||
**Notion:**
|
||||
```bash
|
||||
# From Notion API
|
||||
export NOTION_TOKEN=secret_...
|
||||
skill-seekers create --database-id DATABASE_ID --name my-notes
|
||||
|
||||
# From a Notion HTML/Markdown export directory
|
||||
skill-seekers create --notion-export-path ./notion-export --name my-notes
|
||||
```
|
||||
|
||||
### How do I convert Word, EPUB, or PowerPoint files?
|
||||
|
||||
```bash
|
||||
# Word document
|
||||
skill-seekers create report.docx --name quarterly-report
|
||||
|
||||
# EPUB book
|
||||
skill-seekers create handbook.epub --name dev-handbook
|
||||
|
||||
# PowerPoint presentation
|
||||
skill-seekers create slides.pptx --name training-deck
|
||||
|
||||
# Or use auto-detection for any of them
|
||||
skill-seekers create report.docx
|
||||
skill-seekers create handbook.epub
|
||||
skill-seekers create slides.pptx
|
||||
```
|
||||
|
||||
### How do I parse an OpenAPI/Swagger spec?
|
||||
|
||||
```bash
|
||||
# From a local YAML/JSON file
|
||||
skill-seekers create api-spec.yaml --name my-api
|
||||
|
||||
# Auto-detection works too
|
||||
skill-seekers create api-spec.yaml
|
||||
```
|
||||
|
||||
OpenAPI extraction parses endpoints, schemas, parameters, and examples into a structured API reference skill.
|
||||
|
||||
### How do I extract content from RSS feeds or man pages?
|
||||
|
||||
```bash
|
||||
# RSS/Atom feed
|
||||
skill-seekers create https://blog.example.com/feed.xml --name blog-feed
|
||||
|
||||
# Man page
|
||||
skill-seekers create grep.1 --name grep-manual
|
||||
```
|
||||
|
||||
### How do I import from Slack or Discord?
|
||||
|
||||
```bash
|
||||
# From a Slack export directory
|
||||
skill-seekers create --chat-export-path ./slack-export --platform slack --name team-knowledge
|
||||
|
||||
# From a Discord export directory
|
||||
skill-seekers create --chat-export-path ./discord-export --platform discord --name server-archive
|
||||
```
|
||||
|
||||
### Can I combine multiple sources?
|
||||
|
||||
Yes! Unified multi-source scraping:
|
||||
|
||||
**Create unified config** (`configs/unified/my-framework.json`):
|
||||
```json
|
||||
{
|
||||
"name": "my-framework",
|
||||
"sources": {
|
||||
"documentation": {
|
||||
"type": "docs",
|
||||
"base_url": "https://docs.example.com"
|
||||
},
|
||||
"github": {
|
||||
"type": "github",
|
||||
"repo_url": "https://github.com/org/repo"
|
||||
},
|
||||
"pdf": {
|
||||
"type": "pdf",
|
||||
"pdf_path": "manual.pdf"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Run unified scraping:**
|
||||
```bash
|
||||
skill-seekers create --config configs/unified/my-framework.json
|
||||
```
|
||||
|
||||
### How do I upload skills to platforms?
|
||||
|
||||
```bash
|
||||
# Upload to Claude AI
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
skill-seekers upload output/react-claude.zip --target claude
|
||||
|
||||
# Upload to Google Gemini
|
||||
export GOOGLE_API_KEY=AIza...
|
||||
skill-seekers upload output/react-gemini.tar.gz --target gemini
|
||||
|
||||
# Upload to OpenAI ChatGPT
|
||||
export OPENAI_API_KEY=sk-...
|
||||
skill-seekers upload output/react-openai.zip --target openai
|
||||
```
|
||||
|
||||
**Or use complete workflow (uploads by default):**
|
||||
```bash
|
||||
skill-seekers install --config react --target claude
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Platform-Specific Questions
|
||||
|
||||
### What's the difference between platforms?
|
||||
|
||||
| Feature | Claude AI | Google Gemini | OpenAI ChatGPT | Markdown |
|
||||
|---------|-----------|---------------|----------------|----------|
|
||||
| Format | ZIP + YAML | tar.gz | ZIP | ZIP |
|
||||
| Upload API | Projects API | Corpora API | Vector Stores | N/A |
|
||||
| Model | Sonnet 4.5 | Gemini 2.0 Flash | GPT-4o | N/A |
|
||||
| Max Size | 32MB | 10MB | 512MB | N/A |
|
||||
| Use Case | Claude Code | Grounded Gen | ChatGPT Custom | Export |
|
||||
|
||||
**Choose based on:**
|
||||
- Claude AI: Best for Claude Code integration
|
||||
- Google Gemini: Best for Grounded Generation in Gemini
|
||||
- OpenAI ChatGPT: Best for ChatGPT Custom GPTs
|
||||
- MiniMax/Kimi/DeepSeek/Qwen: Best for Chinese LLM ecosystem
|
||||
- OpenRouter/Together/Fireworks: Best for multi-model routing or open-source model access
|
||||
- Markdown: Generic export for other tools
|
||||
|
||||
### Can I use multiple platforms at once?
|
||||
|
||||
Yes! Package and upload to all platforms:
|
||||
|
||||
```bash
|
||||
# Package for all platforms
|
||||
for platform in claude gemini openai minimax kimi deepseek qwen openrouter together fireworks markdown; do
|
||||
skill-seekers package output/react/ --target $platform
|
||||
done
|
||||
|
||||
# Install + upload per platform (one --target per run)
|
||||
for platform in claude gemini openai; do
|
||||
skill-seekers install --config react --target $platform
|
||||
done
|
||||
```
|
||||
|
||||
### How do I use skills in Claude Code?
|
||||
|
||||
1. **Install skill to Claude Code directory:**
|
||||
```bash
|
||||
skill-seekers install-agent output/react/ --agent claude
|
||||
```
|
||||
|
||||
2. **Use in Claude Code:**
|
||||
```
|
||||
Use the react skill to explain React hooks
|
||||
```
|
||||
|
||||
3. **Or upload to Claude AI:**
|
||||
```bash
|
||||
skill-seekers upload output/react-claude.zip --target claude
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Features & Capabilities
|
||||
|
||||
### What is AI enhancement?
|
||||
|
||||
AI enhancement transforms basic skills (2-3/10 quality) into production-ready skills (8-9/10 quality) using LLMs.
|
||||
|
||||
**Two Modes (via AgentClient):**
|
||||
1. **API Mode:** Multi-provider AI API calls -- Anthropic, Moonshot/Kimi, Gemini, OpenAI (fast, costs ~$0.15-0.30)
|
||||
2. **LOCAL Mode:** Any supported coding agent -- Claude Code, Kimi, Codex, Copilot, OpenCode, or custom (free with agent subscription)
|
||||
|
||||
**What it improves:**
|
||||
- Better organization and structure
|
||||
- Clearer explanations
|
||||
- More examples and use cases
|
||||
- Better cross-references
|
||||
- Improved searchability
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# API mode (if ANTHROPIC_API_KEY is set)
|
||||
skill-seekers enhance output/react/
|
||||
|
||||
# LOCAL mode (free! — auto-selected when no API key is set; pick the agent)
|
||||
skill-seekers enhance output/react/ --agent claude
|
||||
|
||||
# Background mode
|
||||
skill-seekers enhance output/react/ --background
|
||||
skill-seekers enhance-status output/react/ --watch
|
||||
```
|
||||
|
||||
### What are C3.x features?
|
||||
|
||||
C3.x features are advanced codebase analysis capabilities:
|
||||
|
||||
- **C3.1:** Design pattern detection (Singleton, Factory, Strategy, etc.)
|
||||
- **C3.2:** Test example extraction (real usage examples from tests)
|
||||
- **C3.3:** How-to guide generation (educational guides from test workflows)
|
||||
- **C3.4:** Configuration pattern extraction (env vars, config files)
|
||||
- **C3.5:** Architectural overview (system architecture analysis)
|
||||
- **C3.6:** AI enhancement (Claude API integration for insights)
|
||||
- **C3.7:** Architectural pattern detection (MVC, MVVM, Repository, etc.)
|
||||
- **C3.8:** Standalone codebase scraping (300+ line SKILL.md from code alone)
|
||||
|
||||
**Enable C3.x:**
|
||||
```bash
|
||||
# All C3.x features enabled by default
|
||||
skill-seekers create --directory /path/to/repo
|
||||
|
||||
# Skip specific features
|
||||
skill-seekers create --directory . --skip-patterns --skip-how-to-guides
|
||||
```
|
||||
|
||||
### What are router skills?
|
||||
|
||||
Router skills help Claude navigate large documentation (>500 pages) by providing a table of contents and keyword index.
|
||||
|
||||
**When to use:**
|
||||
- Documentation with 500+ pages
|
||||
- Complex multi-section docs
|
||||
- Large API references
|
||||
|
||||
**Generate router (from split sub-skill configs):**
|
||||
```bash
|
||||
python -m skill_seekers.cli.generate_router configs/godot_*.json --name godot
|
||||
```
|
||||
(Also available as the `generate_router` MCP tool.)
|
||||
|
||||
### What preset configurations are available?
|
||||
|
||||
**24 preset configs:**
|
||||
- Web: react, vue, angular, svelte, nextjs
|
||||
- Python: django, flask, fastapi, sqlalchemy, pytest
|
||||
- Game Dev: godot, pygame, unity
|
||||
- DevOps: docker, kubernetes, terraform, ansible
|
||||
- Unified: react-unified, vue-unified, nextjs-unified, etc.
|
||||
|
||||
**List all:**
|
||||
```bash
|
||||
skill-seekers estimate --all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Scraping is very slow, how can I speed it up?
|
||||
|
||||
**Solutions:**
|
||||
1. **Use async mode** (2-3x faster):
|
||||
```bash
|
||||
skill-seekers create --config react --async
|
||||
```
|
||||
|
||||
2. **Increase rate limit** (faster requests):
|
||||
```json
|
||||
{
|
||||
"rate_limit": 0.1 // Faster (but may hit rate limits)
|
||||
}
|
||||
```
|
||||
|
||||
3. **Limit pages**:
|
||||
```json
|
||||
{
|
||||
"max_pages": 100 // Stop after 100 pages
|
||||
}
|
||||
```
|
||||
|
||||
### Why are some pages missing?
|
||||
|
||||
**Common Causes:**
|
||||
1. **URL patterns exclude them**
|
||||
2. **Max pages limit reached**
|
||||
3. **BFS didn't reach them**
|
||||
|
||||
**Solutions:**
|
||||
```bash
|
||||
# Check URL patterns in config
|
||||
{
|
||||
"url_patterns": {
|
||||
"include": ["/docs/"], // Make sure your pages match
|
||||
"exclude": [] // Remove overly broad exclusions
|
||||
}
|
||||
}
|
||||
|
||||
# Increase max pages
|
||||
{
|
||||
"max_pages": 1000 // Default is 500
|
||||
}
|
||||
|
||||
# Use verbose mode to see what's being scraped
|
||||
skill-seekers create --config react --verbose
|
||||
```
|
||||
|
||||
### How do I fix "NetworkError: Connection failed"?
|
||||
|
||||
**Solutions:**
|
||||
1. **Check internet connection**
|
||||
2. **Verify URL is accessible**:
|
||||
```bash
|
||||
curl -I https://docs.example.com
|
||||
```
|
||||
|
||||
3. **Increase timeout**:
|
||||
```json
|
||||
{
|
||||
"timeout": 30 // 30 seconds
|
||||
}
|
||||
```
|
||||
|
||||
4. **Check rate limiting**:
|
||||
```json
|
||||
{
|
||||
"rate_limit": 1.0 // Slower requests
|
||||
}
|
||||
```
|
||||
|
||||
### Tests are failing, what should I do?
|
||||
|
||||
**Quick fixes:**
|
||||
```bash
|
||||
# Ensure package is installed
|
||||
pip install -e ".[all-llms,dev]"
|
||||
|
||||
# Clear caches
|
||||
rm -rf .pytest_cache/ **/__pycache__/
|
||||
|
||||
# Run specific failing test
|
||||
pytest tests/test_file.py::test_name -vv
|
||||
|
||||
# Check for missing dependencies
|
||||
pip install -e ".[all-llms,dev]"
|
||||
```
|
||||
|
||||
**If still failing:**
|
||||
1. Check [Troubleshooting Guide](../TROUBLESHOOTING.md)
|
||||
2. Report issue on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers/issues)
|
||||
|
||||
---
|
||||
|
||||
## MCP Server Questions
|
||||
|
||||
### How do I start the MCP server?
|
||||
|
||||
```bash
|
||||
# stdio mode (Claude Code, VS Code + Cline)
|
||||
skill-seekers-mcp
|
||||
|
||||
# HTTP mode (Cursor, Windsurf, IntelliJ)
|
||||
skill-seekers-mcp --transport http --port 8765
|
||||
```
|
||||
|
||||
### What MCP tools are available?
|
||||
|
||||
**40 MCP tools:**
|
||||
|
||||
*Core Tools (9):*
|
||||
1. `list_configs` - List preset configurations
|
||||
2. `generate_config` - Generate config from docs URL
|
||||
3. `validate_config` - Validate config structure
|
||||
4. `estimate_pages` - Estimate page count
|
||||
5. `scrape_docs` - Scrape documentation
|
||||
6. `package_skill` - Package to .zip (supports `--format` and `--target`)
|
||||
7. `upload_skill` - Upload to platform (supports `--target`)
|
||||
8. `enhance_skill` - AI enhancement
|
||||
9. `install_skill` - Complete workflow
|
||||
|
||||
*Extended Tools (10):*
|
||||
10. `scrape_github` - GitHub analysis
|
||||
11. `scrape_pdf` - PDF extraction
|
||||
12. `unified_scrape` - Multi-source scraping
|
||||
13. `merge_sources` - Merge docs + code
|
||||
14. `detect_conflicts` - Find discrepancies
|
||||
15. `split_config` - Split large configs
|
||||
16. `generate_router` - Generate router skills
|
||||
17. `add_config_source` - Register git repos
|
||||
18. `fetch_config` - Fetch configs from git
|
||||
19. `list_config_sources` - List registered sources
|
||||
20. `remove_config_source` - Remove config source
|
||||
|
||||
*Vector DB Tools (4):*
|
||||
21. `export_to_chroma` - Export to ChromaDB
|
||||
22. `export_to_weaviate` - Export to Weaviate
|
||||
23. `export_to_faiss` - Export to FAISS
|
||||
24. `export_to_qdrant` - Export to Qdrant
|
||||
|
||||
*Cloud Tools (3):*
|
||||
25. `cloud_upload` - Upload to S3/GCS/Azure
|
||||
26. `cloud_download` - Download from cloud storage
|
||||
|
||||
### How do I configure MCP for Claude Code?
|
||||
|
||||
**Add to `claude_desktop_config.json`:**
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"skill-seekers": {
|
||||
"command": "skill-seekers-mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Restart Claude Code**, then use:
|
||||
```
|
||||
Use skill-seekers MCP tools to scrape React documentation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Questions
|
||||
|
||||
### Can I use Skill Seekers programmatically?
|
||||
|
||||
Yes! Full API for Python integration:
|
||||
|
||||
```python
|
||||
from skill_seekers.cli.doc_scraper import scrape_all, build_skill
|
||||
from skill_seekers.cli.adaptors import get_adaptor
|
||||
|
||||
# Scrape documentation
|
||||
pages = scrape_all(
|
||||
base_url='https://docs.example.com',
|
||||
selectors={'main_content': 'article'},
|
||||
config={'name': 'example'}
|
||||
)
|
||||
|
||||
# Build skill
|
||||
skill_path = build_skill(
|
||||
config_name='example',
|
||||
output_dir='output/example'
|
||||
)
|
||||
|
||||
# Package for platform
|
||||
adaptor = get_adaptor('claude')
|
||||
package_path = adaptor.package(skill_path, 'output/')
|
||||
```
|
||||
|
||||
**See:** [API Reference](reference/API_REFERENCE.md)
|
||||
|
||||
### How do I create custom configurations?
|
||||
|
||||
**Create config file** (`configs/my-framework.json`):
|
||||
```json
|
||||
{
|
||||
"name": "my-framework",
|
||||
"description": "My custom framework documentation",
|
||||
"base_url": "https://docs.example.com/",
|
||||
"selectors": {
|
||||
"main_content": "article", // CSS selector
|
||||
"title": "h1",
|
||||
"code_blocks": "pre code"
|
||||
},
|
||||
"url_patterns": {
|
||||
"include": ["/docs/", "/api/"],
|
||||
"exclude": ["/blog/", "/changelog/"]
|
||||
},
|
||||
"categories": {
|
||||
"getting_started": ["intro", "quickstart"],
|
||||
"api": ["api", "reference"]
|
||||
},
|
||||
"rate_limit": 0.5,
|
||||
"max_pages": 500
|
||||
}
|
||||
```
|
||||
|
||||
**Use config:**
|
||||
```bash
|
||||
skill-seekers create --config configs/my-framework.json
|
||||
```
|
||||
|
||||
### Can I contribute preset configs?
|
||||
|
||||
Yes! We welcome config contributions:
|
||||
|
||||
1. **Create config** in `configs/` directory
|
||||
2. **Test it** thoroughly:
|
||||
```bash
|
||||
skill-seekers create --config configs/your-framework.json
|
||||
```
|
||||
3. **Submit PR** on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers)
|
||||
|
||||
**Guidelines:**
|
||||
- Name: `{framework-name}.json`
|
||||
- Include all required fields
|
||||
- Add to appropriate category
|
||||
- Test with real documentation
|
||||
|
||||
### How do I debug scraping issues?
|
||||
|
||||
```bash
|
||||
# Verbose output
|
||||
skill-seekers create --config react --verbose
|
||||
|
||||
# Dry run (no actual scraping)
|
||||
skill-seekers create --config react --dry-run
|
||||
|
||||
# Single page test
|
||||
skill-seekers create https://docs.example.com/intro --max-pages 1
|
||||
|
||||
# Check config (also available as the validate_config MCP tool)
|
||||
python -m skill_seekers.cli.config_validator configs/react.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting More Help
|
||||
|
||||
### Where can I find documentation?
|
||||
|
||||
**Main Documentation:**
|
||||
- [README](../README.md) - Project overview
|
||||
- [Usage Guide](guides/USAGE.md) - Detailed usage
|
||||
- [API Reference](reference/API_REFERENCE.md) - Programmatic usage
|
||||
- [Troubleshooting](../TROUBLESHOOTING.md) - Common issues
|
||||
|
||||
**Guides:**
|
||||
- [MCP Setup](guides/MCP_SETUP.md)
|
||||
- [Testing Guide](guides/TESTING_GUIDE.md)
|
||||
- [Migration Guide](guides/MIGRATION_GUIDE.md)
|
||||
- [Quick Reference](QUICK_REFERENCE.md)
|
||||
|
||||
### How do I report bugs?
|
||||
|
||||
1. **Check existing issues:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
2. **Create new issue** with:
|
||||
- Skill Seekers version (`skill-seekers --version`)
|
||||
- Python version (`python --version`)
|
||||
- Operating system
|
||||
- Config file (if relevant)
|
||||
- Error message and stack trace
|
||||
- Steps to reproduce
|
||||
|
||||
### How do I request features?
|
||||
|
||||
1. **Check roadmap:** [ROADMAP.md](../ROADMAP.md)
|
||||
2. **Create feature request:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
3. **Join discussions:** https://github.com/yusufkaraaslan/Skill_Seekers/discussions
|
||||
|
||||
### Is there a community?
|
||||
|
||||
Yes!
|
||||
- **GitHub Discussions:** https://github.com/yusufkaraaslan/Skill_Seekers/discussions
|
||||
- **Issue Tracker:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2
|
||||
|
||||
---
|
||||
|
||||
**Version:** 3.6.0
|
||||
**Last Updated:** 2026-03-15
|
||||
**Questions? Ask on [GitHub Discussions](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)**
|
||||
@@ -0,0 +1,933 @@
|
||||
# Kubernetes Deployment Guide
|
||||
|
||||
Complete guide for deploying Skill Seekers on Kubernetes.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Quick Start with Helm](#quick-start-with-helm)
|
||||
- [Manual Deployment](#manual-deployment)
|
||||
- [Configuration](#configuration)
|
||||
- [Scaling](#scaling)
|
||||
- [High Availability](#high-availability)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Ingress & Load Balancing](#ingress--load-balancing)
|
||||
- [Storage](#storage)
|
||||
- [Security](#security)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Kubernetes Cluster
|
||||
|
||||
**Minimum requirements:**
|
||||
- Kubernetes v1.21+
|
||||
- kubectl configured
|
||||
- 2 nodes (minimum)
|
||||
- 4 CPU cores total
|
||||
- 8 GB RAM total
|
||||
|
||||
**Cloud providers:**
|
||||
- **AWS:** EKS (Elastic Kubernetes Service)
|
||||
- **GCP:** GKE (Google Kubernetes Engine)
|
||||
- **Azure:** AKS (Azure Kubernetes Service)
|
||||
- **Local:** Minikube, kind, k3s
|
||||
|
||||
### 2. Required Tools
|
||||
|
||||
```bash
|
||||
# kubectl
|
||||
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
|
||||
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
|
||||
|
||||
# Helm 3
|
||||
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
|
||||
|
||||
# Verify installations
|
||||
kubectl version --client
|
||||
helm version
|
||||
```
|
||||
|
||||
### 3. Cluster Access
|
||||
|
||||
```bash
|
||||
# Verify cluster connection
|
||||
kubectl cluster-info
|
||||
kubectl get nodes
|
||||
|
||||
# Create namespace
|
||||
kubectl create namespace skillseekers
|
||||
kubectl config set-context --current --namespace=skillseekers
|
||||
```
|
||||
|
||||
## Quick Start with Helm
|
||||
|
||||
### 1. Install with Default Values
|
||||
|
||||
```bash
|
||||
# Add Helm repository (when available)
|
||||
helm repo add skillseekers https://charts.skillseekers.io
|
||||
helm repo update
|
||||
|
||||
# Install release
|
||||
helm install skillseekers skillseekers/skillseekers \
|
||||
--namespace skillseekers \
|
||||
--create-namespace
|
||||
|
||||
# Or install from local chart
|
||||
helm install skillseekers ./helm/skillseekers \
|
||||
--namespace skillseekers \
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
### 2. Install with Custom Values
|
||||
|
||||
```bash
|
||||
# Create values file
|
||||
cat > values-prod.yaml <<EOF
|
||||
replicaCount: 3
|
||||
|
||||
secrets:
|
||||
anthropicApiKey: "sk-ant-..."
|
||||
githubToken: "ghp_..."
|
||||
openaiApiKey: "sk-..."
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
hosts:
|
||||
- host: api.skillseekers.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls:
|
||||
- secretName: skillseekers-tls
|
||||
hosts:
|
||||
- api.skillseekers.example.com
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 70
|
||||
EOF
|
||||
|
||||
# Install with custom values
|
||||
helm install skillseekers ./helm/skillseekers \
|
||||
--namespace skillseekers \
|
||||
--create-namespace \
|
||||
--values values-prod.yaml
|
||||
```
|
||||
|
||||
### 3. Helm Commands
|
||||
|
||||
```bash
|
||||
# List releases
|
||||
helm list -n skillseekers
|
||||
|
||||
# Get status
|
||||
helm status skillseekers -n skillseekers
|
||||
|
||||
# Upgrade release
|
||||
helm upgrade skillseekers ./helm/skillseekers \
|
||||
--namespace skillseekers \
|
||||
--values values-prod.yaml
|
||||
|
||||
# Rollback
|
||||
helm rollback skillseekers 1 -n skillseekers
|
||||
|
||||
# Uninstall
|
||||
helm uninstall skillseekers -n skillseekers
|
||||
```
|
||||
|
||||
## Manual Deployment
|
||||
|
||||
### 1. Secrets
|
||||
|
||||
Create secrets for API keys:
|
||||
|
||||
```yaml
|
||||
# secrets.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: skillseekers-secrets
|
||||
namespace: skillseekers
|
||||
type: Opaque
|
||||
stringData:
|
||||
ANTHROPIC_API_KEY: "sk-ant-..."
|
||||
GITHUB_TOKEN: "ghp_..."
|
||||
OPENAI_API_KEY: "sk-..."
|
||||
VOYAGE_API_KEY: "..."
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f secrets.yaml
|
||||
```
|
||||
|
||||
### 2. ConfigMap
|
||||
|
||||
```yaml
|
||||
# configmap.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: skillseekers-config
|
||||
namespace: skillseekers
|
||||
data:
|
||||
MCP_TRANSPORT: "http"
|
||||
MCP_PORT: "8765"
|
||||
LOG_LEVEL: "INFO"
|
||||
CACHE_TTL: "86400"
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f configmap.yaml
|
||||
```
|
||||
|
||||
### 3. Deployment
|
||||
|
||||
```yaml
|
||||
# deployment.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
labels:
|
||||
app: skillseekers
|
||||
component: mcp-server
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: skillseekers
|
||||
component: mcp-server
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: skillseekers
|
||||
component: mcp-server
|
||||
spec:
|
||||
containers:
|
||||
- name: mcp-server
|
||||
image: skillseekers:2.9.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 8765
|
||||
name: http
|
||||
protocol: TCP
|
||||
env:
|
||||
- name: MCP_TRANSPORT
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillseekers-config
|
||||
key: MCP_TRANSPORT
|
||||
- name: MCP_PORT
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillseekers-config
|
||||
key: MCP_PORT
|
||||
- name: ANTHROPIC_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillseekers-secrets
|
||||
key: ANTHROPIC_API_KEY
|
||||
- name: GITHUB_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillseekers-secrets
|
||||
key: GITHUB_TOKEN
|
||||
resources:
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8765
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8765
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 2
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /app/data
|
||||
- name: cache
|
||||
mountPath: /app/cache
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: skillseekers-data
|
||||
- name: cache
|
||||
emptyDir: {}
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f deployment.yaml
|
||||
```
|
||||
|
||||
### 4. Service
|
||||
|
||||
```yaml
|
||||
# service.yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
labels:
|
||||
app: skillseekers
|
||||
component: mcp-server
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 8765
|
||||
targetPort: 8765
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
app: skillseekers
|
||||
component: mcp-server
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f service.yaml
|
||||
```
|
||||
|
||||
### 5. Verify Deployment
|
||||
|
||||
```bash
|
||||
# Check pods
|
||||
kubectl get pods -n skillseekers
|
||||
|
||||
# Check services
|
||||
kubectl get svc -n skillseekers
|
||||
|
||||
# Check logs
|
||||
kubectl logs -n skillseekers -l app=skillseekers --tail=100 -f
|
||||
|
||||
# Port forward for testing
|
||||
kubectl port-forward -n skillseekers svc/skillseekers-mcp 8765:8765
|
||||
|
||||
# Test endpoint
|
||||
curl http://localhost:8765/health
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. Resource Requests & Limits
|
||||
|
||||
```yaml
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m # Guaranteed CPU
|
||||
memory: 1Gi # Guaranteed memory
|
||||
limits:
|
||||
cpu: 2000m # Maximum CPU
|
||||
memory: 4Gi # Maximum memory
|
||||
```
|
||||
|
||||
### 2. Environment Variables
|
||||
|
||||
```yaml
|
||||
env:
|
||||
# From ConfigMap
|
||||
- name: LOG_LEVEL
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillseekers-config
|
||||
key: LOG_LEVEL
|
||||
|
||||
# From Secret
|
||||
- name: ANTHROPIC_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillseekers-secrets
|
||||
key: ANTHROPIC_API_KEY
|
||||
|
||||
# Direct value
|
||||
- name: MCP_TRANSPORT
|
||||
value: "http"
|
||||
```
|
||||
|
||||
### 3. Multi-Environment Setup
|
||||
|
||||
```bash
|
||||
# Development
|
||||
helm install skillseekers-dev ./helm/skillseekers \
|
||||
--namespace skillseekers-dev \
|
||||
--values values-dev.yaml
|
||||
|
||||
# Staging
|
||||
helm install skillseekers-staging ./helm/skillseekers \
|
||||
--namespace skillseekers-staging \
|
||||
--values values-staging.yaml
|
||||
|
||||
# Production
|
||||
helm install skillseekers-prod ./helm/skillseekers \
|
||||
--namespace skillseekers-prod \
|
||||
--values values-prod.yaml
|
||||
```
|
||||
|
||||
## Scaling
|
||||
|
||||
### 1. Manual Scaling
|
||||
|
||||
```bash
|
||||
# Scale deployment
|
||||
kubectl scale deployment skillseekers-mcp -n skillseekers --replicas=5
|
||||
|
||||
# Verify
|
||||
kubectl get pods -n skillseekers
|
||||
```
|
||||
|
||||
### 2. Horizontal Pod Autoscaler (HPA)
|
||||
|
||||
```yaml
|
||||
# hpa.yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: skillseekers-mcp
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 50
|
||||
periodSeconds: 60
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 0
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 100
|
||||
periodSeconds: 15
|
||||
- type: Pods
|
||||
value: 2
|
||||
periodSeconds: 15
|
||||
selectPolicy: Max
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f hpa.yaml
|
||||
|
||||
# Monitor autoscaling
|
||||
kubectl get hpa -n skillseekers --watch
|
||||
```
|
||||
|
||||
### 3. Vertical Pod Autoscaler (VPA)
|
||||
|
||||
```yaml
|
||||
# vpa.yaml
|
||||
apiVersion: autoscaling.k8s.io/v1
|
||||
kind: VerticalPodAutoscaler
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
spec:
|
||||
targetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: skillseekers-mcp
|
||||
updatePolicy:
|
||||
updateMode: "Auto"
|
||||
resourcePolicy:
|
||||
containerPolicies:
|
||||
- containerName: mcp-server
|
||||
minAllowed:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
maxAllowed:
|
||||
cpu: 4000m
|
||||
memory: 8Gi
|
||||
```
|
||||
|
||||
## High Availability
|
||||
|
||||
### 1. Pod Disruption Budget
|
||||
|
||||
```yaml
|
||||
# pdb.yaml
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: skillseekers
|
||||
component: mcp-server
|
||||
```
|
||||
|
||||
### 2. Pod Anti-Affinity
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
labelSelector:
|
||||
matchExpressions:
|
||||
- key: app
|
||||
operator: In
|
||||
values:
|
||||
- skillseekers
|
||||
topologyKey: kubernetes.io/hostname
|
||||
```
|
||||
|
||||
### 3. Node Affinity
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: node-role
|
||||
operator: In
|
||||
values:
|
||||
- worker
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 1
|
||||
preference:
|
||||
matchExpressions:
|
||||
- key: node-type
|
||||
operator: In
|
||||
values:
|
||||
- high-cpu
|
||||
```
|
||||
|
||||
### 4. Multi-Zone Deployment
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: topology.kubernetes.io/zone
|
||||
whenUnsatisfiable: DoNotSchedule
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: skillseekers
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### 1. Prometheus Metrics
|
||||
|
||||
```yaml
|
||||
# servicemonitor.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: skillseekers
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
path: /metrics
|
||||
```
|
||||
|
||||
### 2. Grafana Dashboard
|
||||
|
||||
```bash
|
||||
# Import dashboard
|
||||
kubectl apply -f grafana/dashboard.json
|
||||
```
|
||||
|
||||
### 3. Logging with Fluentd
|
||||
|
||||
```yaml
|
||||
# fluentd-configmap.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: fluentd-config
|
||||
data:
|
||||
fluent.conf: |
|
||||
<source>
|
||||
@type tail
|
||||
path /var/log/containers/skillseekers*.log
|
||||
pos_file /var/log/fluentd-skillseekers.pos
|
||||
tag kubernetes.*
|
||||
format json
|
||||
</source>
|
||||
<match **>
|
||||
@type elasticsearch
|
||||
host elasticsearch
|
||||
port 9200
|
||||
</match>
|
||||
```
|
||||
|
||||
## Ingress & Load Balancing
|
||||
|
||||
### 1. Nginx Ingress
|
||||
|
||||
```yaml
|
||||
# ingress.yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: skillseekers
|
||||
namespace: skillseekers
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: nginx
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
nginx.ingress.kubernetes.io/rate-limit: "100"
|
||||
nginx.ingress.kubernetes.io/ssl-redirect: "true"
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- api.skillseekers.example.com
|
||||
secretName: skillseekers-tls
|
||||
rules:
|
||||
- host: api.skillseekers.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: skillseekers-mcp
|
||||
port:
|
||||
number: 8765
|
||||
```
|
||||
|
||||
### 2. TLS with cert-manager
|
||||
|
||||
```bash
|
||||
# Install cert-manager
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
|
||||
|
||||
# Create ClusterIssuer
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-prod
|
||||
spec:
|
||||
acme:
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
email: admin@example.com
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-prod
|
||||
solvers:
|
||||
- http01:
|
||||
ingress:
|
||||
class: nginx
|
||||
EOF
|
||||
```
|
||||
|
||||
## Storage
|
||||
|
||||
### 1. Persistent Volume
|
||||
|
||||
```yaml
|
||||
# pv.yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: skillseekers-data
|
||||
spec:
|
||||
capacity:
|
||||
storage: 50Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: standard
|
||||
hostPath:
|
||||
path: /mnt/skillseekers-data
|
||||
```
|
||||
|
||||
### 2. Persistent Volume Claim
|
||||
|
||||
```yaml
|
||||
# pvc.yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: skillseekers-data
|
||||
namespace: skillseekers
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
storageClassName: standard
|
||||
```
|
||||
|
||||
### 3. StatefulSet (for stateful workloads)
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: skillseekers-cache
|
||||
spec:
|
||||
serviceName: skillseekers-cache
|
||||
replicas: 3
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: [ "ReadWriteOnce" ]
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
### 1. Network Policies
|
||||
|
||||
```yaml
|
||||
# networkpolicy.yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: skillseekers-mcp
|
||||
namespace: skillseekers
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: skillseekers
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: skillseekers
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8765
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443 # HTTPS
|
||||
- protocol: TCP
|
||||
port: 80 # HTTP
|
||||
```
|
||||
|
||||
### 2. Pod Security Policy
|
||||
|
||||
```yaml
|
||||
# psp.yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: skillseekers-restricted
|
||||
spec:
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
requiredDropCapabilities:
|
||||
- ALL
|
||||
volumes:
|
||||
- 'configMap'
|
||||
- 'emptyDir'
|
||||
- 'projected'
|
||||
- 'secret'
|
||||
- 'persistentVolumeClaim'
|
||||
runAsUser:
|
||||
rule: 'MustRunAsNonRoot'
|
||||
seLinux:
|
||||
rule: 'RunAsAny'
|
||||
fsGroup:
|
||||
rule: 'RunAsAny'
|
||||
```
|
||||
|
||||
### 3. RBAC
|
||||
|
||||
```yaml
|
||||
# rbac.yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: skillseekers
|
||||
namespace: skillseekers
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: skillseekers
|
||||
namespace: skillseekers
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps", "secrets"]
|
||||
verbs: ["get", "list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: skillseekers
|
||||
namespace: skillseekers
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: skillseekers
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: skillseekers
|
||||
namespace: skillseekers
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Pods Not Starting
|
||||
|
||||
```bash
|
||||
# Check pod status
|
||||
kubectl get pods -n skillseekers
|
||||
|
||||
# Describe pod
|
||||
kubectl describe pod <pod-name> -n skillseekers
|
||||
|
||||
# Check events
|
||||
kubectl get events -n skillseekers --sort-by='.lastTimestamp'
|
||||
|
||||
# Check logs
|
||||
kubectl logs <pod-name> -n skillseekers
|
||||
```
|
||||
|
||||
#### 2. Image Pull Errors
|
||||
|
||||
```bash
|
||||
# Check image pull secrets
|
||||
kubectl get secrets -n skillseekers
|
||||
|
||||
# Create image pull secret
|
||||
kubectl create secret docker-registry regcred \
|
||||
--docker-server=registry.example.com \
|
||||
--docker-username=user \
|
||||
--docker-password=password \
|
||||
-n skillseekers
|
||||
|
||||
# Use in pod spec
|
||||
spec:
|
||||
imagePullSecrets:
|
||||
- name: regcred
|
||||
```
|
||||
|
||||
#### 3. Resource Constraints
|
||||
|
||||
```bash
|
||||
# Check node resources
|
||||
kubectl top nodes
|
||||
|
||||
# Check pod resources
|
||||
kubectl top pods -n skillseekers
|
||||
|
||||
# Increase resources
|
||||
kubectl edit deployment skillseekers-mcp -n skillseekers
|
||||
```
|
||||
|
||||
#### 4. Service Not Accessible
|
||||
|
||||
```bash
|
||||
# Check service
|
||||
kubectl get svc -n skillseekers
|
||||
kubectl describe svc skillseekers-mcp -n skillseekers
|
||||
|
||||
# Check endpoints
|
||||
kubectl get endpoints -n skillseekers
|
||||
|
||||
# Port forward
|
||||
kubectl port-forward svc/skillseekers-mcp 8765:8765 -n skillseekers
|
||||
```
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# Execute command in pod
|
||||
kubectl exec -it <pod-name> -n skillseekers -- /bin/bash
|
||||
|
||||
# Copy files from pod
|
||||
kubectl cp skillseekers/<pod-name>:/app/data ./data
|
||||
|
||||
# Check pod networking
|
||||
kubectl exec <pod-name> -n skillseekers -- nslookup google.com
|
||||
|
||||
# View full pod spec
|
||||
kubectl get pod <pod-name> -n skillseekers -o yaml
|
||||
|
||||
# Restart deployment
|
||||
kubectl rollout restart deployment skillseekers-mcp -n skillseekers
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always set resource requests and limits**
|
||||
2. **Use namespaces for environment separation**
|
||||
3. **Enable autoscaling for variable workloads**
|
||||
4. **Implement health checks (liveness & readiness)**
|
||||
5. **Use Secrets for sensitive data**
|
||||
6. **Enable monitoring and logging**
|
||||
7. **Implement Pod Disruption Budgets for HA**
|
||||
8. **Use RBAC for access control**
|
||||
9. **Enable Network Policies**
|
||||
10. **Regular backup of persistent volumes**
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Review [PRODUCTION_DEPLOYMENT.md](./PRODUCTION_DEPLOYMENT.md) for general guidelines
|
||||
- See [DOCKER_DEPLOYMENT.md](./DOCKER_DEPLOYMENT.md) for container-specific details
|
||||
- Check [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common issues
|
||||
|
||||
---
|
||||
|
||||
**Need help?** Open an issue on [GitHub](https://github.com/yusufkaraaslan/Skill_Seekers/issues).
|
||||
@@ -0,0 +1,827 @@
|
||||
# Production Deployment Guide
|
||||
|
||||
Complete guide for deploying Skill Seekers in production environments.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Installation](#installation)
|
||||
- [Configuration](#configuration)
|
||||
- [Deployment Options](#deployment-options)
|
||||
- [Monitoring & Observability](#monitoring--observability)
|
||||
- [Security](#security)
|
||||
- [Scaling](#scaling)
|
||||
- [Backup & Disaster Recovery](#backup--disaster-recovery)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### System Requirements
|
||||
|
||||
**Minimum:**
|
||||
- CPU: 2 cores
|
||||
- RAM: 4 GB
|
||||
- Disk: 10 GB
|
||||
- Python: 3.10+
|
||||
|
||||
**Recommended (for production):**
|
||||
- CPU: 4+ cores
|
||||
- RAM: 8+ GB
|
||||
- Disk: 50+ GB SSD
|
||||
- Python: 3.12+
|
||||
|
||||
### Dependencies
|
||||
|
||||
**Required:**
|
||||
```bash
|
||||
# System packages (Ubuntu/Debian)
|
||||
sudo apt update
|
||||
sudo apt install -y python3.12 python3.12-venv python3-pip \
|
||||
git curl wget build-essential libssl-dev
|
||||
|
||||
# System packages (RHEL/CentOS)
|
||||
sudo yum install -y python312 python312-devel git curl wget \
|
||||
gcc gcc-c++ openssl-devel
|
||||
```
|
||||
|
||||
**Optional (for specific features):**
|
||||
```bash
|
||||
# OCR support (PDF scraping)
|
||||
sudo apt install -y tesseract-ocr
|
||||
|
||||
# Cloud storage
|
||||
# (Install provider-specific SDKs via pip)
|
||||
|
||||
# Embedding generation
|
||||
# (GPU support requires CUDA)
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Production Installation
|
||||
|
||||
```bash
|
||||
# Create dedicated user
|
||||
sudo useradd -m -s /bin/bash skillseekers
|
||||
sudo su - skillseekers
|
||||
|
||||
# Create virtual environment
|
||||
python3.12 -m venv /opt/skillseekers/venv
|
||||
source /opt/skillseekers/venv/bin/activate
|
||||
|
||||
# Install package
|
||||
pip install --upgrade pip
|
||||
pip install skill-seekers[all]
|
||||
|
||||
# Verify installation
|
||||
skill-seekers --version
|
||||
```
|
||||
|
||||
### 2. Configuration Directory
|
||||
|
||||
```bash
|
||||
# Create config directory
|
||||
mkdir -p ~/.config/skill-seekers/{configs,output,logs,cache}
|
||||
|
||||
# Set permissions
|
||||
chmod 700 ~/.config/skill-seekers
|
||||
```
|
||||
|
||||
### 3. Environment Variables
|
||||
|
||||
Create `/opt/skillseekers/.env`:
|
||||
|
||||
```bash
|
||||
# API Keys
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
GOOGLE_API_KEY=AIza...
|
||||
OPENAI_API_KEY=sk-...
|
||||
VOYAGE_API_KEY=...
|
||||
|
||||
# GitHub Tokens (use skill-seekers config --github for multiple)
|
||||
GITHUB_TOKEN=ghp_...
|
||||
|
||||
# Cloud Storage (optional)
|
||||
AWS_ACCESS_KEY_ID=...
|
||||
AWS_SECRET_ACCESS_KEY=...
|
||||
GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs-key.json
|
||||
AZURE_STORAGE_CONNECTION_STRING=...
|
||||
|
||||
# MCP Server
|
||||
MCP_TRANSPORT=http
|
||||
MCP_PORT=8765
|
||||
|
||||
# Sync Monitoring (optional)
|
||||
SYNC_WEBHOOK_URL=https://...
|
||||
SLACK_WEBHOOK_URL=https://hooks.slack.com/...
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
LOG_FILE=/var/log/skillseekers/app.log
|
||||
```
|
||||
|
||||
**Security Note:** Never commit `.env` files to version control!
|
||||
|
||||
```bash
|
||||
# Secure the env file
|
||||
chmod 600 /opt/skillseekers/.env
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### 1. GitHub Configuration
|
||||
|
||||
Use the interactive configuration wizard:
|
||||
|
||||
```bash
|
||||
skill-seekers config --github
|
||||
```
|
||||
|
||||
This will:
|
||||
- Add GitHub personal access tokens
|
||||
- Configure rate limit strategies
|
||||
- Test token validity
|
||||
- Support multiple profiles (work, personal, etc.)
|
||||
|
||||
### 2. API Keys Configuration
|
||||
|
||||
```bash
|
||||
skill-seekers config --api-keys
|
||||
```
|
||||
|
||||
Configure:
|
||||
- Claude API (Anthropic)
|
||||
- Gemini API (Google)
|
||||
- OpenAI API
|
||||
- Voyage AI (embeddings)
|
||||
|
||||
### 3. Connection Testing
|
||||
|
||||
```bash
|
||||
skill-seekers config --test
|
||||
```
|
||||
|
||||
Verifies:
|
||||
- ✅ GitHub token(s) validity and rate limits
|
||||
- ✅ Claude API connectivity
|
||||
- ✅ Gemini API connectivity
|
||||
- ✅ OpenAI API connectivity
|
||||
- ✅ Cloud storage access (if configured)
|
||||
|
||||
## Deployment Options
|
||||
|
||||
### Option 1: Systemd Service (Recommended)
|
||||
|
||||
Create `/etc/systemd/system/skillseekers-mcp.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Skill Seekers MCP Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=skillseekers
|
||||
Group=skillseekers
|
||||
WorkingDirectory=/opt/skillseekers
|
||||
EnvironmentFile=/opt/skillseekers/.env
|
||||
ExecStart=/opt/skillseekers/venv/bin/python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=skillseekers-mcp
|
||||
|
||||
# Security
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/opt/skillseekers /var/log/skillseekers
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**Enable and start:**
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable skillseekers-mcp
|
||||
sudo systemctl start skillseekers-mcp
|
||||
sudo systemctl status skillseekers-mcp
|
||||
```
|
||||
|
||||
### Option 2: Docker Deployment
|
||||
|
||||
See [Docker Deployment Guide](./DOCKER_DEPLOYMENT.md) for detailed instructions.
|
||||
|
||||
**Quick Start:**
|
||||
|
||||
```bash
|
||||
# Build image
|
||||
docker build -t skillseekers:latest .
|
||||
|
||||
# Run container
|
||||
docker run -d \
|
||||
--name skillseekers-mcp \
|
||||
-p 8765:8765 \
|
||||
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
|
||||
-e GITHUB_TOKEN=$GITHUB_TOKEN \
|
||||
-v /opt/skillseekers/data:/app/data \
|
||||
--restart unless-stopped \
|
||||
skillseekers:latest
|
||||
```
|
||||
|
||||
### Option 3: Kubernetes Deployment
|
||||
|
||||
See [Kubernetes Deployment Guide](./KUBERNETES_DEPLOYMENT.md) for detailed instructions.
|
||||
|
||||
**Quick Start:**
|
||||
|
||||
```bash
|
||||
# Install with Helm
|
||||
helm install skillseekers ./helm/skillseekers \
|
||||
--namespace skillseekers \
|
||||
--create-namespace \
|
||||
--set secrets.anthropicApiKey=$ANTHROPIC_API_KEY \
|
||||
--set secrets.githubToken=$GITHUB_TOKEN
|
||||
```
|
||||
|
||||
### Option 4: Docker Compose
|
||||
|
||||
See [Docker Compose Guide](./DOCKER_COMPOSE.md) for multi-service deployment.
|
||||
|
||||
```bash
|
||||
# Start all services
|
||||
docker-compose up -d
|
||||
|
||||
# Check status
|
||||
docker-compose ps
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### 1. Health Checks
|
||||
|
||||
**MCP Server Health:**
|
||||
|
||||
```bash
|
||||
# HTTP transport
|
||||
curl http://localhost:8765/health
|
||||
|
||||
# Expected response:
|
||||
{
|
||||
"status": "healthy",
|
||||
"version": "2.9.0",
|
||||
"uptime": 3600,
|
||||
"tools": 25
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Logging
|
||||
|
||||
**Configure structured logging:**
|
||||
|
||||
```python
|
||||
# config/logging.yaml
|
||||
version: 1
|
||||
formatters:
|
||||
json:
|
||||
format: '{"time":"%(asctime)s","level":"%(levelname)s","msg":"%(message)s"}'
|
||||
handlers:
|
||||
file:
|
||||
class: logging.handlers.RotatingFileHandler
|
||||
filename: /var/log/skillseekers/app.log
|
||||
maxBytes: 10485760 # 10MB
|
||||
backupCount: 5
|
||||
formatter: json
|
||||
loggers:
|
||||
skill_seekers:
|
||||
level: INFO
|
||||
handlers: [file]
|
||||
```
|
||||
|
||||
**Log aggregation options:**
|
||||
- **ELK Stack:** Elasticsearch + Logstash + Kibana
|
||||
- **Grafana Loki:** Lightweight log aggregation
|
||||
- **CloudWatch Logs:** For AWS deployments
|
||||
- **Stackdriver:** For GCP deployments
|
||||
|
||||
### 3. Metrics
|
||||
|
||||
**Prometheus metrics endpoint:**
|
||||
|
||||
```bash
|
||||
# Add to MCP server
|
||||
from prometheus_client import start_http_server, Counter, Histogram
|
||||
|
||||
# Metrics
|
||||
scraping_requests = Counter('scraping_requests_total', 'Total scraping requests')
|
||||
scraping_duration = Histogram('scraping_duration_seconds', 'Scraping duration')
|
||||
|
||||
# Start metrics server
|
||||
start_http_server(9090)
|
||||
```
|
||||
|
||||
**Key metrics to monitor:**
|
||||
- Request rate
|
||||
- Response time (p50, p95, p99)
|
||||
- Error rate
|
||||
- Memory usage
|
||||
- CPU usage
|
||||
- Disk I/O
|
||||
- GitHub API rate limit remaining
|
||||
- Claude API token usage
|
||||
|
||||
### 4. Alerting
|
||||
|
||||
**Example Prometheus alert rules:**
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: skillseekers
|
||||
rules:
|
||||
- alert: HighErrorRate
|
||||
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
|
||||
for: 5m
|
||||
annotations:
|
||||
summary: "High error rate detected"
|
||||
|
||||
- alert: HighMemoryUsage
|
||||
expr: process_resident_memory_bytes > 2e9 # 2GB
|
||||
for: 10m
|
||||
annotations:
|
||||
summary: "Memory usage above 2GB"
|
||||
|
||||
- alert: GitHubRateLimitLow
|
||||
expr: github_rate_limit_remaining < 100
|
||||
for: 1m
|
||||
annotations:
|
||||
summary: "GitHub rate limit low"
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
### 1. API Key Management
|
||||
|
||||
**Best Practices:**
|
||||
|
||||
✅ **DO:**
|
||||
- Store keys in environment variables or secret managers
|
||||
- Use different keys for dev/staging/prod
|
||||
- Rotate keys regularly (quarterly minimum)
|
||||
- Use least-privilege IAM roles for cloud services
|
||||
- Monitor key usage for anomalies
|
||||
|
||||
❌ **DON'T:**
|
||||
- Commit keys to version control
|
||||
- Share keys via email/Slack
|
||||
- Use production keys in development
|
||||
- Grant overly broad permissions
|
||||
|
||||
**Recommended Secret Managers:**
|
||||
- **Kubernetes Secrets** (for K8s deployments)
|
||||
- **AWS Secrets Manager** (for AWS)
|
||||
- **Google Secret Manager** (for GCP)
|
||||
- **Azure Key Vault** (for Azure)
|
||||
- **HashiCorp Vault** (cloud-agnostic)
|
||||
|
||||
### 2. Network Security
|
||||
|
||||
**Firewall Rules:**
|
||||
|
||||
```bash
|
||||
# Allow only necessary ports
|
||||
sudo ufw enable
|
||||
sudo ufw allow 22/tcp # SSH
|
||||
sudo ufw allow 8765/tcp # MCP server (if public)
|
||||
sudo ufw deny incoming
|
||||
sudo ufw allow outgoing
|
||||
```
|
||||
|
||||
**Reverse Proxy (Nginx):**
|
||||
|
||||
```nginx
|
||||
# /etc/nginx/sites-available/skillseekers
|
||||
server {
|
||||
listen 80;
|
||||
server_name api.skillseekers.example.com;
|
||||
|
||||
# Redirect to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name api.skillseekers.example.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/api.skillseekers.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/api.skillseekers.example.com/privkey.pem;
|
||||
|
||||
# Security headers
|
||||
add_header Strict-Transport-Security "max-age=31536000" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
|
||||
# Rate limiting
|
||||
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
|
||||
limit_req zone=api burst=20 nodelay;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:8765;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Timeouts
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. TLS/SSL
|
||||
|
||||
**Let's Encrypt (free certificates):**
|
||||
|
||||
```bash
|
||||
# Install certbot
|
||||
sudo apt install certbot python3-certbot-nginx
|
||||
|
||||
# Obtain certificate
|
||||
sudo certbot --nginx -d api.skillseekers.example.com
|
||||
|
||||
# Auto-renewal (cron)
|
||||
0 12 * * * /usr/bin/certbot renew --quiet
|
||||
```
|
||||
|
||||
### 4. Authentication & Authorization
|
||||
|
||||
**API Key Authentication (optional):**
|
||||
|
||||
```python
|
||||
# Add to MCP server
|
||||
from fastapi import Security, HTTPException
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
security = HTTPBearer()
|
||||
|
||||
async def verify_token(credentials: HTTPAuthorizationCredentials = Security(security)):
|
||||
token = credentials.credentials
|
||||
if token != os.getenv("API_SECRET_KEY"):
|
||||
raise HTTPException(status_code=401, detail="Invalid token")
|
||||
return token
|
||||
```
|
||||
|
||||
## Scaling
|
||||
|
||||
### 1. Vertical Scaling
|
||||
|
||||
**Increase resources:**
|
||||
|
||||
```yaml
|
||||
# Kubernetes resource limits
|
||||
resources:
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: "4Gi"
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: "8Gi"
|
||||
```
|
||||
|
||||
### 2. Horizontal Scaling
|
||||
|
||||
**Deploy multiple instances:**
|
||||
|
||||
```bash
|
||||
# Kubernetes HPA (Horizontal Pod Autoscaler)
|
||||
kubectl autoscale deployment skillseekers-mcp \
|
||||
--cpu-percent=70 \
|
||||
--min=2 \
|
||||
--max=10
|
||||
```
|
||||
|
||||
**Load Balancing:**
|
||||
|
||||
```nginx
|
||||
# Nginx load balancer
|
||||
upstream skillseekers {
|
||||
least_conn;
|
||||
server 10.0.0.1:8765;
|
||||
server 10.0.0.2:8765;
|
||||
server 10.0.0.3:8765;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
location / {
|
||||
proxy_pass http://skillseekers;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Database/Storage Scaling
|
||||
|
||||
**Distributed caching:**
|
||||
|
||||
```python
|
||||
# Redis for distributed cache
|
||||
import redis
|
||||
|
||||
cache = redis.Redis(host='redis.example.com', port=6379, db=0)
|
||||
```
|
||||
|
||||
**Object storage:**
|
||||
- Use S3/GCS/Azure Blob for skill packages
|
||||
- Enable CDN for static assets
|
||||
- Use read replicas for databases
|
||||
|
||||
### 4. Rate Limit Management
|
||||
|
||||
**Multiple GitHub tokens:**
|
||||
|
||||
```bash
|
||||
# Configure multiple profiles
|
||||
skill-seekers config --github
|
||||
|
||||
# Automatic token rotation on rate limit
|
||||
# (handled by rate_limit_handler.py)
|
||||
```
|
||||
|
||||
## Backup & Disaster Recovery
|
||||
|
||||
### 1. Data Backup
|
||||
|
||||
**What to backup:**
|
||||
- Configuration files (`~/.config/skill-seekers/`)
|
||||
- Generated skills (`output/`)
|
||||
- Database/cache (if applicable)
|
||||
- Logs (for forensics)
|
||||
|
||||
**Backup script:**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# /opt/skillseekers/scripts/backup.sh
|
||||
|
||||
BACKUP_DIR="/backups/skillseekers"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Create backup
|
||||
tar -czf "$BACKUP_DIR/backup_$TIMESTAMP.tar.gz" \
|
||||
~/.config/skill-seekers \
|
||||
/opt/skillseekers/output \
|
||||
/opt/skillseekers/.env
|
||||
|
||||
# Retain last 30 days
|
||||
find "$BACKUP_DIR" -name "backup_*.tar.gz" -mtime +30 -delete
|
||||
|
||||
# Upload to S3 (optional)
|
||||
aws s3 cp "$BACKUP_DIR/backup_$TIMESTAMP.tar.gz" \
|
||||
s3://backups/skillseekers/
|
||||
```
|
||||
|
||||
**Schedule backups:**
|
||||
|
||||
```bash
|
||||
# Crontab
|
||||
0 2 * * * /opt/skillseekers/scripts/backup.sh
|
||||
```
|
||||
|
||||
### 2. Disaster Recovery Plan
|
||||
|
||||
**Recovery steps:**
|
||||
|
||||
1. **Provision new infrastructure**
|
||||
```bash
|
||||
# Deploy from backup
|
||||
terraform apply
|
||||
```
|
||||
|
||||
2. **Restore configuration**
|
||||
```bash
|
||||
tar -xzf backup_20250207.tar.gz -C /
|
||||
```
|
||||
|
||||
3. **Verify services**
|
||||
```bash
|
||||
skill-seekers config --test
|
||||
systemctl status skillseekers-mcp
|
||||
```
|
||||
|
||||
4. **Test functionality**
|
||||
```bash
|
||||
skill-seekers create --config configs/test.json --max-pages 10
|
||||
```
|
||||
|
||||
**RTO/RPO targets:**
|
||||
- **RTO (Recovery Time Objective):** < 2 hours
|
||||
- **RPO (Recovery Point Objective):** < 24 hours
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. High Memory Usage
|
||||
|
||||
**Symptoms:**
|
||||
- OOM kills
|
||||
- Slow performance
|
||||
- Swapping
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Check memory usage
|
||||
ps aux --sort=-%mem | head -10
|
||||
|
||||
# Reduce parallel workers
|
||||
skill-seekers create --config config.json --workers 2
|
||||
|
||||
# Enable memory limits
|
||||
docker run --memory=4g skillseekers:latest
|
||||
```
|
||||
|
||||
#### 2. GitHub Rate Limits
|
||||
|
||||
**Symptoms:**
|
||||
- `403 Forbidden` errors
|
||||
- "API rate limit exceeded" messages
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Check rate limit
|
||||
curl -H "Authorization: token $GITHUB_TOKEN" \
|
||||
https://api.github.com/rate_limit
|
||||
|
||||
# Add more tokens
|
||||
skill-seekers config --github
|
||||
|
||||
# Use rate limit strategy
|
||||
# (automatic with multi-token config)
|
||||
```
|
||||
|
||||
#### 3. Slow Scraping
|
||||
|
||||
**Symptoms:**
|
||||
- Long scraping times
|
||||
- Timeouts
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Enable async scraping (2-3x faster)
|
||||
skill-seekers create --config config.json --async
|
||||
|
||||
# Increase concurrency
|
||||
# (adjust in config: "concurrency": 10)
|
||||
|
||||
# Re-runs reuse cached data automatically (use --fresh to discard)
|
||||
skill-seekers create --config config.json
|
||||
```
|
||||
|
||||
#### 4. API Errors
|
||||
|
||||
**Symptoms:**
|
||||
- `401 Unauthorized`
|
||||
- `429 Too Many Requests`
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Verify API keys
|
||||
skill-seekers config --test
|
||||
|
||||
# Check API key validity
|
||||
# Claude API: https://console.anthropic.com/
|
||||
# OpenAI: https://platform.openai.com/api-keys
|
||||
# Google: https://console.cloud.google.com/apis/credentials
|
||||
|
||||
# Rotate keys if compromised
|
||||
```
|
||||
|
||||
#### 5. Service Won't Start
|
||||
|
||||
**Symptoms:**
|
||||
- systemd service fails
|
||||
- Container exits immediately
|
||||
|
||||
**Solutions:**
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
journalctl -u skillseekers-mcp -n 100
|
||||
|
||||
# Or for Docker
|
||||
docker logs skillseekers-mcp
|
||||
|
||||
# Common causes:
|
||||
# - Missing environment variables
|
||||
# - Port already in use
|
||||
# - Permission issues
|
||||
|
||||
# Verify config
|
||||
skill-seekers config --show
|
||||
```
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable detailed logging:
|
||||
|
||||
```bash
|
||||
# Set debug level
|
||||
export LOG_LEVEL=DEBUG
|
||||
|
||||
# Run with verbose output
|
||||
skill-seekers create --config config.json --verbose
|
||||
```
|
||||
|
||||
### Getting Help
|
||||
|
||||
**Community Support:**
|
||||
- GitHub Issues: https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
- Documentation: https://skillseekersweb.com/
|
||||
|
||||
**Log Collection:**
|
||||
|
||||
```bash
|
||||
# Collect diagnostic info
|
||||
tar -czf skillseekers-debug.tar.gz \
|
||||
/var/log/skillseekers/ \
|
||||
~/.config/skill-seekers/configs/ \
|
||||
/opt/skillseekers/.env
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### 1. Scraping Performance
|
||||
|
||||
**Optimization techniques:**
|
||||
|
||||
```python
|
||||
# Enable async scraping
|
||||
"async_scraping": true,
|
||||
"concurrency": 20, # Adjust based on resources
|
||||
|
||||
# Optimize selectors
|
||||
"selectors": {
|
||||
"main_content": "article", # More specific = faster
|
||||
"code_blocks": "pre code"
|
||||
}
|
||||
|
||||
# Enable caching
|
||||
"use_cache": true,
|
||||
"cache_ttl": 86400 # 24 hours
|
||||
```
|
||||
|
||||
### 2. Embedding Performance
|
||||
|
||||
**GPU acceleration (if available):**
|
||||
|
||||
```python
|
||||
# Use GPU for sentence-transformers
|
||||
pip install sentence-transformers[gpu]
|
||||
|
||||
# Configure
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
```
|
||||
|
||||
**Batch processing:**
|
||||
|
||||
```python
|
||||
# Generate embeddings in batches
|
||||
generator.generate_batch(texts, batch_size=32)
|
||||
```
|
||||
|
||||
### 3. Storage Performance
|
||||
|
||||
**Use SSD for:**
|
||||
- SQLite databases
|
||||
- Cache directories
|
||||
- Log files
|
||||
|
||||
**Use object storage for:**
|
||||
- Skill packages
|
||||
- Backup archives
|
||||
- Large datasets
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Review** deployment option that fits your infrastructure
|
||||
2. **Configure** monitoring and alerting
|
||||
3. **Set up** backups and disaster recovery
|
||||
4. **Test** failover procedures
|
||||
5. **Document** your specific deployment
|
||||
6. **Train** your team on operations
|
||||
|
||||
---
|
||||
|
||||
**Need help?** See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) or open an issue on GitHub.
|
||||
@@ -0,0 +1,231 @@
|
||||
# Skill Seekers Documentation
|
||||
|
||||
> **Complete documentation for Skill Seekers v3.7.0**
|
||||
|
||||
---
|
||||
|
||||
## Welcome!
|
||||
|
||||
This is the official documentation for **Skill Seekers** - the universal tool for converting **18 source types** (documentation sites, GitHub repos, PDFs, videos, Word docs, EPUB books, Jupyter notebooks, local HTML, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence, Notion, Slack/Discord, and local codebases) into AI-ready skills for 21+ platforms.
|
||||
|
||||
---
|
||||
|
||||
## Where Should I Start?
|
||||
|
||||
### 🚀 I'm New Here
|
||||
|
||||
Start with our **Getting Started** guides:
|
||||
|
||||
1. [Installation](getting-started/01-installation.md) - Install Skill Seekers
|
||||
2. [Quick Start](getting-started/02-quick-start.md) - Create your first skill in 3 commands
|
||||
3. [Your First Skill](getting-started/03-your-first-skill.md) - Complete walkthrough
|
||||
4. [Next Steps](getting-started/04-next-steps.md) - Where to go from here
|
||||
5. [Scan a Project](getting-started/05-scan-a-project.md) - Bootstrap configs from a codebase
|
||||
|
||||
### 📖 I Want to Learn
|
||||
|
||||
Explore our **User Guides**:
|
||||
|
||||
- [Core Concepts](user-guide/01-core-concepts.md) - How Skill Seekers works
|
||||
- [Scraping Guide](user-guide/02-scraping.md) - All scraping options
|
||||
- [Enhancement Guide](user-guide/03-enhancement.md) - AI enhancement explained
|
||||
- [Packaging Guide](user-guide/04-packaging.md) - Export to platforms
|
||||
- [Workflows Guide](user-guide/05-workflows.md) - Enhancement workflows
|
||||
- [Troubleshooting](user-guide/06-troubleshooting.md) - Common issues
|
||||
|
||||
### 📚 I Need Reference
|
||||
|
||||
Look up specific information:
|
||||
|
||||
- [CLI Reference](reference/CLI_REFERENCE.md) - All 19 commands
|
||||
- [MCP Reference](reference/MCP_REFERENCE.md) - 40 MCP tools
|
||||
- [Config Format](reference/CONFIG_FORMAT.md) - JSON specification
|
||||
- [Environment Variables](reference/ENVIRONMENT_VARIABLES.md) - All env vars
|
||||
- [FAQ](FAQ.md) - Frequently asked questions
|
||||
- [Troubleshooting](TROUBLESHOOTING.md) - Full troubleshooting reference
|
||||
|
||||
### 🚀 I'm Ready for Advanced Topics
|
||||
|
||||
Power user features:
|
||||
|
||||
- [MCP Server Setup](advanced/mcp-server.md) - MCP integration
|
||||
- [MCP Tools Deep Dive](advanced/mcp-server.md) - Advanced MCP usage
|
||||
- [Custom Workflows](advanced/custom-workflows.md) - Create workflows
|
||||
- [Multi-Source Scraping](advanced/multi-source.md) - Combine sources
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### The 3 Commands
|
||||
|
||||
```bash
|
||||
# 1. Install
|
||||
pip install skill-seekers
|
||||
|
||||
# 2. Create skill
|
||||
skill-seekers create https://docs.django.com/
|
||||
|
||||
# 3. Package for Claude
|
||||
skill-seekers package output/django --target claude
|
||||
```
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# Create from any source (auto-detects type)
|
||||
skill-seekers create https://docs.django.com/
|
||||
skill-seekers create facebook/react
|
||||
skill-seekers create manual.pdf
|
||||
skill-seekers create notebook.ipynb
|
||||
|
||||
# Scan a project for tech stack — emits one config per framework
|
||||
skill-seekers scan ./my-react-app --out ./configs/scanned/
|
||||
|
||||
# Enhance skill
|
||||
skill-seekers enhance output/my-skill/
|
||||
|
||||
# Package for platform
|
||||
skill-seekers package output/my-skill/ --target claude
|
||||
|
||||
# Upload
|
||||
skill-seekers upload output/my-skill-claude.zip
|
||||
|
||||
# Install complete workflow
|
||||
skill-seekers install --config react --target claude
|
||||
|
||||
# Doctor / diagnostics
|
||||
skill-seekers doctor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── README.md # This file - start here
|
||||
├── ARCHITECTURE.md # How docs are organized
|
||||
├── UML_ARCHITECTURE.md # Software architecture (UML diagrams)
|
||||
├── UNIFICATION_PLAN.md # Grand Unification refactor plan + phase results
|
||||
├── BUG_AUDIT.md # Full-codebase bug audit (historical record)
|
||||
│
|
||||
├── getting-started/ # For new users
|
||||
│ ├── 01-installation.md
|
||||
│ ├── 02-quick-start.md
|
||||
│ ├── 03-your-first-skill.md
|
||||
│ ├── 04-next-steps.md
|
||||
│ └── 05-scan-a-project.md
|
||||
│
|
||||
├── user-guide/ # Common tasks
|
||||
│ ├── 01-core-concepts.md
|
||||
│ ├── 02-scraping.md
|
||||
│ ├── 03-enhancement.md
|
||||
│ ├── 04-packaging.md
|
||||
│ ├── 05-workflows.md
|
||||
│ └── 06-troubleshooting.md
|
||||
│
|
||||
├── guides/ # How-to guides
|
||||
│ ├── MCP_SETUP.md
|
||||
│ ├── MIGRATION_GUIDE.md
|
||||
│ ├── TESTING_GUIDE.md
|
||||
│ └── UPLOAD_GUIDE.md
|
||||
│
|
||||
├── integrations/ # Platform integrations
|
||||
│ ├── LANGCHAIN.md
|
||||
│ ├── LLAMA_INDEX.md
|
||||
│ ├── CURSOR.md
|
||||
│ └── ...
|
||||
│
|
||||
├── features/ # Feature deep-dives
|
||||
│ ├── BOOTSTRAP_SKILL.md
|
||||
│ ├── UNIFIED_SCRAPING.md
|
||||
│ └── ENHANCEMENT.md
|
||||
│
|
||||
├── reference/ # Technical reference
|
||||
│ ├── CLI_REFERENCE.md # 19 commands
|
||||
│ ├── MCP_REFERENCE.md # 40 MCP tools
|
||||
│ ├── CONFIG_FORMAT.md # JSON spec
|
||||
│ └── ENVIRONMENT_VARIABLES.md
|
||||
│
|
||||
├── advanced/ # Power user topics
|
||||
│ ├── mcp-server.md
|
||||
│ ├── custom-workflows.md
|
||||
│ └── multi-source.md
|
||||
│
|
||||
├── archive/ # Legacy docs
|
||||
├── blog/ # Blog posts
|
||||
├── case-studies/ # Case studies
|
||||
├── plans/ # Feature plans
|
||||
├── roadmap/ # Roadmap
|
||||
├── strategy/ # Strategy docs
|
||||
└── zh-CN/ # Chinese translations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## By Use Case
|
||||
|
||||
### I Want to Build AI Skills
|
||||
|
||||
For Claude, Gemini, ChatGPT:
|
||||
|
||||
1. [Quick Start](getting-started/02-quick-start.md)
|
||||
2. [Enhancement Guide](user-guide/03-enhancement.md)
|
||||
3. [Workflows Guide](user-guide/05-workflows.md)
|
||||
|
||||
### I Want to Build RAG Pipelines
|
||||
|
||||
For LangChain, LlamaIndex, vector DBs:
|
||||
|
||||
1. [Core Concepts](user-guide/01-core-concepts.md)
|
||||
2. [Packaging Guide](user-guide/04-packaging.md)
|
||||
3. [MCP Reference](reference/MCP_REFERENCE.md)
|
||||
|
||||
### I Want AI Coding Assistance
|
||||
|
||||
For Cursor, Windsurf, Cline, Roo, Aider, Bolt, Kilo, Continue, Kimi Code:
|
||||
|
||||
1. [Your First Skill](getting-started/03-your-first-skill.md)
|
||||
2. [Local Codebase Analysis](user-guide/02-scraping.md#local-codebase-analysis)
|
||||
3. `skill-seekers install-agent --agent cursor`
|
||||
|
||||
---
|
||||
|
||||
## Version Information
|
||||
|
||||
- **Current Version:** 3.7.0
|
||||
- **Last Updated:** 2026-06-11
|
||||
- **Source Types:** 18
|
||||
- **Python Required:** 3.10+
|
||||
|
||||
---
|
||||
|
||||
## Contributing to Documentation
|
||||
|
||||
Found an issue? Want to improve docs?
|
||||
|
||||
1. Edit files in the `docs/` directory
|
||||
2. Follow the existing structure
|
||||
3. Submit a PR
|
||||
|
||||
See [Contributing Guide](../CONTRIBUTING.md) for details.
|
||||
|
||||
---
|
||||
|
||||
## External Links
|
||||
|
||||
- **Main Repository:** https://github.com/yusufkaraaslan/Skill_Seekers
|
||||
- **Website:** https://skillseekersweb.com/
|
||||
- **PyPI:** https://pypi.org/project/skill-seekers/
|
||||
- **Issues:** https://github.com/yusufkaraaslan/Skill_Seekers/issues
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT License - see [LICENSE](../LICENSE) file.
|
||||
|
||||
---
|
||||
|
||||
*Happy skill building! 🚀*
|
||||
|
After Width: | Height: | Size: 108 KiB |
|
After Width: | Height: | Size: 141 KiB |
|
After Width: | Height: | Size: 568 KiB |
|
After Width: | Height: | Size: 750 KiB |
|
After Width: | Height: | Size: 458 KiB |
|
After Width: | Height: | Size: 341 KiB |
|
After Width: | Height: | Size: 130 KiB |
|
After Width: | Height: | Size: 491 KiB |
|
After Width: | Height: | Size: 214 KiB |
|
After Width: | Height: | Size: 113 KiB |
|
After Width: | Height: | Size: 261 KiB |
|
After Width: | Height: | Size: 156 KiB |
|
After Width: | Height: | Size: 127 KiB |
|
After Width: | Height: | Size: 264 KiB |
|
After Width: | Height: | Size: 116 KiB |