docs: preserve upstream English README
CI / Lint & Test (Python 3.12) (push) Has been cancelled
CI / Lint & Test (Python 3.13) (push) Has been cancelled
CI / Lint & Test (Python 3.14) (push) Has been cancelled
CI / DCO Check (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / Lint & Test (Python 3.12) (push) Has been cancelled
CI / Lint & Test (Python 3.13) (push) Has been cancelled
CI / Lint & Test (Python 3.14) (push) Has been cancelled
CI / DCO Check (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
This commit is contained in:
+761
@@ -0,0 +1,761 @@
|
||||
# SkillSpector
|
||||
|
||||
**Security scanner for AI agent skills.** Detect vulnerabilities, malicious patterns, and security risks before installing agent skills.
|
||||
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
## Overview
|
||||
|
||||
AI agent skills (used by Claude Code, Codex CLI, Gemini CLI, etc.) execute with implicit trust and minimal vetting. Research shows that **26.1% of skills contain vulnerabilities** and **5.2% show likely malicious intent**.
|
||||
|
||||
SkillSpector helps you answer: **"Is this skill safe to install?"**
|
||||
|
||||
## Documentation
|
||||
|
||||
- **[Development guide](docs/DEVELOPMENT.md)** — Architecture, package layout, and how to extend the analyzer pipeline.
|
||||
- **[Pi extension](docs/PI_EXTENSION.md)** — Install SkillSpector as a Pi tool for scanning skills from inside agent sessions.
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-format input**: Scan Git repos, URLs, zip files, directories, or single files
|
||||
- **68 vulnerability patterns** across 17 categories: prompt injection, data exfiltration, privilege escalation, supply chain, excessive agency, output handling, system prompt leakage, memory poisoning, tool misuse, rogue agent, anti-refusal, trigger abuse, dangerous code (AST), taint tracking, YARA signatures, MCP least privilege, and MCP tool poisoning
|
||||
- **Two-stage analysis**: Fast static analysis + optional LLM semantic evaluation
|
||||
- **Live vulnerability lookups**: SC4 queries [OSV.dev](https://osv.dev) for real-time CVE data with automatic offline fallback
|
||||
- **Multiple output formats**: Terminal, JSON, Markdown, and SARIF reports
|
||||
- **Risk scoring**: 0-100 score with severity labels and clear recommendations
|
||||
- **Baseline / false-positive suppression**: Accept known findings via a glob-rule or fingerprint baseline so re-scans surface only *new* issues ([docs](docs/SUPPRESSION.md))
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Installation
|
||||
|
||||
Create and activate a virtual environment first (all `make` targets assume the venv is active). Use **uv** or **pip**; the Makefile uses `uv` if available, otherwise `pip`.
|
||||
|
||||
**Quick install with uv (CLI-only):**
|
||||
|
||||
```bash
|
||||
uv tool install git+https://github.com/NVIDIA/skillspector.git
|
||||
# Update later: uv tool update skillspector
|
||||
```
|
||||
|
||||
If you plan to run `skillspector mcp`, install the MCP extra at install time:
|
||||
|
||||
```bash
|
||||
uv tool install 'skillspector[mcp] @ git+https://github.com/NVIDIA/skillspector.git'
|
||||
```
|
||||
|
||||
**From source:**
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/NVIDIA/skillspector.git
|
||||
cd skillspector
|
||||
|
||||
# Create and activate virtual environment
|
||||
uv venv .venv && source .venv/bin/activate
|
||||
# or: python3 -m venv .venv && source .venv/bin/activate
|
||||
|
||||
# Install for production use
|
||||
make install
|
||||
|
||||
# Or install with development dependencies
|
||||
make install-dev
|
||||
```
|
||||
|
||||
### Docker (no Python required)
|
||||
|
||||
Run SkillSpector without installing Python by building it locally from the included [Dockerfile](Dockerfile). The image is based on the Docker Official Python `3.12-slim-bookworm` image.
|
||||
|
||||
**Build the image:**
|
||||
|
||||
```bash
|
||||
make docker-build
|
||||
# or: docker build -t skillspector .
|
||||
```
|
||||
|
||||
**Scan a local directory** by mounting your current directory into `/scan`, the container's working directory:
|
||||
|
||||
```bash
|
||||
docker run --rm -v "$PWD:/scan" skillspector scan ./my-skill/ --no-llm
|
||||
```
|
||||
|
||||
**Scan with LLM analysis** by passing credentials with a local `.env` file:
|
||||
|
||||
```bash
|
||||
cat > .env <<'EOF'
|
||||
SKILLSPECTOR_PROVIDER=anthropic
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
EOF
|
||||
```
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v "$PWD:/scan" \
|
||||
--env-file .env \
|
||||
skillspector scan ./my-skill/
|
||||
```
|
||||
|
||||
Or pass credentials directly from your shell environment:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v "$PWD:/scan" \
|
||||
-e SKILLSPECTOR_PROVIDER=anthropic \
|
||||
-e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
|
||||
skillspector scan ./my-skill/
|
||||
```
|
||||
|
||||
**Write a report to the host filesystem** by writing to the mounted directory:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
-v "$PWD:/scan" \
|
||||
skillspector scan ./my-skill/ --no-llm --format json --output report.json
|
||||
```
|
||||
|
||||
**Optional alias** for repeated static scans:
|
||||
|
||||
```bash
|
||||
alias skillspector-docker='docker run --rm -v "$PWD:/scan" skillspector'
|
||||
skillspector-docker scan ./my-skill/ --no-llm
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Scan a local skill directory
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Scan a single SKILL.md file
|
||||
skillspector scan ./SKILL.md
|
||||
|
||||
# Scan a Git repository
|
||||
skillspector scan https://github.com/user/my-skill
|
||||
|
||||
# Scan a zip file
|
||||
skillspector scan ./my-skill.zip
|
||||
```
|
||||
|
||||
### Output Formats
|
||||
|
||||
```bash
|
||||
# Terminal output (default) - pretty formatted
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# JSON output - machine readable
|
||||
skillspector scan ./my-skill/ --format json --output report.json
|
||||
|
||||
# Markdown output - for documentation
|
||||
skillspector scan ./my-skill/ --format markdown --output report.md
|
||||
|
||||
# SARIF output - for CI/CD integration and IDE tooling
|
||||
skillspector scan ./my-skill/ --format sarif --output report.sarif
|
||||
```
|
||||
|
||||
### Batch Scanning
|
||||
|
||||
Scan entire directories of skills in parallel from `contrib/batch_scan/`:
|
||||
|
||||
```bash
|
||||
python -m contrib.batch_scan.batch_scan ./my-skills/ --no-llm
|
||||
python -m contrib.batch_scan.batch_scan ./my-skills/ --workers 20 -f json -o report.json
|
||||
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 20
|
||||
```
|
||||
|
||||
Supports multilingual detection (zh/ja/ko) and terminal/JSON/Markdown output.
|
||||
|
||||
For LLM scans with higher concurrency, configure multiple API keys following
|
||||
[`.env.example`](contrib/batch_scan/.env.example) — the pool improves throughput
|
||||
and resilience, provided the keys don't share an account-level rate limit.
|
||||
|
||||
See the [contrib guide](contrib/batch_scan/docs/) for details.
|
||||
|
||||
> **Note on LLM support:** The default configuration targets DeepSeek as the
|
||||
> cheapest public option. DeepSeek-Chat is
|
||||
> [expected to sunset](https://api-docs.deepseek.com/), and the contributor
|
||||
> does not have hardware to test against local models. The batch scanner was
|
||||
> originally tested with OpenAI-compatible endpoints — DeepSeek's lack of
|
||||
> structured-output support required manual JSON-parsing patches. If you can
|
||||
> contribute a more universal backend (Ollama, vLLM, or a different provider),
|
||||
> PRs are very welcome.
|
||||
|
||||
### Suppressing False Positives (baseline)
|
||||
|
||||
Suppress known/accepted findings so the risk score reflects only un-triaged
|
||||
issues and re-scans surface only *new* findings. See the
|
||||
[suppression guide](docs/SUPPRESSION.md) for the full reference.
|
||||
|
||||
```bash
|
||||
# Accept all current findings into a baseline (run once), then commit it.
|
||||
skillspector baseline ./my-skill/ -o .skillspector-baseline.yaml
|
||||
|
||||
# Scan against the baseline — only NEW findings are reported and scored.
|
||||
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml
|
||||
|
||||
# Review what was suppressed (still excluded from the score).
|
||||
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-suppressed
|
||||
```
|
||||
|
||||
A baseline can also use drift-tolerant glob rules (by rule id, file path, or
|
||||
message) — see [`.skillspector-baseline.example.yaml`](.skillspector-baseline.example.yaml).
|
||||
|
||||
### LLM Analysis
|
||||
|
||||
For the best results, configure an OpenAI-compatible LLM endpoint for
|
||||
semantic analysis. Pick a provider with `SKILLSPECTOR_PROVIDER`; each
|
||||
ships its own bundled default model. SkillSpector also works against
|
||||
local OpenAI-compatible servers (Ollama, vLLM, llama.cpp) and managed
|
||||
inference gateways.
|
||||
|
||||
| Provider (`SKILLSPECTOR_PROVIDER`) | Credential env var | Endpoint | Default model |
|
||||
| ---------- | ---- | ---- | ---- |
|
||||
| `openai` | `OPENAI_API_KEY` (+ optional `OPENAI_BASE_URL`) | api.openai.com (or any OpenAI-compatible URL) | `gpt-5.4` |
|
||||
| `anthropic` | `ANTHROPIC_API_KEY` | api.anthropic.com | `claude-opus-4-6` |
|
||||
| `anthropic_proxy` | `ANTHROPIC_PROXY_API_KEY` + `ANTHROPIC_PROXY_ENDPOINT_URL` | Any Vertex-style raw-predict proxy | `claude-sonnet-4-6` |
|
||||
| `bedrock` | `AWS_PROFILE` (optional) + `AWS_REGION` — SigV4 via boto3 | AWS Bedrock Runtime | `us.anthropic.claude-sonnet-4-6-20250915-v1:0` |
|
||||
| `nv_build` | `NVIDIA_INFERENCE_KEY` | build.nvidia.com | `deepseek-ai/deepseek-v4-flash` |
|
||||
| `claude_cli` | _(none — uses local CLI auth)_ | local `claude` binary | `claude-sonnet-4-6` |
|
||||
| `codex_cli` | _(none — uses local CLI auth)_ | local `codex` binary | `o4-mini` |
|
||||
|
||||
```bash
|
||||
# Stock OpenAI
|
||||
export SKILLSPECTOR_PROVIDER=openai
|
||||
export OPENAI_API_KEY=sk-...
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Anthropic
|
||||
export SKILLSPECTOR_PROVIDER=anthropic
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Anthropic via Vertex-style proxy (corporate gateways, GCP Vertex AI)
|
||||
export SKILLSPECTOR_PROVIDER=anthropic_proxy
|
||||
export ANTHROPIC_PROXY_ENDPOINT_URL=https://my-gateway.example.com/models/claude-sonnet-4-6:streamRawPredict
|
||||
export ANTHROPIC_PROXY_API_KEY=your-bearer-token
|
||||
export SKILLSPECTOR_MODEL=claude-sonnet-4-6
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# AWS Bedrock (Claude via SigV4)
|
||||
export SKILLSPECTOR_PROVIDER=bedrock
|
||||
# Optional: select an AWS named profile. When unset, the standard
|
||||
# boto3 credential chain (env vars, instance metadata, SSO, etc.) resolves.
|
||||
# export AWS_PROFILE=my-profile
|
||||
export AWS_REGION=us-west-2 # default if unset
|
||||
# Default model: us.anthropic.claude-sonnet-4-6-20250915-v1:0
|
||||
# Override with any Bedrock model ID, cross-region inference-profile
|
||||
# ID, or your own application-inference-profile ARN:
|
||||
# export SKILLSPECTOR_MODEL=us.anthropic.claude-opus-4-6-20250915-v1:0
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# NVIDIA build.nvidia.com
|
||||
export SKILLSPECTOR_PROVIDER=nv_build
|
||||
export NVIDIA_INFERENCE_KEY=nvapi-...
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Local Claude CLI — no API key; uses your existing `claude auth login` session
|
||||
# Requires: claude CLI installed and authenticated (claude auth login)
|
||||
export SKILLSPECTOR_PROVIDER=claude_cli
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Local Codex CLI — no API key; uses your existing `codex login` session
|
||||
# Requires: codex CLI installed and authenticated
|
||||
export SKILLSPECTOR_PROVIDER=codex_cli
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Local Ollama or any OpenAI-compatible endpoint
|
||||
export SKILLSPECTOR_PROVIDER=openai
|
||||
export OPENAI_API_KEY=ollama
|
||||
export OPENAI_BASE_URL=http://localhost:11434/v1
|
||||
export SKILLSPECTOR_MODEL=llama3.1:8b
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Override the provider's default model
|
||||
export SKILLSPECTOR_MODEL=gpt-5.2
|
||||
skillspector scan ./my-skill/
|
||||
|
||||
# Skip LLM analysis (faster, static analysis only)
|
||||
skillspector scan ./my-skill/ --no-llm
|
||||
```
|
||||
|
||||
### MCP Server
|
||||
|
||||
Run SkillSpector as a [Model Context Protocol](https://modelcontextprotocol.io)
|
||||
server so any MCP-capable agent (Claude Code, Codex CLI, Gemini CLI) or remote
|
||||
runtime can call scanning as a tool and **gate skill/MCP installs on the
|
||||
result** — turning SkillSpector into a runtime guardrail instead of an
|
||||
out-of-band audit step.
|
||||
|
||||
`skillspector mcp` requires `skillspector[mcp]`.
|
||||
|
||||
```bash
|
||||
# Install, or reinstall if you already used the CLI-only path
|
||||
uv tool install --force 'skillspector[mcp] @ git+https://github.com/NVIDIA/skillspector.git'
|
||||
|
||||
# FastMCP stdio transport for local CLI agents
|
||||
skillspector mcp
|
||||
|
||||
# streamable HTTP/SSE transport for remote / A2A callers
|
||||
skillspector mcp --transport http --host 127.0.0.1 --port 8000
|
||||
```
|
||||
|
||||
The stdio transport is the current FastMCP path for local CLI agents, and the
|
||||
initialize hang reported in issue #199 still applies there.
|
||||
|
||||
The server exposes a single tool:
|
||||
|
||||
- **`scan_skill(target, use_llm=true, output_format="json")`** — scans a Git
|
||||
URL, file URL, `.zip`, `.md` file, or directory and returns a structured
|
||||
verdict: `risk_score` (0-100), `severity`, `recommendation`,
|
||||
`safe_to_install`, and `findings`. It also reports `llm_used` / `scan_mode`
|
||||
so a low score from a static-only scan is never mistaken for a clean full
|
||||
scan.
|
||||
|
||||
Register it with Claude Code via:
|
||||
|
||||
```bash
|
||||
claude mcp add skillspector -- skillspector mcp
|
||||
```
|
||||
|
||||
> **Security — HTTP transport trust model**
|
||||
>
|
||||
> The HTTP transport ships **without authentication**. Any caller that can
|
||||
> reach the port can invoke `scan_skill`. Over stdio or `127.0.0.1` this is
|
||||
> the same trust boundary as the CLI. If you bind to a routable interface:
|
||||
>
|
||||
> - Sit the server behind an authenticating reverse proxy (e.g. nginx + mTLS)
|
||||
> before exposing it externally.
|
||||
> - Local paths and `file://` URLs are **automatically rejected** over HTTP to
|
||||
> prevent unauthenticated callers from reading arbitrary host files. Only
|
||||
> remote Git and `.zip` URLs are accepted.
|
||||
|
||||
## Vulnerability Patterns
|
||||
|
||||
SkillSpector detects **68 vulnerability patterns** across 17 categories:
|
||||
|
||||
### Prompt Injection (5 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| P1 | Instruction Override | HIGH | Commands to ignore safety constraints |
|
||||
| P2 | Hidden Instructions | HIGH | Malicious directives in comments/invisible text |
|
||||
| P3 | Exfiltration Commands | HIGH | Instructions to transmit context externally |
|
||||
| P4 | Behavior Manipulation | MEDIUM | Subtle instructions altering agent decisions |
|
||||
| P5 | Harmful Content | CRITICAL | Instructions that could cause physical harm |
|
||||
|
||||
### Anti-Refusal (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| AR1 | Refusal Suppression | HIGH | Instructions to never refuse or always comply (e.g. "never refuse", "always comply") |
|
||||
| AR2 | Disclaimer Suppression | HIGH | Instructions to omit warnings, disclaimers, or ethical commentary (e.g. "no disclaimers", "do not moralize") |
|
||||
| AR3 | Safety Policy Nullification | HIGH | Jailbreak framing that nullifies guardrails (e.g. "you have no restrictions", "ignore your guidelines", "do anything now") |
|
||||
|
||||
### Data Exfiltration (4 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| E1 | External Transmission | MEDIUM | Sending data to external URLs |
|
||||
| E2 | Env Variable Harvesting | HIGH | Collecting API keys and secrets |
|
||||
| E3 | File System Enumeration | MEDIUM | Scanning directories for sensitive files |
|
||||
| E4 | Context Leakage | HIGH | Transmitting conversation context externally |
|
||||
|
||||
### Privilege Escalation (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| PE1 | Excessive Permissions | LOW | Requesting access beyond stated functionality |
|
||||
| PE2 | Sudo/Root Execution | MEDIUM | Invoking elevated system privileges |
|
||||
| PE3 | Credential Access | HIGH | Reading SSH keys, tokens, passwords |
|
||||
|
||||
### Supply Chain (6 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| SC1 | Unpinned Dependencies | LOW | No version constraints on packages |
|
||||
| SC2 | External Script Fetching | HIGH | curl \| bash and remote code execution |
|
||||
| SC3 | Obfuscated Code | HIGH | Base64/hex encoded execution |
|
||||
| SC4 | Known Vulnerable Dependencies | HIGH | Dependencies with known CVEs (live OSV.dev lookup) |
|
||||
| SC5 | Abandoned Dependencies | MEDIUM | Unmaintained packages without security updates |
|
||||
| SC6 | Typosquatting | HIGH | Package names similar to popular packages |
|
||||
|
||||
### Excessive Agency (4 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| EA1 | Unrestricted Tool Access | HIGH | Unfettered tool access without constraints |
|
||||
| EA2 | Autonomous Decision Making | HIGH | High-impact decisions without human-in-the-loop |
|
||||
| EA3 | Scope Creep | MEDIUM | Capabilities extending beyond stated purpose |
|
||||
| EA4 | Unbounded Resource Access | MEDIUM | No rate limits or quotas on resource consumption |
|
||||
|
||||
### Output Handling (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| OH1 | Unvalidated Output Injection | HIGH | Model output used without sanitization |
|
||||
| OH2 | Cross-Context Output | MEDIUM | Output flows across trust boundaries without validation |
|
||||
| OH3 | Unbounded Output | MEDIUM | No limits on output size or generation rate |
|
||||
|
||||
### System Prompt Leakage (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| P6 | Direct Leakage | HIGH | Instructions that expose system prompts or internal rules |
|
||||
| P7 | Indirect Extraction | MEDIUM | Extraction via rephrasing, translation, or side-channels |
|
||||
| P8 | Tool-Based Exfiltration | HIGH | System prompts exfiltrated via file writes or network requests |
|
||||
|
||||
### Memory Poisoning (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| MP1 | Persistent Context Injection | HIGH | Content designed to persist across interactions |
|
||||
| MP2 | Context Window Stuffing | MEDIUM | Filler content displacing safety constraints |
|
||||
| MP3 | Memory Manipulation | HIGH | Tampering with agent memory or stored state |
|
||||
|
||||
### Tool Misuse (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| TM1 | Tool Parameter Abuse | HIGH | Crafted parameters for unintended behavior (shell=True, --force) |
|
||||
| TM2 | Chaining Abuse | HIGH | Tool chains that bypass individual safety checks |
|
||||
| TM3 | Unsafe Defaults | MEDIUM | Overly permissive defaults (disabled TLS, no auth) |
|
||||
|
||||
### Rogue Agent (2 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| RA1 | Self-Modification | CRITICAL | Modifying own code or configuration at runtime |
|
||||
| RA2 | Session Persistence | HIGH | Unauthorized persistence via cron jobs or startup scripts |
|
||||
|
||||
### Trigger Abuse (3 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| TR1 | Overly Broad Trigger | MEDIUM | Trigger patterns matching common words |
|
||||
| TR2 | Shadow Command Trigger | HIGH | Triggers that shadow built-in commands or other skills |
|
||||
| TR3 | Keyword Baiting Trigger | MEDIUM | Generic triggers designed to maximize activation |
|
||||
|
||||
### Behavioral AST (9 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| AST1 | exec() Call | CRITICAL | Direct exec() enabling arbitrary code execution |
|
||||
| AST2 | eval() Call | HIGH | Direct eval() evaluating arbitrary expressions |
|
||||
| AST3 | Dynamic Import | HIGH | \_\_import\_\_() loading arbitrary modules at runtime |
|
||||
| AST4 | subprocess Call | HIGH | External command execution via subprocess |
|
||||
| AST5 | os.system / exec-family | HIGH | Shell commands via os module |
|
||||
| AST6 | compile() Call | MEDIUM | Code object creation from strings |
|
||||
| AST7 | Dynamic getattr() | MEDIUM | Arbitrary attribute access with non-literal names |
|
||||
| AST8 | Dangerous Execution Chain | CRITICAL | exec/eval combined with dynamic source (network, encoded data) |
|
||||
| AST9 | Reflective getattr() Sink | HIGH | Reflective exec via `getattr(os,'system')` / `getattr(builtins,'exec')` that evades AST1/AST5 |
|
||||
|
||||
### Taint Tracking (5 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| TT1 | Direct Taint Flow | HIGH | Data flows directly from a source to a sink without sanitization |
|
||||
| TT2 | Variable-Mediated Taint Flow | MEDIUM | Data flows from source to sink through intermediate variables |
|
||||
| TT3 | Credential Exfiltration Chain | CRITICAL | Credentials (env vars, secrets) flow to network output sinks |
|
||||
| TT4 | File Read to Network Exfiltration | HIGH | File contents flow to network output sinks |
|
||||
| TT5 | External Input to Code Execution | CRITICAL | Network or user input flows to exec/eval/subprocess sinks |
|
||||
|
||||
### YARA Signatures (4 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| YR1 | Malware Match | CRITICAL | YARA rule match for known malware signatures |
|
||||
| YR2 | Webshell Match | CRITICAL | YARA rule match for webshell patterns |
|
||||
| YR3 | Cryptominer Match | HIGH | YARA rule match for crypto mining indicators |
|
||||
| YR4 | Hack Tool / Exploit Match | HIGH | YARA rule match for hack tools or exploit code |
|
||||
|
||||
### MCP Least Privilege (4 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| LP1 | Underdeclared Capability | HIGH | Code uses capabilities not listed in declared permissions |
|
||||
| LP2 | Wildcard Permission | MEDIUM | Permission list contains wildcards (\*, all, full, any) |
|
||||
| LP3 | Missing Permission Declaration | MEDIUM | No permissions field but code has detectable capabilities |
|
||||
| LP4 | Overdeclared Permission | LOW | Permission declared but no corresponding code capability found |
|
||||
|
||||
### MCP Tool Poisoning (4 patterns)
|
||||
|
||||
| ID | Pattern | Severity | Description |
|
||||
|----|---------|----------|-------------|
|
||||
| TP1 | Hidden Instructions | HIGH | Hidden directives in metadata (HTML comments, zero-width chars, base64, data URIs) |
|
||||
| TP2 | Unicode Deception | HIGH | Homoglyphs, RTL overrides, mixed-script identifiers in tool metadata |
|
||||
| TP3 | Parameter Description Injection | MEDIUM | Injection patterns in parameter definitions (overrides, system tokens, malicious defaults) |
|
||||
| TP4 | Description-Behavior Mismatch | MEDIUM | Declared tool description does not match actual code behavior (LLM-powered) |
|
||||
|
||||
All detected patterns are listed in the tables above.
|
||||
|
||||
## Risk Scoring
|
||||
|
||||
### Score Calculation
|
||||
|
||||
- **CRITICAL issues**: +50 points
|
||||
- **HIGH issues**: +25 points
|
||||
- **MEDIUM issues**: +10 points
|
||||
- **LOW issues**: +5 points
|
||||
- **Executable scripts**: 1.3x multiplier
|
||||
|
||||
### Severity Levels
|
||||
|
||||
| Score | Severity | Recommendation |
|
||||
|-------|----------|----------------|
|
||||
| 0-20 | LOW | SAFE |
|
||||
| 21-50 | MEDIUM | CAUTION |
|
||||
| 51-80 | HIGH | DO NOT INSTALL |
|
||||
| 81-100 | CRITICAL | DO NOT INSTALL |
|
||||
|
||||
## Example Output
|
||||
|
||||
### Terminal Output
|
||||
|
||||
```
|
||||
SkillSpector Security Report v2.0.0
|
||||
|
||||
Skill: suspicious-skill
|
||||
Source: ./suspicious-skill/
|
||||
Scanned: 2026-01-29 10:30:00 UTC
|
||||
|
||||
Risk Assessment
|
||||
Metric Value
|
||||
Score 78/100
|
||||
Severity HIGH
|
||||
Recommendation DO NOT INSTALL
|
||||
|
||||
Components (3)
|
||||
File Type Lines Executable
|
||||
SKILL.md markdown 142 No
|
||||
scripts/sync.py python 87 Yes
|
||||
requirements.txt text 3 No
|
||||
|
||||
Issues (2)
|
||||
|
||||
HIGH: Env Variable Harvesting (E2)
|
||||
Location: scripts/sync.py:23
|
||||
Finding: for key, val in os.environ.items():...
|
||||
Confidence: 94%
|
||||
Explanation: This code collects environment variables containing
|
||||
API keys and secrets, then sends them to an external server.
|
||||
|
||||
HIGH: External Transmission (E1)
|
||||
Location: scripts/sync.py:45
|
||||
Finding: requests.post("https://api.skill.io/env"...
|
||||
Confidence: 89%
|
||||
Explanation: Data is being sent to an external server. Combined
|
||||
with env harvesting above, this indicates credential exfiltration.
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Required |
|
||||
|----------|-------------|----------|
|
||||
| `SKILLSPECTOR_PROVIDER` | Active LLM provider: `openai`, `anthropic`, `anthropic_proxy`, `bedrock`, `nv_build`, `claude_cli`, `codex_cli`, or `gemini_cli`. Each provider has its own bundled `model_registry.yaml` and default model (see the LLM Analysis table above). Defaults to `nv_build`. | Optional |
|
||||
| `NVIDIA_INFERENCE_KEY` | Credential for the `nv_build` provider (build.nvidia.com). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=nv_build` |
|
||||
| `OPENAI_API_KEY` | Credential for the OpenAI provider (`SKILLSPECTOR_PROVIDER=openai`). Also serves as the tier-2 fallback in the credential waterfall when the active provider returns no credentials. | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=openai` |
|
||||
| `OPENAI_BASE_URL` | Override the OpenAI endpoint (e.g. point at Ollama). | Optional |
|
||||
| `ANTHROPIC_API_KEY` | Credential for the Anthropic provider (`SKILLSPECTOR_PROVIDER=anthropic`). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=anthropic` |
|
||||
| `ANTHROPIC_PROXY_ENDPOINT_URL` | Full endpoint URL for the Anthropic proxy provider (Vertex-style raw-predict). | Required when `SKILLSPECTOR_PROVIDER=anthropic_proxy` |
|
||||
| `ANTHROPIC_PROXY_API_KEY` | Bearer token for the Anthropic proxy provider. | Required when `SKILLSPECTOR_PROVIDER=anthropic_proxy` |
|
||||
| `ANTHROPIC_PROXY_API_VERSION` | `anthropic_version` value sent in the request body (default: `vertex-2023-10-16`). | Optional |
|
||||
| `AWS_PROFILE` | Named AWS profile for the Bedrock provider — authenticates via SigV4 through boto3. When unset, the standard boto3 credential chain (env vars, instance metadata, SSO, etc.) resolves. | Optional (used when `SKILLSPECTOR_PROVIDER=bedrock`) |
|
||||
| `AWS_REGION` | AWS region for the Bedrock Runtime endpoint. Defaults to `us-west-2`. | Optional (used when `SKILLSPECTOR_PROVIDER=bedrock`) |
|
||||
| `SKILLSPECTOR_MODEL` | Override the active provider's default model. See the LLM Analysis table for each provider's default. | Optional |
|
||||
| `SKILLSPECTOR_MODEL_REGISTRY` | Override the bundled per-provider YAML registry (`src/skillspector/providers/<provider>/model_registry.yaml`) with a custom path. | Optional |
|
||||
| `SKILLSPECTOR_LOG_LEVEL` | Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` (default: `WARNING`). | Optional |
|
||||
|
||||
> **CLI providers** (`claude_cli`, `codex_cli`): No API key is needed. Authentication is managed entirely by the agent CLI's own login session (`claude auth login` / `codex login`). SkillSpector never reads or forwards API keys when these providers are active. The subprocess is run in a hardened sandbox: tools disabled, no MCP, read-only sandbox mode (codex), and untrusted skill content is delivered only via stdin.
|
||||
|
||||
### CLI Options
|
||||
|
||||
```bash
|
||||
skillspector scan --help
|
||||
|
||||
Options:
|
||||
-f, --format [terminal|json|markdown|sarif] Output format [default: terminal]
|
||||
-o, --output PATH Output file path
|
||||
--no-llm Skip LLM analysis (static only)
|
||||
--yara-rules-dir PATH Extra YARA rules directory
|
||||
-b, --baseline PATH Suppress findings listed in a baseline
|
||||
--show-suppressed List baseline-suppressed findings
|
||||
-V, --verbose Show detailed progress
|
||||
--help Show this message and exit
|
||||
|
||||
# Generate a baseline of all current findings (see docs/SUPPRESSION.md)
|
||||
skillspector baseline <path> [-o FILE] [--no-llm] [--reason TEXT]
|
||||
```
|
||||
|
||||
## Integrating SkillSpector
|
||||
|
||||
SkillSpector is built to be driven by other tools (CI pipelines, install gates, editor integrations). Its exit code and JSON output are a stable contract.
|
||||
|
||||
### Exit codes
|
||||
|
||||
`skillspector scan` exits with:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Scan completed, `risk_score` ≤ 50 (recommendation `SAFE` or `CAUTION`) |
|
||||
| `1` | Scan completed, `risk_score` > 50 (recommendation `DO_NOT_INSTALL`) |
|
||||
| `2` | Error (bad input, unreadable source, internal failure) |
|
||||
|
||||
> The exit code collapses `SAFE` and `CAUTION` into `0`. To act differently on them (e.g. *warn* on `CAUTION` but *block* on `DO_NOT_INSTALL`), read the `recommendation` field from the JSON output rather than relying on the exit code.
|
||||
|
||||
### Machine-readable output
|
||||
|
||||
`--format json` produces a JSON report; with no `--output`/`-o` it is written to stdout:
|
||||
|
||||
```bash
|
||||
skillspector scan ./my-skill/ --format json
|
||||
```
|
||||
|
||||
The top-level shape is (this example shows a full LLM-backed scan; with `--no-llm`, `metadata.llm_requested` is `false`):
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "name": "...", "source": "...", "scanned_at": "<ISO 8601>" },
|
||||
"risk_assessment": { "score": 0, "severity": "LOW", "recommendation": "SAFE" },
|
||||
"components": [ { "path": "...", "type": "...", "lines": 0, "executable": false, "size_bytes": 0 } ],
|
||||
"issues": [ { "id": "...", "category": "...", "severity": "...", "confidence": 0.0, "location": { "file": "...", "start_line": 0 } } ],
|
||||
"metadata": { "has_executable_scripts": false, "skillspector_version": "...", "llm_requested": true, "llm_available": true }
|
||||
}
|
||||
```
|
||||
|
||||
- `risk_assessment.severity` ∈ `LOW | MEDIUM | HIGH | CRITICAL`.
|
||||
- `risk_assessment.recommendation` ∈ `SAFE | CAUTION | DO_NOT_INSTALL`, mapped from severity: `LOW → SAFE`, `MEDIUM → CAUTION`, `HIGH`/`CRITICAL → DO_NOT_INSTALL`.
|
||||
- `metadata.llm_error` appears only when LLM analysis was requested but unavailable.
|
||||
- The full per-issue shape is defined by `Finding.to_dict()` in [models.py](src/skillspector/models.py); rely on the fields above and treat any additional fields as best-effort.
|
||||
|
||||
For CI/IDE tooling, `--format sarif` emits SARIF 2.1.0.
|
||||
|
||||
### Recommended gate mapping
|
||||
|
||||
When using SkillSpector as an install gate, map the recommendation to an action:
|
||||
|
||||
| `recommendation` | Suggested action |
|
||||
|------------------|------------------|
|
||||
| `SAFE` | allow |
|
||||
| `CAUTION` | prompt / warn the user |
|
||||
| `DO_NOT_INSTALL` | block |
|
||||
|
||||
SkillSpector computes the score band and recommendation; how strict the gate is (e.g. whether `CAUTION` blocks in CI) is a policy decision for the integrating tool.
|
||||
|
||||
## Development
|
||||
|
||||
### Setup
|
||||
|
||||
All `make` targets assume a virtual environment is already created and activated. The Makefile uses **uv** if available, else **pip**.
|
||||
|
||||
```bash
|
||||
# Clone, create venv, activate, install dev dependencies
|
||||
git clone https://github.com/NVIDIA/skillspector.git
|
||||
cd skillspector
|
||||
uv venv .venv && source .venv/bin/activate
|
||||
# or: python3 -m venv .venv && source .venv/bin/activate
|
||||
make install-dev
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Run tests with coverage
|
||||
make test-cov
|
||||
|
||||
# Run linting
|
||||
make lint
|
||||
|
||||
# Format code
|
||||
make format
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
SkillSpector uses a two-stage detection pipeline:
|
||||
|
||||
### Stage 1: Static Analysis
|
||||
- Fast regex-based pattern matching across 11 static analyzers
|
||||
- AST-based behavioral analysis detecting dangerous calls (exec, eval, subprocess, etc.)
|
||||
- Live vulnerability lookups via OSV.dev for known CVEs in dependencies
|
||||
- Scans all files in the skill
|
||||
- High recall (catches most issues)
|
||||
- Moderate precision (some false positives)
|
||||
|
||||
### Stage 2: LLM Semantic Analysis (Optional)
|
||||
- Evaluates context and intent
|
||||
- Filters false positives
|
||||
- Provides human-readable explanations
|
||||
- Improves precision to ~87%
|
||||
|
||||
The LLM prompt includes anti-jailbreak protections to prevent malicious skills from manipulating the analysis.
|
||||
|
||||
## Live Vulnerability Lookups (SC4)
|
||||
|
||||
SC4 uses the [OSV.dev](https://osv.dev) API to check dependencies against the full Open Source Vulnerabilities database — covering tens of thousands of advisories across PyPI and npm.
|
||||
|
||||
- **No API key required** — OSV.dev is free and unauthenticated.
|
||||
- **Batch queries** — all dependencies are checked in a single HTTP call.
|
||||
- **Automatic fallback** — if OSV.dev is unreachable (air-gapped/offline), a small built-in fallback list is used.
|
||||
- **Caching** — results are cached in-memory for 1 hour to avoid redundant API calls during a session.
|
||||
|
||||
The tool requires outbound HTTPS access to `api.osv.dev` for live vulnerability data. When that is not available, findings are limited to the static fallback list.
|
||||
|
||||
## Trust model and data egress
|
||||
|
||||
SkillSpector is defense-in-depth, not a sandbox. Know what it does and does not do before relying on it:
|
||||
|
||||
- **It never executes the scanned skill.** All analysis is static (regex, Python AST, YARA) plus optional LLM evaluation of file *contents* — the skill's code is never run.
|
||||
- **LLM analysis sends file contents to the configured provider.** When LLM analysis is enabled (the default), file contents are sent to the active `SKILLSPECTOR_PROVIDER` endpoint. Use `--no-llm` to keep contents local (static analysis only).
|
||||
- **SC4 sends dependency names to OSV.dev.** The supply-chain check queries [OSV.dev](https://osv.dev) with the package names and versions the skill declares, to look up known CVEs. This is fundamental to the check and runs even with `--no-llm`. It sends dependency coordinates (not file contents), requires no API key, and falls back to a bundled list when OSV.dev is unreachable.
|
||||
- **It does not sandbox the host.** SkillSpector flags risky patterns *before* you install a skill; it does not contain or isolate a skill you choose to install anyway.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Non-English content**: May miss patterns in other languages
|
||||
- **Image-based attacks**: Cannot analyze text in images
|
||||
- **Encrypted/binary code**: Cannot analyze compiled or encrypted content
|
||||
- **Runtime behavior**: Static analysis only, no dynamic execution
|
||||
- **Offline SC4**: Without network access to `api.osv.dev`, SC4 uses a small static fallback list
|
||||
|
||||
## Research Background
|
||||
|
||||
Based on research from "Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale" (Liu et al., 2026):
|
||||
|
||||
- **Dataset**: 42,447 skills from major marketplaces
|
||||
- **Vulnerable**: 26.1% contain at least one vulnerability
|
||||
- **High-severity**: 5.2% show likely malicious intent
|
||||
- **Key finding**: Skills with executable scripts are 2.12x more likely to be vulnerable
|
||||
|
||||
## Python API Integration
|
||||
|
||||
```python
|
||||
from skillspector import graph
|
||||
|
||||
# Invoke the LangGraph workflow
|
||||
result = graph.invoke({
|
||||
"input_path": "/path/to/skill",
|
||||
"output_format": "json", # terminal, json, markdown, or sarif
|
||||
"use_llm": True, # False for static-only analysis
|
||||
})
|
||||
|
||||
# Access results
|
||||
print(f"Risk Score: {result['risk_score']}/100")
|
||||
print(f"Severity: {result['risk_severity']}")
|
||||
print(f"Recommendation: {result['risk_recommendation']}")
|
||||
|
||||
for finding in result["filtered_findings"]:
|
||||
print(f"[{finding['severity']}] {finding['rule_id']}: {finding['message']}")
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache License 2.0 - see [LICENSE](LICENSE) for details.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please read our contributing guidelines and submit pull requests.
|
||||
|
||||
## Support
|
||||
|
||||
- **Issues**: [GitHub Issues](https://github.com/NVIDIA/skillspector/issues)
|
||||
Reference in New Issue
Block a user