wehub-resource-sync
2114ccd278
CI / Lint & Test (Python 3.13) (push) Failing after 2s
CI / Lint & Test (Python 3.14) (push) Failing after 1s
CI / Lint & Test (Python 3.12) (push) Failing after 2s
CI / DCO Check (push) Has been skipped
Scorecard supply-chain security / Scorecard analysis (push) Failing after 2s
120 lines
5.3 KiB
Markdown
120 lines
5.3 KiB
Markdown
# Baseline / False-Positive Suppression
|
|
|
|
SkillSpector's analyzers — especially the LLM semantic ones — can produce
|
|
findings that are correct in general but not actionable for *your* skills
|
|
(framework/architectural patterns, first-party tooling conventions, accepted
|
|
lab practices). A **baseline** lets you suppress those known findings so that:
|
|
|
|
- the risk score reflects only **un-triaged** issues,
|
|
- re-scans surface only **new** findings (incremental CI/CD), and
|
|
- every suppression carries an auditable **reason**.
|
|
|
|
Suppressed findings never count toward the risk score and are excluded from the
|
|
SARIF results. They are shown in the terminal/Markdown report only when you pass
|
|
`--show-suppressed`, and are always listed (machine-readable) in the JSON report
|
|
under `suppressed` / `suppressed_count`.
|
|
|
|
> Addresses [issue #88](https://github.com/NVIDIA/SkillSpector/issues/88).
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
# 1. Accept all current findings into a baseline (run once).
|
|
skillspector baseline ./my-skill/ -o .skillspector-baseline.yaml
|
|
|
|
# 2. Commit the baseline, then scan against it. Only NEW findings are reported.
|
|
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml
|
|
|
|
# Review what was suppressed.
|
|
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-suppressed
|
|
```
|
|
|
|
## CLI
|
|
|
|
| Command / option | Description |
|
|
|------------------|-------------|
|
|
| `skillspector baseline <path> [-o FILE] [--no-llm] [--reason TEXT]` | Scan and write a baseline that fingerprint-suppresses every current finding. Default output: `.skillspector-baseline.yaml`. |
|
|
| `skillspector scan <path> --baseline FILE` (`-b`) | Suppress findings matching the baseline before scoring/reporting. |
|
|
| `skillspector scan <path> --baseline FILE --show-suppressed` | Also list the suppressed findings (they still don't affect the score). |
|
|
|
|
A missing or malformed baseline file exits with code 2.
|
|
|
|
## Baseline file format
|
|
|
|
YAML or JSON (the `.json` extension selects JSON output when generating). Two
|
|
complementary mechanisms:
|
|
|
|
```yaml
|
|
version: 1
|
|
|
|
rules: # human-authored, glob-based, drift-tolerant
|
|
- id: "SQP-1" # glob over the finding's rule id
|
|
reason: "Trigger-phrase breadth is a description nit, not a vuln"
|
|
- id: "SSD-2"
|
|
path: "example-skill/SKILL.md" # glob over the finding's file
|
|
message: "*example false-positive phrase*" # glob over the finding's message
|
|
reason: "False positive: benign trigger phrase, not an instruction"
|
|
|
|
fingerprints: # machine-generated, exact
|
|
- hash: "sha256:1a2b3c4d5e6f7081"
|
|
rule_id: "SDI-2" # informational (for humans reading the file)
|
|
file: "example-skill/SKILL.md"
|
|
reason: "Accepted — reads its own environment for context"
|
|
```
|
|
|
|
### `rules` — glob suppression
|
|
|
|
A finding is suppressed when **every** field a rule specifies matches it;
|
|
unspecified fields match anything. Use this for:
|
|
|
|
- **Global pattern suppression** — `id: "SQP-1"` (or `id: "SQP-*"`) drops a rule
|
|
or rule family across all skills.
|
|
- **Skill/file-scoped suppression** — add `path:` (and optionally `message:`) to
|
|
scope the suppression to a specific skill, file, or message.
|
|
|
|
Field reference:
|
|
|
|
| Field | Matches against | Notes |
|
|
|-------|-----------------|-------|
|
|
| `id` (or `rule_id`) | `Finding.rule_id` | glob |
|
|
| `path` (or `file`) | `Finding.file` | glob; `*` crosses `/`, `**` is an alias for `*` |
|
|
| `message` | `Finding.message` | glob, case-insensitive; wrap a keyword in `*` for substring |
|
|
| `reason` | — | required; recorded in reports and audits |
|
|
|
|
Glob matching uses Python's [`fnmatch`](https://docs.python.org/3/library/fnmatch.html),
|
|
so `*` matches across path separators (`*SKILL.md` matches `a/b/SKILL.md`).
|
|
Rules are **drift-tolerant**: they keep working after line numbers shift or
|
|
content is reworded.
|
|
|
|
### `fingerprints` — exact suppression
|
|
|
|
Each entry is the stable hash of one finding
|
|
(`sha256(rule_id|file|start_line|end_line|message)`, truncated). Generated by
|
|
`skillspector baseline`. Because the hash includes the line span and message,
|
|
editing a skill so a finding moves or is reworded changes its fingerprint —
|
|
**regenerate the baseline** after material changes, or prefer `rules` for
|
|
suppressions you want to survive edits.
|
|
|
|
An entry may be a bare string (`"sha256:..."`) or a mapping with `hash`,
|
|
optional `reason`, and informational `rule_id` / `file`.
|
|
|
|
## How it fits the pipeline
|
|
|
|
Suppression is applied in the **report node** (`skillspector/nodes/report.py`),
|
|
the single place where findings are scored and formatted, so the CLI and any
|
|
future REST API behave identically. The CLI loads the baseline file into a
|
|
`skillspector.suppression.Baseline` and passes it via graph state
|
|
(`state["baseline"]`, `state["show_suppressed"]`); the report node partitions
|
|
findings into kept vs. suppressed via
|
|
`skillspector.suppression.partition_findings`.
|
|
|
|
## Recommended workflow
|
|
|
|
1. Triage the first scan. For genuine false positives, prefer a `rules` entry
|
|
with a clear `reason` (drift-tolerant). For "accept everything as-is right
|
|
now", run `skillspector baseline` to fingerprint them.
|
|
2. Commit the baseline file to the repo.
|
|
3. In CI, run `skillspector scan <path> --baseline <file>`; the build fails
|
|
(exit 1) only when a **new** finding pushes the risk score above threshold.
|
|
4. Periodically review with `--show-suppressed` and prune stale entries.
|