chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,640 @@
|
||||
---
|
||||
sidebar_label: CI/CD
|
||||
description: Integrate ModelAudit into GitHub Actions, GitLab CI, Jenkins, and CircleCI pipelines for automated ML model security scanning with SARIF output and deployment gates
|
||||
keywords:
|
||||
[
|
||||
modelaudit ci cd,
|
||||
github actions model scanning,
|
||||
gitlab ci model security,
|
||||
jenkins model scanning,
|
||||
circleci model security,
|
||||
automated model scanning,
|
||||
sarif integration,
|
||||
model security automation,
|
||||
production deployment gates,
|
||||
scheduled security scans,
|
||||
strict mode validation,
|
||||
exit codes,
|
||||
security pipeline integration,
|
||||
github code scanning,
|
||||
model scan workflow,
|
||||
continuous security,
|
||||
ml security automation,
|
||||
automated vulnerability scanning,
|
||||
]
|
||||
---
|
||||
|
||||
# CI/CD Integration
|
||||
|
||||
ModelAudit integrates into CI/CD pipelines to automatically scan ML model files for security vulnerabilities before deployment.
|
||||
|
||||
## Exit Codes
|
||||
|
||||
ModelAudit uses specific exit codes for CI/CD automation:
|
||||
|
||||
- **0**: No security issues found ✅
|
||||
- **1**: Security issues detected (warnings or critical findings) 🟡
|
||||
- **2**: Operational errors or inconclusive scans (file access, installation, timeouts, or no scanned files) 🔴
|
||||
|
||||
In CI/CD pipelines, exit code 1 indicates findings that should be reviewed. Only exit code 2 represents actual scan failures.
|
||||
|
||||
## GitHub Actions
|
||||
|
||||
### Scan Changed Model Files
|
||||
|
||||
Scan model files modified in pull requests:
|
||||
|
||||
```yaml
|
||||
name: Model Security Scan
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
|
||||
jobs:
|
||||
scan-models:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install Promptfoo and ModelAudit
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
|
||||
- name: Get changed model files
|
||||
id: changed-files
|
||||
run: |
|
||||
MODEL_EXTENSIONS='pkl|pickle|dill|pth|pt|ckpt|checkpoint|orbax-checkpoint|h5|hdf5|keras|onnx|pb|tflite|safetensors|gguf|ggml|ggmf|ggjt|ggla|ggsa|bin|engine|plan|msgpack|flax|orbax|jax|joblib|skops|npy|npz|pmml|zip|tar|7z|json|yaml|yml|xml|toml|config|jinja|j2|template|bst|model|ubj'
|
||||
CHANGED=$(git diff --name-only --diff-filter=ACM \
|
||||
${{ github.event.pull_request.base.sha }} ${{ github.sha }} | \
|
||||
grep -Ei "\.(${MODEL_EXTENSIONS})$" || true)
|
||||
|
||||
if [ -z "$CHANGED" ]; then
|
||||
echo "has_changes=false" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "$CHANGED"
|
||||
echo "has_changes=true" >> $GITHUB_OUTPUT
|
||||
echo "$CHANGED" > changed_files.txt
|
||||
fi
|
||||
|
||||
- name: Scan changed models
|
||||
if: steps.changed-files.outputs.has_changes == 'true'
|
||||
run: |
|
||||
mkdir -p scan_results
|
||||
while IFS= read -r file; do
|
||||
if [ -f "$file" ]; then
|
||||
echo "Scanning: $file"
|
||||
report_id=$(printf '%s' "$file" | sha256sum | cut -d ' ' -f1)
|
||||
promptfoo scan-model "$file" \
|
||||
--format json \
|
||||
--output "scan_results/${report_id}.json" || true
|
||||
fi
|
||||
done < changed_files.txt
|
||||
|
||||
- name: Upload scan results
|
||||
if: steps.changed-files.outputs.has_changes == 'true'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: model-scan-results
|
||||
path: scan_results/
|
||||
|
||||
- name: Check for critical issues
|
||||
if: steps.changed-files.outputs.has_changes == 'true'
|
||||
run: |
|
||||
jq -es '
|
||||
all(.[];
|
||||
type == "object" and
|
||||
(.issues | type == "array") and
|
||||
(.checks | type == "array") and
|
||||
(.has_errors | type == "boolean") and
|
||||
(.failed_checks | type == "number") and
|
||||
(.has_errors == false) and
|
||||
(.failed_checks == 0) and
|
||||
([.checks[]? | select(.status == "failed")] | length == 0) and
|
||||
([.issues[]? | select(.severity == "critical" or .severity == "error")] | length == 0)
|
||||
)
|
||||
' scan_results/*.json >/dev/null
|
||||
|
||||
WARNING_COUNT=$(jq -es '[.[] | .issues[]? | select(.severity == "warning")] | length' scan_results/*.json)
|
||||
|
||||
if [ "$WARNING_COUNT" -gt 0 ]; then
|
||||
echo "⚠️ Found warnings in $WARNING_COUNT file(s)"
|
||||
exit 0
|
||||
else
|
||||
echo "✅ No security issues detected"
|
||||
fi
|
||||
```
|
||||
|
||||
### Upload to GitHub Advanced Security
|
||||
|
||||
Use SARIF format to integrate with GitHub Code Scanning:
|
||||
|
||||
```yaml
|
||||
name: Model Security Scan with SARIF
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- '**/*.pkl'
|
||||
- '**/*.pth'
|
||||
- '**/*.h5'
|
||||
- '**/*.onnx'
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
scan-models:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
security-events: write
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install tools
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
|
||||
- name: Scan models directory
|
||||
id: scan
|
||||
continue-on-error: true
|
||||
run: |
|
||||
promptfoo scan-model models/ \
|
||||
--no-write \
|
||||
--format sarif \
|
||||
--output modelaudit.sarif
|
||||
|
||||
- name: Upload SARIF to GitHub
|
||||
if: always()
|
||||
uses: github/codeql-action/upload-sarif@v3
|
||||
with:
|
||||
sarif_file: modelaudit.sarif
|
||||
category: model-security
|
||||
|
||||
- name: Fail if scan found issues or errors
|
||||
if: steps.scan.outcome == 'failure'
|
||||
run: exit 1
|
||||
```
|
||||
|
||||
### Scheduled Security Scans
|
||||
|
||||
Run periodic scans on all models:
|
||||
|
||||
```yaml
|
||||
name: Weekly Model Security Audit
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 2 * * 0' # Sundays at 2 AM UTC
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
comprehensive-scan:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install tools
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit[all]
|
||||
|
||||
- name: Comprehensive model scan
|
||||
continue-on-error: true
|
||||
run: |
|
||||
promptfoo scan-model models/ \
|
||||
--format json \
|
||||
--output scan_results.json \
|
||||
--strict \
|
||||
--sbom sbom.json
|
||||
|
||||
- name: Generate SBOM
|
||||
run: |
|
||||
echo "Software Bill of Materials generated: sbom.json"
|
||||
|
||||
- name: Check for critical issues
|
||||
id: check-issues
|
||||
run: |
|
||||
if ! jq -e '
|
||||
type == "object" and
|
||||
(.issues | type == "array") and
|
||||
(.checks | type == "array") and
|
||||
(.has_errors | type == "boolean") and
|
||||
(.failed_checks | type == "number") and
|
||||
(.has_errors == false) and
|
||||
(.failed_checks == 0) and
|
||||
([.checks[]? | select(.status == "failed")] | length == 0) and
|
||||
([.issues[]? | select(.severity == "critical" or .severity == "error")] | length == 0)
|
||||
' scan_results.json >/dev/null; then
|
||||
echo "critical=true" >> $GITHUB_OUTPUT
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Create issue if scan fails closed
|
||||
if: failure() && steps.check-issues.outputs.critical == 'true'
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const results = JSON.parse(fs.readFileSync('scan_results.json', 'utf8'));
|
||||
|
||||
const issues = Array.isArray(results.issues) ? results.issues : [];
|
||||
const criticalIssues = issues
|
||||
.filter(i => i.severity === 'critical' || i.severity === 'error')
|
||||
.map(i => `- ${i.message}`)
|
||||
.join('\n');
|
||||
const failedCheckCount = results.failed_checks || 0;
|
||||
const findingsSummary = criticalIssues || `- Failed checks: ${failedCheckCount}`;
|
||||
|
||||
await github.rest.issues.create({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
title: '🚨 Critical Model Security Issues Detected',
|
||||
body: `Weekly security scan found non-clean results:\n\n${findingsSummary}`,
|
||||
labels: ['security', 'critical', 'model-audit']
|
||||
});
|
||||
|
||||
- name: Upload artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: weekly-audit-results
|
||||
path: |
|
||||
scan_results.json
|
||||
sbom.json
|
||||
```
|
||||
|
||||
### Strict Mode for Production
|
||||
|
||||
Block deployments on any security findings:
|
||||
|
||||
```yaml
|
||||
name: Production Model Validation
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'models/production/**'
|
||||
|
||||
jobs:
|
||||
strict-validation:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '24'
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install tools
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit[all]
|
||||
|
||||
- name: Strict security scan
|
||||
run: |
|
||||
promptfoo scan-model models/production/ \
|
||||
--strict \
|
||||
--format json \
|
||||
--output results.json
|
||||
```
|
||||
|
||||
The scan step fails if strict mode finds warnings or critical issues.
|
||||
|
||||
## GitLab CI
|
||||
|
||||
Example `.gitlab-ci.yml`:
|
||||
|
||||
```yaml
|
||||
model-security-scan:
|
||||
stage: test
|
||||
image: node:24
|
||||
|
||||
before_script:
|
||||
- apt-get update && apt-get install -y python3 python3-pip
|
||||
- npm install -g promptfoo
|
||||
- pip3 install modelaudit
|
||||
|
||||
script:
|
||||
- |
|
||||
MODEL_EXTENSIONS='pkl|pickle|dill|pth|pt|ckpt|checkpoint|orbax-checkpoint|h5|hdf5|keras|onnx|pb|tflite|safetensors|gguf|ggml|ggmf|ggjt|ggla|ggsa|bin|engine|plan|msgpack|flax|orbax|jax|joblib|skops|npy|npz|pmml|zip|tar|7z|json|yaml|yml|xml|toml|config|jinja|j2|template|bst|model|ubj'
|
||||
CHANGED=$(git diff --name-only --diff-filter=ACM $CI_COMMIT_BEFORE_SHA $CI_COMMIT_SHA | \
|
||||
grep -Ei "\.(${MODEL_EXTENSIONS})$" || true)
|
||||
|
||||
if [ -n "$CHANGED" ]; then
|
||||
mkdir -p scan_results
|
||||
printf '%s\n' "$CHANGED" > changed_files.txt
|
||||
scan_status=0
|
||||
while IFS= read -r file; do
|
||||
if [ -f "$file" ]; then
|
||||
report_id=$(printf '%s' "$file" | sha256sum | cut -d ' ' -f1)
|
||||
file_status=0
|
||||
promptfoo scan-model "$file" \
|
||||
--format json \
|
||||
--output "scan_results/${report_id}.json" || file_status=$?
|
||||
if [ "$file_status" -gt "$scan_status" ]; then scan_status=$file_status; fi
|
||||
fi
|
||||
done < changed_files.txt
|
||||
exit "$scan_status"
|
||||
else
|
||||
echo "No model files changed"
|
||||
fi
|
||||
|
||||
artifacts:
|
||||
paths:
|
||||
- scan_results/
|
||||
when: always
|
||||
expire_in: 30 days
|
||||
|
||||
only:
|
||||
- merge_requests
|
||||
- main
|
||||
```
|
||||
|
||||
## Jenkins
|
||||
|
||||
Example Jenkinsfile:
|
||||
|
||||
```groovy
|
||||
pipeline {
|
||||
agent any
|
||||
|
||||
stages {
|
||||
stage('Setup') {
|
||||
steps {
|
||||
sh '''
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
'''
|
||||
}
|
||||
}
|
||||
|
||||
stage('Scan Models') {
|
||||
steps {
|
||||
script {
|
||||
def changed = sh(
|
||||
script: '''
|
||||
MODEL_EXTENSIONS='pkl|pickle|dill|pth|pt|ckpt|checkpoint|orbax-checkpoint|h5|hdf5|keras|onnx|pb|tflite|safetensors|gguf|ggml|ggmf|ggjt|ggla|ggsa|bin|engine|plan|msgpack|flax|orbax|jax|joblib|skops|npy|npz|pmml|zip|tar|7z|json|yaml|yml|xml|toml|config|jinja|j2|template|bst|model|ubj'
|
||||
git diff --name-only HEAD~1 HEAD | \
|
||||
grep -Ei "\\.(${MODEL_EXTENSIONS})$" || true
|
||||
''',
|
||||
returnStdout: true
|
||||
).trim()
|
||||
|
||||
if (changed) {
|
||||
changed.split('\n').each { file ->
|
||||
def reportFile = "scan_${file.bytes.encodeHex().toString()}.json"
|
||||
withEnv(["MODEL_FILE=${file}", "REPORT_FILE=${reportFile}"]) {
|
||||
sh '''
|
||||
promptfoo scan-model "$MODEL_FILE" \
|
||||
--format json \
|
||||
--output "$REPORT_FILE"
|
||||
'''
|
||||
}
|
||||
}
|
||||
|
||||
// Check for critical issues
|
||||
def critical = sh(
|
||||
script: '''
|
||||
jq -es '
|
||||
all(.[];
|
||||
type == "object" and
|
||||
(.issues | type == "array") and
|
||||
(.checks | type == "array") and
|
||||
(.has_errors | type == "boolean") and
|
||||
(.failed_checks | type == "number") and
|
||||
(.has_errors == false) and
|
||||
(.failed_checks == 0) and
|
||||
([.checks[]? | select(.status == "failed")] | length == 0) and
|
||||
([.issues[]? | select(.severity == "critical" or .severity == "error")] | length == 0)
|
||||
)
|
||||
' scan_*.json >/dev/null
|
||||
''',
|
||||
returnStatus: true
|
||||
)
|
||||
|
||||
if (critical != 0) {
|
||||
error("Found model scan failures or critical security issues")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
post {
|
||||
always {
|
||||
archiveArtifacts artifacts: 'scan_*.json', allowEmptyArchive: true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## CircleCI
|
||||
|
||||
Example `.circleci/config.yml`:
|
||||
|
||||
```yaml
|
||||
version: 2.1
|
||||
|
||||
jobs:
|
||||
model-scan:
|
||||
docker:
|
||||
- image: cimg/node:lts
|
||||
|
||||
steps:
|
||||
- checkout
|
||||
|
||||
- run:
|
||||
name: Install tools
|
||||
command: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
|
||||
- run:
|
||||
name: Scan changed models
|
||||
command: |
|
||||
MODEL_EXTENSIONS='pkl|pickle|dill|pth|pt|ckpt|checkpoint|orbax-checkpoint|h5|hdf5|keras|onnx|pb|tflite|safetensors|gguf|ggml|ggmf|ggjt|ggla|ggsa|bin|engine|plan|msgpack|flax|orbax|jax|joblib|skops|npy|npz|pmml|zip|tar|7z|json|yaml|yml|xml|toml|config|jinja|j2|template|bst|model|ubj'
|
||||
CHANGED=$(git diff --name-only origin/main...HEAD | \
|
||||
grep -Ei "\.(${MODEL_EXTENSIONS})$" || true)
|
||||
|
||||
if [ -n "$CHANGED" ]; then
|
||||
echo "$CHANGED" | while read -r file; do
|
||||
if [ -f "$file" ]; then
|
||||
promptfoo scan-model "$file" --format json >> scan_results.json
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
- store_artifacts:
|
||||
path: scan_results.json
|
||||
destination: model-scan-results
|
||||
|
||||
workflows:
|
||||
security:
|
||||
jobs:
|
||||
- model-scan:
|
||||
filters:
|
||||
branches:
|
||||
only:
|
||||
- main
|
||||
- develop
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Scan Strategy
|
||||
|
||||
- **Pull Requests**: Scan only changed files for fast feedback
|
||||
- **Main Branch**: Run comprehensive scans on merge
|
||||
- **Scheduled Scans**: Weekly or daily full audits
|
||||
- **Production Gate**: Use `--strict` mode to block deployments
|
||||
|
||||
### Performance
|
||||
|
||||
```yaml
|
||||
# Cache dependencies
|
||||
- name: Cache npm and pip
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
~/.npm
|
||||
~/.cache/pip
|
||||
key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json', '**/requirements.txt') }}
|
||||
|
||||
# Parallel scanning
|
||||
- name: Parallel scan
|
||||
run: |
|
||||
find models/ -name "*.pkl" -print0 | \
|
||||
xargs -0 -P 4 -I {} promptfoo scan-model {} --format json --output {}.json
|
||||
```
|
||||
|
||||
### Security
|
||||
|
||||
- Store credentials as encrypted secrets
|
||||
- Use read-only tokens when possible
|
||||
- Rotate credentials regularly
|
||||
- Audit access to scan results
|
||||
|
||||
### Timeout Configuration
|
||||
|
||||
For large models (8GB+):
|
||||
|
||||
```yaml
|
||||
- name: Scan large model
|
||||
run: |
|
||||
promptfoo scan-model large_model.bin \
|
||||
--timeout 1800 \
|
||||
--verbose \
|
||||
--format json \
|
||||
--output results.json
|
||||
```
|
||||
|
||||
### Retry Logic
|
||||
|
||||
```yaml
|
||||
- name: Scan with retry
|
||||
uses: nick-fields/retry@v2
|
||||
with:
|
||||
timeout_minutes: 10
|
||||
max_attempts: 3
|
||||
command: promptfoo scan-model models/ --format json --output results.json
|
||||
```
|
||||
|
||||
### Notifications
|
||||
|
||||
```yaml
|
||||
- name: Notify on critical issues
|
||||
if: failure()
|
||||
uses: slackapi/slack-github-action@v1
|
||||
with:
|
||||
payload: |
|
||||
{
|
||||
"text": "🚨 Critical model security issues detected!",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": "*Model Security Scan Failed*\nRepository: ${{ github.repository }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Details>"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
env:
|
||||
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Verbose Output
|
||||
|
||||
```bash
|
||||
promptfoo scan-model models/ --verbose
|
||||
```
|
||||
|
||||
### File Size Limits
|
||||
|
||||
```bash
|
||||
# Set maximum file size
|
||||
promptfoo scan-model models/ --max-size 1GB
|
||||
```
|
||||
|
||||
### Dry Run
|
||||
|
||||
```bash
|
||||
# Preview scan without processing
|
||||
promptfoo scan-model models/ --dry-run
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Scanner Reference](./scanners.md) - Security checks performed
|
||||
- [Advanced Usage](./usage.md) - Cloud storage, authentication, remote sources
|
||||
- [Overview](./index.md) - Getting started with ModelAudit
|
||||
@@ -0,0 +1,350 @@
|
||||
---
|
||||
title: ModelAudit - Static Security Scanner for ML Models
|
||||
description: Scan AI/ML models for security vulnerabilities, malicious code, and backdoors. Supports PyTorch, TensorFlow, ONNX, Keras, and 30+ model formats.
|
||||
keywords:
|
||||
[
|
||||
model security,
|
||||
AI security,
|
||||
ML security scanning,
|
||||
static analysis,
|
||||
malicious model detection,
|
||||
pytorch security,
|
||||
tensorflow security,
|
||||
model vulnerability scanner,
|
||||
]
|
||||
sidebar_label: Overview
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Model Scanning
|
||||
|
||||
## Overview
|
||||
|
||||
ModelAudit is a lightweight static security scanner for machine learning models accessible through Promptfoo. It scans AI/ML models for potential security risks before deployment.
|
||||
|
||||
Promptfoo provides a wrapper command `promptfoo scan-model` that integrates ModelAudit scanning capabilities.
|
||||
|
||||
|
||||
|
||||
Promptfoo also includes a UI that allows you to set up a scan:
|
||||
|
||||
|
||||
|
||||
And displays the results:
|
||||
|
||||
|
||||
|
||||
## Purpose
|
||||
|
||||
AI/ML models can introduce security risks through:
|
||||
|
||||
- Malicious code embedded in pickled models
|
||||
- Suspicious TensorFlow operations
|
||||
- Potentially unsafe Keras Lambda layers
|
||||
- Dangerous pickle opcodes
|
||||
- Encoded payloads hidden in model structures
|
||||
- Risky configurations in model architectures
|
||||
- Malicious content in ZIP archives
|
||||
- Embedded executables in binary model files
|
||||
- Hidden credentials (API keys, tokens, passwords)
|
||||
- Network communication patterns (URLs, IPs, sockets)
|
||||
- JIT/Script execution in TorchScript and ONNX models
|
||||
|
||||
ModelAudit helps identify these risks before models are deployed to production environments, ensuring a more secure AI pipeline.
|
||||
|
||||
## Installation
|
||||
|
||||
### Using Promptfoo
|
||||
|
||||
The easiest way to use ModelAudit is through Promptfoo:
|
||||
|
||||
```bash
|
||||
# Install Promptfoo globally
|
||||
npm install -g promptfoo
|
||||
|
||||
# Install modelaudit dependency
|
||||
pip install modelaudit
|
||||
```
|
||||
|
||||
### Standalone Installation
|
||||
|
||||
You can also install ModelAudit directly:
|
||||
|
||||
```bash
|
||||
# Basic installation
|
||||
pip install modelaudit
|
||||
|
||||
# With optional dependencies for specific model formats
|
||||
pip install modelaudit[tensorflow,h5,pytorch]
|
||||
|
||||
# For all dependencies
|
||||
pip install modelaudit[all]
|
||||
|
||||
# Or install specific components:
|
||||
pip install modelaudit[tensorflow,h5,pytorch] # Core ML frameworks
|
||||
pip install modelaudit[cloud,mlflow] # Remote model access
|
||||
pip install modelaudit[numpy1] # NumPy 1.x compatibility
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull from GitHub Container Registry
|
||||
docker pull ghcr.io/promptfoo/modelaudit:latest
|
||||
|
||||
# Use specific variants
|
||||
docker pull ghcr.io/promptfoo/modelaudit:latest-full # All ML frameworks
|
||||
docker pull ghcr.io/promptfoo/modelaudit:latest-tensorflow # TensorFlow only
|
||||
|
||||
# Run with Docker
|
||||
docker run --rm -v $(pwd):/data ghcr.io/promptfoo/modelaudit:latest scan /data/model.pkl
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Command Structure
|
||||
|
||||
```bash
|
||||
promptfoo scan-model [OPTIONS] PATH...
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Scan a single model file
|
||||
promptfoo scan-model model.pkl
|
||||
|
||||
# Scan a model directly from HuggingFace without downloading
|
||||
promptfoo scan-model https://huggingface.co/bert-base-uncased
|
||||
promptfoo scan-model hf://microsoft/resnet-50
|
||||
|
||||
# Scan from cloud storage
|
||||
promptfoo scan-model s3://my-bucket/model.pt
|
||||
promptfoo scan-model gs://my-bucket/model.h5
|
||||
|
||||
# Scan from MLflow registry
|
||||
promptfoo scan-model models:/MyModel/1
|
||||
|
||||
# Scan multiple models and directories
|
||||
promptfoo scan-model model.pkl model2.h5 models_directory
|
||||
|
||||
# Export results to JSON
|
||||
promptfoo scan-model model.pkl --format json --output results.json
|
||||
|
||||
# Export results to SARIF for security tool integration
|
||||
promptfoo scan-model model.pkl --no-write --format sarif --output results.sarif
|
||||
|
||||
# Add custom blacklist patterns
|
||||
promptfoo scan-model model.pkl --blacklist "unsafe_model" --blacklist "malicious_net"
|
||||
|
||||
# Enable verbose output
|
||||
promptfoo scan-model model.pkl --verbose
|
||||
|
||||
# Set file size limits
|
||||
promptfoo scan-model models/ --max-size 1GB
|
||||
|
||||
# Generate Software Bill of Materials
|
||||
promptfoo scan-model model.pkl --sbom sbom.json
|
||||
|
||||
# Enable strict mode for security-critical scans
|
||||
promptfoo scan-model model.pkl --strict
|
||||
|
||||
# List scanner IDs and run only selected scanners
|
||||
promptfoo scan-model --list-scanners
|
||||
promptfoo scan-model models/ --scanners pickle,tf_savedmodel
|
||||
|
||||
# Preview scan without actually processing
|
||||
promptfoo scan-model model.pkl --dry-run
|
||||
```
|
||||
|
||||
See the [Advanced Usage](./usage.md) guide for detailed authentication setup for cloud storage, JFrog, and other remote sources.
|
||||
|
||||
:::info Alternative Installation and Usage
|
||||
|
||||
- **Standalone**: Install modelaudit directly using `pip install modelaudit`. `modelaudit scan` behaves the same as `promptfoo scan-model`.
|
||||
- **Web Interface**: For a GUI experience, use `promptfoo view` and navigate to `/model-audit` for visual scanning and configuration.
|
||||
|
||||
:::
|
||||
|
||||
### Options
|
||||
|
||||
| Option | Description |
|
||||
| ------------------- | --------------------------------------------------------------- |
|
||||
| `--blacklist`, `-b` | Additional blacklist patterns to check against model names |
|
||||
| `--format`, `-f` | Output format (`text` \| `json` \| `sarif`) [default: text] |
|
||||
| `--output`, `-o` | Output file path (prints to stdout if not specified) |
|
||||
| `--timeout`, `-t` | Scan timeout in seconds [default: 300] |
|
||||
| `--verbose`, `-v` | Enable verbose output |
|
||||
| `--max-size` | Maximum total size to scan (e.g., `500MB`, `1GB`) |
|
||||
| `--sbom` | Generate CycloneDX Software Bill of Materials with license info |
|
||||
| `--strict` | Fail on warnings; enable stricter validation |
|
||||
| `--dry-run` | Preview scan without processing files |
|
||||
| `--scanners` | Only run selected scanners |
|
||||
| `--exclude-scanner` | Exclude scanners from the default scanner set |
|
||||
| `--list-scanners` | Print available scanner IDs and class names |
|
||||
| `--quiet` | Suppress non-critical output |
|
||||
| `--progress` | Force-enable progress reporting |
|
||||
| `--no-cache` | Disable caching of downloaded files |
|
||||
| `--no-write` | Skip writing results to database |
|
||||
|
||||
## Web Interface
|
||||
|
||||
Promptfoo includes a web interface for ModelAudit at `/model-audit` with visual path selection, real-time progress tracking, and detailed results visualization.
|
||||
|
||||
**Access:** Run `promptfoo view` and navigate to `http://localhost:15500/model-audit`
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- Visual file/directory selection with current working directory context
|
||||
- GUI configuration for all scan options (blacklist patterns, timeouts, file limits)
|
||||
- Live scanning progress and tabbed results display with severity color coding
|
||||
- Scan history and automatic installation detection
|
||||
|
||||
## Supported Formats
|
||||
|
||||
ModelAudit supports scanning 30+ specialized file format scanners across major ML frameworks:
|
||||
|
||||
### Model Formats
|
||||
|
||||
| Format | Extensions | Description |
|
||||
| ------------------------- | ---------------------------------------------------- | ------------------------------------------------------- |
|
||||
| **PyTorch** | `.pt`, `.pth`, `.bin` | PyTorch model files and checkpoints |
|
||||
| **TensorFlow SavedModel** | `.pb`, directories | TensorFlow's standard model format |
|
||||
| **TensorFlow Lite** | `.tflite` | Mobile-optimized TensorFlow models |
|
||||
| **TensorRT** | `.engine`, `.plan` | NVIDIA GPU-optimized inference engines |
|
||||
| **Keras** | `.h5`, `.keras`, `.hdf5` | Keras/TensorFlow models in HDF5 format |
|
||||
| **ONNX** | `.onnx` | Open Neural Network Exchange format |
|
||||
| **SafeTensors** | `.safetensors` | Hugging Face's secure tensor format |
|
||||
| **GGUF/GGML** | `.gguf`, `.ggml`, `.ggmf`, `.ggjt`, `.ggla`, `.ggsa` | Quantized models (LLaMA, Mistral, etc.) |
|
||||
| **Flax/JAX** | `.msgpack`, `.flax`, `.orbax`, `.jax` | JAX-based model formats |
|
||||
| **JAX Checkpoints** | `.ckpt`, `.checkpoint`, `.orbax-checkpoint` | JAX training checkpoints |
|
||||
| **Pickle** | `.pkl`, `.pickle`, `.dill` | Python serialization (includes Dill) |
|
||||
| **Joblib** | `.joblib` | Scikit-learn and general ML serialization |
|
||||
| **NumPy** | `.npy`, `.npz` | NumPy array storage formats |
|
||||
| **PMML** | `.pmml` | Predictive Model Markup Language (XML) |
|
||||
| **ZIP Archives** | `.zip` | Compressed model archives with recursive scanning |
|
||||
| **Container Manifests** | `.manifest` | OCI/Docker layer scanning |
|
||||
| **Binary Files** | `.bin` | Auto-detected format (PyTorch, ONNX, SafeTensors, etc.) |
|
||||
|
||||
### Remote Sources
|
||||
|
||||
| Source | URL Format | Example |
|
||||
| ------------------------ | ---------------------------------------------------- | ------------------------------------------------------- |
|
||||
| **HuggingFace Hub** | `https://huggingface.co/`, `https://hf.co/`, `hf://` | `hf://microsoft/resnet-50` |
|
||||
| **Amazon S3** | `s3://` | `s3://my-bucket/model.pt` |
|
||||
| **Google Cloud Storage** | `gs://` | `gs://my-bucket/model.h5` |
|
||||
| **Cloudflare R2** | `r2://` | `r2://my-bucket/model.safetensors` |
|
||||
| **MLflow Registry** | `models:/` | `models:/MyModel/1` |
|
||||
| **JFrog Artifactory** | `https://*.jfrog.io/` | `https://company.jfrog.io/artifactory/models/model.pkl` |
|
||||
| **DVC** | `.dvc` files | `model.pkl.dvc` |
|
||||
|
||||
## Security Checks Performed
|
||||
|
||||
The scanner looks for various security issues, including:
|
||||
|
||||
- **Malicious Code**: Detecting potentially dangerous code in pickled models
|
||||
- **Suspicious Operations**: Identifying risky TensorFlow operations and custom ONNX operators
|
||||
- **Unsafe Layers**: Finding potentially unsafe Keras Lambda layers
|
||||
- **Blacklisted Names**: Checking for models with names matching suspicious patterns
|
||||
- **Dangerous Serialization**: Detecting unsafe pickle opcodes, nested pickle payloads, and decode-exec chains
|
||||
- **Enhanced Dill/Joblib Security**: ML-aware scanning with format validation and bypass prevention
|
||||
- **Encoded Payloads**: Looking for suspicious strings that might indicate hidden code
|
||||
- **Risky Configurations**: Identifying dangerous settings in model architectures
|
||||
- **XML Security**: Detecting XXE attacks and malicious content in PMML files
|
||||
- **Embedded Executables**: Detecting Windows PE, Linux ELF, and macOS Mach-O files
|
||||
- **Container Security**: Scanning model files within OCI/Docker container layers
|
||||
- **Compression Attacks**: Detecting zip bombs and decompression attacks
|
||||
- **Weight Anomalies**: Statistical analysis to detect potential backdoors
|
||||
- **Format Integrity**: Validating file format structure
|
||||
- **License Compliance**: Detecting AGPL obligations and commercial restrictions
|
||||
- **DVC Integration**: Automatic resolution and scanning of DVC-tracked models
|
||||
- **Secrets Detection**: Finding embedded API keys, tokens, and credentials
|
||||
- **Network Analysis**: Detecting URLs, IPs, and socket usage that could enable data exfiltration
|
||||
- **JIT Code Detection**: Scanning TorchScript, ONNX custom ops, and other JIT-compiled code
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
The scan results are classified by severity:
|
||||
|
||||
- **CRITICAL**: Definite security concerns that should be addressed immediately
|
||||
- **WARNING**: Potential issues that require review
|
||||
- **INFO**: Informational findings, not necessarily security concerns
|
||||
- **DEBUG**: Additional details (only shown with `--verbose`)
|
||||
|
||||
Some issues include a "Why" explanation to help understand the security risk:
|
||||
|
||||
```
|
||||
1. suspicious_model.pkl (pos 28): [CRITICAL] Suspicious module reference found: posix.system
|
||||
Why: The 'os' module provides direct access to operating system functions.
|
||||
```
|
||||
|
||||
## Integration in Workflows
|
||||
|
||||
ModelAudit is particularly useful in CI/CD pipelines when incorporated with Promptfoo:
|
||||
|
||||
```bash
|
||||
# Example CI/CD script segment
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit
|
||||
promptfoo scan-model --format json --output scan-results.json ./models/
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Security issues found in models! Check scan-results.json"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### Exit Codes
|
||||
|
||||
ModelAudit returns specific exit codes for automation:
|
||||
|
||||
- **0**: No security issues found ✅
|
||||
- **1**: Security issues detected (warnings or critical) 🟡
|
||||
- **2**: Operational errors or inconclusive scans occurred (installation, file access, timeouts, no scanned files, etc.) 🔴
|
||||
|
||||
:::tip CI/CD Best Practice
|
||||
In CI/CD pipelines, exit code 1 indicates findings that should be reviewed but don't necessarily block deployment. Only exit code 2 represents actual scan failures.
|
||||
:::
|
||||
|
||||
## Requirements
|
||||
|
||||
ModelAudit is included with Promptfoo, but specific model formats may require additional dependencies:
|
||||
|
||||
```bash
|
||||
# For TensorFlow models
|
||||
pip install tensorflow
|
||||
|
||||
# For PyTorch models
|
||||
pip install torch
|
||||
|
||||
# For Keras models with HDF5
|
||||
pip install h5py
|
||||
|
||||
# For YAML configuration scanning
|
||||
pip install pyyaml
|
||||
|
||||
# For SafeTensors support
|
||||
pip install safetensors
|
||||
|
||||
# For HuggingFace URL scanning
|
||||
pip install huggingface-hub
|
||||
|
||||
# For cloud storage scanning
|
||||
pip install boto3 google-cloud-storage
|
||||
|
||||
# For MLflow registry scanning
|
||||
pip install mlflow
|
||||
```
|
||||
|
||||
### NumPy Compatibility
|
||||
|
||||
ModelAudit supports both NumPy 1.x and 2.x. If you encounter NumPy compatibility issues:
|
||||
|
||||
```bash
|
||||
# Force NumPy 1.x if needed for full compatibility
|
||||
pip install modelaudit[numpy1]
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Advanced Usage](./usage.md)
|
||||
- [Scanner Reference](./scanners.md)
|
||||
@@ -0,0 +1,697 @@
|
||||
---
|
||||
description: Complete guide to ModelAudit's security scanners for different ML model formats including PyTorch, TensorFlow, Keras, ONNX, GGUF, and more.
|
||||
keywords:
|
||||
[
|
||||
modelaudit,
|
||||
model security,
|
||||
AI security,
|
||||
ML security scanning,
|
||||
pickle scanner,
|
||||
pytorch security,
|
||||
tensorflow security,
|
||||
keras security,
|
||||
onnx security,
|
||||
model vulnerability detection,
|
||||
malicious code detection,
|
||||
backdoor detection,
|
||||
model file scanning,
|
||||
]
|
||||
sidebar_label: Scanners
|
||||
sidebar_position: 200
|
||||
---
|
||||
|
||||
# ModelAudit Scanners
|
||||
|
||||
ModelAudit includes specialized scanners for different model formats and file types. Each scanner is designed to identify specific security issues relevant to that format.
|
||||
|
||||
## Selecting Scanners
|
||||
|
||||
Use `--list-scanners` to see the scanner IDs supported by your installed ModelAudit version. Then pass IDs or class names to `--scanners` to run only those scanners, or use `--exclude-scanner` to remove scanners from the default set.
|
||||
|
||||
```bash
|
||||
promptfoo scan-model --list-scanners
|
||||
promptfoo scan-model models/ --scanners pickle,tf_savedmodel
|
||||
promptfoo scan-model models/ --exclude-scanner weight_distribution
|
||||
```
|
||||
|
||||
The web UI exposes the same catalog in Advanced Scan Options.
|
||||
|
||||
## Pickle Scanner
|
||||
|
||||
**File types:** `.pkl`, `.pickle`, `.dill`, `.bin` (when containing pickle data), `.pt`, `.pth`, `.ckpt`
|
||||
|
||||
The Pickle Scanner analyzes Python pickle files for security risks, which are common in many ML frameworks. It supports standard pickle files as well as dill-serialized files (an extended pickle format).
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious module imports (e.g., `os`, `subprocess`, `sys`)
|
||||
- Dangerous functions (e.g., `eval`, `exec`, `system`)
|
||||
- Malicious pickle opcodes (REDUCE, INST, OBJ, NEWOBJ, STACK_GLOBAL)
|
||||
- Encoded payloads and suspicious string patterns
|
||||
- Embedded executables in binary content
|
||||
- ML context detection to reduce false positives
|
||||
- Network communication patterns (URLs, IPs, sockets)
|
||||
- Embedded credentials (API keys, tokens, passwords)
|
||||
- JIT/Script execution patterns
|
||||
|
||||
**Why it matters:**
|
||||
Pickle files are a common serialization format for ML models but can execute arbitrary code during unpickling. Attackers can craft malicious pickle files that execute harmful commands when loaded.
|
||||
|
||||
## TensorFlow SavedModel Scanner
|
||||
|
||||
**File types:** `.pb` files and SavedModel directories
|
||||
|
||||
This scanner examines TensorFlow models saved in the SavedModel format.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious TensorFlow operations that could access files or the system
|
||||
- Python function calls embedded in the graph
|
||||
- Operations that allow arbitrary code execution (e.g., `PyFunc`)
|
||||
- File I/O operations that might access unexpected locations
|
||||
- Execution operations that could run system commands
|
||||
|
||||
**Why it matters:**
|
||||
TensorFlow models can contain operations that interact with the filesystem or execute arbitrary code, which could be exploited if a malicious model is loaded.
|
||||
|
||||
## TensorFlow Lite Scanner
|
||||
|
||||
**File types:** `.tflite`
|
||||
|
||||
This scanner examines TensorFlow Lite model files, which are optimized for mobile and embedded devices.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Custom operations that could contain malicious code
|
||||
- Flex delegate operations that enable full TensorFlow ops execution
|
||||
- Model metadata that could contain executable content
|
||||
- Suspicious operator configurations or patterns
|
||||
- Buffer validation to detect tampering
|
||||
|
||||
**Why it matters:**
|
||||
While TensorFlow Lite models are generally safer than full TensorFlow models due to their limited operator set, they can still include custom operations or use the Flex delegate to access the full TensorFlow runtime, potentially introducing security risks. Malicious actors could embed harmful code in custom ops or metadata.
|
||||
|
||||
## TensorRT Scanner
|
||||
|
||||
**File types:** `.engine`, `.plan`
|
||||
|
||||
This scanner examines NVIDIA TensorRT engine files, which are optimized inference engines for NVIDIA GPUs.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious file paths (`/tmp/`, `../`) that might indicate unauthorized access
|
||||
- Embedded shared library references (`.so` files) that could contain malicious code
|
||||
- Script execution patterns (`exec`, `eval`) that could run arbitrary code
|
||||
- Unauthorized plugin references that might load malicious extensions
|
||||
|
||||
**Why it matters:**
|
||||
TensorRT engines can contain custom plugins and operations. While generally safer than pickle files, they could be crafted to include malicious plugins or reference unauthorized system resources.
|
||||
|
||||
## Keras H5 Scanner
|
||||
|
||||
**File types:** `.h5`, `.hdf5`, `.keras`
|
||||
|
||||
This scanner analyzes Keras models stored in HDF5 format.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Unsafe Lambda layers that could contain arbitrary Python code
|
||||
- Suspicious layer configurations with embedded code
|
||||
- Custom layers or metrics that might execute malicious code
|
||||
- Dangerous string patterns in model configurations
|
||||
|
||||
**Why it matters:**
|
||||
Keras models with Lambda layers can contain arbitrary Python code that executes when the model is loaded or run. This could be exploited to execute malicious code on the host system.
|
||||
|
||||
## Keras ZIP Scanner
|
||||
|
||||
**File types:** `.keras`
|
||||
|
||||
This scanner analyzes ZIP-based Keras model files (new `.keras` format introduced in Keras 3).
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Unsafe Lambda layers with base64-encoded Python code
|
||||
- Suspicious layer configurations and custom objects
|
||||
- Python files or executables embedded in the ZIP archive
|
||||
- Dangerous patterns in model configuration JSON
|
||||
|
||||
**Why it matters:**
|
||||
The new `.keras` ZIP format stores Lambda layers as base64-encoded functions that execute during inference. Malicious actors could embed arbitrary code in these layers or hide executables within the archive structure.
|
||||
|
||||
## ONNX Scanner
|
||||
|
||||
**File types:** `.onnx`
|
||||
|
||||
This scanner examines ONNX (Open Neural Network Exchange) model files for security issues and integrity problems.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Custom operators that might contain malicious functionality
|
||||
- External data file references and path traversal attempts
|
||||
- Tensor size and data integrity validation
|
||||
- File size mismatches that could indicate tampering
|
||||
|
||||
**Why it matters:**
|
||||
ONNX models can reference external data files and custom operators. Malicious actors could exploit these features to include harmful custom operations or manipulate external data references to access unauthorized files on the system.
|
||||
|
||||
## OpenVINO Scanner
|
||||
|
||||
**File types:** `.xml`, `.bin` (OpenVINO IR format)
|
||||
|
||||
This scanner examines Intel OpenVINO Intermediate Representation (IR) model files.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious custom layer configurations
|
||||
- External data references with path traversal attempts
|
||||
- Malformed XML structure or oversized files
|
||||
- Dangerous layer types that could access system resources
|
||||
- Plugin references that might load unauthorized code
|
||||
|
||||
**Why it matters:**
|
||||
OpenVINO models consist of XML topology files and binary weight files. The XML can contain custom layer definitions or external references that could be exploited to execute malicious code or access unauthorized files.
|
||||
|
||||
## PyTorch Zip Scanner
|
||||
|
||||
**File types:** `.pt`, `.pth`
|
||||
|
||||
This scanner examines PyTorch model files, which are ZIP archives containing pickled data.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Malicious pickle files embedded within the PyTorch model
|
||||
- Python code files included in the model archive
|
||||
- Executable scripts or binaries bundled with the model
|
||||
- Suspicious serialization patterns in the embedded pickles
|
||||
|
||||
**Why it matters:**
|
||||
PyTorch models are essentially ZIP archives containing pickled objects, which can include malicious code. The scanner unpacks these archives and applies pickle security checks to the contents.
|
||||
|
||||
## ExecuTorch Scanner
|
||||
|
||||
**File types:** `.pte`, `.pt` (ExecuTorch archives)
|
||||
|
||||
This scanner examines PyTorch ExecuTorch model files designed for mobile and edge deployment.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Embedded pickle files within ExecuTorch archives
|
||||
- Python code or executables bundled in the archive
|
||||
- Suspicious serialization patterns
|
||||
- Custom operators that might contain malicious functionality
|
||||
- Dangerous metadata or configuration data
|
||||
|
||||
**Why it matters:**
|
||||
ExecuTorch models package PyTorch models for edge devices but can still contain pickled data and embedded code. Mobile deployment environments are often resource-constrained and may have limited security monitoring, making them attractive targets.
|
||||
|
||||
## GGUF/GGML Scanner
|
||||
|
||||
**File types:** `.gguf`, `.ggml`, `.ggmf`, `.ggjt`, `.ggla`, `.ggsa`
|
||||
|
||||
This scanner validates GGUF (GPT-Generated Unified Format) and GGML model files commonly used for large language models like LLaMA, Alpaca, and other quantized models.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Header validation**: Verifies file format integrity and header structure
|
||||
- **Metadata security**: Scans JSON metadata for suspicious content and path traversal attempts
|
||||
- **Tensor integrity**: Validates tensor dimensions, types, and data alignment
|
||||
- **Resource limits**: Enforces security limits to prevent denial-of-service attacks
|
||||
- **Compression validation**: Checks for reasonable tensor sizes and prevents decompression bombs
|
||||
|
||||
**Why it matters:**
|
||||
GGUF/GGML files are increasingly popular for distributing large language models. While generally safer than pickle formats, they can still contain malicious metadata or be crafted to cause resource exhaustion attacks. The scanner ensures these files are structurally sound and don't contain hidden threats.
|
||||
|
||||
## Joblib Scanner
|
||||
|
||||
**File types:** `.joblib`
|
||||
|
||||
This scanner analyzes joblib serialized files, which are commonly used by ML libraries for model persistence.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Compression bomb detection**: Identifies files with suspicious compression ratios that could cause resource exhaustion
|
||||
- **Embedded pickle analysis**: Decompresses and scans embedded pickle content for malicious code
|
||||
- **Size limits**: Enforces maximum decompressed size limits to prevent memory exhaustion
|
||||
- **Format validation**: Distinguishes between ZIP archives and compressed pickle data
|
||||
|
||||
**Why it matters:**
|
||||
Joblib files often contain compressed pickle data, inheriting the same security risks as pickle files. Additionally, malicious actors could craft compression bombs that consume excessive memory or CPU resources when loaded. The scanner provides safe decompression with security limits.
|
||||
|
||||
## Skops Scanner
|
||||
|
||||
**File types:** `.skops`, `.pkl` (skops format)
|
||||
|
||||
This scanner detects known vulnerabilities in scikit-learn models saved with the skops library.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **CVE-2025-54412**: Remote code execution via malicious sklearn estimator
|
||||
- **CVE-2025-54413**: Arbitrary code execution through sklearn.compose.ColumnTransformer
|
||||
- **CVE-2025-54886**: Code execution via callable arguments in estimators
|
||||
- Version detection for vulnerable skops versions (< 0.12.0)
|
||||
- Dangerous sklearn components (Pipeline, ColumnTransformer, FunctionTransformer)
|
||||
- Malicious callable arguments in estimator configurations
|
||||
|
||||
**Why it matters:**
|
||||
Skops versions before 0.12.0 contain multiple critical vulnerabilities allowing remote code execution through specially crafted sklearn estimators. Attackers can embed malicious callables in pipelines, transformers, or estimator parameters that execute when the model is loaded or used.
|
||||
|
||||
## Flax/JAX Scanner
|
||||
|
||||
**File types:** `.msgpack`, `.flax`, `.orbax`, `.jax`
|
||||
|
||||
This scanner analyzes Flax/JAX model files serialized in MessagePack format and other JAX-specific formats.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious MessagePack structures that could exploit deserializers
|
||||
- Embedded code objects or executable content
|
||||
- Malformed or oversized data structures that could cause resource exhaustion
|
||||
- Potentially dangerous nested objects or recursive structures
|
||||
- Unusual data types that might indicate tampering
|
||||
|
||||
**Why it matters:**
|
||||
Flax models serialized as msgpack files can potentially contain embedded code or malicious data structures. While MessagePack is generally safer than pickle, it can still be exploited through carefully crafted payloads that target specific deserializer vulnerabilities or cause denial-of-service attacks through resource exhaustion.
|
||||
|
||||
## JAX Checkpoint Scanner
|
||||
|
||||
**File types:** `.ckpt`, `.checkpoint`, `.orbax-checkpoint`, `.pickle` (when in JAX context)
|
||||
|
||||
This scanner analyzes JAX checkpoint files in various serialization formats, including Orbax checkpoints and JAX-specific pickle files.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Dangerous JAX operations like experimental callbacks (`jax.experimental.host_callback.call`)
|
||||
- Custom restore functions in Orbax checkpoint metadata
|
||||
- Dangerous pickle opcodes in JAX-serialized files
|
||||
- Directory-based checkpoint structure validation
|
||||
- Resource limits to prevent denial-of-service attacks
|
||||
|
||||
**Why it matters:**
|
||||
JAX checkpoints can contain custom restore functions or experimental callbacks that could be exploited. Orbax checkpoints may include metadata with arbitrary restore functions that execute during model loading.
|
||||
|
||||
## NumPy Scanner
|
||||
|
||||
**File types:** `.npy`, `.npz`
|
||||
|
||||
This scanner validates NumPy binary array files for integrity issues and potential security risks.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Array validation**: Checks array dimensions and data types for malicious manipulation
|
||||
- **Header integrity**: Validates NumPy file headers and magic numbers
|
||||
- **Dangerous data types**: Detects potentially harmful data types like object arrays
|
||||
- **Size validation**: Prevents loading of excessively large arrays that could cause memory exhaustion
|
||||
- **Dimension limits**: Enforces reasonable limits on array dimensions to prevent DoS attacks
|
||||
|
||||
**Why it matters:**
|
||||
While NumPy files are generally safer than pickle files, they can still be crafted maliciously. Object arrays can contain arbitrary Python objects (including code), and extremely large arrays can cause denial-of-service attacks. The scanner ensures arrays are safe to load and don't contain hidden threats.
|
||||
|
||||
## OCI Layer Scanner
|
||||
|
||||
**File types:** `.manifest` (with `.tar.gz` layer references)
|
||||
|
||||
This scanner examines OCI (Open Container Initiative) and Docker manifest files that contain embedded model files in compressed layers.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Layer extraction**: Safely extracts and scans model files from `.tar.gz` layers
|
||||
- **Manifest validation**: Parses JSON and YAML manifest formats
|
||||
- **Recursive scanning**: Applies appropriate scanners to model files found within container layers
|
||||
- **Path validation**: Prevents directory traversal attacks during layer extraction
|
||||
|
||||
**Why it matters:**
|
||||
Container images are increasingly used to distribute ML models and datasets. These containers can contain multiple layers with various file types, potentially hiding malicious models within what appears to be a legitimate container image. The scanner ensures that all model files within container layers are safe.
|
||||
|
||||
## Manifest Scanner
|
||||
|
||||
**File types:** `.json`, `.yaml`, `.yml`, `.xml`, `.toml`, `.config`, etc.
|
||||
|
||||
This scanner analyzes model configuration files and manifests.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Blacklisted model names that might indicate known vulnerable models
|
||||
- Suspicious configuration patterns related to:
|
||||
- Network access (URLs, endpoints, webhooks)
|
||||
- File system access (paths, directories, file operations)
|
||||
- Code execution (commands, scripts, shell access)
|
||||
- Credentials (passwords, tokens, secrets)
|
||||
- Framework-specific patterns in popular ML library configurations
|
||||
|
||||
**Why it matters:**
|
||||
Model configuration files can contain settings that lead to insecure behavior, such as downloading content from untrusted sources, accessing sensitive files, or executing commands.
|
||||
|
||||
## Text Scanner
|
||||
|
||||
**File types:** `.txt`, `.md`, `.markdown`, `.rst`
|
||||
|
||||
This scanner analyzes ML-specific text files like vocabulary lists, README files, and model documentation.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Unusually large text files that may indicate data hiding
|
||||
- File type identification (vocabulary, labels, documentation)
|
||||
- Basic content validation for ML-related text files
|
||||
|
||||
**Why it matters:**
|
||||
Text files in ML repositories like vocab.txt, labels.txt, and README files should follow expected patterns. Deviations may indicate tampering or hidden data.
|
||||
|
||||
## Jinja2 Template Scanner
|
||||
|
||||
**File types:** `.gguf`, `.json`, `.yaml`, `.yml`, `.jinja`, `.j2`, `.template`
|
||||
|
||||
This scanner detects template injection vulnerabilities in Jinja2 templates embedded in model files and configurations.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Server-side template injection (SSTI) patterns
|
||||
- Dangerous Jinja2 filters or functions (e.g., `eval`, `exec`, `import`)
|
||||
- Unrestricted variable access that could leak sensitive data
|
||||
- Code execution patterns in template expressions
|
||||
- CVE-2024-34359 exploitation in GGUF chat templates
|
||||
|
||||
**Why it matters:**
|
||||
Jinja2 templates in GGUF models, tokenizer configs, and deployment files can execute arbitrary code when processed. CVE-2024-34359 affects llama-cpp-python when loading malicious chat templates. Template injection allows full system compromise.
|
||||
|
||||
## Metadata Scanner
|
||||
|
||||
**File types:** `README.md`, `MODEL_CARD.md`, `METADATA.md`, model card files
|
||||
|
||||
This scanner analyzes model documentation and metadata files for security concerns.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Embedded credentials or API keys in documentation
|
||||
- Suspicious URLs or download links
|
||||
- References to malicious code repositories
|
||||
- Known vulnerable model versions or components
|
||||
- Misleading or deceptive model descriptions
|
||||
- Missing or inadequate security disclosures
|
||||
|
||||
**Why it matters:**
|
||||
Model documentation often contains setup instructions, example code, and configuration that may reference malicious resources or expose sensitive information. Attackers can embed malicious download links or credentials in README files that users may execute without careful review.
|
||||
|
||||
## PyTorch Binary Scanner
|
||||
|
||||
**File types:** `.bin` (raw PyTorch tensor files)
|
||||
|
||||
This scanner examines raw PyTorch binary tensor files that contain serialized weight data. It performs binary content scanning to detect various threats.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Embedded code patterns (imports, function calls, eval/exec)
|
||||
- Executable file signatures (Windows PE with DOS stub validation, Linux ELF, macOS Mach-O)
|
||||
- Shell script shebangs that might indicate embedded scripts
|
||||
- Blacklisted patterns specified in configuration
|
||||
- Suspiciously small files that might not be valid tensor data
|
||||
- Validation of tensor structure
|
||||
- PE file detection with MS-DOS stub signature validation
|
||||
|
||||
**Why it matters:**
|
||||
While `.bin` files typically contain raw tensor data, attackers could embed malicious code or executables within these files. The scanner performs deep content analysis with PE file detection (including DOS stub validation) to detect such threats.
|
||||
|
||||
## ZIP Archive Scanner
|
||||
|
||||
**File types:** `.zip`, `.npz`
|
||||
|
||||
This scanner examines ZIP archives and their contents recursively.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Directory traversal attacks:** Detects entries with paths containing ".." or absolute paths that could overwrite system files
|
||||
- **Zip bombs:** Identifies files with suspicious compression ratios (>100x) that could cause resource exhaustion
|
||||
- **Nested archives:** Scans ZIP files within ZIP files up to a configurable depth to prevent infinite recursion attacks
|
||||
- **Malicious content:** Each file within the archive is scanned with its appropriate scanner (e.g., pickle files with PickleScanner)
|
||||
- **Resource limits:** Enforces maximum number of entries and file sizes to prevent denial-of-service attacks
|
||||
|
||||
**Why it matters:**
|
||||
ZIP archives are commonly used to distribute models and datasets. Malicious actors can craft ZIP files that exploit extraction vulnerabilities, contain malware, or cause resource exhaustion. This scanner ensures that archives are safe to extract and that their contents don't pose security risks.
|
||||
|
||||
## TAR Scanner
|
||||
|
||||
**File types:** `.tar`, `.tar.gz`, `.tgz`, `.tar.bz2`
|
||||
|
||||
This scanner examines TAR archives and their contents recursively.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Directory traversal attacks with paths containing ".." or absolute paths
|
||||
- Symlink attacks that could point to sensitive system files
|
||||
- TAR bombs with excessive file counts or decompression ratios
|
||||
- Malicious content within archived files (scans with appropriate scanners)
|
||||
- Resource limits to prevent denial-of-service attacks
|
||||
|
||||
**Why it matters:**
|
||||
TAR archives are commonly used in Linux environments and container images to distribute models. They can contain symlinks that point outside the extraction directory, potentially allowing access to sensitive files. TAR bombs can exhaust system resources during extraction.
|
||||
|
||||
## 7-Zip Scanner
|
||||
|
||||
**File types:** `.7z`
|
||||
|
||||
This scanner examines 7-Zip archives and their contents.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Directory traversal attacks in entry paths
|
||||
- Compression bombs with suspicious ratios
|
||||
- Encrypted archives that might hide malicious content
|
||||
- Nested archives to prevent infinite recursion
|
||||
- Resource limits for memory and CPU usage
|
||||
- Malicious content within archived files
|
||||
|
||||
**Why it matters:**
|
||||
7-Zip archives offer high compression ratios, making them attractive for compression bomb attacks. They support encryption, which can hide malicious content from initial inspection. The scanner safely extracts and analyzes 7z files with appropriate security limits.
|
||||
|
||||
## Weight Distribution Scanner
|
||||
|
||||
**File types:** `.pt`, `.pth`, `.h5`, `.keras`, `.hdf5`, `.pb`, `.onnx`, `.safetensors`
|
||||
|
||||
This scanner analyzes neural network weight distributions to detect potential backdoors or trojaned models by identifying statistical anomalies.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Outlier neurons:** Detects output neurons with abnormally high weight magnitudes using Z-score analysis
|
||||
- **Dissimilar weight vectors:** Identifies neurons whose weight patterns are significantly different from others in the same layer (using cosine similarity)
|
||||
- **Extreme weight values:** Flags neurons containing unusually large individual weight values that deviate from the layer's distribution
|
||||
- **Final layer focus:** Prioritizes analysis of classification heads and output layers where backdoors are typically implemented
|
||||
|
||||
**Configuration options:**
|
||||
|
||||
- `z_score_threshold`: Controls sensitivity for outlier detection (default: 3.0, higher for LLMs)
|
||||
- `cosine_similarity_threshold`: Minimum similarity required between neurons (default: 0.7)
|
||||
- `weight_magnitude_threshold`: Threshold for extreme weight detection (default: 3.0 standard deviations)
|
||||
- `llm_vocab_threshold`: Vocabulary size threshold to identify LLM models (default: 10,000)
|
||||
- `enable_llm_checks`: Whether to perform checks on large language models (default: false)
|
||||
|
||||
**Why it matters:**
|
||||
Backdoored or trojaned models often contain specific neurons that activate on trigger inputs. These malicious neurons typically have weight patterns that are statistically anomalous compared to benign neurons. By analyzing weight distributions, this scanner can detect models that have been tampered with to include hidden behaviors.
|
||||
|
||||
**Special handling for LLMs:**
|
||||
Large language models with vocabulary layers (>10,000 outputs) use more conservative thresholds due to their naturally varied weight distributions. LLM checking is disabled by default but can be enabled via configuration.
|
||||
|
||||
## SafeTensors Scanner
|
||||
|
||||
**File types:** `.safetensors`, `.bin` (when containing SafeTensors data)
|
||||
|
||||
This scanner examines SafeTensors format files, which are designed to be a safer alternative to pickle files.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **Header validation**: Verifies SafeTensors format structure and JSON header integrity
|
||||
- **Metadata security**: Scans metadata for suspicious content, encoded payloads, and unusually large sections
|
||||
- **Tensor validation**: Validates tensor offsets, sizes, and data type consistency
|
||||
- **Offset integrity**: Ensures tensor data offsets are contiguous and within file bounds
|
||||
|
||||
**Why it matters:**
|
||||
While SafeTensors is designed to be safer than pickle files, the metadata section can still contain malicious content. Attackers might try to exploit parsers or include encoded payloads in the metadata. The scanner ensures the format integrity and metadata safety.
|
||||
|
||||
## PaddlePaddle Scanner
|
||||
|
||||
**File types:** `.pdmodel`, `.pdiparams`
|
||||
|
||||
This scanner examines PaddlePaddle model files, including model definitions and parameter files.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious operations in model definitions
|
||||
- Embedded pickle data within PaddlePaddle files
|
||||
- Custom operators that might contain malicious code
|
||||
- Dangerous configuration patterns
|
||||
- Executable content or embedded scripts
|
||||
|
||||
**Why it matters:**
|
||||
PaddlePaddle models can contain custom operators and may use pickle serialization internally. Malicious actors could embed harmful code in model definitions or exploit custom operators to execute unauthorized operations.
|
||||
|
||||
## XGBoost Scanner
|
||||
|
||||
**File types:** `.bst`, `.model`, `.json`, `.ubj`
|
||||
|
||||
This scanner examines XGBoost model files in binary, JSON, and UBJSON formats.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- Suspicious custom objectives or evaluation metrics
|
||||
- Embedded code in JSON configurations
|
||||
- Malformed model structures
|
||||
- Dangerous callback functions
|
||||
- Path traversal in external feature maps
|
||||
- Pickle-based custom functions in binary models
|
||||
|
||||
**Why it matters:**
|
||||
XGBoost models can include custom Python functions for objectives, metrics, and callbacks. In binary format, these are often pickled, inheriting pickle security risks. JSON formats can contain embedded code strings or references to malicious external resources.
|
||||
|
||||
## PMML Scanner
|
||||
|
||||
**File types:** `.pmml`
|
||||
|
||||
This scanner performs security checks on PMML (Predictive Model Markup Language) files to detect potential XML External Entity (XXE) attacks, malicious scripts, and suspicious external references.
|
||||
|
||||
**Key checks:**
|
||||
|
||||
- **XXE Attack Prevention**: Detects `<!DOCTYPE`, `<!ENTITY`, `<!ELEMENT`, and `<!ATTLIST` declarations that could enable XML External Entity attacks
|
||||
- **Safe XML Parsing**: Uses defusedxml when available for secure XML parsing; warns when using unsafe parsers
|
||||
- **Malicious Content Detection**: Scans for suspicious patterns like `<script>`, `eval()`, `exec()`, system commands, and imports
|
||||
- **External Resource References**: Identifies suspicious URLs (HTTP, HTTPS, FTP, file://) in model content
|
||||
- **PMML Structure Validation**: Validates PMML version and root element structure
|
||||
- **Extension Element Analysis**: Performs deep inspection of `<Extension>` elements which can contain arbitrary content
|
||||
|
||||
**Security features:**
|
||||
|
||||
- **XML Security**: Uses defusedxml library when available to prevent XXE and billion laughs attacks
|
||||
- **Content Scanning**: Recursive analysis of all element text content and attributes for malicious patterns
|
||||
- **Well-formedness Validation**: Ensures XML structure integrity and UTF-8 encoding compliance
|
||||
|
||||
**Why it matters:**
|
||||
PMML files are XML-based and can be exploited through XML vulnerabilities like XXE attacks. Extension elements can contain arbitrary content that might execute scripts or access external resources. The scanner ensures PMML files don't contain hidden security threats while maintaining model functionality.
|
||||
|
||||
## Auto Format Detection
|
||||
|
||||
ModelAudit includes comprehensive file format detection for ambiguous file extensions, particularly `.bin` files, which can contain different types of model data:
|
||||
|
||||
- **Pickle format**: Detected by pickle protocol magic bytes (\x80\x02, \x80\x03, etc.)
|
||||
- **SafeTensors format**: Detected by JSON header structure and metadata patterns
|
||||
- **ONNX format**: Detected by ONNX protobuf signatures
|
||||
- **PyTorch ZIP format**: Detected by ZIP magic bytes (PK headers)
|
||||
- **Raw PyTorch tensors**: Default for `.bin` files without other recognizable signatures
|
||||
|
||||
**Detection Features:**
|
||||
|
||||
- **Magic byte analysis**: Reads file headers to determine actual format regardless of extension
|
||||
- **Content-based routing**: Automatically applies the most appropriate scanner based on detected format
|
||||
- **Multi-format support**: Handles cases where files might be misnamed or have generic extensions
|
||||
- **Fallback handling**: Gracefully handles unknown formats with generic binary scanning
|
||||
|
||||
This allows ModelAudit to automatically apply the correct scanner based on the actual file content, not just the extension. When a `.bin` file contains SafeTensors data, the SafeTensors scanner is automatically applied instead of assuming it's a raw binary file.
|
||||
|
||||
## License Checking and Compliance
|
||||
|
||||
ModelAudit includes license detection across all file formats to help organizations identify legal obligations before deployment.
|
||||
|
||||
**Key features:**
|
||||
|
||||
- **License Detection**: Scans headers, LICENSE files, and metadata for license information
|
||||
- **AGPL Warnings**: Alerts about network copyleft obligations
|
||||
- **Commercial Restrictions**: Identifies non-commercial licenses
|
||||
- **Unlicensed Content**: Flags large datasets without clear licensing
|
||||
- **SBOM Generation**: Creates CycloneDX-compliant Software Bill of Materials
|
||||
|
||||
**Example warnings:**
|
||||
|
||||
```text
|
||||
⚠️ AGPL license detected: Component is under AGPL-3.0
|
||||
This may require source code disclosure if used in network services
|
||||
|
||||
🚨 Non-commercial license detected: Creative Commons NonCommercial
|
||||
This component cannot be used for commercial purposes
|
||||
```
|
||||
|
||||
**Generate SBOM:**
|
||||
|
||||
```bash
|
||||
promptfoo scan-model ./models/ --sbom model-sbom.json
|
||||
```
|
||||
|
||||
The SBOM includes component information, license metadata, risk scores, and copyright details in CycloneDX format.
|
||||
|
||||
**Why it matters:**
|
||||
AI/ML projects often combine components with different licenses. AGPL requires source disclosure for network services, non-commercial licenses block commercial use, and unlicensed datasets create legal risks.
|
||||
|
||||
## Network Communication Detection
|
||||
|
||||
ModelAudit includes comprehensive detection of network communication capabilities that could be used for data exfiltration or command & control:
|
||||
|
||||
**Detection capabilities:**
|
||||
|
||||
- **URL patterns**: HTTP(S), FTP, SSH, WebSocket URLs embedded in model data
|
||||
- **IP addresses**: Both IPv4 and IPv6 addresses that could indicate hardcoded endpoints
|
||||
- **Domain names**: Suspicious domain patterns that might be C&C servers
|
||||
- **Network libraries**: Imports of socket, urllib, requests, and 50+ other network libraries
|
||||
- **Network functions**: Calls to urlopen, socket.connect, requests.get, and other network operations
|
||||
- **C&C patterns**: Known command & control patterns like beacon_url, callback_url, exfil_endpoint
|
||||
- **Port numbers**: Common and suspicious port numbers for various protocols
|
||||
|
||||
**Why it matters:**
|
||||
Malicious models could contain embedded network communication code to:
|
||||
|
||||
- Exfiltrate sensitive data from the deployment environment
|
||||
- Download additional payloads or updates
|
||||
- Establish command & control channels
|
||||
- Report telemetry to unauthorized servers
|
||||
|
||||
## Secrets Detection
|
||||
|
||||
ModelAudit scans for embedded credentials and sensitive information:
|
||||
|
||||
**Detection patterns:**
|
||||
|
||||
- **API Keys**: AWS, Azure, GCP, OpenAI, and other service API keys
|
||||
- **Tokens**: JWT tokens, OAuth tokens, GitHub tokens, etc.
|
||||
- **Passwords**: Hardcoded passwords and authentication strings
|
||||
- **Private Keys**: SSH keys, SSL certificates, cryptographic keys
|
||||
- **Database Credentials**: Connection strings with embedded passwords
|
||||
- **Webhook URLs**: Slack, Discord, and other webhook endpoints
|
||||
|
||||
**Why it matters:**
|
||||
Credentials embedded in models could:
|
||||
|
||||
- Expose production infrastructure access
|
||||
- Lead to unauthorized cloud resource usage
|
||||
- Enable lateral movement in compromised systems
|
||||
- Result in data breaches or compliance violations
|
||||
|
||||
## JIT/Script Detection
|
||||
|
||||
ModelAudit detects Just-In-Time compilation and script execution patterns:
|
||||
|
||||
**Detection capabilities:**
|
||||
|
||||
- **TorchScript**: Embedded TorchScript code that could execute arbitrary operations
|
||||
- **ONNX Custom Ops**: Custom operators that might contain malicious functionality
|
||||
- **TensorFlow Eager Execution**: Dynamic execution patterns in TF models
|
||||
- **Compilation Patterns**: eval(), exec(), compile() calls in various contexts
|
||||
- **Script Injections**: JavaScript, Python, or shell script injection attempts
|
||||
|
||||
**Why it matters:**
|
||||
JIT-compiled code can:
|
||||
|
||||
- Bypass static analysis security checks
|
||||
- Execute arbitrary code at runtime
|
||||
- Modify model behavior dynamically
|
||||
- Access system resources without detection
|
||||
|
||||
## HuggingFace URL Support
|
||||
|
||||
ModelAudit can scan models directly from HuggingFace URLs without manual downloading. When a HuggingFace URL is provided, ModelAudit:
|
||||
|
||||
1. **Downloads the model**: Uses the `huggingface-hub` library to download all model files to a temporary directory
|
||||
2. **Scans all files**: Applies appropriate scanners to each file based on its format (config.json, pytorch_model.bin, model.safetensors, etc.)
|
||||
3. **Cleans up**: Automatically removes downloaded files after scanning
|
||||
|
||||
**Supported URL formats:**
|
||||
|
||||
- `https://huggingface.co/user/model`
|
||||
- `https://hf.co/user/model`
|
||||
- `hf://user/model`
|
||||
|
||||
This feature requires the `huggingface-hub` package to be installed.
|
||||
@@ -0,0 +1,659 @@
|
||||
---
|
||||
title: ModelAudit Advanced Usage
|
||||
sidebar_label: Advanced Usage
|
||||
sidebar_position: 120
|
||||
description: Automate LLM model scanning across cloud providers with remote storage integration, CI/CD workflows, and programmatic controls for advanced security testing
|
||||
keywords:
|
||||
[
|
||||
modelaudit,
|
||||
model scanning,
|
||||
cloud storage,
|
||||
remote model scanning,
|
||||
huggingface,
|
||||
s3 model scanning,
|
||||
gcs model scanning,
|
||||
mlflow,
|
||||
jfrog artifactory,
|
||||
dvc,
|
||||
model authentication,
|
||||
ci cd integration,
|
||||
github actions,
|
||||
gitlab ci,
|
||||
programmatic scanning,
|
||||
sarif output,
|
||||
sbom generation,
|
||||
custom scanners,
|
||||
model security automation,
|
||||
]
|
||||
---
|
||||
|
||||
# Advanced Usage
|
||||
|
||||
This page covers advanced ModelAudit features including cloud storage integration, CI/CD workflows, and programmatic usage.
|
||||
|
||||
## Authentication and Configuration
|
||||
|
||||
ModelAudit uses environment variables for authentication with cloud services and model registries.
|
||||
|
||||
### Cloud & Artifact Registry Authentication
|
||||
|
||||
Authentication for all remote services is now handled exclusively via environment variables.
|
||||
|
||||
#### HuggingFace
|
||||
|
||||
- `HF_TOKEN`: Your HuggingFace Hub token for accessing private models.
|
||||
|
||||
```bash
|
||||
# Authenticate for private models
|
||||
export HF_TOKEN=your_token_here
|
||||
promptfoo scan-model hf://your-org/private-model
|
||||
```
|
||||
|
||||
#### JFrog Artifactory
|
||||
|
||||
- `JFROG_URL`: The base URL of your JFrog Artifactory instance.
|
||||
- `JFROG_API_TOKEN` or `JFROG_ACCESS_TOKEN`: Your API or access token.
|
||||
|
||||
```bash
|
||||
# Authenticate using an API token
|
||||
export JFROG_URL="https://your-domain.jfrog.io"
|
||||
export JFROG_API_TOKEN="your-api-token"
|
||||
promptfoo scan-model "https://your-domain.jfrog.io/artifactory/repo/model.pkl"
|
||||
```
|
||||
|
||||
#### MLflow Model Registry
|
||||
|
||||
- `MLFLOW_TRACKING_URI`: The URI of your MLflow tracking server.
|
||||
- `MLFLOW_TRACKING_USERNAME` / `MLFLOW_TRACKING_PASSWORD`: Credentials for MLflow authentication.
|
||||
|
||||
```bash
|
||||
# Authenticate with MLflow
|
||||
export MLFLOW_TRACKING_URI="https://your-mlflow-server.com"
|
||||
export MLFLOW_TRACKING_USERNAME="your-username"
|
||||
export MLFLOW_TRACKING_PASSWORD="your-password"
|
||||
promptfoo scan-model models:/model-name/version
|
||||
```
|
||||
|
||||
#### Amazon S3
|
||||
|
||||
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`: Standard AWS credentials.
|
||||
|
||||
```bash
|
||||
export AWS_ACCESS_KEY_ID="your-access-key"
|
||||
export AWS_SECRET_ACCESS_KEY="your-secret-key"
|
||||
export AWS_DEFAULT_REGION="us-east-1"
|
||||
promptfoo scan-model s3://my-bucket/model.pkl
|
||||
```
|
||||
|
||||
#### Google Cloud Storage
|
||||
|
||||
- `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key file.
|
||||
|
||||
```bash
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
|
||||
promptfoo scan-model gs://my-bucket/model.pt
|
||||
```
|
||||
|
||||
#### Cloudflare R2
|
||||
|
||||
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`: Your R2 credentials.
|
||||
- `AWS_ENDPOINT_URL`: The S3-compatible endpoint for your R2 account.
|
||||
|
||||
```bash
|
||||
export AWS_ACCESS_KEY_ID="your-r2-access-key"
|
||||
export AWS_SECRET_ACCESS_KEY="your-r2-secret-key"
|
||||
export AWS_ENDPOINT_URL="https://your-account.r2.cloudflarestorage.com"
|
||||
promptfoo scan-model r2://my-bucket/model.safetensors
|
||||
```
|
||||
|
||||
### Migration from Deprecated Flags
|
||||
|
||||
The following CLI flags have been **removed** and replaced by environment variables. Attempting to use them will result in an error.
|
||||
|
||||
| Removed Flag | Replacement Environment Variable |
|
||||
| ---------------------- | ------------------------------------ |
|
||||
| `--jfrog-api-token` | `JFROG_API_TOKEN` |
|
||||
| `--jfrog-access-token` | `JFROG_ACCESS_TOKEN` |
|
||||
| `--registry-uri` | `JFROG_URL` or `MLFLOW_TRACKING_URI` |
|
||||
|
||||
**Why the change?** Using environment variables is more secure than passing secrets as CLI flags, as it prevents them from being exposed in shell history or process lists. This aligns with industry best practices for managing credentials.
|
||||
|
||||
## Remote Model Scanning
|
||||
|
||||
ModelAudit can scan models directly from various remote sources without manual downloading.
|
||||
|
||||
### HuggingFace
|
||||
|
||||
Scan public or private models from the HuggingFace Hub.
|
||||
|
||||
```bash
|
||||
# Public model
|
||||
promptfoo scan-model https://huggingface.co/bert-base-uncased
|
||||
|
||||
# Private model (requires HF_TOKEN to be set)
|
||||
promptfoo scan-model hf://your-org/private-model
|
||||
```
|
||||
|
||||
### Cloud Storage (S3, GCS, R2)
|
||||
|
||||
Scan models stored in cloud buckets. See the [Authentication](#authentication-and-configuration) section for setup.
|
||||
|
||||
```bash
|
||||
# Scan from S3
|
||||
promptfoo scan-model s3://my-bucket/model.pkl
|
||||
|
||||
# Scan from Google Cloud Storage
|
||||
promptfoo scan-model gs://my-bucket/model.pt
|
||||
|
||||
# Scan from Cloudflare R2
|
||||
promptfoo scan-model r2://my-bucket/model.safetensors
|
||||
```
|
||||
|
||||
### Model Registries (MLflow, JFrog)
|
||||
|
||||
Scan models from MLflow or JFrog Artifactory. See the [Authentication](#authentication-and-configuration) section for setup.
|
||||
|
||||
```bash
|
||||
# Scan from MLflow
|
||||
promptfoo scan-model models:/MyModel/Latest
|
||||
|
||||
# Scan from JFrog Artifactory
|
||||
promptfoo scan-model "https://your-domain.jfrog.io/artifactory/models/model.pkl"
|
||||
```
|
||||
|
||||
### DVC Integration
|
||||
|
||||
ModelAudit automatically resolves DVC pointer files:
|
||||
|
||||
```bash
|
||||
# Scans the actual model file referenced by the .dvc file
|
||||
promptfoo scan-model model.pkl.dvc
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
ModelAudit's behavior can be customized through command-line options. While configuration files are not currently supported, you can achieve similar results using CLI flags:
|
||||
|
||||
```bash
|
||||
# Set blacklist patterns
|
||||
promptfoo scan-model models/ \
|
||||
--blacklist "deepseek" \
|
||||
--blacklist "qwen" \
|
||||
--blacklist "unsafe_model"
|
||||
|
||||
# Set resource limits
|
||||
promptfoo scan-model models/ \
|
||||
--max-size 1GB \
|
||||
--timeout 600
|
||||
|
||||
# Combine multiple options
|
||||
promptfoo scan-model models/ \
|
||||
--blacklist "suspicious_pattern" \
|
||||
--max-size 1GB \
|
||||
--timeout 600 \
|
||||
--verbose
|
||||
|
||||
# Enable strict mode for enhanced security validation
|
||||
promptfoo scan-model model.pkl --strict
|
||||
|
||||
# Discover available scanner IDs and class names
|
||||
promptfoo scan-model --list-scanners
|
||||
|
||||
# Run only selected scanners by ID
|
||||
promptfoo scan-model models/ --scanners pickle,tf_savedmodel
|
||||
|
||||
# Repeat scanner-selection flags when that is clearer in scripts
|
||||
promptfoo scan-model models/ --scanners pickle --scanners tf_savedmodel
|
||||
|
||||
# Exclude a scanner from the default scanner set
|
||||
promptfoo scan-model models/ --exclude-scanner weight_distribution
|
||||
|
||||
# Strict mode with additional output options
|
||||
promptfoo scan-model models/ \
|
||||
--strict \
|
||||
--no-write \
|
||||
--format sarif \
|
||||
--output security-scan.sarif
|
||||
```
|
||||
|
||||
Use `--scanners` for focused triage scans. Use `--exclude-scanner` when the default scanner set is right except for a noisy or irrelevant scanner.
|
||||
|
||||
### Sharing Results
|
||||
|
||||
When connected to promptfoo Cloud, model audit results are automatically shared by default. This provides a web-based interface to view, analyze, and collaborate on scan results.
|
||||
|
||||
```bash
|
||||
# Results are automatically shared when cloud is enabled
|
||||
promptfoo scan-model models/
|
||||
|
||||
# Explicitly enable sharing
|
||||
promptfoo scan-model models/ --share
|
||||
|
||||
# Disable sharing for this scan
|
||||
promptfoo scan-model models/ --no-share
|
||||
|
||||
# Disable sharing globally via environment variable
|
||||
export PROMPTFOO_DISABLE_SHARING=true
|
||||
promptfoo scan-model models/
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions
|
||||
|
||||
```yaml
|
||||
# .github/workflows/model-security.yml
|
||||
name: Model Security Scan
|
||||
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'models/**'
|
||||
- '**.pkl'
|
||||
- '**.h5'
|
||||
- '**.pb'
|
||||
- '**.pt'
|
||||
- '**.pth'
|
||||
|
||||
jobs:
|
||||
scan:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
npm install -g promptfoo
|
||||
pip install modelaudit[all]
|
||||
|
||||
- name: Scan models
|
||||
run: promptfoo scan-model models/ --no-write --format sarif --output model-scan.sarif
|
||||
|
||||
- name: Upload SARIF to GitHub Advanced Security
|
||||
uses: github/codeql-action/upload-sarif@v3
|
||||
if: always()
|
||||
with:
|
||||
sarif_file: model-scan.sarif
|
||||
category: model-security
|
||||
|
||||
- name: Upload scan results as artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: model-scan-results
|
||||
path: model-scan.sarif
|
||||
```
|
||||
|
||||
### GitLab CI
|
||||
|
||||
```yaml
|
||||
# .gitlab-ci.yml
|
||||
model_security_scan:
|
||||
stage: test
|
||||
image: python:3.10
|
||||
script:
|
||||
- pip install modelaudit[all]
|
||||
- npm install -g promptfoo
|
||||
- |
|
||||
scan_status=0
|
||||
promptfoo scan-model models/ --format json --output scan-results.json || scan_status=$?
|
||||
if [ "$scan_status" -gt 1 ]; then exit "$scan_status"; fi
|
||||
- jq -e '[.issues[]? | select(.severity == "critical")] | length == 0' scan-results.json
|
||||
artifacts:
|
||||
paths:
|
||||
- scan-results.json
|
||||
when: always
|
||||
only:
|
||||
changes:
|
||||
- models/**
|
||||
- '**/*.pkl'
|
||||
- '**/*.h5'
|
||||
- '**/*.pb'
|
||||
- '**/*.pt'
|
||||
- '**/*.pth'
|
||||
```
|
||||
|
||||
### Pre-commit Hook
|
||||
|
||||
```yaml
|
||||
# .pre-commit-config.yaml
|
||||
repos:
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: modelaudit
|
||||
name: ModelAudit
|
||||
entry: promptfoo scan-model
|
||||
language: system
|
||||
files: '\.(pkl|h5|pb|pt|pth|keras|hdf5|json|yaml|yml|zip|onnx|safetensors|bin|tflite|msgpack|pmml|joblib|npy|gguf|ggml)$'
|
||||
pass_filenames: true
|
||||
```
|
||||
|
||||
## Programmatic Usage
|
||||
|
||||
You can use ModelAudit programmatically in your Python code:
|
||||
|
||||
```python
|
||||
from modelaudit.core import scan_model_directory_or_file
|
||||
|
||||
# Scan a single model
|
||||
results = scan_model_directory_or_file("path/to/model.pkl")
|
||||
|
||||
# Scan a HuggingFace model URL
|
||||
results = scan_model_directory_or_file("https://huggingface.co/bert-base-uncased")
|
||||
|
||||
# Check for issues
|
||||
if results["issues"]:
|
||||
print(f"Found {len(results['issues'])} issues:")
|
||||
for issue in results["issues"]:
|
||||
print(f"- {issue['severity'].upper()}: {issue['message']}")
|
||||
else:
|
||||
print("No issues found!")
|
||||
|
||||
# Scan with custom configuration
|
||||
config = {
|
||||
"blacklist_patterns": ["unsafe_model", "malicious_net"],
|
||||
"max_file_size": 1073741824, # 1GB
|
||||
"timeout": 600 # 10 minutes
|
||||
}
|
||||
|
||||
results = scan_model_directory_or_file("path/to/models/", **config)
|
||||
```
|
||||
|
||||
## JSON Output Format
|
||||
|
||||
When using `--format json`, ModelAudit outputs structured results:
|
||||
|
||||
```bash
|
||||
promptfoo scan-model model.pkl --format json --output results.json
|
||||
```
|
||||
|
||||
Use the [ModelAudit JSON Schema](/schemas/modelaudit/modelaudit-scan-result.schema.json) and [example result](/examples/modelaudit/modelaudit-scan-result.example.json) to validate `--format json` output.
|
||||
|
||||
The result includes scan statistics, scanned assets, file metadata, `issues`, and `checks`. Treat `issues` as findings. `checks` records performed checks and may include a failed check for the same finding, so do not count both arrays as separate findings.
|
||||
|
||||
Consumers should tolerate omitted optional fields and new scanner-specific fields, especially within `details` and `file_metadata`.
|
||||
|
||||
Use the standalone ModelAudit CLI to export the installed version's scanner catalog when you need the current scanner IDs and dependency metadata:
|
||||
|
||||
```bash
|
||||
modelaudit scan --list-scanners --format json
|
||||
```
|
||||
|
||||
## SARIF Output Format
|
||||
|
||||
ModelAudit supports SARIF (Static Analysis Results Interchange Format) 2.1.0 output for seamless integration with security tools and CI/CD pipelines:
|
||||
|
||||
```bash
|
||||
# Output SARIF to stdout
|
||||
promptfoo scan-model model.pkl --format sarif
|
||||
|
||||
# Save SARIF to file
|
||||
promptfoo scan-model model.pkl --no-write --format sarif --output results.sarif
|
||||
|
||||
# Scan multiple models with SARIF output
|
||||
promptfoo scan-model models/ --no-write --format sarif --output scan-results.sarif
|
||||
```
|
||||
|
||||
### SARIF Structure
|
||||
|
||||
The SARIF output includes:
|
||||
|
||||
- **Rules**: Unique security patterns detected (e.g., pickle issues, dangerous imports)
|
||||
- **Results**: Individual findings with severity levels, locations, and fingerprints
|
||||
- **Artifacts**: Information about scanned files including hashes
|
||||
- **Tool Information**: ModelAudit version and capabilities
|
||||
- **Invocation Details**: Command-line arguments and scan statistics
|
||||
|
||||
Example SARIF output structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json",
|
||||
"version": "2.1.0",
|
||||
"runs": [
|
||||
{
|
||||
"tool": {
|
||||
"driver": {
|
||||
"name": "ModelAudit",
|
||||
"version": "0.2.3",
|
||||
"rules": [
|
||||
{
|
||||
"id": "MA-PICKLE-ISSUE",
|
||||
"name": "Pickle Security Issue",
|
||||
"defaultConfiguration": {
|
||||
"level": "error",
|
||||
"rank": 90.0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"ruleId": "MA-PICKLE-ISSUE",
|
||||
"level": "error",
|
||||
"message": {
|
||||
"text": "Suspicious module reference found: os.system"
|
||||
},
|
||||
"locations": [
|
||||
{
|
||||
"physicalLocation": {
|
||||
"artifactLocation": {
|
||||
"uri": "model.pkl"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Severity Mapping
|
||||
|
||||
ModelAudit severities are mapped to SARIF levels:
|
||||
|
||||
- `CRITICAL` → `error`
|
||||
- `WARNING` → `warning`
|
||||
- `INFO` → `note`
|
||||
- `DEBUG` → `none`
|
||||
|
||||
### Integration with Security Tools
|
||||
|
||||
SARIF output enables integration with:
|
||||
|
||||
#### GitHub Advanced Security
|
||||
|
||||
```yaml
|
||||
# .github/workflows/security.yml
|
||||
- name: Scan models
|
||||
id: scan
|
||||
continue-on-error: true
|
||||
run: promptfoo scan-model models/ --no-write --format sarif --output model-scan.sarif
|
||||
|
||||
- name: Upload SARIF to GitHub
|
||||
if: always()
|
||||
uses: github/codeql-action/upload-sarif@v3
|
||||
with:
|
||||
sarif_file: model-scan.sarif
|
||||
category: model-security
|
||||
|
||||
- name: Fail if scan found issues or errors
|
||||
if: steps.scan.outcome == 'failure'
|
||||
run: exit 1
|
||||
```
|
||||
|
||||
#### Azure DevOps
|
||||
|
||||
```yaml
|
||||
# azure-pipelines.yml
|
||||
- script: |
|
||||
promptfoo scan-model models/ --no-write --format sarif --output $(Build.ArtifactStagingDirectory)/model-scan.sarif
|
||||
displayName: 'Scan models'
|
||||
|
||||
- task: PublishSecurityAnalysisLogs@3
|
||||
inputs:
|
||||
ArtifactName: 'CodeAnalysisLogs'
|
||||
ArtifactType: 'Container'
|
||||
AllTools: false
|
||||
ToolLogsNotFoundAction: 'Standard'
|
||||
```
|
||||
|
||||
#### VS Code SARIF Viewer
|
||||
|
||||
```bash
|
||||
# Generate SARIF for local viewing
|
||||
promptfoo scan-model . --no-write --format sarif --output scan.sarif
|
||||
|
||||
# Open in VS Code with SARIF Viewer extension
|
||||
code scan.sarif
|
||||
```
|
||||
|
||||
#### Static Analysis Platforms
|
||||
|
||||
SARIF output is compatible with:
|
||||
|
||||
- SonarQube/SonarCloud (via import)
|
||||
- Fortify
|
||||
- Checkmarx
|
||||
- CodeQL
|
||||
- Snyk
|
||||
- And many other SARIF-compatible tools
|
||||
|
||||
## Software Bill of Materials (SBOM)
|
||||
|
||||
Generate CycloneDX-compliant SBOMs with license information:
|
||||
|
||||
```bash
|
||||
promptfoo scan-model models/ --sbom model-sbom.json
|
||||
```
|
||||
|
||||
The SBOM includes:
|
||||
|
||||
- Component information (files, types, sizes, checksums)
|
||||
- License metadata (detected licenses, copyright holders)
|
||||
- Risk scoring based on scan findings
|
||||
- Model/dataset classification
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Missing Dependencies**
|
||||
|
||||
```
|
||||
Error: h5py not installed, cannot scan Keras H5 files
|
||||
```
|
||||
|
||||
Solution: Install the required dependencies:
|
||||
|
||||
```bash
|
||||
pip install h5py tensorflow
|
||||
```
|
||||
|
||||
2. **Timeout Errors**
|
||||
|
||||
```
|
||||
Error: Scan timeout after 300 seconds
|
||||
```
|
||||
|
||||
Solution: Increase the timeout:
|
||||
|
||||
```bash
|
||||
promptfoo scan-model model.pkl --timeout 7200 # 2 hours for very large models
|
||||
```
|
||||
|
||||
3. **File Size Limits**
|
||||
|
||||
```
|
||||
Warning: File too large to scan
|
||||
```
|
||||
|
||||
Solution: Increase the maximum file size:
|
||||
|
||||
```bash
|
||||
promptfoo scan-model model.pkl --max-size 3GB
|
||||
```
|
||||
|
||||
4. **Unknown Format**
|
||||
|
||||
```
|
||||
Warning: Unknown or unhandled format
|
||||
```
|
||||
|
||||
Solution: Ensure the file is in a supported format.
|
||||
|
||||
## Extending ModelAudit
|
||||
|
||||
### Creating Custom Scanners
|
||||
|
||||
You can create custom scanners by extending the `BaseScanner` class:
|
||||
|
||||
```python
|
||||
from modelaudit.scanners.base import BaseScanner, ScanResult, IssueSeverity
|
||||
|
||||
class CustomModelScanner(BaseScanner):
|
||||
"""Scanner for custom model format"""
|
||||
name = "custom_format"
|
||||
description = "Scans custom model format for security issues"
|
||||
supported_extensions = [".custom", ".mymodel"]
|
||||
|
||||
@classmethod
|
||||
def can_handle(cls, path: str) -> bool:
|
||||
"""Check if this scanner can handle the given path"""
|
||||
return path.endswith(tuple(cls.supported_extensions))
|
||||
|
||||
def scan(self, path: str) -> ScanResult:
|
||||
"""Scan the model file for security issues"""
|
||||
result = self._create_result()
|
||||
scan_success = True
|
||||
|
||||
try:
|
||||
# Your custom scanning logic here
|
||||
with open(path, 'rb') as f:
|
||||
content = f.read()
|
||||
|
||||
if b'malicious_pattern' in content:
|
||||
result.add_issue(
|
||||
"Suspicious pattern found",
|
||||
severity=IssueSeverity.WARNING,
|
||||
location=path,
|
||||
details={"pattern": "malicious_pattern"}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
result.add_issue(
|
||||
f"Error scanning file: {str(e)}",
|
||||
severity=IssueSeverity.CRITICAL,
|
||||
location=path,
|
||||
details={"exception": str(e)}
|
||||
)
|
||||
scan_success = False
|
||||
|
||||
result.finish(success=scan_success)
|
||||
return result
|
||||
```
|
||||
|
||||
To integrate your custom scanner, add it to the scanner registry in `modelaudit/scanners/__init__.py`:
|
||||
|
||||
```python
|
||||
# In modelaudit/scanners/__init__.py
|
||||
from .custom_scanner import CustomModelScanner
|
||||
|
||||
# Add to the registry
|
||||
registry.register(
|
||||
"custom_format",
|
||||
lambda: CustomModelScanner,
|
||||
description="Custom model format scanner",
|
||||
module="modelaudit.scanners.custom_scanner"
|
||||
)
|
||||
```
|
||||
|
||||
Custom scanners require integration into the ModelAudit package structure and cannot be dynamically registered at runtime. For production use, consider contributing your scanner to the ModelAudit project.
|
||||
Reference in New Issue
Block a user