wehub/diegosouzapw--omniroute
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:12 +08:00
commit d8dcd5f6d1
8604 changed files with 2479390 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ops/BRANCH_PROTECTION_MAIN.md
+23
View File
@@ -0,0 +1,23 @@
---
title: "Branch Protection — main"
---
# Branch protection — `main` (OpenSSF Scorecard: Branch-Protection)
Owner action. Apply via Settings → Branches → Add rule, or:
```bash
gh api -X PUT repos/diegosouzapw/OmniRoute/branches/main/protection \
--input - <<'JSON'
{ "required_status_checks": { "strict": true, "contexts": ["Quality Ratchet", "Quality Gates (Extended)", "Fast Quality Gates"] },
"enforce_admins": false,
"required_pull_request_reviews": { "required_approving_review_count": 0, "dismiss_stale_reviews": true },
"restrictions": null,
"required_linear_history": false,
"allow_force_pushes": false,
"allow_deletions": false }
JSON
```
Lifts Scorecard Branch-Protection from 0. `enforce_admins:false` keeps the existing
forward-merge flow workable; tighten to `true` once stable.
docs/ops/COVERAGE_PLAN.md
+183
View File
@@ -0,0 +1,183 @@
---
title: "Test Coverage Plan"
version: 3.8.40
lastUpdated: 2026-06-28
---
# Test Coverage Plan
Last updated: 2026-06-28
> Status measured on 2026-05-13: lines 82.58%, statements 82.58%, functions 84.23%, branches 75.22%. Phases 1-5 are complete. Current focus is Phase 6 (>=85%) and Phase 7 (>=90%).
## Baseline
There are multiple coverage numbers depending on how the report is computed. For planning, only one of them is useful.
| Metric | Scope | Statements / Lines | Branches | Functions | Notes |
| -------------------- | ----------------------------------------------------- | -----------------: | -------: | --------: | --------------------------------------------------- |
| Legacy | Old `npm run test:coverage` | 79.42% | 75.15% | 67.94% | Inflated: counts test files and excludes `open-sse` |
| Diagnostic | Source-only, excluding tests and excluding `open-sse` | 68.16% | 63.55% | 64.06% | Useful only to isolate `src/**` |
| Recommended baseline | Source-only, excluding tests and including `open-sse` | 82.58% | 75.22% | 84.23% | This is the project-wide baseline to improve |
The recommended baseline is the number to optimize against.
## Rules
- Coverage targets apply to source files, not to `tests/**`.
- `open-sse/**` is part of the product and must remain in scope.
- New code should not reduce coverage in touched areas.
- Prefer testing behavior and branch outcomes over implementation details.
- Prefer temp SQLite databases and small fixtures over broad mocks for `src/lib/db/**`.
## Current command set
- `npm run test:coverage`
- Main source coverage gate for the unit test suite
- Generates `text-summary`, `html`, `json-summary`, and `lcov`
- `npm run coverage:report`
- Detailed file-by-file report from the latest run
- `npm run test:coverage:legacy`
- Historical comparison only
## Milestones
| Phase | Target | Focus | Status |
| ------- | ---------------------: | ------------------------------------------------- | ----------- |
| Phase 1 | 60% statements / lines | Quick wins and low-risk utility coverage | ✅ Done |
| Phase 2 | 65% statements / lines | DB and route foundations | ✅ Done |
| Phase 3 | 70% statements / lines | Provider validation and usage analytics | ✅ Done |
| Phase 4 | 75% statements / lines | `open-sse` translators and helpers | ✅ Done |
| Phase 5 | 80% statements / lines | `open-sse` handlers and executor branches | ✅ Done |
| Phase 6 | 85% statements / lines | Harder edge cases, branch debt, regression suites | In progress |
| Phase 7 | 90% statements / lines | Final sweep, gap closure, strict ratchet | Pending |
Branches and functions should ratchet upward with each phase, but the primary hard target is statements / lines.
## Priority hotspots
These files have the lowest line coverage today (< 60%) and offer the best return for Phases 6-7. Generated from `coverage/coverage-summary.json` on 2026-05-13:
| # | File | Lines % |
| --- | ------------------------------------------------------------ | ------: |
| 1 | `open-sse/services/compression/validation.ts` | 7.87% |
| 2 | `src/app/api/v1/batches/route.ts` | 9.67% |
| 3 | `src/app/docs/components/FeedbackWidget.tsx` | 9.80% |
| 4 | `open-sse/services/compression/toolResultCompressor.ts` | 10.00% |
| 5 | `src/app/docs/components/DocCodeBlocks.tsx` | 10.63% |
| 6 | `open-sse/services/compression/engines/rtk/lineFilter.ts` | 10.96% |
| 7 | `open-sse/services/specificityRules.ts` | 11.28% |
| 8 | `src/mitm/systemCommands.ts` | 12.19% |
| 9 | `open-sse/services/compression/aggressive.ts` | 12.77% |
| 10 | `src/app/api/v1/batches/[id]/cancel/route.ts` | 12.98% |
| 11 | `open-sse/services/compression/progressiveAging.ts` | 13.26% |
| 12 | `open-sse/services/compression/engines/rtk/smartTruncate.ts` | 13.43% |
| 13 | `open-sse/services/compression/engines/rtk/deduplicator.ts` | 13.51% |
| 14 | `src/lib/cloudAgent/agents/jules.ts` | 13.52% |
| 15 | `open-sse/services/compression/lite.ts` | 14.46% |
| 16 | `src/app/api/v1/rerank/route.ts` | 14.94% |
| 17 | `open-sse/services/compression/preservation.ts` | 15.07% |
| 18 | `src/lib/cloudAgent/agents/codex.ts` | 15.54% |
| 19 | `open-sse/services/tierResolver.ts` | 16.66% |
| 20 | `src/app/docs/components/DocsLazyWrapper.tsx` | 16.66% |
Themes for Phases 6-7:
- `open-sse/services/compression/**` is the densest cluster of low coverage and dominates the remaining gap.
- Batch and rerank API routes (`src/app/api/v1/batches/**`, `src/app/api/v1/rerank/route.ts`) need handler-level tests.
- Cloud agent adapters (`src/lib/cloudAgent/agents/jules.ts`, `codex.ts`) and `tierResolver.ts` need scenario tests.
- Docs UI components and `src/mitm/systemCommands.ts` are lower priority but cheap branch wins.
## Execution checklist
### Phase 1: 56.95% -> 60%
- [x] Fix coverage metric so it reflects source code instead of test files
- [x] Keep a legacy coverage script for comparison
- [x] Record the baseline and hotspots in-repo
- [ ] Add focused tests for low-risk utilities:
- `src/shared/utils/upstreamError.ts`
- `src/shared/utils/fetchTimeout.ts`
- `src/lib/api/errorResponse.ts`
- `src/shared/utils/apiAuth.ts`
- `src/lib/display/names.ts`
- [ ] Add route tests for:
- `src/app/api/settings/require-login/route.ts`
- `src/app/api/providers/[id]/models/route.ts`
### Phase 2: 60% -> 65%
- [ ] Add DB-backed tests for:
- `src/lib/db/modelComboMappings.ts`
- `src/lib/db/settings.ts`
- `src/lib/db/registeredKeys.ts`
- [ ] Cover branch behavior in:
- `src/lib/providers/validation.ts`
- `src/app/api/v1/embeddings/route.ts`
- `src/app/api/v1/moderations/route.ts`
### Phase 3: 65% -> 70%
- [ ] Add usage analytics tests for:
- `src/lib/usage/usageHistory.ts`
- `src/lib/usage/usageStats.ts`
- `src/lib/usage/costCalculator.ts`
- [ ] Expand route coverage for proxy management and settings branches
### Phase 4: 70% -> 75%
- [ ] Cover translator helpers and central translation paths:
- `open-sse/translator/index.ts`
- `open-sse/translator/helpers/*`
- `open-sse/translator/request/*`
- `open-sse/translator/response/*`
### Phase 5: 75% -> 80%
- [ ] Add handler-level tests for:
- `open-sse/handlers/chatCore.ts`
- `open-sse/handlers/responsesHandler.js`
- `open-sse/handlers/imageGeneration.js`
- `open-sse/handlers/embeddings.js`
- [ ] Add executor branch coverage for provider-specific auth, retries, and endpoint overrides
### Phase 6: 80% -> 85%
- [ ] Merge more edge-case suites into the main coverage path
- [ ] Increase function coverage for DB modules with weak constructor/helper coverage
- [ ] Close branch gaps in `settings.ts`, `registeredKeys.ts`, `validation.ts`, and translator helpers
### Phase 7: 85% -> 90%
- [ ] Treat the remaining low-coverage files as blockers
- [ ] Add regression tests for every uncovered production bug fixed during the push to 90%
- [ ] Raise the coverage gate in CI only after the local baseline is stable for at least two consecutive runs
## Ratchet policy
Update `npm run test:coverage` thresholds only after the project actually exceeds the next milestone with a comfortable buffer.
**Current gate:** `npm run test:coverage` enforces **60 statements / 60 lines / 60 functions / 60 branches** (the metric was rebased in Quality-Gates Fase 6A.1 — the earlier 82.58% baseline was inflated because it counted test files and excluded `open-sse`). The `test:coverage:legacy` command preserves the old 50/50/50 metric for historical comparison.
For ad-hoc threshold checks against the latest report use:
```bash
node scripts/check/test-report-summary.mjs --threshold 75
```
Recommended ratchet sequence (order is `statements-lines / branches / functions`):
1. 55/60/55
2. 60/62/58
3. 65/64/62
4. 70/66/66
5. 75/70/72 <-- current gate (75/70/75)
6. 80/75/78
7. 85/80/84
8. 90/85/88
Next ratchet target is `80/75/78` once branch coverage holds above 78% for two consecutive runs.
## Known gap
The current coverage command measures the main Node unit suite and includes source reached from it, including `open-sse`. It does not yet merge Vitest coverage into a single unified report. That merge is worth doing later, but it is not a blocker for starting the 60% -> 80% climb.
docs/ops/DATABASE_GUIDE.md
+626
View File
@@ -0,0 +1,626 @@
---
title: "Database Schema & Operations Guide"
version: 3.8.40
lastUpdated: 2026-06-28
---
# Database Schema & Operations Guide
> **TL;DR**: OmniRoute uses **SQLite with WAL journaling** as its primary store, with **AES-256-GCM** encryption at rest for sensitive fields. This guide covers the schema, migrations, backup/recovery, and operational runbooks.
**Sources:**
- `src/lib/db/core.ts` — singleton + SCHEMA_SQL (17 base tables)
- `src/lib/db/migrationRunner.ts` — versioned migrations
- `src/lib/db/migrations/` — 106 versioned SQL files
- `src/lib/db/encryption.ts` — encryption helpers
- `src/lib/db/backup.ts` — backup export/import
- `src/lib/db/healthCheck.ts` — health diagnostics
---
## Why SQLite?
OmniRoute chose SQLite over PostgreSQL/MySQL for several reasons:
| Factor | SQLite | PostgreSQL |
| --------------- | --------------------------------- | --------------------------------- |
| **Deployment** | Embedded — no separate server | Requires server setup |
| **Encryption** | Application-layer (AES-256-GCM) | Built-in TDE |
| **Performance** | Faster for small/medium workloads | Better for huge concurrent writes |
| **Concurrency** | WAL mode allows concurrent reads | Full MVCC |
| **Backup** | Single-file copy | `pg_dump` or filesystem snapshot |
| **Use case** | Per-user install, embedded | Multi-tenant SaaS |
For **single-user, single-instance** deployments (the primary OmniRoute use case), SQLite is simpler and faster.
### WAL Journaling
`core.ts` opens the database with **WAL (Write-Ahead Logging) mode**:
```ts
// src/lib/db/core.ts
db.pragma("journal_mode = WAL");
db.pragma("busy_timeout = 2000");
db.pragma("synchronous = NORMAL");
// Settings > System & Storage > Cache Size is applied as KiB.
db.pragma("cache_size = -16384");
```
WAL allows **concurrent reads** during writes — important for the dashboard, which queries while requests are being recorded.
---
## Database Location
The SQLite file is stored at:
| OS | Path |
| ------- | -------------------------------------------------------- |
| Linux | `~/.omniroute/storage.sqlite` |
| macOS | `~/.omniroute/storage.sqlite` |
| Windows | `%USERPROFILE%\.omniroute\storage.sqlite` |
| Docker | `/app/data/storage.sqlite` (configurable via `DATA_DIR`) |
Companion files:
- `storage.sqlite-wal` — write-ahead log
- `storage.sqlite-shm` — shared memory file
- `call_logs/` — request payload artifacts (if enabled)
**Override the location:**
```bash
DATA_DIR=/custom/path omniroute
```
---
## Domain Module Architecture
OmniRoute's database has **94 domain modules** in `src/lib/db/`. Each module:
- Owns one or more specific tables
- Exports typed CRUD functions
- Never touches another module's tables
- Uses `getDbInstance()` from `core.ts` to access the DB
### The 94 DB Modules
OmniRoute has **94 module files** in `src/lib/db/`. Below is a sampling of core modules; see the directory listing for the complete list:
| Module | Tables | Responsibility |
| ----------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `providers.ts` | `provider_connections` | OAuth/API key provider registration and credentials |
| `models.ts` | `key_value` (model data) | Model definitions, capabilities, pricing |
| `combos.ts` | `combos` | Combo routing configs and ordering |
| `apiKeys.ts` | `api_keys` | API key lifecycle, scopes, quota tracking |
| `settings.ts` | `key_value`, `api_keys`, `combos` | System configuration and shared KV store |
| `backup.ts` | — | Backup export/import operations |
| `proxies.ts` | `proxy_registry`, `proxy_assignments`, `provider_connections` | Proxy configs and routing rules |
| `prompts.ts` | `prompt_templates` | Reusable prompt templates, versioning |
| `webhooks.ts` | `webhooks` | Event-driven webhook subscriptions and logs |
| `detailedLogs.ts` | `request_detail_logs` | Per-request audit logging (optional, high volume) |
| `domainState.ts` | `domain_*` (5 tables) | Domain budgets, circuit breakers, lockouts, fallback chains, cost history |
| `registeredKeys.ts` | `registered_keys`, `account_key_limits`, `provider_key_limits` | Whitelisted API keys for MCP/A2A |
| `quotaSnapshots.ts` | `quota_snapshots` | Historical quota usage |
| `modelComboMappings.ts` | `model_combo_mappings` | Map models to combo defaults |
| `cliToolState.ts` | `cli_tool_state` | CLI-specific persistent state |
| `encryption.ts` | — | Helpers for encrypting/decrypting fields |
| `readCache.ts` | — | In-memory cache for read-heavy ops |
| `secrets.ts` | `key_value` (encrypted entries) | Encrypted secret storage |
| `stateReset.ts` | — | Wipe/reset DB state for testing |
| `contextHandoffs.ts` | `context_handoffs` | Session context for agent handoff |
| `usage*.ts` | `usage_history`, `call_logs`, `proxy_logs` | Usage tracking |
| `compression*.ts` | `compression_settings`, `compression_combos` | Compression config |
### Module Boundaries
A core architectural rule: **modules don't access each other's tables directly**. To work with another module's data, import the function from that module.
```ts
// ❌ WRONG: direct SQL from another module
db.prepare("SELECT * FROM provider_connections").all();
// ✅ RIGHT: use the providers module function
import { listProviders } from "@/lib/db/providers";
const providers = await listProviders();
```
This rule is enforced by code review — there's no static check, but violations are flagged.
---
## Base Schema (17 tables)
`core.ts` defines the 17 base tables in `SCHEMA_SQL`. These are created by migration `001_initial_schema.sql` and form the core schema.
### Core Tables (created in initial migration)
| Table | Purpose | Key columns |
| -------------------------- | -------------------------------- | ----------------------------------------------------------------------- |
| `provider_connections` | Provider credentials (encrypted) | `id`, `provider`, `auth_type`, `api_key`, `is_active` |
| `provider_nodes` | Provider node routing info | `id`, `type`, `name`, `base_url`, `created_at` |
| `key_value` | General KV store | `namespace`, `key`, `value` |
| `combos` | Routing combo definitions | `id`, `name`, `data`, `sort_order` |
| `api_keys` | API keys for the gateway | `id`, `name`, `key`, `machine_id`, `allowed_models` |
| `db_meta` | Database metadata | `key`, `value` |
| `usage_history` | Request usage records | `id`, `provider`, `model`, `tokens_input`, `tokens_output`, `timestamp` |
| `call_logs` | Request payloads & responses | `id`, `timestamp`, `status`, `model`, `provider`, `latency_ms` |
| `proxy_logs` | Proxy request logs | `id`, `timestamp`, `proxy_type`, `status`, `provider` |
| `domain_fallback_chains` | Model-to-provider chains | `model`, `chain` |
| `domain_budgets` | Per-domain spend budgets | `api_key_id`, `daily_limit_usd`, `warning_threshold`, `reset_interval` |
| `domain_budget_reset_logs` | Budget reset history | `id`, `api_key_id`, `reset_interval`, `previous_spend`, `reset_at` |
| `domain_cost_history` | Per-domain cost tracking | `id`, `api_key_id`, `cost`, `timestamp` |
| `domain_lockout_state` | Domain rate-limit state | `identifier`, `attempts`, `locked_until` |
| `domain_circuit_breakers` | Circuit breaker state per domain | `name`, `state`, `failure_count`, `last_failure_time` |
| `semantic_cache` | LLM response cache | `id`, `signature`, `model`, `prompt_hash`, `response` |
| `quota_snapshots` | Historical quota snapshots | `id`, `provider`, `connection_id`, `window_key`, `remaining_percentage` |
### Additional Tables (added by later migrations)
Subsequent migrations add tables such as:
- `cli_tool_state` (migration 011) — CLI tool state
- `mcp_*` tables — MCP server audit
- `a2a_*` tables — A2A task state
- `usage_*` tables — usage tracking
- `plugin_*` tables — plugin system
- `skill_executions` — skill execution history
- `memory_*` tables — memory system
- `compression_*` tables — compression system
- `webhook_*` tables — webhook delivery log
- `acp_*` tables — Agent Client Protocol
- `oneproxy_*` tables — 1proxy marketplace
- `proxy_assignments` — proxy scope bindings
- `detailed_call_artifacts` — call log artifacts metadata
- `quota_alert_history` — quota alert audit
- `command_code_auth_sessions` — Command Code OAuth sessions
The full list of ~30+ tables is in `src/lib/db/migrations/`.
---
## Migrations
OmniRoute uses **versioned, idempotent migrations** in `src/lib/db/migrations/`. Each migration is a single SQL file named `NNN_description.sql`.
### Migration Naming
```
001_initial_schema.sql
002_mcp_a2a_tables.sql
003_provider_node_custom_paths.sql
...
021_combo_call_log_targets.sql
```
### How Migrations Run
At startup, `migrationRunner.ts`:
1. Creates `_omniroute_migrations` table if not exists
2. Queries for already-applied migrations
3. Applies any new migrations in order, each in a transaction
4. Records each applied migration with timestamp
```ts
// src/lib/db/migrationRunner.ts (simplified)
export async function runMigrations(db: SqliteDatabase, migrationsDir: string) {
const applied = getAppliedMigrations(db);
const available = readMigrationFiles(migrationsDir);
for (const migration of available) {
if (applied.includes(migration.id)) continue;
db.transaction(() => {
db.exec(migration.sql);
recordAppliedMigration(db, migration.id);
})();
}
}
```
### Idempotency
Migrations must be **idempotent** — running them twice should be a no-op:
```sql
-- 004_proxy_registry.sql
CREATE TABLE IF NOT EXISTS proxy_registry (
id TEXT PRIMARY KEY,
host TEXT NOT NULL,
port INTEGER NOT NULL,
...
);
```
Use `IF NOT EXISTS`, `IF EXISTS`, and `OR IGNORE` / `OR REPLACE` clauses liberally.
### Adding a New Migration
1. **Identify the next number**: `ls src/lib/db/migrations/ | tail -1`
2. **Create the file**: `NNN_my_change.sql`
3. **Use safe DDL**: `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN`
4. **Backfill data carefully**: use `UPDATE ... WHERE ...` to handle existing rows
5. **Test on a copy**: never run untested migrations on production
Example:
```sql
-- 022_add_combo_priority.sql
ALTER TABLE combos ADD COLUMN priority INTEGER DEFAULT 100;
UPDATE combos SET priority = 100 WHERE priority IS NULL;
CREATE INDEX IF NOT EXISTS idx_combos_priority ON combos(priority);
```
> **Backwards-incompatible changes** (e.g., dropping columns) are tricky. OmniRoute does NOT support downgrade — once a migration is applied, the schema change is permanent. Plan accordingly.
---
## Encryption at Rest
Sensitive fields (API keys, OAuth tokens, connection strings) are encrypted at rest using **AES-256-GCM**.
### How It Works
```ts
// src/lib/db/encryption.ts (simplified)
const key = deriveKeyFromPassphrase(passphrase, salt);
const iv = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", key, iv);
const encrypted = Buffer.concat([cipher.update(plaintext), cipher.final()]);
const authTag = cipher.getAuthTag();
return { encrypted, iv, authTag };
```
### Where It's Used
- `provider_connections.api_key` — encrypted at application level
- `provider_connections.access_token`, `refresh_token`, `id_token` — encrypted at application level
- `key_value` entries with `namespace = "secrets"` — encrypted at application level
- `proxy_registry.auth` — encrypted at application level (if present)
### Encryption Key
The encryption key is derived from a **passphrase** (set via `STORAGE_ENCRYPTION_KEY` env var) and a **salt** (stored in the DB). Both are required to decrypt data.
```bash
# Generate a secure passphrase
openssl rand -hex 32
# Set in .env
STORAGE_ENCRYPTION_KEY=<your-key>
```
> **Critical**: Losing the encryption key means losing access to all encrypted data. **Back up the key separately from the database**.
### What's NOT Encrypted
For performance reasons, the following are stored in plaintext:
- Provider display names
- Model definitions (already public)
- Routing rules
- Usage records (no PII)
---
## Encryption Caveats (v3.8.16+)
OmniRoute uses **`migrateLegacyEncryptedString()`** to handle two encryption schemes transparently:
- **Legacy** (pre-v3.5.0): XOR-based "encryption" (not real crypto)
- **Current**: AES-256-GCM with proper IV and auth tag
The migration helper detects the legacy format and re-encrypts with the new scheme on first read. This means you can upgrade an old database without losing credentials.
---
## Read Cache
For frequently-read data (models, providers, settings), `readCache.ts` provides an **in-memory cache**:
```ts
// Cached at startup, invalidated on write
const providers = await getCachedProviders(); // Fast, in-memory
const fresh = await listProviders(); // Slow, hits DB
```
| Cached entity | Cache key | TTL |
| ---------------------- | -------------- | ----------- |
| `models` | `models:v1` | Until write |
| `provider_connections` | `providers:v1` | Until write |
| `settings` | `settings:v1` | Until write |
| `combos` | `combos:v1` | Until write |
Cache is invalidated on every write to the corresponding table.
---
## Backup and Recovery
### Manual Backup
```bash
# Use the CLI to create a local backup
omniroute backup create --name pre-migration
# Or via the API
curl -X PUT http://localhost:20128/api/db-backups \
-H "Authorization: Bearer $MANAGEMENT_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "pre-migration"}'
```
The backup file includes:
- All DB tables (serialized to JSON)
- Call log artifacts (base64-encoded, optional)
- Settings + secrets (encrypted)
- Plugin configuration
### Restore
```bash
# Via CLI
omniroute restore pre-migration
# Via API
curl -X POST http://localhost:20128/api/db-backups/restore \
-H "Authorization: Bearer $MANAGEMENT_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "pre-migration"}'
```
> **Warning**: Restore overwrites the entire DB. Stop all clients first.
### Automated Backups
```bash
# Enable automated daily backups via CLI
omniroute backup auto enable --cron "0 2 * * *" --retention 7
```
### SQLite Hot Backup
For zero-downtime backup of a live DB:
```bash
sqlite3 ~/.omniroute/storage.sqlite ".backup /backups/omniroute-hot.db"
```
This uses SQLite's online backup API — safe to run while OmniRoute is running.
---
## Performance Tuning
### WAL Mode
WAL is enabled by default. For high-write workloads, consider:
```sql
PRAGMA wal_autocheckpoint = 1000; -- Checkpoint every 1000 pages
PRAGMA journal_size_limit = 67108864; -- 64MB WAL cap
```
### Indexes
Key indexes for performance (auto-created by migrations):
- `idx_models_provider` — model lookups by provider
- `idx_combo_targets_combo_id` — combo target expansion
- `idx_usage_history_api_key_timestamp` — usage analytics
- `idx_quota_snapshots_api_key_window` — quota tracking
- `idx_call_logs_timestamp` — call log queries
To add a new index, create a migration:
```sql
-- 023_add_my_index.sql
CREATE INDEX IF NOT EXISTS idx_my_table_my_column ON my_table(my_column);
```
### Memory-Mapped I/O
For very large databases (>10GB), memory mapping can be adjusted via SQLite pragma:
```sql
-- Set via SQLite pragma (adjust in core.ts or runtime)
PRAGMA mmap_size = 268435456; -- 256MB
```
### Compaction
Long-running OmniRoute instances benefit from occasional `VACUUM`:
```bash
sqlite3 ~/.omniroute/storage.sqlite "VACUUM;"
```
Run monthly during low-traffic windows. (WAL mode reduces the need, but doesn't eliminate it.)
---
## Health Check
`src/lib/db/healthCheck.ts` provides **DB-level health diagnostics**:
````bash
GET /api/db/health
Returns:
```json
{
"status": "healthy",
"checks": {
"writable": { "status": "pass" },
"integrity": { "status": "pass", "result": "ok" },
"foreign_keys": { "status": "pass", "violations": 0 },
"orphaned_artifacts": { "status": "warn", "count": 12 },
"table_sizes": {
"usage_history": { "rows": 12345, "size_mb": 12.3 },
"call_logs": { "rows": 567, "size_mb": 2.1 }
}
}
}
````
Run `PRAGMA integrity_check` to detect corruption:
```bash
sqlite3 ~/.omniroute/storage.sqlite "PRAGMA integrity_check;"
# Should print: ok
```
If it returns anything other than `ok`, **stop using the database immediately** and restore from backup.
---
## Disaster Recovery
### Scenario 1: WAL File Lost
The `-wal` file is missing but `-shm` and main DB are intact:
```bash
# Recovers automatically on next open
omniroute
```
If SQLite can't auto-recover:
```bash
sqlite3 ~/.omniroute/storage.sqlite ".recover" > recovered.sql
sqlite3 recovered.db < recovered.sql
mv recovered.db ~/.omniroute/storage.sqlite
```
### Scenario 2: Main DB File Corrupted
Restore from backup:
```bash
omniroute sync pull --merge # or: omniroute backup restore <backup-id>
```
### Scenario 3: Encryption Key Lost
**No recovery possible** without the key. The encrypted fields are unreadable. Re-add all providers manually with new credentials.
> **Mitigation**: Always back up the encryption key separately, ideally in a password manager or KMS.
### Scenario 4: Disk Full
SQLite will return `SQLITE_FULL` errors. Free disk space, then:
```bash
# Checkpoint WAL to free up space
sqlite3 ~/.omniroute/storage.sqlite "PRAGMA wal_checkpoint(TRUNCATE);"
```
---
## Common Operations
### Inspect a Table
```bash
sqlite3 ~/.omniroute/storage.sqlite "SELECT * FROM api_keys LIMIT 5;"
```
### Count Rows in All Tables
```bash
sqlite3 ~/.omniroute/storage.sqlite <<EOF
SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%';
EOF
```
### Reset (Wipe) All Data
```bash
# Stop OmniRoute first
omniroute stop
# Delete the DB file
rm ~/.omniroute/storage.sqlite*
# Restart (will recreate empty DB)
omniroute
```
For a **selective** reset (keep providers, wipe usage):
```bash
DELETE FROM usage_history WHERE timestamp < datetime('now', '-30 day');
DELETE FROM call_logs WHERE timestamp < datetime('now', '-30 day');
DELETE FROM proxy_logs WHERE timestamp < datetime('now', '-30 day');
```
### Export Single Table
```bash
sqlite3 ~/.omniroute/storage.sqlite <<EOF
.mode csv
.output api_keys.csv
SELECT * FROM api_keys;
EOF
```
---
## Troubleshooting
### "Database is locked"
Another process is holding a write lock. Either:
- Wait for the other process to finish (check `lsof | grep storage.sqlite`)
- Kill the other process
- If persistent, restart OmniRoute
### "Foreign key constraint failed"
A domain module is violating referential integrity. Check:
- Orphaned rows in dependent tables
- Cascading deletes that didn't propagate
- Recent migration that changed a foreign key
Run `PRAGMA foreign_key_check;` to find violations.
### "Out of memory"
SQLite's memory-mapped I/O is exceeding the OS limit. Reduce via SQLite pragma:
```sql
PRAGMA mmap_size = 134217728; -- 128MB instead of 256MB
```
Or disable:
```sql
PRAGMA mmap_size = 0;
```
### "Migration failed mid-way"
The migration ran in a transaction, so it should have rolled back. If not:
1. **Stop OmniRoute** (prevent further attempts)
2. **Check the DB state** with `sqlite3`
3. **Manually fix** the partial migration
4. **Re-run** OmniRoute (the migration will be retried)
To prevent this, always test migrations on a copy first.
---
## See Also
- [USAGE_QUOTA_GUIDE.md](../guides/USAGE_QUOTA_GUIDE.md) — usage tables
- [MONITORING_GUIDE.md](./MONITORING_GUIDE.md) — health monitoring
- [RELEASE_CHECKLIST.md](./RELEASE_CHECKLIST.md) — release flow
- Source: `src/lib/db/` (80+ files, ~25K LOC)
docs/ops/FLY_IO_DEPLOYMENT_GUIDE.md
+498
View File
@@ -0,0 +1,498 @@
---
title: "OmniRoute Fly.io Deployment Guide"
version: 3.8.40
lastUpdated: 2026-06-28
---
# OmniRoute Fly.io Deployment Guide
This document describes the actual deployment process for OmniRoute on Fly.io, covering two scenarios:
- Deploying the current project to Fly.io for the first time
- Publishing subsequent code updates
- New projects following the same deployment workflow
This guide is based on a verified working configuration for the current project. The application name is `omniroute`.
---
## 1. Deployment Goals
- Platform: Fly.io
- Deployment method: Local `flyctl` direct publish
- Runtime: Using the existing `Dockerfile` and `fly.toml` in the repository
- Data persistence: Fly Volume mounted to `/data`
- Access URL: `https://omniroute.fly.dev/`
---
## 2. Current Project Key Configuration
The `fly.toml` in the current repository has been confirmed to contain the following key items:
```toml
app = 'omniroute'
primary_region = 'sin'
[[mounts]]
source = 'data'
destination = '/data'
[processes]
app = 'node run-standalone.mjs'
[http_service]
internal_port = 20128
[env]
TZ = "Asia/Shanghai"
HOST = "0.0.0.0"
HOSTNAME = "0.0.0.0"
BIND = "0.0.0.0"
```
Notes:
- `app = 'omniroute'` determines which Fly application the deployment targets
- `destination = '/data'` determines the persistent volume mount directory
- This project must set `DATA_DIR=/data`, otherwise the database and keys will be written to the container's temporary directory
---
## 3. Prerequisites
### 3.1 Installing the Fly CLI
Windows PowerShell:
```powershell
pwsh -Command "iwr https://fly.io/install.ps1 -useb | iex"
```
If the install script fails in your environment, you can also manually download the `flyctl` binary and add it to your `PATH`.
### 3.2 Logging in to Your Fly Account
```powershell
flyctl auth login
```
### 3.3 Verifying Login Status
```powershell
flyctl auth whoami
flyctl version
```
---
## 4. First-Time Deployment of the Current Project
### 4.1 Clone the Code and Enter the Directory
```powershell
git clone https://github.com/diegosouzapw/OmniRoute.git
cd OmniRoute
```
### 4.2 Confirm the Application Name
Open `fly.toml` and verify the following line:
```toml
app = 'omniroute'
```
If you are deploying to your own new application, you can change it to a globally unique name, for example:
```toml
app = 'omniroute-yourname'
```
Note:
- Make sure the application you see in the console matches the `app` value in `fly.toml`
- If you previously used a different name, such as `oroute`, do not confuse it with `omniroute`
### 4.3 Create the Application
If the application does not yet exist:
```powershell
flyctl apps create omniroute
```
If you changed the application name, replace `omniroute` with your chosen name.
### 4.4 First Deploy
```powershell
flyctl deploy
```
---
## 5. Required Parameters
This project recommends configuring at least the following parameters on Fly.io.
### 5.1 Verified Parameters
These parameters have been used in actual deployments on the current `omniroute` application:
- `API_KEY_SECRET`
- `DATA_DIR`
- `JWT_SECRET`
- `MACHINE_ID_SALT`
- `NEXT_PUBLIC_BASE_URL`
- `OMNIROUTE_WS_BRIDGE_SECRET` (required in production — used for WebSocket bridge authentication)
- `STORAGE_ENCRYPTION_KEY`
### 5.2 About `INITIAL_PASSWORD`
The current project does not set `INITIAL_PASSWORD` because this deployment does not require it.
If it is not set:
- The startup log will indicate the default password is `CHANGEME`
- You should change the login password in system settings as soon as possible after deployment
If you want to initialize the backend password unattended, you can add it later:
- `INITIAL_PASSWORD`
---
## 6. Recommended Parameters
### 6.1 Secrets Configuration
The following variables are recommended for Fly Secrets:
| Variable | Recommendation | Description |
| ----------------------------- | ---------------------- | ----------------------------------------------------- |
| `API_KEY_SECRET` | Required | Used for API Key generation and validation |
| `JWT_SECRET` | Required | Used for login sessions and JWT signing |
| `OMNIROUTE_WS_BRIDGE_SECRET` | Required in production | WebSocket bridge authentication secret |
| `STORAGE_ENCRYPTION_KEY` | Strongly recommended | Encrypts sensitive connection information at rest |
| `MACHINE_ID_SALT` | Recommended | Generates a stable machine identifier |
| `INITIAL_PASSWORD` | Optional | Sets the initial backend password at first deployment |
| OAuth/API private credentials | As needed | External platform authentication configuration |
### 6.2 Recommended Values for the Current Project
| Variable | Recommended Value |
| ---------------------- | --------------------------- |
| `DATA_DIR` | `/data` |
| `NEXT_PUBLIC_BASE_URL` | `https://omniroute.fly.dev` |
Notes:
- `DATA_DIR=/data` is critical and must match the Fly Volume mount point
- `NEXT_PUBLIC_BASE_URL` is used by the scheduler, frontend callbacks, and similar scenarios
### 6.3 OAuth Callback URL Configuration
If you need to enable OAuth-based providers (e.g. Antigravity, Gemini, Cursor) on the Fly.io deployment, make sure of the following two points:
1. **Set `NEXT_PUBLIC_BASE_URL` to your public HTTPS domain**
```powershell
flyctl secrets set NEXT_PUBLIC_BASE_URL=https://omniroute.fly.dev -a omniroute
```
If you are using a custom domain, replace it with the corresponding domain (e.g. `https://omniroute.yourdomain.com`).
2. **Configure the callback URL on the provider console**
All OAuth providers share the single callback path `/callback` — there is NO per-provider callback route:
```text
<NEXT_PUBLIC_BASE_URL>/callback
```
For example, regardless of Gemini, Antigravity, Cursor, or GitLab Duo:
- `https://omniroute.fly.dev/callback`
If `NEXT_PUBLIC_BASE_URL` does not match the callback URL registered with the provider, the OAuth flow will fail at the browser redirect step.
---
## 7. One-Command Secret Setup
The following commands generate secure random values and write all required parameters for the current project to Fly Secrets in one step.
Notes:
- Does not include `INITIAL_PASSWORD`
- Intended for the current project `omniroute`
```powershell
$apiKeySecret = [Convert]::ToHexString((1..32 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 })).ToLower()
$jwtSecret = [Convert]::ToHexString((1..64 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 })).ToLower()
$machineIdSalt = [Convert]::ToHexString((1..32 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 })).ToLower()
$storageKey = [Convert]::ToHexString((1..32 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 })).ToLower()
$wsBridgeSecret = [Convert]::ToHexString((1..32 | ForEach-Object { Get-Random -Minimum 0 -Maximum 256 })).ToLower()
flyctl secrets set `
API_KEY_SECRET=$apiKeySecret `
JWT_SECRET=$jwtSecret `
MACHINE_ID_SALT=$machineIdSalt `
STORAGE_ENCRYPTION_KEY=$storageKey `
OMNIROUTE_WS_BRIDGE_SECRET=$wsBridgeSecret `
DATA_DIR=/data `
NEXT_PUBLIC_BASE_URL=https://omniroute.fly.dev `
-a omniroute
```
On Linux / macOS, you can also use `openssl rand -hex 32`:
```bash
flyctl secrets set OMNIROUTE_WS_BRIDGE_SECRET=$(openssl rand -hex 32) -a omniroute
```
Notes:
- `OMNIROUTE_WS_BRIDGE_SECRET` is required in production; missing it will break the WebSocket bridge handshake
If you also want to set an initial password:
```powershell
flyctl secrets set INITIAL_PASSWORD=your-strong-password -a omniroute
```
---
## 8. Viewing Current Parameters
```powershell
flyctl secrets list -a omniroute
```
If the `Secrets` page in the console does not show the expected variables, check:
- That you are viewing the `omniroute` application
- That the `app` value in `fly.toml` matches the application in the console
---
## 9. Subsequent Updates and Releases
After code updates, the release process is straightforward:
```powershell
git pull
flyctl deploy
```
If you only need to update parameters without changing code:
```powershell
flyctl secrets set KEY=value -a omniroute
```
Fly will automatically perform a rolling update of machines.
### 9.1 Tracking Upstream Repository Updates While Preserving Your Fork's `fly.toml`
If the current repository is a fork and you want to sync updates from the upstream `https://github.com/diegosouzapw/OmniRoute`, follow the workflow below.
First, verify your remotes:
```powershell
git remote -v
```
You should see at least:
- `origin` pointing to your own fork
- `upstream` pointing to the original repository
If `upstream` is not configured, add it:
```powershell
git remote add upstream https://github.com/diegosouzapw/OmniRoute.git
```
Before syncing with upstream, fetch the latest commits and tags:
```powershell
git fetch upstream --tags
```
Check the current version and upstream tags:
```powershell
git describe --tags --always
git show --no-patch --oneline v3.4.7
```
> Note: The current project version is `v3.8.0`. The `v3.4.7` references below are kept as historical examples only. For actual releases, use `:latest` or the current version tag (e.g. `:v3.8.0`).
If you want to merge the latest upstream `main` while forcefully keeping your fork's `fly.toml`, follow this workflow:
```powershell
git merge upstream/main
git checkout HEAD~1 -- fly.toml
git add -- fly.toml
git commit -m "chore(deploy): keep fork fly.toml"
git push origin main
```
Notes:
- `git merge upstream/main` syncs the latest code from the original repository
- `git checkout HEAD~1 -- fly.toml` restores your fork's own `fly.toml` from before the merge
- If upstream did not modify `fly.toml`, this step will not introduce any differences
- If upstream did modify `fly.toml`, this step ensures your Fly application name, volume mount, region, and other fork-specific deployment configuration are not overwritten
If you want to align with a specific release tag (e.g. `v3.4.7`), first verify that the tag is already included in `upstream/main`:
```powershell
git merge-base --is-ancestor v3.4.7 upstream/main
```
A successful return means `upstream/main` already contains that version; you can simply merge `upstream/main`.
### 9.2 Standard Release Sequence After Syncing Upstream
After syncing with the original repository, follow this recommended release order:
1. `git fetch upstream --tags`
2. `git merge upstream/main`
3. Restore the fork's `fly.toml`
4. `git push origin main`
5. `flyctl deploy`
6. `flyctl status -a omniroute`
7. `flyctl logs --no-tail -a omniroute`
This is the actual workflow used when upgrading the current project to `v3.4.7` (the example refers to a historical version; the current actual version is `v3.8.0`).
---
## 10. Post-Deployment Checks
### 10.1 Check Application Status
```powershell
flyctl status -a omniroute
```
### 10.2 View Startup Logs
```powershell
flyctl logs --no-tail -a omniroute
```
### 10.3 Verify Site Accessibility
```powershell
try {
(Invoke-WebRequest -Uri "https://omniroute.fly.dev" -MaximumRedirection 5 -UseBasicParsing).StatusCode
} catch {
if ($_.Exception.Response) {
$_.Exception.Response.StatusCode.value__
} else {
throw
}
}
```
A return value of `200` indicates the site is responding normally.
---
## 11. Success Indicators
After a successful deployment, the logs should show content similar to:
```text
[bootstrap] Secrets persisted to: /data/server.env
[DB] SQLite database ready: /data/storage.sqlite
```
These two points are critical:
- `/data/server.env` confirms the runtime secrets are written to the persistent volume
- `/data/storage.sqlite` confirms the database is written to the persistent volume
If you see `/app/data/...` instead, `DATA_DIR` is misconfigured and must be corrected immediately.
---
## 12. Common Issues
### 12.1 `Secrets` Page Is Empty
There are usually two reasons:
- You have not yet run `flyctl secrets set`
- You are viewing a different application (e.g. `oroute` instead of `omniroute`)
### 12.2 `flyctl deploy` Reports `app not found`
Create the application first:
```powershell
flyctl apps create omniroute
```
### 12.3 `fly.toml` Parsing Fails
Check the following:
- Whether there are garbled characters in comments
- Whether TOML quoting and indentation are correct
### 12.4 Data Is Not Persisting
Verify both of the following:
- `fly.toml` contains `destination = '/data'`
- `DATA_DIR` is set to `/data`
### 12.5 Can It Run Without `INITIAL_PASSWORD`?
Yes, it can run. It will fall back to the default `CHANGEME` password. It is recommended to change the backend password as soon as possible in production.
---
## 13. Reusing for New Projects
If you are deploying a new project following this document, you only need to change these items:
1. Change the `app` value in `fly.toml`
2. Change `NEXT_PUBLIC_BASE_URL`
3. Keep `DATA_DIR=/data`
4. Regenerate `API_KEY_SECRET`, `JWT_SECRET`, `MACHINE_ID_SALT`, and `STORAGE_ENCRYPTION_KEY`
5. After the first deployment, verify that logs are written to `/data`
Do not reuse keys from a previous project.
---
## 14. Minimal Release Checklist for the Current Project
The most commonly used commands for subsequent releases are:
```powershell
flyctl auth whoami
flyctl status -a omniroute
flyctl secrets list -a omniroute
flyctl deploy
flyctl logs --no-tail -a omniroute
```
For a normal release, the core command is simply:
```powershell
flyctl deploy
```
For a first-time deployment in a new environment, the core steps are:
1. `flyctl auth login`
2. `flyctl apps create omniroute`
3. `flyctl secrets set ... -a omniroute`
4. `flyctl deploy`
5. `flyctl logs --no-tail -a omniroute`
docs/ops/MATURITY_REEVAL.md
+115
View File
@@ -0,0 +1,115 @@
---
title: "Quality-Gate Maturity Re-evaluation (Fase 9)"
---
# Maturity Re-evaluation — post-Waves 0–3 (Quality-Gate v2)
> **What this document is.** A re-measurement of the quality-gates system maturity
> **after** Waves 0–3 of the Quality-Gate v2 program, compared to the baseline recorded in
> [`QUALITY_GATE_PLAYBOOK.md`](./QUALITY_GATE_PLAYBOOK.md) (2026-06-16). Measures what changed,
> against DSOMM L5 / OpenSSF Scorecard 9 / SLSA L3, separating what is **CI-measurable**
> (already delivered / deliverable by code) from what is **process/owner** (organization settings).
>
> **Date:** 2026-06-30. Generated from the actual state of the repository, not from memory.
> **Benchmarks:** OWASP DSOMM · OpenSSF Scorecard · SLSA · SonarQube "Clean as You Code".
---
## 1. Updated verdict
**Overall grade: A− → A ("Advanced", top ~5%).** The **two biggest structural weaknesses**
of the 06-16 baseline — the _fast-gates gap_ and the _mutation-score-not-a-ratchet_ — have been **closed**.
The residual gaps for "absolute maximum" are almost all **owner/infra-gated** (branch-protection,
SLSA L3, CodeQL advanced); the code side of the program is essentially complete.
| Reference framework | Baseline 06-16 | Now 06-30 | Movement | Evidence |
| --------------------------------- | ------------------------------ | ----------------------------------------------------------------- | -------- | --------------------------------------------------------------------- |
| **OWASP DSOMM** (5 levels) | L3→L4 | **L4** in _Test Intensity_ and _Static Depth_; solid L3 in others | ▲ | blocking mutation-ratchet + deterministic suite at merge gate |
| **OpenSSF Scorecard** | ~7–8/10 | ~7–8/10 (unchanged — gate is the **owner**) | = | missing Branch-Protection on `main` (owner setting) + actions pinning |
| **SLSA** | L2→L3 | **L2** (approaching L3) | = | missing hermetic/reproducible builder (infra/owner) |
| **SonarQube "Clean as You Code"** | Aligned with caveat | Aligned with caveat | = | _sprawl_ caveat (~46+ gates) persists — ROI review pending |
| **Quality-Ratchet pattern** | Exemplar | **Exemplar+** | ▲ | new `dedicatedGate` for `mutationScore` (direction up) |
| **Mutation testing** | "Almost there" (not a ratchet) | **Active ratchet** | ▲▲ | `check-mutation-ratchet.mjs` + seeded baseline + blocking nightly job |
---
## 2. Deltas since 2026-06-16 (what Waves 0–3 delivered)
### 2.1 🔴→✅ Fast-gates gap CLOSED (was structural weakness #1)
The baseline warned: `quality.yml` (PR→`release/**`) ran **only filesystem gates** — no
typecheck, tests, or build —, so deterministic regressions only exploded on PR→`main`.
**Today** `.github/workflows/quality.yml` runs, in the _Fast Quality Gates_ job: `typecheck:core`,
**blocking impacted unit tests (TIA) with fail-safe to the full suite**, the
vitest fast-path, and unit shards. The gate now runs **where the merge happens** (shift-left),
exactly the cross-cutting principle the playbook prescribes.
### 2.2 🟠→✅ Mutation score became a RATCHET (was weakness #3 / P0 #1)
The strongest antidote against coverage-gaming was **advisory**. **Today**:
- `scripts/check/check-mutation-ratchet.mjs` (advisory by default, `--ratchet` blocking, graceful skip);
- `config/quality/quality-baseline.json` has seeded `mutationScore.<module>` entries (`direction: up`, `dedicatedGate`);
- `.github/workflows/nightly-mutation.yml` has the **"Mutation score ratchet (blocking)"** job that unifies batch reports and ratchets merged per-module scores.
Result: the per-module mutation score **cannot regress** — coverage has ceased to be a vanity metric.
### 2.3 ✅ Quick-win gates (Phase 6A/7) delivered
- **a11y axe-core "fake-green" fixed:** `@axe-core/playwright` in devDeps; `a11y.spec.ts` with conditional `REQUIRE_AXE` skip; job in `nightly-resilience.yml`.
- **complexity scans `bin/`+`electron`:** `check-complexity.mjs` includes those directories in `ESLINT_ARGS`.
- **tracked-artifacts in pre-commit + pre-push:** `.husky/pre-commit` + `pre-push` block accidentally tracked artifacts.
---
## 3. The 12 categories — status (delta-focused)
| # | Category | Status 06-30 |
| --- | -------------------------------- | ---------------------------------------------------------------------------------------- |
| 1 | Style & formatting | ✅ unchanged (Prettier+ESLint lint-staged) |
| 2 | Types | ✅ **reinforced** — `typecheck:core` now also in the PR→release gate |
| 3 | Tests (intensity) | ✅ **reinforced** — mutation testing became a ratchet; deterministic suite at merge gate |
| 4 | Test policy (anti-gaming) | ✅ unchanged (pr-test-policy/test-masking/pr-evidence) |
| 5 | Complexity & health | ✅ **reinforced** — complexity scans bin/electron |
| 6 | Static security (SAST+secrets) | 🟡 CodeQL default-setup (advanced = owner); semgrep cloud not versioned |
| 7 | Supply-chain (deps) | ✅ unchanged (osv/audit/Trivy/Dependabot + allowlist) |
| 8 | Supply-chain (build/release) | 🟡 SLSA L2 (L3 = hermetic builder, owner/infra) |
| 9 | Contracts & API | 🟡 oasdiff/osv advisory (candidates for blocking-with-scope, P1) |
| 10 | Docs & i18n (anti-rot) | ✅ **reinforced** — `fabricated-docs --strict` blocking (exit 0 verified) |
| 11 | Anti-hallucination / consistency | ✅ unchanged (known-symbols/fetch-targets/docs-symbols/db-rules) |
| 12 | Resilience & domain | ✅ unchanged (chaos/heap/k6/promptfoo/garak nightly) |
---
## 4. Residual gaps for "absolute maximum"
### 4.1 CI-measurable / deliverable by code (this program's backlog)
- **P1 — osv/oasdiff → blocking with the right scope:** osv only `CRITICAL`+fixable (two-step like Trivy); oasdiff blocks contract-breaking changes.
- **P1 — `require-tighten` blocking (end of cycle):** locks metric gains (prevents loosening the baseline without recording).
- **P1/P2 — ROI review / gate sprawl:** consolidate doc-sync micro-gates; measure per-gate timing in `ci-summary` (combats fatigue — SonarQube/DORA caveat). Deferred ROI merges (unified complexity; unified `/api` anti-hallucination) fall here.
- **P2 — CodeQL config committed + semgrep versioned:** more control/reproducibility.
### 4.2 Process / owner (CI cannot move — organization settings)
- **Branch-protection on `main`** (raises Scorecard, closes the DSOMM gap). See [`BRANCH_PROTECTION_MAIN.md`](./BRANCH_PROTECTION_MAIN.md).
- **CodeQL Default → Advanced setup.**
- **SLSA L3** — hermetic/reproducible builder (GitHub SLSA generator). Stretch (diminishing returns).
### 4.3 Explicitly out of scope
- **DSOMM L5** is largely **org-level / process** (not CI-encodable).
- **SLSA L4** (bit-for-bit reproducibility) is a declared stretch goal.
---
## 5. Deferred / removed items (tail housekeeping)
- **`semcheck.yaml` (LLM layer for semantic drift docs↔code) — REMOVED.** It was **orphaned**
(no workflow/script invoked it) and had stale counts in the rules. Deterministic coverage
already exists (`check:fabricated-docs --strict` + `check:docs-counts-sync` + `check:docs-symbols`),
and the _gate sprawl_ caveat discourages adding an LLM advisory gate with recurring cost.
It may be re-introduced in the future as an opt-in nightly job if semantic drift becomes a real problem.
- **`agent-lsp` scaffold — DEFERRED / opt-in not enabled.** Exists as a mention in docs
(`docs/architecture/QUALITY_GATES.md`, CHANGELOG) but **without wiring** and without `.mcp.json.example`
in the repo. Remains as a documented opt-in scaffold; it is not an active gate nor a maturity gap.
docs/ops/MONITORING_GUIDE.md
+476
View File
@@ -0,0 +1,476 @@
---
title: "Monitoring & Observability Guide"
version: 3.8.40
lastUpdated: 2026-06-28
---
# Monitoring & Observability Guide
> **TL;DR**: OmniRoute ships with built-in health monitoring, provider autopilot, quota tracking, and observability hooks. This guide covers the dashboard, alerts, and troubleshooting.
**Sources:**
- `src/lib/monitoring/observability.ts` — observability snapshot
- `src/lib/monitoring/comboHealthAutopilot.ts` — combo health autopilot
- `src/lib/monitoring/providerHealthAutopilot.ts` — provider autopilot
- `src/lib/monitoring/providerHealthMatrix.ts` — provider health matrix
- `src/lib/localHealthCheck.ts` — local health check
- `src/lib/tokenHealthCheck.ts` — token refresh health
- `src/lib/proxyHealth.ts` — proxy health cache (covered in PROXY_GUIDE.md)
---
## Overview
OmniRoute has **3 layers of monitoring**:
```
┌──────────────────────────────────────────────────────────────┐
│ Layer 1: System Health (server-level) │
│ ├─ localHealthCheck.ts — DB, ports, native deps │
│ ├─ db/healthCheck.ts — integrity, FK, orphaned artifacts │
│ └─ Dashboard: /dashboard/health │
├──────────────────────────────────────────────────────────────┤
│ Layer 2: Provider Health (per-provider resilience) │
│ ├─ providerHealthAutopilot.ts — circuit breaker, cooldowns │
│ ├─ providerHealthMatrix.ts — health scores by provider/model │
│ └─ Dashboard: /dashboard/providers │
├──────────────────────────────────────────────────────────────┤
│ Layer 3: Live Observability (runtime snapshots) │
│ ├─ observability.ts — circuit breakers, sessions, quota │
│ ├─ tokenHealthCheck.ts — OAuth token refresh health │
│ └─ MCP tools: omniroute_get_health, omniroute_get_session_snapshot │
└──────────────────────────────────────────────────────────────┘
```
---
## Dashboard Pages
### `/dashboard/health` (System Health)
The top-level health dashboard shows:
| Section | What it shows |
| -------------------- | -------------------------------------------------- |
| **Server status** | Uptime, version, port, active connections |
| **Database** | Connection, integrity, WAL size, recent migrations |
| **Provider summary** | Active count, healthy count, breaker open count |
| **Quota monitors** | Active sessions, alerting, exhausted |
| **Recent errors** | Last 10 errors with stack traces |
| **Resource usage** | Memory, CPU, heap pressure indicator |
### `/dashboard/providers` (Provider Health)
Per-provider dashboard:
| Column | Description |
| ----------- | ------------------------------------- |
| Provider | Provider ID + display name |
| Health | Green/yellow/red status |
| Circuit | Open/closed/half-open state |
| Connections | Count of connections, last refresh |
| Models | Available models, health per model |
| Cost | Today's cost, 7-day trend |
| Errors | Last 24h error count, top error class |
Click a provider to see:
- Recent requests with latency breakdown
- Per-connection health scores
- Per-model lockouts
- Autopilot recommendations
### `/dashboard/quota` (Quota Tracking)
For each API key:
- Current usage vs limit (progress bar)
- Quota trend (30-day chart)
- Next reset time
- Alert history
### `/dashboard/combos` (Combo Health)
Per-combo:
- Strategy + targets
- Health per target
- Recent fallback events
- Success rate (24h, 7d, 30d)
---
## Health Check API
> **Note:** Only `GET /api/monitoring/health` is exposed as a REST endpoint. All other monitoring data (provider health, autopilot issues, quota monitors, token health, latency) is accessed via the **MCP tool** `observability_snapshot` or the **dashboard** pages — there are no dedicated REST routes for these.
### System Health
```bash
GET /api/monitoring/health
```
Response:
```json
{
"status": "healthy",
"version": "3.8.16",
"uptime": 123456,
"checks": {
"database": { "status": "pass", "latency_ms": 2 },
"writeable": { "status": "pass" },
"integrity": { "status": "pass", "result": "ok" },
"foreign_keys": { "status": "pass", "violations": 0 },
"heap_pressure": { "status": "pass", "usage_mb": 142, "threshold_mb": 512 },
"active_sessions": 12,
"providers": {
"total": 7,
"healthy": 6,
"degraded": 1,
"down": 0
}
}
}
```
### Provider Health
> **No REST endpoint.** Provider health data is available via the MCP tool `observability_snapshot` or the dashboard `/dashboard/providers` page.
### Provider Detail
> **No REST endpoint.** Per-provider detail is available via the dashboard `/dashboard/providers` page.
---
## Provider Health Autopilot
The `providerHealthAutopilot.ts` module is a **self-healing system** that:
1. Detects provider issues (circuit open, cooldowns, lockouts, quota warnings)
2. Generates **recommended actions** to resolve them
3. Optionally **auto-executes** low-risk actions
### Issue Types Detected
| Issue kind | Severity | Example condition |
| ---------------------------- | -------- | ------------------------------------- |
| `provider_circuit_open` | critical | Circuit breaker open after 5 failures |
| `provider_circuit_half_open` | warning | Circuit testing recovery |
| `connection_cooldown` | warning | Connection in cooldown after 429 |
| `stale_connection_error` | warning | Last refresh failed 30+ minutes ago |
| `terminal_connection_error` | critical | OAuth revoked, key invalid |
| `inactive_connection` | info | Connection disabled in settings |
| `model_lockout` | warning | Specific model in quarantine |
| `quota_monitor_warning` | warning | Quota at 80%+ usage |
### Action Types Generated
| Action | Risk | Description |
| ------------------------------ | ------ | ----------------------------------- |
| `clear_provider_breaker` | medium | Reset the circuit breaker to closed |
| `clear_connection_cooldown` | low | Remove cooldown from a connection |
| `clear_stale_connection_error` | low | Clear stale error flag |
| `clear_model_lockout` | low | Re-enable a quarantined model |
| `reactivate_connection` | medium | Re-enable a deactivated connection |
| `deactivate_connection` | high | Disable a problematic connection |
### API
> **No REST endpoint.** Autopilot issues are available via the MCP tool `observability_snapshot` or the dashboard. The autopilot runs internally; its behavior is configured via the settings DB (per-connection `autopilotMode` field), not environment variables — `grep -rn` for an autopilot-mode env var returns zero hits.
### Autopilot Mode
The autopilot operates in **manual mode** by default — it detects issues and generates recommended actions, but does not auto-apply them. Actions can be applied via the dashboard.
---
## Combo Health Autopilot
`comboHealthAutopilot.ts` is the **combo-specific** equivalent of the provider autopilot. It:
- Detects unhealthy combos
- Recommends target reordering
- Suggests disabling broken targets
- Auto-removes dead targets after N failures
### Combo Issue Examples
```
Combo "always-on" (priority strategy)
├─ Target 1: openai/gpt-5 (healthy)
├─ Target 2: anthropic/claude-opus-4-6 (⚠️ model lockout until 14:00)
└─ Target 3: kiro/claude-sonnet-4-5 (healthy)
Recommended action: Reorder — move kiro above anthropic until lockout expires
```
---
## Quota Monitors
`observability.ts` exposes **per-session quota monitors** for subscription providers (Claude Code, Codex, GitHub Copilot):
```ts
interface QuotaMonitorSnapshot {
sessionId: string;
provider: string;
accountId: string;
status: "starting" | "idle" | "healthy" | "warning" | "exhausted" | "error";
lastQuotaPercent: number | null; // 0-100
lastQuotaUsed: number | null;
lastQuotaTotal: number | null;
lastResetAt: string | null;
nextPollAt: string | null;
totalPolls: number;
totalAlerts: number;
consecutiveFailures: number;
}
```
### Status Meanings
| Status | When | UI action |
| ----------- | ------------------------ | --------------------------------- |
| `starting` | Initial poll in progress | Spinner |
| `idle` | No recent activity | Hidden from dashboard |
| `healthy` | Quota > 50% remaining | Green dot |
| `warning` | Quota < 50% remaining | Yellow alert |
| `exhausted` | Quota = 0% | Red block, route to next provider |
| `error` | Polling failed | Red dot, retry soon |
### API
> **No REST endpoint.** Quota monitor data is available via the MCP tool `observability_snapshot` or the dashboard.
---
## Observability Snapshot
The MCP tool `observability_snapshot` returns a **complete system snapshot** for AI agents:
```json
{
"circuitBreakers": [
{
"name": "openai",
"state": "closed",
"failureCount": 0,
"lastFailureTime": null,
"retryAfterMs": null
}
],
"sessions": [
{
"sessionId": "sess-123",
"createdAt": 1234567890,
"lastActive": 1234567999,
"requestCount": 42,
"connectionId": "conn-456",
"ageMs": 109
}
],
"quotaMonitors": {
/* see above */
},
"uptime": 12345,
"version": "3.8.16"
}
```
Agents use this to make **routing decisions** — for example, "if openai's circuit is open, route to anthropic first".
---
## Token Health Check
OAuth providers (Claude Code, GitHub Copilot, Cursor) need **periodic token refresh**. `src/lib/tokenHealthCheck.ts` runs a background scheduler:
- **Sweep tick**: every 60 seconds (sweep in `TICK_MS = 60 * 1000` at `src/lib/tokenHealthCheck.ts:30`)
- **Per-connection health check interval**: default 60 minutes (`DEFAULT_HEALTH_CHECK_INTERVAL_MIN = 60`); configurable via the settings DB
- **Pre-emptive refresh on 401**: handled by the per-connection interceptor
### Token Health Status
```ts
interface TokenHealth {
connectionId: string;
provider: string;
status: "valid" | "expiring_soon" | "expired" | "refresh_failed";
expiresAt: string;
lastRefresh: string;
nextRefresh: string;
consecutiveFailures: number;
}
```
### Configuration
Token health check configuration is handled internally by `tokenHealthCheck.ts`.
### Token Health
> **No REST endpoint.** Token health data is available via the dashboard or the MCP tool `observability_snapshot`.
---
## Alerting
### Built-in Channels
OmniRoute supports **3 alert channels**:
| Channel | Setup | Use case |
| ---------------- | ------------- | ---------------------------- |
| Dashboard banner | Always on | In-app notifications |
| Webhook | Configure URL | Slack, Discord, PagerDuty |
| Log | Default | For external log aggregation |
### Webhook Configuration
> **Note:** Webhook alerting configuration is handled via the dashboard Settings page. See the Settings UI for webhook URL, event filtering, and payload customization.
### Alert Types
| Alert | When | Default severity |
| ---------------------------- | -------------------------------- | ---------------- |
| `provider_circuit_open` | Circuit opens | critical |
| `provider_circuit_half_open` | Circuit testing recovery | info |
| `quota_warning` | Quota at 80%+ | warning |
| `quota_exhausted` | Quota at 100% | critical |
| `token_refresh_failed` | 3+ consecutive refresh failures | warning |
| `token_expired` | Token past expiry | critical |
| `combo_target_unhealthy` | Combo target in cooldown for 1h+ | warning |
| `db_integrity_warning` | FK violations > 0 | warning |
| `heap_pressure` | Heap usage > 80% of threshold | warning |
---
## Performance Metrics
### Tracked Metrics
| Metric | Type | Source |
| ----------------------- | --------- | ------------------------------- |
| `request_count` | counter | `services/usage.ts` |
| `request_latency_ms` | histogram | `services/usage.ts` |
| `tokens_consumed` | counter | `services/usage.ts` |
| `cost_usd` | counter | `services/usage.ts` |
| `provider_errors` | counter | `services/errorClassifier.ts` |
| `circuit_state_changes` | counter | `services/resilience.ts` |
| `cache_hits` | counter | `services/signatureCache.ts` |
| `compression_savings` | histogram | `services/compression/stats.ts` |
| `quota_used` | gauge | `services/quotaMonitor.ts` |
| `memory_used_mb` | gauge | `observability.ts` |
### Latency Percentiles (p50/p95/p99)
> **No REST endpoint.** Latency percentile data is available via the dashboard `/dashboard/health` page. Prometheus/OpenTelemetry export is planned for v3.9.
### Prometheus / OpenTelemetry Export (Phase 2)
Planned for v3.9: native export to Prometheus, OpenTelemetry, Datadog.
For now, scrape `/api/monitoring/health` with any HTTP-based monitoring system (Prometheus blackbox exporter, Datadog HTTP check, etc.).
---
## Alerting Recipes
### Slack
> **Note:** Webhook alerting is configured through the dashboard Settings page — there are no dedicated webhook env vars (`grep -rn` returns zero hits). See the Settings UI for webhook URL, event filtering, and payload customization.
### Discord
> Webhook alerting uses the same Settings UI flow as Slack. Discord accepts the same JSON payload shape.
### PagerDuty
> Webhook alerting uses the same Settings UI flow. PagerDuty Events API v2 routing keys are configured in the Settings UI.
### Custom Webhook (JSON)
> Any HTTP endpoint that accepts POST with JSON body will work. Configure the URL in the Settings UI.
---
## Dashboard Configuration
### Customize the Health Dashboard
Create a `~/.omniroute/dashboard.json`:
```json
{
"health": {
"sections": ["server_status", "database", "providers", "quota_monitors", "recent_errors"],
"refresh_interval_ms": 5000
}
}
```
### Pin a Provider to the Top
```json
{
"health": {
"pinned_providers": ["openai", "anthropic"]
}
}
```
---
## Troubleshooting
### "Provider says healthy but requests fail"
1. Check the **autopilot issues** — maybe a model is locked out
2. Look at **recent errors** for the specific error class
3. Try the **connection test** in the provider card
4. Check if the provider is **rate-limited at upstream** (not visible locally)
### "Quota says healthy but I see 429s"
- 429 means the provider says you've used your quota
- OmniRoute's quota tracking may be **stale** — the provider's truth is upstream
- Quota data refreshes automatically via the internal quota monitor
### "Combo is failing but all targets look healthy"
- Check **combo health** dashboard for target ordering issues
- Look at **fallback events** — maybe the combo is exhausting too quickly
- Verify the **strategy** matches your use case (priority vs round-robin vs auto)
### "Database health check is failing"
- Run `sqlite3 ~/.omniroute/storage.sqlite "PRAGMA integrity_check;"`
- If "ok" — false alarm, the health check is being too strict
- If anything else — **stop OmniRoute** and follow the [disaster recovery guide](./DATABASE_GUIDE.md#disaster-recovery)
### "Memory heap pressure is critical"
```bash
# Check current heap
node -e "console.log(process.memoryUsage())"
# Trigger manual GC (if --expose-gc)
node --expose-gc -e "global.gc(); console.log(process.memoryUsage())"
# Reduce concurrent requests (set via the dashboard Settings page, not an env var)
# There is no `MAX_CONCURRENT_REQUESTS` env var — configure it in Settings → Concurrency.
```
---
## See Also
- [USAGE_QUOTA_GUIDE.md](../guides/USAGE_QUOTA_GUIDE.md) — usage & cost tracking
- [DATABASE_GUIDE.md](./DATABASE_GUIDE.md) — DB schema + health
- [PROXY_GUIDE.md](./PROXY_GUIDE.md) — proxy health (separate cache)
- [ARCHITECTURE.md](../architecture/ARCHITECTURE.md) — system architecture
- [RESILIENCE_GUIDE.md](../architecture/RESILIENCE_GUIDE.md) — circuit breaker details
- Source: `src/lib/monitoring/` (4 files, 2121 LOC)
docs/ops/PROXY_GUIDE.md
+824
View File
@@ -0,0 +1,824 @@
---
title: "🌐 OmniRoute Proxy Guide"
version: 3.8.40
lastUpdated: 2026-06-28
---
# 🌐 OmniRoute Proxy Guide
> **Bypass geographic blocks, protect your identity, and route AI traffic through any proxy — with zero configuration complexity.**
OmniRoute includes a full-featured proxy management system that lets you route upstream AI provider traffic through HTTP, HTTPS, or SOCKS5 proxies. Whether you're in a blocked region, need IP rotation, or want stealth fingerprinting — this guide covers everything.
---
## Table of Contents
- [Why Use Proxies?](#why-use-proxies)
- [Architecture Overview](#architecture-overview)
- [4-Level Proxy System](#4-level-proxy-system)
- [Proxy Registry (CRUD)](#proxy-registry-crud)
- [1proxy Free Marketplace](#1proxy-free-proxy-marketplace)
- [Proxy Rotation](#proxy-rotation)
- [Anti-Detection & Stealth](#anti-detection--stealth)
- [Upstream Proxy Modes](#upstream-proxy-modes)
- [Dashboard UI](#dashboard-ui)
- [API Reference](#api-reference)
- [Environment Variables](#environment-variables)
- [Troubleshooting](#troubleshooting)
---
## Why Use Proxies?
Many AI providers restrict access by geographic region. Developers in **Russia, China, Iran, Cuba, Turkey**, and other countries encounter errors like:
```
unsupported_country_region_territory
```
Even outside blocked regions, proxies are useful for:
| Use Case | Description |
| --------------------- | --------------------------------------------------------------- |
| **Geographic bypass** | Access OpenAI, Anthropic, Codex, Copilot from blocked countries |
| **IP rotation** | Distribute requests across multiple IPs to avoid rate limiting |
| **Privacy** | Hide your real IP from upstream providers |
| **Compliance** | Route traffic through specific jurisdictions |
| **Testing** | Simulate requests from different regions |
---
## Architecture Overview
```
┌───────────────────────────────────────────────────────────────┐
│ OmniRoute Server │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ Proxy │ │ Proxy │ │ Proxy │ │
│ │ Registry │───▶│ Dispatcher │───▶│ Fetch (undici) │ │
│ │ (SQLite) │ │ (cached) │ │ │ │
│ └─────────────┘ └──────────────┘ └────────┬─────────┘ │
│ ▲ │ │
│ │ ▼ │
│ ┌──────┴──────┐ ┌──────────────────┐ │
│ │ 1proxy Sync │ │ Upstream │ │
│ │ (free pool) │ │ Provider API │ │
│ └─────────────┘ └──────────────────┘ │
└───────────────────────────────────────────────────────────────┘
```
### Key Components
| Component | File | Role |
| -------------------- | -------------------------------------------- | ---------------------------------------------------------- |
| **Proxy Registry** | `src/lib/db/proxies.ts` | CRUD for proxy entries + scope assignments |
| **Proxy Dispatcher** | `open-sse/utils/proxyDispatcher.ts` | Creates `undici` ProxyAgent/SOCKS dispatchers with caching |
| **Proxy Fetch** | `open-sse/utils/proxyFetch.ts` | Wraps `fetch()` with proxy dispatcher injection |
| **Settings Route** | `src/app/api/settings/proxy/route.ts` | Legacy proxy config API (GET/PUT/DELETE) |
| **Management Route** | `src/app/api/v1/management/proxies/route.ts` | Registry CRUD API (GET/POST/PATCH/DELETE) |
| **1proxy DB** | `src/lib/db/oneproxy.ts` | Free proxy marketplace persistence |
| **1proxy Sync** | `src/lib/oneproxySync.ts` | Fetches proxies from 1proxy API |
| **1proxy Rotator** | `src/lib/oneproxyRotator.ts` | Rotation strategies (quality/random/sequential) |
---
## 4-Level Proxy System
OmniRoute supports proxy configuration at **four independent scopes**, resolved in priority order:
```
Priority Resolution Order (highest → lowest):
1. 🔵 Account/Connection Proxy → per API key / OAuth connection
2. 🟡 Provider Proxy → per provider (e.g., all OpenAI traffic)
3. 🟠 Combo Proxy → per combo/routing configuration
4. 🟢 Global Proxy → all traffic, all providers
```
### How Resolution Works
When OmniRoute sends a request to an upstream provider, it calls `resolveProxyForConnectionFromRegistry()` which checks each level in order:
1. **Account-level** — Is there a proxy assigned to this specific connection ID?
2. **Provider-level** — Is there a proxy assigned to this provider (e.g., `openai`)?
3. **Global-level** — Is there a global proxy configured?
4. **No proxy** — Direct connection to the provider.
The first match wins. This means you can set a global proxy as a fallback but override it for specific providers or connections.
### What Gets Proxied
| Traffic Type | Proxied? | Notes |
| -------------------- | -------- | --------------------------------------------- |
| Chat completions | ✅ | All `/v1/chat/completions` requests |
| Embeddings | ✅ | `/v1/embeddings` |
| Image generation | ✅ | `/v1/images/generations` |
| Audio (TTS/STT) | ✅ | `/v1/audio/*` |
| OAuth token exchange | ✅ | Solves `unsupported_country_region_territory` |
| Connection tests | ✅ | "Test Connection" button uses proxy |
| Token refresh | ✅ | Background OAuth renewal |
| Model sync | ✅ | Model listing and discovery |
---
## Proxy Registry (CRUD)
The proxy registry is a SQLite table (`proxy_registry`) that stores all your proxies. Each proxy has:
| Field | Type | Description |
| ---------- | ------- | ----------------------------------- |
| `id` | UUID | Unique identifier |
| `name` | String | Human-readable label |
| `type` | String | Protocol: `http`, `https`, `socks5` |
| `host` | String | Proxy hostname or IP |
| `port` | Integer | Port number |
| `username` | String | Auth username (encrypted at rest) |
| `password` | String | Auth password (encrypted at rest) |
| `region` | String | Geographic region label |
| `notes` | String | Free-text notes |
| `status` | String | `active` or `inactive` |
| `source` | String | `manual` or `oneproxy` |
### Creating a Proxy
**Via Dashboard:**
1. Go to **Settings → Proxy**
2. Click **Add Proxy**
3. Fill in the type, host, port, and optional auth credentials
4. Save
**Via API:**
```bash
curl -X POST http://localhost:20128/api/v1/management/proxies \
-H "Content-Type: application/json" \
-d '{
"name": "US Proxy",
"type": "http",
"host": "proxy.example.com",
"port": 8080,
"username": "user",
"password": "pass",
"region": "US"
}'
```
### Updating a Proxy
```bash
curl -X PATCH http://localhost:20128/api/v1/management/proxies \
-H "Content-Type: application/json" \
-d '{
"id": "proxy-uuid-here",
"host": "new-proxy.example.com",
"port": 9090
}'
```
> **Note:** Credentials are preserved unless you explicitly send non-empty replacements. Sending empty strings for `username`/`password` will keep the stored values.
### Deleting a Proxy
```bash
# Fails if proxy is assigned to any scope
curl -X DELETE "http://localhost:20128/api/v1/management/proxies?id=proxy-uuid"
# Force delete (removes assignments too)
curl -X DELETE "http://localhost:20128/api/v1/management/proxies?id=proxy-uuid&force=1"
```
### Listing Proxies
```bash
curl "http://localhost:20128/api/v1/management/proxies?limit=50&offset=0"
```
### Assigning Proxies to Scopes
```bash
# Assign to global scope
curl -X PUT http://localhost:20128/api/settings/proxy \
-H "Content-Type: application/json" \
-d '{"level": "global", "proxy": {"type":"http","host":"proxy.example.com","port":8080}}'
# Assign to a specific provider
curl -X PUT http://localhost:20128/api/settings/proxy \
-H "Content-Type: application/json" \
-d '{"level": "provider", "id": "openai", "proxy": {"type":"socks5","host":"socks.example.com","port":1080}}'
# Assign to a specific connection/key
curl -X PUT http://localhost:20128/api/settings/proxy \
-H "Content-Type: application/json" \
-d '{"level": "key", "id": "connection-uuid", "proxy": {"type":"http","host":"key-proxy.com","port":3128}}'
```
### Resolving Effective Proxy
Check which proxy would be used for a given connection:
```bash
curl "http://localhost:20128/api/settings/proxy?resolve=connection-uuid"
```
Returns the resolved proxy with its level (`account`, `provider`, or `global`) and source.
### Bulk Assignment
Assign one proxy to multiple providers or connections at once:
```bash
curl -X POST http://localhost:20128/api/v1/management/proxies/bulk-assign \
-H "Content-Type: application/json" \
-d '{
"scope": "provider",
"scopeIds": ["openai", "anthropic", "codex"],
"proxyId": "proxy-uuid"
}'
```
### Import/Export
Proxies are included in the **Backup/Restore** system. When you export your OmniRoute configuration:
1. Go to **Dashboard → Settings → Backup**
2. Click **Export** — proxy registry and assignments are included
3. To restore, click **Import** and upload the backup file
The proxy registry also supports **upsert by host+port** — if you import a proxy that already exists (same host and port), it updates instead of creating a duplicate.
### Legacy Migration
If you configured proxies in an older version (pre-registry), OmniRoute automatically migrates them:
```
Legacy key_value store → proxy_registry + proxy_assignments
```
This happens once on first startup after upgrade. Use `migrateLegacyProxyConfigToRegistry({ force: true })` to re-run.
---
## 1proxy Free Proxy Marketplace
> 🆕 **Contributed by [@oyi77](https://github.com/oyi77)** — PR [#1847](https://github.com/diegosouzapw/OmniRoute/pull/1847) (Issue [#1788](https://github.com/diegosouzapw/OmniRoute/issues/1788))
OmniRoute integrates with the **[1proxy](https://1proxy-api.aitradepulse.com)** community platform to provide access to **hundreds of free, validated proxies** from around the world. This is perfect for users who don't have their own proxy infrastructure.
### How It Works
```
┌─────────────┐ Sync ┌─────────────────┐ Rotate ┌──────────┐
│ 1proxy API │ ────────────▶ │ proxy_registry │ ────────────▶ │ Provider │
│ (external) │ up to 500 │ source=oneproxy │ by quality │ API │
└─────────────┘ proxies └─────────────────┘ └──────────┘
```
1. **Sync** — OmniRoute fetches validated proxies from the 1proxy API
2. **Store** — Proxies are saved in the same `proxy_registry` table with `source = 'oneproxy'`
3. **Filter** — Filter by protocol, country, quality score
4. **Rotate** — Pick the best proxy using quality, random, or sequential strategies
5. **Auto-degrade** — Failed proxies get their quality score reduced; below threshold → marked inactive
### Syncing Proxies
**Via Dashboard:**
1. Go to **Settings → 1proxy** tab
2. Click **"Sync Now"**
3. View stats: total proxies, active count, average quality, by-country breakdown
**Via API:**
```bash
# Trigger sync
curl -X POST http://localhost:20128/api/settings/oneproxy \
-H "Content-Type: application/json" \
-d '{}'
# Response:
# { "success": true, "added": 127, "updated": 45, "failed": 2, "total": 172 }
```
### Filtering Proxies
```bash
# Filter by protocol
curl "http://localhost:20128/api/settings/oneproxy?protocol=socks5"
# Filter by country
curl "http://localhost:20128/api/settings/oneproxy?countryCode=US"
# Filter by minimum quality score
curl "http://localhost:20128/api/settings/oneproxy?minQuality=80"
# Combine filters
curl "http://localhost:20128/api/settings/oneproxy?protocol=http&countryCode=DE&minQuality=70"
```
### Proxy Quality Scores
Each 1proxy proxy comes with metadata:
| Field | Description |
| --------------- | -------------------------------------------- |
| `qualityScore` | 0-100 rating from 1proxy validation |
| `latencyMs` | Measured network latency |
| `anonymity` | `transparent`, `anonymous`, or `elite` |
| `googleAccess` | Whether the proxy can access Google services |
| `countryCode` | Two-letter ISO country code |
| `lastValidated` | Timestamp of last validation |
Quality scores are dynamically adjusted:
- **Failed requests** reduce the score by 10 points
- **Score drops to ≤10** → proxy is marked `inactive`
- Inactive proxies are excluded from rotation
### Rotation Strategies
```bash
# Rotate by quality (best proxy first) — default
curl -X POST http://localhost:20128/api/settings/oneproxy/rotate \
-H "Content-Type: application/json" \
-d '{"strategy": "quality"}'
# Random rotation
curl -X POST http://localhost:20128/api/settings/oneproxy/rotate \
-d '{"strategy": "random"}'
# Sequential (least recently validated first)
curl -X POST http://localhost:20128/api/settings/oneproxy/rotate \
-d '{"strategy": "sequential"}'
```
### Circuit Breaker
The 1proxy sync has a built-in circuit breaker:
- After **5 consecutive sync failures**, further sync attempts are blocked
- Reset with: `resetOneproxyCircuitBreaker()` or restart the server
- Sync status is available at `GET /api/settings/oneproxy?action=status`
### Clearing 1proxy Proxies
```bash
# Delete a single 1proxy proxy
curl -X DELETE "http://localhost:20128/api/settings/oneproxy?id=proxy-uuid"
# Clear ALL 1proxy proxies (manual proxies are untouched)
curl -X DELETE "http://localhost:20128/api/settings/oneproxy?clearAll=1"
```
---
## Anti-Detection & Stealth
OmniRoute doesn't just route traffic through a proxy — it makes the traffic look legitimate:
### TLS Fingerprint Spoofing
Uses `wreq-js` to generate browser-like TLS fingerprints, bypassing bot detection systems that flag non-browser TLS handshakes.
### CLI Fingerprint Matching
The **CLI Fingerprint Toggle** (`Settings → Security`) reorders HTTP headers and JSON body fields to match the exact signature of native CLI binaries (Claude Code, Codex, etc.). This works **on top of** the proxy:
```
Your IP (blocked) → Proxy IP (US) → Provider API
+ TLS spoof
+ CLI fingerprint
```
You get both **IP masking** and **request authenticity** simultaneously.
### Proxy IP Preservation
Color-coded badges in the dashboard show which proxy level is active:
| Badge | Level | Meaning |
| ----- | ---------- | ----------------------------------------- |
| 🟢 | Global | All traffic goes through this proxy |
| 🟡 | Provider | Only this provider's traffic is proxied |
| 🔵 | Connection | This specific key/account uses this proxy |
The badge also shows the resolved proxy IP for verification.
---
## Upstream Proxy Modes
For providers that use the CLIProxyAPI pattern, OmniRoute supports three upstream proxy modes:
| Mode | Description |
| ------------- | -------------------------------------------------- |
| `native` | OmniRoute handles proxy routing directly (default) |
| `cliproxyapi` | Delegates to an external CLIProxyAPI instance |
| `fallback` | Tries native first, falls back to CLIProxyAPI |
Configure per-provider:
```bash
curl -X PUT "http://localhost:20128/api/upstream-proxy/openai" \
-H "Content-Type: application/json" \
-d '{"mode": "native", "enabled": true}'
```
---
## Dashboard UI
### Settings → Proxy Tab
- **Global proxy** configuration (set once for all traffic)
- **Per-provider proxy** overrides
- **Per-connection proxy** assignments
- **Connection test** through configured proxy
- **Color-coded badges** showing active proxy level
### Settings → 1proxy Tab
- **Sync Now** button to fetch free proxies
- **Stats cards**: Total, Active, Avg Quality, Last Sync
- **Filters**: Protocol, Country Code, Min Quality
- **Proxy table** with host, protocol, country, quality score, latency, anonymity, Google access
- **Sync status** panel with success/failure tracking and consecutive failure count
- **Clear All** to remove all 1proxy entries
---
## API Reference
### Proxy Settings API
| Method | Endpoint | Description |
| -------- | ---------------------------------------------- | ----------------------- |
| `GET` | `/api/settings/proxy` | Get full proxy config |
| `GET` | `/api/settings/proxy?level=global` | Get global proxy |
| `GET` | `/api/settings/proxy?level=provider&id=openai` | Get provider proxy |
| `GET` | `/api/settings/proxy?resolve=connectionId` | Resolve effective proxy |
| `PUT` | `/api/settings/proxy` | Update proxy config |
| `DELETE` | `/api/settings/proxy?level=provider&id=openai` | Remove proxy at level |
### Proxy Registry API
| Method | Endpoint | Description |
| -------- | ------------------------------------------------- | --------------------- |
| `GET` | `/api/v1/management/proxies` | List all proxies |
| `GET` | `/api/v1/management/proxies?id=uuid` | Get proxy by ID |
| `GET` | `/api/v1/management/proxies?id=uuid&where_used=1` | Get proxy assignments |
| `POST` | `/api/v1/management/proxies` | Create proxy |
| `PATCH` | `/api/v1/management/proxies` | Update proxy |
| `DELETE` | `/api/v1/management/proxies?id=uuid` | Delete proxy |
| `DELETE` | `/api/v1/management/proxies?id=uuid&force=1` | Force delete |
| `POST` | `/api/v1/management/proxies/bulk-assign` | Bulk assign |
| `GET` | `/api/v1/management/proxies/assignments` | List assignments |
| `GET` | `/api/v1/management/proxies/health` | Proxy health stats |
### Tunnels API
For exposing your OmniRoute instance to the public internet (Cloudflare/ngrok/Tailscale) instead of routing outbound through a proxy, see [TUNNELS_GUIDE.md](./TUNNELS_GUIDE.md). The tunnel REST API lives under `/api/tunnels/{cloudflared,ngrok,tailscale}/*` and is orthogonal to the outbound proxy chain documented above.
### 1proxy API
| Method | Endpoint | Description |
| -------- | -------------------------------------- | ----------------------- |
| `GET` | `/api/settings/oneproxy` | List 1proxy proxies |
| `GET` | `/api/settings/oneproxy?action=stats` | Get stats + sync status |
| `GET` | `/api/settings/oneproxy?action=status` | Get sync status only |
| `POST` | `/api/settings/oneproxy` | Trigger sync |
| `POST` | `/api/settings/oneproxy/rotate` | Rotate to next proxy |
| `DELETE` | `/api/settings/oneproxy?id=uuid` | Delete one |
| `DELETE` | `/api/settings/oneproxy?clearAll=1` | Clear all |
### Upstream Proxy API
| Method | Endpoint | Description |
| -------- | --------------------------------- | ---------------------------- |
| `GET` | `/api/upstream-proxy/:providerId` | Get upstream proxy config |
| `PUT` | `/api/upstream-proxy/:providerId` | Set upstream proxy mode |
| `DELETE` | `/api/upstream-proxy/:providerId` | Remove upstream proxy config |
---
## Environment Variables
| Variable | Default | Description |
| -------------------------------- | ------------------------------------- | -------------------------------------------------------------- |
| `ENABLE_SOCKS5_PROXY` | `true` | Enable SOCKS5 proxy support (default `true` in `.env.example`) |
| `ONEPROXY_ENABLED` | `true` | Enable 1proxy integration |
| `ONEPROXY_API_URL` | `https://1proxy-api.aitradepulse.com` | 1proxy API endpoint |
| `ONEPROXY_MAX_PROXIES` | `500` | Maximum proxies to sync |
| `ONEPROXY_MIN_QUALITY_THRESHOLD` | `50` | Minimum quality score to import |
---
## Troubleshooting
### "SOCKS5 proxy is disabled"
Set `ENABLE_SOCKS5_PROXY=true` in your `.env` file and restart.
### "socket hang up" errors through proxy
This is normal with cheap proxies that drop idle connections. OmniRoute already handles this by:
- Disabling keep-alive on proxy connections (`keepAliveTimeout: 1`)
- Disabling pipelining (`pipelining: 0`)
- Caching dispatchers to avoid repeated handshakes
If it persists, try a different proxy or use the 1proxy rotation feature.
### "unsupported_country_region_territory" during OAuth
Make sure the proxy is configured **before** starting the OAuth flow. OmniRoute routes OAuth token exchange through the configured proxy. Set a global or provider-level proxy first, then connect.
### Proxy not being used
Check the resolution order:
1. Verify with `GET /api/settings/proxy?resolve=your-connection-id`
2. Check if the proxy `status` is `active` (not `inactive`)
3. Ensure the proxy assignment scope matches your connection
### 1proxy sync failing
Check the sync status:
```bash
curl "http://localhost:20128/api/settings/oneproxy?action=status"
```
If `consecutiveFailures >= 5`, the circuit breaker has tripped. Restart the server to reset, or wait for manual reset.
---
## Database Schema
### `proxy_registry` Table
```sql
CREATE TABLE proxy_registry (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
type TEXT NOT NULL DEFAULT 'http',
host TEXT NOT NULL,
port INTEGER NOT NULL,
username TEXT DEFAULT '',
password TEXT DEFAULT '',
region TEXT,
notes TEXT,
status TEXT DEFAULT 'active',
source TEXT NOT NULL DEFAULT 'manual', -- 'manual' or 'oneproxy'
quality_score INTEGER, -- 0-100 (1proxy only)
latency_ms INTEGER, -- milliseconds (1proxy only)
anonymity TEXT, -- transparent/anonymous/elite
google_access INTEGER DEFAULT 0, -- can access Google? (1proxy)
last_validated TEXT, -- ISO timestamp (1proxy)
country_code TEXT, -- ISO 2-letter code (1proxy)
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
```
### `proxy_assignments` Table
```sql
CREATE TABLE proxy_assignments (
id INTEGER PRIMARY KEY AUTOINCREMENT,
proxy_id TEXT NOT NULL REFERENCES proxy_registry(id),
scope TEXT NOT NULL, -- 'global', 'provider', 'account', 'combo'
scope_id TEXT, -- provider ID, connection ID, or combo ID
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
UNIQUE(scope, scope_id)
);
```
---
## Proxy Health Checking (v3.8.16+)
OmniRoute's **proxy fast-fail** mechanism (`src/lib/proxyHealth.ts`) detects dead proxies in <2s via a quick TCP connection check, then **caches the result** to avoid per-request overhead.
### How It Works
```
Request ──▶ ProxyHealthCache.get(url)
│
├─ Cache hit + fresh? ──▶ return cached status
│
└─ Cache miss / stale? ──▶ TCP connect to host:port
(timeout: FAST_FAIL_TIMEOUT_MS)
──▶ cache for HEALTH_CACHE_TTL_MS
──▶ return result
```
Without this, a dead proxy would block every request for the full `PROXY_TIMEOUT_MS` (default 30s) before failing.
### Tunable Environment Variables
| Variable | Default | Purpose |
| ---------------------------- | ------- | --------------------------------------- |
| `PROXY_FAST_FAIL_TIMEOUT_MS` | `2000` | TCP connection timeout per health check |
| `PROXY_HEALTH_CACHE_TTL_MS` | `30000` | How long a health result is cached |
**Recommended values:**
| Scenario | Fast-fail timeout | Cache TTL | Reasoning |
| --------------------------- | ----------------- | --------- | --------------------------------------------------------------- |
| High-throughput API gateway | 1500ms | 60000ms | Aggressive fail-fast, longer cache to reduce checks |
| Geo-distributed nodes | 3000ms | 15000ms | Slower networks need more time; shorter cache for fast failover |
| Dev / testing | 1000ms | 10000ms | Quick iteration on local proxies |
| Stealth / anti-detection | 2500ms | 45000ms | Avoid rapid probing that could trigger rate limits |
### Inspecting Proxy Health
```ts
import { getAllProxyHealthStatuses, invalidateProxyHealth } from "omniroute/proxyHealth";
const statuses = getAllProxyHealthStatuses();
for (const s of statuses) {
console.log(`${s.proxyUrl} → healthy=${s.healthy}, stale=${s.stale}`);
}
// Force re-check a specific proxy
invalidateProxyHealth("http://user:pass@1.2.3.4:8080");
```
The `stale` flag is `true` when the cache entry has exceeded `HEALTH_CACHE_TTL_MS` and the next request will trigger a fresh check.
### Per-Proxy Type Defaults
The health check uses sensible defaults based on the URL scheme:
| Scheme | Default port |
| -------------------------- | ------------ |
| `http://` | 8080 |
| `https://` | 443 |
| `socks5://` / `socks5h://` | 1080 |
Custom ports in the URL (`http://host:9999`) always take precedence over the scheme default.
---
## Proxy Analytics & Observability
OmniRoute tracks per-proxy usage to help operators diagnose routing patterns, latency spikes, and recurring failures.
### What's Tracked
For every request through a configured proxy, OmniRoute records:
| Metric | Description |
| ------------ | ----------------------------------------------- |
| `proxy_url` | Full proxy URL (with auth credentials masked) |
| `provider` | Upstream provider ID (openai, anthropic, etc.) |
| `latency_ms` | Total round-trip time including proxy handshake |
| `connect_ms` | TCP connect time only |
| `status` | HTTP status code from upstream |
| `error` | Error class if request failed |
| `timestamp` | ISO 8601 UTC |
### Accessing the Data
```bash
# Recent proxy events
curl -H "Authorization: Bearer $OMNIROUTE_KEY" \
"http://localhost:20128/api/usage/proxy-logs?limit=100"
```
The real endpoint is `/api/usage/proxy-logs` (see `src/app/api/usage/proxy-logs/route.ts`). This endpoint supports:
- `GET /api/usage/proxy-logs` — retrieve proxy logs
- `DELETE /api/usage/proxy-logs` — clear all proxy logs
Aggregate stats can be queried directly from the `proxy_logs` table via SQL if needed. The dashboard UI may offer aggregate views.
### Common Patterns
**Detect a flapping proxy** (alternates between success/failure):
```sql
SELECT proxy_url,
COUNT(*) AS total,
SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) AS errors,
ROUND(100.0 * SUM(CASE WHEN status >= 500 THEN 1 ELSE 0 END) / COUNT(*), 1) AS error_pct
FROM proxy_logs
WHERE timestamp > datetime('now', '-1 hour')
GROUP BY proxy_url
HAVING error_pct > 5
ORDER BY error_pct DESC;
```
**Find slow proxies** (p95 latency > 2s):
```sql
WITH ranked AS (
SELECT proxy_url, latency_ms,
PERCENT_RANK() OVER (PARTITION BY proxy_url ORDER BY latency_ms) AS pct
FROM proxy_logs
WHERE timestamp > datetime('now', '-24 hour')
)
SELECT proxy_url, latency_ms
FROM ranked
WHERE pct >= 0.95
ORDER BY latency_ms DESC;
```
---
## Rotation Strategy Decision Tree
When multiple proxies are assigned to a scope, OmniRoute uses a **rotation strategy** to pick which one to use for each request. The strategy is configured at the scope level (global, per-provider, per-account, per-combo).
### Available Strategies
| Strategy | When to use | Trade-off |
| ------------------- | ------------------------------------- | ----------------------------------------------------- |
| `quality` (default) | Production with mixed-quality proxies | Favors high-rated proxies; may starve low-rated ones |
| `random` | Load distribution, privacy | Even distribution; ignores quality signals |
| `sequential` | Debugging, deterministic testing | Cycles through proxies in order; easy to reason about |
### Decision Tree
```
Do you have quality scores for your proxies?
│
┌───────────┴───────────┐
│ │
YES NO
│ │
Are all proxies │
roughly equal │
in quality? │
│ │
┌────┴────┐ │
│ │ │
YES NO Use
│ │ `random`
│ │ (even spread
│ │ builds quality
│ │ data over time)
│ │
│ Use `quality`
│ (best for
│ mixed quality)
│
Use `random`
(spread load
evenly)
```
### Configuring Rotation Strategy
```ts
import { rotateOneproxyProxy } from "omniroute/oneproxyRotator";
// In a one-off script
const proxy = await rotateOneproxyProxy({ strategy: "quality" });
if (proxy) {
console.log(`Selected: ${proxy.host}:${proxy.port}, quality=${proxy.qualityScore}`);
}
```
### Resetting Sequential Index
When using `sequential` strategy, the internal index accumulates. To reset:
```ts
import { resetSequentialIndex } from "omniroute/oneproxyRotator";
resetSequentialIndex();
```
Useful when:
- Restarting a load test
- Recovering from a proxy outage (so you don't cycle through dead ones first)
- Manually rebalancing after adding new proxies
### Marking a Proxy as Failed
When a proxy consistently fails, mark it manually so the rotator will skip it:
```ts
import { failOneproxyProxy } from "omniroute/oneproxyRotator";
const removed = await failOneproxyProxy("1.2.3.4", 8080);
if (removed) {
console.log("Proxy marked as failed; rotator will skip it");
}
```
The proxy is **not deleted** — it's marked unhealthy and won't be selected until the next successful health check (via `proxyHealth.ts`) or manual reset.
---
> 📖 **Related documentation:**
>
> - [User Guide](../guides/USER_GUIDE.md) — General setup and configuration
> - [API Reference](../reference/API_REFERENCE.md) — Full API documentation
> - [Environment Config](../reference/ENVIRONMENT.md) — All environment variables
docs/ops/QUALITY_GATE_PLAYBOOK.md
+311
View File
@@ -0,0 +1,311 @@
---
title: "Quality Gate Playbook"
---
# Quality-Gate System — Critical Assessment, Catalog and Replication Playbook
> **What this document is.** A critical assessment of OmniRoute's quality-gate system,
> compared to industry best practices, **plus** a comprehensive catalog of all quality
> checkpoints and a **tool-agnostic replication plan** to apply the same system to
> any project. Generated on 2026-06-16 from the real repository state (not from memory).
>
> Benchmarks: OWASP DSOMM · OpenSSF Scorecard · SLSA · SonarQube "Clean as You Code" ·
> Quality-Ratchet pattern · DORA 2024 · OWASP LLM Top 10 (2025) · mutation-testing best practices.
---
## Part 1 — Verdict and Maturity Classification
**Overall grade: A− / "Advanced". Top ~5–10% of projects.** The system independently
implements several patterns that the industry explicitly names — which is the strongest
alignment signal (we didn't copy a checklist; we converged on the right practices).
| Reference framework | Where we stand | Grade |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
| **OWASP DSOMM** (5 levels, 5 dimensions) | Solid Level 3, reaching 4 in _Test Intensity_ and _Static Depth_. Most orgs sit at 1–2. | **L3→L4** |
| **OpenSSF Scorecard** (18 checks) | We pass CI-Tests, Code-Review, Dependency-Update-Tool, Fuzzing, SAST, Signed-Releases (provenance), Token-Permissions, Vulnerabilities, Dangerous-Workflow. **Gaps:** Branch-Protection on `main` OFF; some actions not pinned. | **~7–8/10** |
| **SLSA** (4 levels) | `npm publish --provenance` + `id-token: write` + GitHub-hosted build = **L2**, approaching L3. Missing hardened/hermetic builder for L3+. | **L2→L3** |
| **SonarQube "Clean as You Code"** | Identical philosophy: the ratchet gates _non-regression_ (new code doesn't worsen the metric). **Divergence:** Sonar recommends **few** conditions; we have ~46 gates (fatigue risk). | **Aligned, with caveat** |
| **Quality-Ratchet pattern** | Reference implementation: ratchet + `dedicatedGate` + `tightenSlack` + `--require-tighten` + graceful-skip. More sophisticated than most public examples. | **Exemplary** |
| **DORA 2024** | Very strong on _stability_ axis. Risk: heavy gates can cost _lead time_ — mitigated by fast-gates split, but with coverage gap (see Part 2). | **Strong (stability)** |
| **OWASP LLM Top 10 (2025)** | We cover risk #1 (prompt-injection) with runtime guard + promptfoo (eval) + garak (red-team). Standard industry tools. | **Covered** |
| **Mutation testing** | Stryker nightly, thresholds 70/50, 8 critical modules. Industry consensus (60% existing / 80% new, nightly) — **we beat it**. **Gap:** score is not yet a ratchet. | **Almost there** |
---
## Part 2 — Critical Assessment (strengths + honest weaknesses)
### Strengths (what's above average)
1. **Multi-metric ratchet engine.** The heart of the system. 24 metrics in `quality-baseline.json`
- 4 dedicated baselines, each with direction (`up`/`down`), tolerance (`eps`), slack
(`tightenSlack`) and `dedicatedGate` flag. Things that get fixed **stay** fixed — it's the
antidote to codebase entropy.
2. **Defense-in-depth for supply-chain.** SAST (CodeQL/Sonar) + secrets (gitleaks with
`useDefault`) + SCA (osv/npm-audit/Trivy/Dependabot) + licenses + lockfile + SBOM + SLSA
provenance + Scorecard + workflow hardening (zizmor). Few codebases have this complete stack.
3. **Antidotes against Goodhart's Law.** Coverage as a target is a classic anti-pattern
("when the measure becomes the target, it ceases to be a good measure"). We have the
counterweights: **mutation testing** (measures whether the test catches the bug, not just
whether it executes the line), **`check-test-masking`** (blocks weakening asserts to pass),
**per-module coverage floors** (forces testing HIGH-risk code, not just the easy parts) and
**`check-pr-evidence`** (Hard Rule #18).
4. **Anti-hallucination / consistency gates.** A rare and valuable category: `check-known-symbols`,
`check-fetch-targets`, `check-openapi-routes`, `check-docs-symbols` ensure that docs, specs and
string dispatches point to living symbols. Catches "rot" that lint/test don't.
5. **Advisory→blocking lifecycle.** New gates enter as advisory (don't block merges while
maturing), then become blocking at cycle end. Reduces friction without losing the ceiling.
6. **Graceful skip when infra is missing.** Scanners (`--ratchet`) exit `exit 0` if the binary/network
fails — missing infra never blocks a legitimate PR. Mature engineering.
7. **Codified culture.** Hard Rules + `trust-but-verify` + stale-allowlist + evidence-gate
turn discipline into automated verification.
### Honest weaknesses (real gaps)
1. **🔴 The fast-gates split is a structural hole.** `quality.yml` (PR→`release/**`) runs **only
filesystem gates** — no typecheck, no tests, no build, no coverage. A typecheck/test regression
passes in a release PR and only blows up on the forward-merge to `main`. The motivation
(speed) is valid, but the gate should be where the merge happens (shift-left). **Largest
pending structural fix.**
2. **🟠 Gate sprawl/fatigue risk.** ~46 gates + 25 jobs is A LOT. Sonar itself warns:
too many conditions cause "gate fatigue" and priority debates, with risk of a gate being
ignored. DORA warns that heavy gates cost lead-time. We mitigate with advisory tiers and
non-absolute ratchets, but a **periodic ROI review per gate** is missing (some micro-gates for
doc-sync are consolidatable).
3. **🟠 Mutation score is not yet a ratchet.** The strongest antidote against coverage-gaming is
**advisory**. It's the highest-value pending item (and already 90% built).
4. **🟡 Advisories that should block (with the right scope).** `osv` (vulnCount) and `oasdiff` are
advisory despite frozen baselines. osv-advisory makes sense (a new CVE on an old dep would block
an unrelated PR) — but there's a middle ground (block only CRITICAL+fixable, as we did with
Trivy). oasdiff advisory means a contract-breaking change can pass.
5. **🟡 Runtime security is nightly-only.** schemathesis/garak/promptfoo/chaos/k6 run at night.
Correct decision (slow, need a live server), but a PR can introduce an injection-guard regression
that only gets caught the following night.
6. **🟡 Branch-protection on `main` is OFF.** `BRANCH_LOCK_TOKEN` locks _release_ branches, but
`main` itself is unprotected. Scorecard/DSOMM ding. Owner action required.
7. **🟡 CodeQL default-setup; semgrep not codified.** default-setup works (0 alerts), but a
committed `codeql.yml` gives more control; semgrep runs via an external cloud platform, not
versioned in the repo.
---
## Part 3 — Complete Catalog of Quality Checkpoints (portable)
The 12 categories below are the "quality system" in reusable form. Each lists the
**objective** (what to protect), the **tools we use** and the **tool-agnostic equivalent**
to replicate on any stack.
### 1. Style & formatting (deterministic, fast)
- **OmniRoute:** Prettier + ESLint via lint-staged (pre-commit), 2-spaces/double-quotes/100col.
- **Generic:** one auto-fixable formatter + one linter, running in pre-commit on staged files.
### 2. Types
- **OmniRoute:** `typecheck:core` (blocking) + `typecheck:noimplicit:core` (advisory) + `type-coverage` ratchet 92.17% + per-file any-budget.
- **Generic:** strict typecheck in CI + ratcheted type-coverage metric + per-file `any`/escape-hatch budget.
### 3. Tests (intensity)
- **OmniRoute:** 2 non-overlapping runners (Node native + vitest), 8 shards, global coverage 60/60/60/60 + ratchet ~76% + **8 per-module floors for critical modules** + nightly property tests + **mutation testing** nightly.
- **Generic:** test runner(s) + **absolute** coverage floor (anti-zero) + coverage **ratchet** (anti-regression) + **per-module floors for high-risk code** (anti-Goodhart) + property-based for pure logic + **mutation testing** nightly as the real measure of test quality.
### 4. Test policy (anti-gaming)
- **OmniRoute:** `pr-test-policy` (prod code requires a test), `check-test-masking` (blocks weakened asserts), `pr-evidence` (success claim requires evidence block), `test-discovery` (every test collected by a runner).
- **Generic:** "new code ⇒ new test" gate + assert-removed/tautology detector + evidence requirement (TDD or living test) + guarantee that no test is orphaned outside the globs.
### 5. Complexity & code health (ratchets)
- **OmniRoute:** ESLint-warnings (3769↓), jscpd duplication (5.72%↓), cyclomatic+max-lines complexity (1800↓), cognitive complexity sonarjs (753↓), dead-code/unused-exports knip (339↓), per-file file-size (frozen, shrink-only), circular-deps (custom Tarjan, blocking).
- **Generic:** ratchet every health metric (warnings, duplication, cyclomatic **and** cognitive complexity, dead code, file size, import cycles). Direction always "don't regress".
### 6. Static security (SAST + secrets)
- **OmniRoute:** CodeQL (ratchet alerts = 0), gitleaks (`[extend] useDefault=true` — critical!), SonarQube, custom security rules (public-creds, error-helper, route-guard-membership, route-validation).
- **Generic:** SAST (CodeQL/Sonar/semgrep) with alert ratchet + secrets scanner with **inherited default ruleset** (custom config that overrides the default = blind) + project-specific Hard Rule security gates.
### 7. Supply-chain (dependencies)
- **OmniRoute:** osv-scanner + npm-audit + Trivy + Dependabot (SCA), license-checker (SPDX allowlist), lockfile-lint (HTTPS+sha512+registry), `check-deps` anti-slopsquatting (allowlist + age ≥72h).
- **Generic:** multi-source SCA + license allowlist + lockfile integrity check + dependency allowlist with age/typosquatting check + grouped update bot.
### 8. Supply-chain (build & release)
- **OmniRoute:** SBOM (CycloneDX + syft), SLSA provenance (`--provenance`), OpenSSF Scorecard (weekly), workflow hardening (zizmor: artipacked→`persist-credentials:false`, cache-poisoning, token-permissions).
- **Generic:** generate SBOM on publish + signed provenance (SLSA L2+) + scheduled Scorecard + harden all workflows (minimum-privilege tokens, no persisted credentials on non-pusher checkout, actions pinned by SHA).
### 9. Contracts & API
- **OmniRoute:** oasdiff (breaking-change OpenAPI), schemathesis (contract fuzz nightly), openapi-coverage (% documented routes, ratchet 38.3%), openapi-security-tiers (spec vs route-guard).
- **Generic:** breaking-change contract diff (oasdiff/buf) + property-based fuzz against the spec (schemathesis) + ratcheted documentation coverage + spec↔code consistency.
### 10. Docs & i18n (anti-rot)
- **OmniRoute:** docs-sync (mirrored versions), docs-counts-sync (numbers in docs vs code), env-doc-sync, doc-links, fabricated-docs, cli-i18n, i18n-ui-coverage (`--threshold=65` + ratchet 80.1%).
- **Generic:** sync versions/counts/env-vars between docs and code (gate, not trust) + validate internal links + ratcheted i18n coverage.
### 11. Anti-hallucination / consistency (the rare category)
- **OmniRoute:** known-symbols (string dispatch ⇒ living symbol), provider-consistency, fetch-targets (client fetch ⇒ real route), docs-symbols, db-rules (Hard Rules #2/#5), migration-numbering.
- **Generic:** for every "duplicated source of truth" (registry, string dispatch, cross-layer references), a gate that proves both sides match. Catches the rot that typecheck/test don't.
### 12. Resilience & domain (product-specific)
- **OmniRoute:** chaos (fault-injection), heap-growth (leak), k6 (soak), promptfoo+garak (LLM red-team OWASP LLM Top 10), the 3 resilience laws (circuit-breaker/cooldown/lockout).
- **Generic:** identify the failure modes of **your** domain and have a gate (even if nightly) for each. For AI apps: injection red-team. For distributed systems: chaos + leak + soak.
---
## Part 4 — Replication Plan for Any Project
Build in **phases**, each delivering value on its own. Don't try all 12 categories at once —
that causes exactly the gate fatigue Part 2 warns about. Every new gate enters **advisory** and
becomes **blocking** when stable.
### The reusable centerpiece: the "anatomy of a ratchet gate"
The entire system revolves around this 3-file pattern. Copy it first:
1. **`baseline.json`** — the frozen metric value + `direction` (`up`/`down`) + `eps` (anti-flake) + `tightenSlack` + `dedicatedGate`.
2. **`collect-metrics.<ext>`** — runs the tool, extracts the number, writes `metrics.json`.
3. **`check-ratchet.<ext>`** — compares `metrics.json` vs `baseline.json`; `exit 1` **only** if regressed beyond `eps`; `exit 0` (graceful skip) if the tool/infra was missing; with `--require-tighten`, `exit 1` if it **improved** without updating the baseline (locks in the gain).
With this in place, **every** new metric (coverage, complexity, warnings, SAST alerts, bundle size, mutation score…) is just one line in the baseline.
### Phase 0 — Foundation (week 1)
CI exists; formatter + linter + typecheck + 1 test runner + **absolute** coverage floor
(e.g., 60%). Pre-commit runs fast auto-fixable checks. _Output: no PR breaks the basics._
### Phase 1 — The ratchet engine (week 2) — **the foundation of everything**
Implement the 3 files above. Freeze baselines for: warnings, coverage, complexity, duplication,
dead code, file size. _Output: the codebase can only improve from here._
### Phase 2 — Static depth (week 3)
SAST (CodeQL/Sonar/semgrep) with alert ratchet; secrets scanner (**inherit the default ruleset**);
SCA (osv/Dependabot) + license allowlist + lockfile-lint. _Output: known vulnerabilities and
leaked secrets don't pass._
### Phase 3 — Build supply-chain (week 4)
SBOM on publish + signed provenance (SLSA L2) + scheduled Scorecard + workflow hardening
(zizmor: minimum tokens, no persisted credentials, pinned actions). _Output: traceable and
tamper-proof releases._
### Phase 4 — Test intensity (week 5–6)
2nd runner if useful; **per-module coverage floors for critical modules** (anti-Goodhart);
property-based for pure logic; **mutation testing nightly** → when the 1st score arrives, make
`mutationScore` a ratchet. _Output: coverage stops being a vanity metric; tests provably catch bugs._
### Phase 5 — Contract & dynamic (week 7)
If there's a public API: oasdiff (breaking-change, **blocking**) + schemathesis (nightly fuzz).
DAST/red-team nightly as appropriate for the domain. _Output: contracts don't break silently._
### Phase 6 — Anti-hallucination & domain (week 8)
One consistency gate for each "duplicated truth" in the project. Domain-specific failure-mode
gates (for AI: injection red-team). _Output: structural rot and domain failures have a safety net._
### Phase 7 — Governance (ongoing)
- Advisory→blocking cycle for every new gate.
- `stale-allowlist`: every suppression has a justification + issue; obsolete suppression is caught.
- `evidence-gate`: success claim in a PR requires proof (test or living test).
- **Quarterly ROI review per gate** (kill/defund those that don't pay back — fights fatigue).
- Promote your project's Hard Rules into executable gates.
### Cross-cutting principles (non-negotiable)
- **Ratchet, not absolute.** Gate _non-regression_, not a fixed number (except anti-zero floors).
- **Absolute floor + ratchet together.** The floor prevents collapse; the ratchet prevents slow erosion.
- **Anti-Goodhart by design.** Every target metric needs a counterweight (coverage ⇒ mutation + anti-masking; per-module floors to force testing the hard code).
- **Graceful skip.** Missing infra never blocks; only real regression blocks.
- **`dedicatedGate` for expensive metrics.** Metrics that need an external binary get their own script (with skip), outside the synchronous central ratchet.
- **Gate where the merge happens.** Don't leave a gap between the fast gate and the actual merge (the lesson from the fast-gates split).
- **Few blocking gates, well-chosen.** Sonar/DORA: too many conditions = fatigue. Prefer advisory + ratchet over a wall of blocking gates.
---
## Part 5 — Recommended improvements (prioritized, compatible)
**P0 — highest ROI, almost ready**
1. **Mutation score ratchet** (after the 1st nightly Stryker produces values). Key antidote against coverage-Goodhart; ~90% done.
2. **Close the fast-gates hole** — add typecheck + impacted tests to `quality.yml` (PR→release).
3. **Branch-protection on `main`** (owner setting) — boosts Scorecard, closes the DSOMM gap.
**P1 — valuable** 4. **osv/oasdiff → blocking with the right scope** — osv only CRITICAL+fixable (two-step like Trivy); oasdiff blocks breaking-changes. 5. **`require-tighten` → blocking** (end of cycle) — locks in metric gains. 6. **ROI/timing review per-gate** in `ci-summary` — find and prune slow/low-value gates.
**P2 — diminishing returns** 7. **SLSA L3** — hermetic/reproducible builder (GitHub SLSA generator) if you want to move up from L2. 8. **Committed CodeQL config + versioned semgrep** — more control/reproducibility. 9. **Per-PR DAST smoke** — fast subset of schemathesis/promptfoo on highest-risk endpoints (not just nightly). 10. **Flakiness dashboard + DORA metrics** — ensure gates aren't eroding speed.
---
## Part 6 — Concrete release lessons (gates to add in Phase 9)
> This section records real incidents from release closures where a gate **was missing**,
> with concrete evidence and the proposed gate. Each item is a candidate for Part 5.
### Lesson v3.8.27 (2026-06-17) — the "fast-gates hole" lets deterministic regressions reach release day
**What happened.** During the v3.8.27 `/generate-release`, the release PR (`release/v3.8.27` → `main`)
was the **first** execution of the full `ci.yml` matrix in the integrated cycle. Result: 12 failures
at once — **3 deterministic tests** + ~9 flakes/env. None were live product regressions, but
all went unnoticed because cycle PRs enter `release/**` via the **Fast QG
(`quality.yml`)**, which does NOT run the full unit suite, nor `pr-test-policy` (test-masking), nor the
full integration suite, nor schema parity checking. The 3 deterministic ones:
1. **Test outdated by UI change** — `permissions modal switch buttons declare button type`:
#4034 added a 4th switch (a11y `type="button"` maintained); the test's `=== 3` count became
outdated. Static analysis should have caught this in the #4034 PR.
2. **Test outdated by packaging change** — `findMissingArtifactPaths ... root runtime files`:
`dist/http-method-guard.cjs` became a legitimate required-path; the test's expected list became
outdated.
3. **Lossy modularization divergence (most serious)** — `settings schemas accept ... unprefixed
toggle`: the **modularized** `updateSettingsSchema` (`schemas/settings.ts`, created by #3988) diverged
from the canonical one (`settingsSchemas.ts`): **45 fields vs 85 — 40 dropped + 6 divergent (qdrant\*)**. It was
**dead-code** (runtime uses the canonical one), so no live impact, but only a hand-written parity
test caught it. #4030 restored 16 analogous drops from #3988/#3993, but this one slipped through.
**Proposed gates (Phase 9):**
- **G1 — Actually close the fast-gates hole (extends P0 #2).** In `quality.yml` (PR→`release/**`),
beyond typecheck + impacted tests, run **`pr-test-policy` (test-masking) + the full deterministic
unit suite** (or at least the static/parity files, which are fast and non-flaky).
This way, outdated tests and assert removal are caught in the PR that introduces them — not on
release day. Keep integration/e2e out (slow/flaky), but the deterministic layer CANNOT stay only
in PR→main.
- **G2 — Modularization parity gate (NEW, not covered today).** A check that, for each symbol
re-exported by a modularized barrel (`src/shared/validation/schemas/*`, `providerRegistry`
modules, etc.), compares the **shape** (`z.object` keys, registry entries) against the canonical
source and **fails on divergence** (dropped/extra field). Would have caught the 40-field drop from
#3988 in that very PR. Generalizes the hand-written parity tests (which only exist where someone
remembered to write them). Cheap: imports both and diffs `Object.keys(shape)`.
- **G3 — Deterministic flake triage (support).** LiveWS-startup and the integration-combo/breaker
tests fail due to server timeout/cascade in CI (env), not logic. Mark these as
`known-flaky` (quarantined with issue) so the release-PR red is **only real signals**, not noise
masking deterministic regressions in the middle.
**Principle:** _the gate has to run where the merge happens_ (already in "Cross-cutting principles"). The
v3.8.27 incident shows this also applies to the **deterministic test layer**, not just lint/typecheck —
otherwise the debt of outdated tests + lossy modularization only appears in PR→main, in batch, at
the worst moment.
---
## Sources (industry best practices)
- OWASP DevSecOps Maturity Model (DSOMM) — https://dsomm.owasp.org/about
- OpenSSF Scorecard / SLSA — https://openssf.org · https://slsa.dev
- SonarQube "Clean as You Code" — https://docs.sonarsource.com/sonarqube-server/latest/user-guide/clean-as-you-code
- Quality Ratchets (LeadDev) — https://leaddev.com/software-quality/introducing-quality-ratchets-tool-managing-complex-systems
- Continuous Code Improvement Using Ratcheting (Greiner) — https://robertgreiner.com/continuous-code-improvement-using-ratcheting/
- DORA 2024 State of DevOps — https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report
- Mutation testing best practices (Stryker) — https://stryker-mutator.io
- Coverage as anti-pattern (Goodhart) — https://www.industriallogic.com/blog/code-coverage-complications/
- OWASP Top 10 for LLM Applications (2025) — https://owasp.org/www-project-top-10-for-large-language-model-applications/
- Contract testing (oasdiff/schemathesis) — https://www.oasdiff.com · https://schemathesis.readthedocs.io
docs/ops/RELEASE_CHECKLIST.md
+326
View File
@@ -0,0 +1,326 @@
---
title: "Release Checklist"
version: 3.8.40
lastUpdated: 2026-06-28
---
# Release Checklist
> **Last updated:** 2026-06-28 — v3.8.40
> Streamlined release flow that leverages Claude Code skills for automation.
>
> **Keep the queue/branch green between releases:** see [RELEASE_GREEN.md](./RELEASE_GREEN.md)
> (`/green-prs` family + `npm run check:release-green` + `/babysit` + nightly). Running
> this periodically — and especially **before** this checklist — makes the release PR start green.
## TL;DR
```bash
# 1. Bump version + generate CHANGELOG (skill)
/version-bump-cc patch # or minor/major
# 2. Run quality gate locally
npm run check # lint + tests
npm run test:coverage # full coverage gate (60/60/60/60)
# 3. Build & smoke
npm run build
npm run test:e2e # optional but recommended
# 4. Generate release (skill)
/generate-release-cc
# 5. Deploy (skill)
/deploy-vps-both-cc # or akamai-cc / local-cc
# 6. Capture release evidences (skill)
/capture-release-evidences-cc
```
## Detailed Checklist
### Pre-release
- [ ] All PRs targeted to this release are merged to `release/vX.Y.0`
- [ ] All open Linear/issue items for this version are closed or pushed to next milestone
- [ ] CI green on `release/vX.Y.0` branch
- [ ] No `TODO(release)` markers in code: `grep -r "TODO(release)" src/ open-sse/`
- [ ] Docker base image up to date (currently `node:24.15.0-trixie-slim`)
### Version & Changelog
- [ ] Run `/version-bump-cc <patch|minor|major>` (Claude Code skill)
- Bumps `package.json`, `electron/package.json`
- Regenerates `CHANGELOG.md` from git commits since last tag
- Updates README.md badges
- [ ] Manually review CHANGELOG.md and clean up commit messages if needed
- [ ] Ensure the latest semver section in `CHANGELOG.md` equals `package.json` version
- [ ] Keep `## [Unreleased]` as the first changelog section for upcoming work
- [ ] Update `docs/openapi.yaml` → `info.version` must equal `package.json` version
### Code Quality
- [ ] `npm run lint` — 0 errors (warnings are pre-existing)
- [ ] `npm run typecheck:core` — clean
- [ ] `npm run typecheck:noimplicit:core` — clean (strict)
- [ ] `npm run check:cycles` — no circular deps
- [ ] `npm run check:any-budget:t11` — within budget
- [ ] `npm run check:route-validation:t06` — clean
- [ ] `npm run check:node-runtime` — supported runtime floor met (`>=22.22.2 <23`, `>=24.0.0 <27`, per `SUPPORTED_NODE_RANGE` in `src/shared/utils/nodeRuntimeSupport.ts`; aligned with `package.json` `engines`)
### Testing
- [ ] `npm run test:unit` — pass
- [ ] `npm run test:vitest` — pass (MCP server, autoCombo, cache)
- [ ] `npm run test:coverage` — gate 60/60/60/60 satisfied (statements/lines/functions/branches)
- [ ] `npm run test:integration` — pass (if changes touch DB / handlers)
- [ ] `npm run test:combo:matrix` — pass (combo strategy matrix: proves all 17 routing strategies' selection decisions deterministically; run when touching combo routing, strategy resolution, or fallback logic)
- [ ] `RUN_COMBO_LIVE=1 npm run test:combo:live` — **optional/manual** (gated real-upstream smoke; sources a read-only DB snapshot from VPS `root@192.168.0.15`; hits real providers, costs credits; never runs in CI; skips cleanly without the gate)
- [ ] `npm run test:combo:live:vps` — **optional/manual** (Phase-3 VPS live smoke: 7 HTTP scenarios against the live `.15` server via plain Node ESM; requires `ssh root@192.168.0.15`; creates/deletes only `__live_test__*` combos; hits real providers; never runs in CI)
- [ ] `npm run test:e2e` — pass (UI changes)
- [ ] `npm run test:protocols:e2e` — pass (MCP/A2A changes)
- [ ] `npm run test:ecosystem` — pass
### Hooks (Husky validated)
Husky hooks live in `.husky/` and run automatically on git operations.
- **pre-commit:** `npx lint-staged + node scripts/check/check-docs-sync.mjs + npm run check:any-budget:t11`
- **pre-push:** fast deterministic gates — `npm run check:any-budget:t11 && npm run check:tracked-artifacts` (activated 2026-06-13). Intentionally excludes `test:unit` (slow; covered by the CI `test-unit` job).
- Run `npm run test:unit` manually before pushing release branches.
If a hook fails: fix the underlying issue, don't bypass with `--no-verify`.
### Conventional Commits
All release-bound commits must follow `type(scope): subject` format.
**Valid types:** `feat`, `fix`, `refactor`, `docs`, `test`, `chore`, `perf`, `style`, `ci`
**Valid scopes:** `db`, `sse`, `oauth`, `dashboard`, `api`, `cli`, `docker`, `ci`, `mcp`, `a2a`, `memory`, `skills`, `cloud-agent`, `guardrails`, `compression`, `auto-combo`, `resilience`, `providers`, `executors`, `translator`, `domain`, `authz`
Breaking changes: add `BREAKING CHANGE:` footer or `!` after the scope (e.g. `feat(api)!: drop /v0`).
### Documentation
- [ ] `npm run check:docs-sync` passes (auto-run by pre-commit)
- [ ] `npm run check:docs-all` passes (umbrella: docs-sync + docs-counts + env-doc-sync + deprecated-versions + doc-links)
- [ ] `npm run check:env-doc-sync` exits 0 — code ↔ `.env.example` ↔ `docs/reference/ENVIRONMENT.md` env contract is intact
- [ ] `npm run check:doc-links` exits 0 — no broken internal markdown references after restructuring
- [ ] `docs/architecture/ARCHITECTURE.md` reviewed for storage/runtime drift
- [ ] `docs/guides/TROUBLESHOOTING.md` reviewed for env var and operational drift
- [ ] If `.env.example` changed: `docs/reference/ENVIRONMENT.md` updated
- [ ] If new feature has a UI: `docs/guides/USER_GUIDE.md` mentions it
- [ ] If new feature has API: `docs/reference/API_REFERENCE.md` + `docs/openapi.yaml` updated
- [ ] If new feature is a module: dedicated `docs/<MODULE>.md` exists
- [ ] If breaking change: `docs/guides/TROUBLESHOOTING.md` has migration note
### i18n
- [ ] `npm run i18n:check` exits 0 — translation state (`.i18n-state.json`) in sync with source docs (no drifted sources in strict mode; warn-mode advisory is acceptable for last-minute doc touch-ups, but should be 0 before tagging)
- [ ] `npm run i18n:check-ui-coverage` exits 0 — every UI locale at or above the 80% coverage floor
- [ ] `npm run i18n:sync-ui:dry` reports 0 missing keys across all 42 locales
- [ ] If source English docs changed, run `npm run i18n:run` (requires `OMNIROUTE_TRANSLATION_API_KEY` in `.env`) before tagging
- [ ] Translation contributions can be deferred to next release if minor (track in CHANGELOG)
### Database Migrations
- [ ] If `src/lib/db/migrations/` has new files:
- [ ] Each migration is idempotent (`CREATE TABLE IF NOT EXISTS`, etc.)
- [ ] Migrations wrapped in transactions
- [ ] Numbered correctly (no gaps in sequence)
- [ ] Test on fresh install: delete `~/.omniroute/omniroute.db` and run `npm run dev`
- [ ] Test on existing install: backup DB, run migration, verify schema
- [ ] WAL files (`-wal`, `-shm`) handled correctly if migration rewrites tables
### Provider Catalog (Zod-validated)
- [ ] `src/shared/constants/providers.ts` Zod schema valid at load time
- [ ] All providers have required fields (`id`, `label`, `kind`, etc.)
- [ ] `freeNote` provided for new free providers
- [ ] OAuth providers have `oauthConfig` registered in `src/lib/oauth/constants/oauth.ts`
- [ ] If new provider added: corresponding executor in `open-sse/executors/`
- [ ] If non-OpenAI format: translator in `open-sse/translator/`
- [ ] Models registered in `open-sse/config/providerRegistry.ts`
- [ ] Unit tests in `tests/unit/` cover provider classification and routing
### Desktop (Electron)
If `electron/` changed:
- [ ] `npm run electron:smoke:packaged` passes
- [ ] Builds tested for at least one of `:win`, `:mac`, `:linux`
- [ ] Code signing certs not expired (if signing)
- [ ] `electron/package.json` version matches root `package.json`
- [ ] Auto-update channel pointer updated if releasing to `stable`
### Build Layout
The repository uses three distinct output directories — never mix them up:
| Directory | Purpose | Tracked? |
| --------- | -------------------------------------------------------- | --------------- |
| `src/` | Application source (TypeScript / TSX) | Yes |
| `.build/` | Build intermediates — `next build` output (`distDir`) | No (gitignored) |
| `dist/` | Shippable npm bundle — assembled by `assembleStandalone` | No (gitignored) |
> **Operator note:** the remote VPS image directory remains `/usr/lib/node_modules/omniroute/app/`.
> Only the **in-repo** build output moved (`app/` → `dist/`). The deploy skills rsync
> `dist/` contents into the remote `app/` dir — no VPS path changes required.
**Single-build flow:**
```
npm run build:release
└─ rm -rf .build dist (clean)
└─ next build → .build/next/ (intermediates)
└─ assembleStandalone (copies standalone + static + public + natives → dist/)
└─ writes dist/BUILD_SHA (HEAD sentinel)
```
Do NOT run `npm run build` followed by a separate `npm run build:cli` for deploy — use
`npm run build:release` which does a clean rebuild + sentinel in one command.
### Artifact Validation
- [ ] `npm run build:release` succeeds and `dist/BUILD_SHA` == `git rev-parse --short HEAD`
- [ ] `npm run check:pack-artifact` clean — no `app.__qa_backup`, `scripts/scratch`, `package-lock.json`, or other local residue
- [ ] `dist/server.js` exists after build
### Tagging & Release
- [ ] Run `/generate-release-cc` (Claude Code skill):
- Creates tag `vX.Y.Z`
- Pushes tag and branch
- Opens GitHub Release with changelog body
- Attaches Electron installers (if built)
- [ ] Or manually:
```bash
git tag -a vX.Y.Z -m "Release vX.Y.Z"
git push origin vX.Y.Z
gh release create vX.Y.Z --notes-from-tag
```
### Deploy
Deploy skills use the light rsync flow — no `npm pack`, no `npm i -g`:
- [ ] Use deploy skill that matches target:
- `/deploy-vps-local-cc` — local VPS (192.168.0.15)
- `/deploy-vps-akamai-cc` — Akamai VPS (69.164.221.35)
- `/deploy-vps-both-cc` — both
- [ ] Before deploying, confirm `dist/BUILD_SHA` == `git rev-parse --short HEAD`
- [ ] Build must run where `node_modules` is real (main checkout or `npm ci`'d worktree — NOT a symlinked worktree)
- [ ] Smoke test deployed instance:
- Open `/dashboard/health` → check version string matches release
- Run a `/v1/chat/completions` request against a known provider
- Verify `/api/monitoring/health` returns `CLOSED` circuit breakers
- Confirm MCP transports respond (`/mcp` HTTP, `/mcp-sse` SSE)
### Post-release
- [ ] Run `/capture-release-evidences-cc` (Claude Code skill)
- Captures WebP screenshots/recordings of new features
- Attaches to release notes / blog post
- [ ] Update GitHub Discussions / Discord with release announcement
- [ ] Open milestone for next version
- [ ] If critical: pin discussion or post in `news.json` for in-app banner
## Embedded Services smoke (v3.8.4+)
Before shipping any release that includes embedded services changes, verify:
### Fresh-DB boot (catches migration collisions — added after v3.8.4 hotfix)
- [ ] `DATA_DIR=$(mktemp -d) npm start &` — wait 10 s for boot
- [ ] `curl -s http://127.0.0.1:20128/api/services/9router/status | jq '.tool'` returns `"9router"` (NOT 404, NOT 500). Confirms migration `071_services.sql` applied + row seeded.
- [ ] `sqlite3 $DATA_DIR/storage.sqlite "PRAGMA table_info(version_manager);" | grep -E "provider_expose|logs_buffer_path|last_sync_at"` returns 3 rows.
- [ ] `sqlite3 $DATA_DIR/storage.sqlite "PRAGMA table_info(webhooks);" | grep -E "kind|metadata_encrypted"` returns 2 rows (validates `070_webhooks_kind_metadata.sql` applied).
- [ ] `node --import tsx/esm --test tests/unit/db/no-migration-collisions.test.ts` passes — guards against future collisions.
### 9Router
- [ ] `POST /api/services/9router/install` returns 200 with `installedVersion` in under 2 min
- [ ] `POST /api/services/9router/start` returns 200 and `state: "running"` in under 30 s
- [ ] `GET /api/services/9router/status` reports `health: "healthy"`
- [ ] `POST /v1/chat/completions` with `"model": "9router/auto/..."` returns 200 (end-to-end routing through 9Router)
- [ ] `GET /dashboard/providers/services/9router/embed/dashboard` renders the 9Router native UI inside the proxy (no direct `127.0.0.1:port` iframe)
- [ ] `POST /api/services/9router/rotate-key` returns `{ keyRotated: true }` and service restarts cleanly
- [ ] `POST /api/services/9router/stop` returns 200 and `state: "stopped"`
- [ ] `GET /api/services/9router/logs?tail=50` returns SSE stream with `snapshot` event containing recent lines
- [ ] Install in environment without `npm` in PATH returns 500 with a friendly (non-stack-trace) error message
### CLIProxyAPI
- [ ] `POST /api/services/cliproxy/install` returns 200 in under 2 min
- [ ] `POST /api/services/cliproxy/start` returns 200 and `state: "running"` in under 30 s
- [ ] `GET /api/services/cliproxy/status` reports `health: "healthy"`
- [ ] `POST /api/services/cliproxy/stop` returns 200 and `state: "stopped"`
- [ ] `GET /api/services/cliproxy/logs?tail=50` returns SSE stream
### Security regression
- [ ] `curl -H "X-Forwarded-For: 1.2.3.4" http://localhost:20128/api/services/9router/start` returns `403 LOCAL_ONLY`
- [ ] `curl -H "X-Forwarded-For: 1.2.3.4" http://localhost:20128/api/services/cliproxy/start` returns `403 LOCAL_ONLY`
- [ ] Error responses from `/api/services/*` do not contain `err.stack` or absolute file paths
## v3.8.0+ checks
Before shipping any v3.8.x release, verify these additional items:
- [ ] `omniroute --tray` boots on macOS (systray2 installed into `~/.omniroute/runtime/`)
- [ ] `omniroute --tray` boots on Linux (requires DISPLAY; graceful error if not set)
- [ ] `omniroute --tray` boots on Windows (PowerShell NotifyIcon, no extra binaries)
- [ ] `omniroute config tray enable` creates autostart entry; disable removes it
- [ ] `npm install -g omniroute@<this-version>` runs postinstall without fatal exit
- [ ] Update path keeps optional deps: `omniroute update --apply` and the auto-updater
run `npm install -g … --include=optional` so `optionalDependencies` (better-sqlite3,
keytar, tls-client, and the llmlingua SLM stack: `@atjsh/llmlingua-2`,
`@huggingface/transformers@3.5.2`, `@tensorflow/tfjs`, `js-tiktoken`) survive an update.
`@huggingface/transformers` stays optional so its `onnxruntime-node` CUDA provider postinstall
cannot abort installation on CUDA 11 hosts. The ultra `modelPath` SLM tier also needs the
tinybert model, auto-downloaded to `${DATA_DIR}/models/llmlingua` on first use. Postinstall
(`scripts/build/colocateOptionals.mjs`) then co-locates the SLM optional closure into
`dist/node_modules` so the worker resolves a SINGLE `@huggingface/transformers` 3.5.2
optional instance — the standalone trace bundles only transformers, not the dynamically-imported
optionals, so without this the worker would load llmlingua-2 against the root's transformers
and the SLM tier would silently fail-open.
- [ ] `omniroute status` works with no `.env` (CLI token path, loopback only)
- [ ] `curl http://localhost:20128/api/shutdown` returns 401 (always-protected route)
- [ ] `curl -H "host: evil.com" http://localhost:20128/api/mcp/sse` returns 401 (loopback guard)
- [ ] SQLite runtime resolves to `bundled` on first run (bundled binary valid for platform)
- [ ] SQLite runtime falls back to `runtime` when `node_modules/better-sqlite3` is deleted
- [ ] Smart MCP filter compresses real `playwright-mcp browser_snapshot` output (≥50% reduction)
- [ ] All 10 `skills/omniroute*/SKILL.md` files are publicly fetchable via raw GitHub URL
- [ ] Onboarding wizard shows "How It Works" tier tour step on fresh setup
- [ ] Home dashboard tier coverage widget shows configured/active counts
---
## Rollback
If release has critical issue:
1. `gh release edit vX.Y.Z --prerelease` (marks as not latest)
2. `git tag -d vX.Y.Z && git push --delete origin vX.Y.Z` (only if not yet adopted by users)
3. Or: hotfix on `release/vX.Y.0` → patch release `vX.Y.(Z+1)`
4. Communicate in GitHub Discussions and Discord immediately
## Hard Rules
- Never commit directly to `main`
- Never use `git push --force` to `main` or `release/*` branches
- Never skip Husky hooks (`--no-verify`)
- Never commit secrets, credentials, or `.env` files
- Coverage must stay ≥60/60/60/60 (statements/lines/functions/branches)
- Always include or update tests when changing production code in `src/`, `open-sse/`, `electron/`, or `bin/`
## Automated Sync Check
Run the docs sync guard locally before opening a PR:
```bash
npm run check:docs-sync
```
CI also runs this check in `.github/workflows/ci.yml` (lint job).
docs/ops/RELEASE_GREEN.md
+89
View File
@@ -0,0 +1,89 @@
---
title: "Release-Green — keeping the queue and release branch green"
---
# Release-Green: keeping the queue and release branch green
## The problem this solves
The **full gate** (`.github/workflows/ci.yml` — unit shards, vitest, ratchets,
`package-artifact`, SonarQube, E2E) runs **only on the release PR** (PR → `main`). PRs targeting
`release/**` receive only the **fast-gates** (`quality.yml`: TIA-impacted tests + typecheck +
lint). Consequence: reds accumulate silently on the release branch and **explode in layers
of ~40 min** at release time, one at a time.
The "release-green family" exists to **anticipate** those reds — validate the equivalent of the full
gate **locally / outside of release**, at any time, so the release PR is already
green on its first CI run.
> **Non-negotiable principle:** none of this blocks the contributor. We do not add a required
> check that fails their PR. The **drift** (ratchets) is for the maintainer to rebaseline at release —
> never a contributor concern. No piece **closes** a PR (credit theft) nor
> **weakens** a test to pass.
## The family (4 pieces) — and how each runs independently
| Piece | What it is | When to run | Scope |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------- |
| **`/green-prs`** (Solution A) | On-demand scan by the maintainer of the **queue of open PRs** | **Independently, periodically** — and especially **before** a `/generate-release` | Entire PR queue → `release/**` |
| **`/validate-release-green`** (Solution C — `npm run check:release-green`) | Validation engine: reproduces the full gate against a branch OR a merge candidate | Independently, at any time | A specific branch or a merge-PR |
| **`/babysit <PR#>`** | Drives **live CI** of **one** PR to green | Independently, per PR | A single PR |
| **`nightly-release-green.yml`** (Solution D) | Automated nightly workflow; opens issue on HARD red | Automatic (cron) | The active release branch |
**Short answer to "is this only for releases?":** **no.** `/green-prs` was designed to
run **periodically, between releases**. Running independently is the normal use — release is just
the moment when running it yields the most value.
## Solution C — `npm run check:release-green` (the engine)
Reproduces release-equivalent validation against the current working tree and classifies each red:
- **HARD** (typecheck, lint errors, unit, vitest, db-rules, public-creds, optional
`package-artifact`) → **real defect**; `exit 1`. Fixed on the source branch (TDD, Rule #18).
- **DRIFT** (eslint **warnings**, cognitive-complexity, file-size) → ratchet drift accumulated in
the cycle, **not the contributor's fault**; it is only reported and **rebaselined by the maintainer at
release**. Drift **never** changes the exit code — so it never blocks anyone.
```bash
npm run check:release-green # current branch (working tree)
node scripts/quality/validate-release-green.mjs --json # structured output
node scripts/quality/validate-release-green.mjs --quick # skips unit+vitest (drift+typecheck+lint only)
node scripts/quality/validate-release-green.mjs --with-build # includes package-artifact (slow)
```
Diagnoses and **reports** only (no auto-fix). The fix-to-green orchestration lives in
`/green-prs` and `/review-prs`.
## Solution A — `/green-prs` (the queue scan)
Procedure (summary — see the `green-prs` skill for details):
1. **Inventory** the queue of open PRs against the active release branch.
2. **Triage** each PR (viable / reject-worthy / needs-author) — reject/needs-author are
**reported, not closed** (the author decides).
3. For each viable PR, in an **isolated worktree** (Rule #19), bring the PR to the release tip and run
`npm run check:release-green`:
- **HARD** → fix **on the contributor's branch** via co-authorship (preserves the author's "Merged" status),
re-run until all HARDs are cleared.
- **DRIFT** → leave it; it will be rebaselined at release.
4. **Report** a PR × (verdict, HARD reds, fixed?, DRIFT, release-green now?) table.
Can **prepare** the queue without merging; only merges when explicitly requested — and never closes a PR.
## Recommended cadence
- Run **`/green-prs` periodically** (e.g., weekly) and **always before a
`/generate-release`**.
- Keep **`nightly-release-green.yml`** (Solution D) as a continuous signal: when it opens a
HARD red issue, it is time for a scan.
- Use **`/validate-release-green`** ad-hoc to check a branch or a specific merge candidate.
- Use **`/babysit <PR#>`** when a specific PR needs to be driven to green on live CI.
## Relationship to release
- `/generate-release` calls validation in **Phase 0 (pre-flight)**: rebaselines DRIFT and fixes
HARD before opening the release PR.
- `/review-prs` uses the release-green gate at the merge decision step (green-before-merge).
The goal of all pieces is the same: **a green release PR on the first CI run**, instead of surfing
reds in 40-minute layers on release day.
docs/ops/SQLITE_RUNTIME.md
+82
View File
@@ -0,0 +1,82 @@
---
title: "SQLite Runtime Resolution"
---
# SQLite Runtime Resolution
OmniRoute resolves its SQLite driver at startup through a 5-step fallback chain:
1. **Bundled `better-sqlite3`** (via `dependencies` in `package.json`)
— fastest, native binary, installed by `npm install` when build tools are present.
2. **Runtime-installed `better-sqlite3`** (in `~/.omniroute/runtime/`)
— installed lazily on first run **OR** by `scripts/build/postinstall.mjs → scripts/postinstall.mjs`.
Validates native `.node` magic bytes (ELF / Mach-O / PE) before loading
to guard against corrupt or wrong-platform binaries.
3. **`node:sqlite`** (Node ≥22.5 stdlib) — no native build needed; used when
both better-sqlite3 paths fail. Limited feature set.
4. **`sql.js`** (WASM) — final fallback. Works everywhere but is slower
and writes data on an interval rather than synchronously.
## Why this complexity?
- **Windows EBUSY**: `npm install -g omniroute@latest` can fail if the previous
version's `better_sqlite3.node` is locked by a running process. The runtime
install in `~/.omniroute/runtime/` sidesteps the global npm cache.
- **No build tools**: Some environments (corporate Windows without VS Build
Tools, minimal Docker images) cannot compile `better-sqlite3`. The runtime
installer resolves a pre-built binary from the npm registry; the fallback
drivers ensure OmniRoute still boots even if that fails.
- **Air-gapped systems**: If the npm registry is unreachable, `node:sqlite`
or `sql.js` guarantee baseline functionality.
## Magic-byte validation
Before loading a runtime-installed `.node` file, OmniRoute reads the first 8
bytes and matches against known platform magics:
| Platform | Bytes (hex) | Label |
| --------------------- | ------------- | ----------- |
| Linux | `7F 45 4C 46` | `elf` |
| macOS 64-bit BE | `FE ED FA CF` | `macho` |
| macOS 64-bit LE | `CF FA ED FE` | `macho-le` |
| macOS fat (universal) | `CA FE BA BE` | `macho-fat` |
| Windows | `4D 5A` (MZ) | `pe` |
A mismatched magic → file is ignored, fallback continues to the next step.
## Checking the active driver
```typescript
import { getDriverInfo } from "@/lib/db/core";
const info = getDriverInfo();
// { source: "bundled" | "runtime" | "runtime-installed-now" | "node-sqlite" | "sql-js",
// kind: "better-sqlite3" | "node-sqlite" | "sql-js" }
```
## Manual control
```bash
# Skip postinstall warm-up (for fast CI installs)
OMNIROUTE_SKIP_POSTINSTALL=1 npm install -g omniroute
# Force-reinstall runtime better-sqlite3
rm -rf ~/.omniroute/runtime
omniroute # will reinstall on next start
# Check what driver is active
omniroute config db-info # (if CLI command exists)
```
## Reference
Implementation:
- `bin/cli/runtime/magicBytes.mjs` — binary magic-byte validation helpers
- `bin/cli/runtime/sqliteRuntime.mjs` — 5-step runtime resolver + lazy installer
- `bin/cli/runtime/index.mjs` — startup orchestrator (`warmUpRuntimes()`)
- `scripts/postinstall.mjs` — npm post-install hook (non-fatal warm-up)
- `src/lib/db/core.ts` — `ensureDbInitialized()` / `getDriverInfo()` exports
docs/ops/TUNNELS_GUIDE.md
+292
View File
@@ -0,0 +1,292 @@
---
title: "Tunnels Guide"
version: 3.8.40
lastUpdated: 2026-06-28
---
# Tunnels Guide
> **Source of truth:** `src/lib/{cloudflaredTunnel,ngrokTunnel,tailscaleTunnel}.ts`, `src/app/api/tunnels/`
> **Last updated:** 2026-06-28 — v3.8.40
OmniRoute can expose its local server (`http://localhost:20128`) to the public
internet via three tunnel backends. This is useful for:
- OAuth callbacks from cloud providers (Antigravity, Gemini, Cursor) that need a
publicly reachable redirect URL.
- Sharing your local instance with teammates without deploying a VM.
- Mobile, remote, or cross-network testing.
All three backends are managed in-process — OmniRoute starts/stops the underlying
binary or SDK from the dashboard or REST API. No reverse-proxy or systemd setup
is required.
## Backends at a glance
| Backend | Persistence | Cost | Setup |
| --------------------------- | ------------------------------------------------------ | ----------------- | ----------------------------------------------- |
| **Cloudflare Quick Tunnel** | Ephemeral (URL changes each restart) | Free | Zero — auto-installs `cloudflared` |
| **ngrok** | Stable while a paid plan or fixed domain is configured | Free tier + paid | Requires ngrok account + authtoken |
| **Tailscale Funnel** | Stable per node within your tailnet | Free for personal | Requires Tailscale install + login + Funnel ACL |
The implementations live in `src/lib/cloudflaredTunnel.ts`,
`src/lib/ngrokTunnel.ts`, and `src/lib/tailscaleTunnel.ts`. All three return a
common-shaped `status` object with `phase`, `running`, `publicUrl`, `apiUrl`,
`targetUrl`, and `lastError` fields, so the dashboard can render them uniformly.
## 1. Cloudflare Tunnel (Quick Tunnel)
`src/lib/cloudflaredTunnel.ts` runs `cloudflared tunnel --url
http://localhost:<apiPort>` as a child process and parses the assigned
`*.trycloudflare.com` URL from stdout.
Key behaviors:
- **Auto-install.** On first use, OmniRoute downloads the latest `cloudflared`
binary from the official GitHub releases (managed install lives under
`DATA_DIR/cloudflared/`). SHA256 of the downloaded asset is verified against the
release manifest before execution.
- **Quick-tunnel only.** The current implementation runs only the
`--url`-style quick tunnel. Named/persistent tunnels (`cloudflared tunnel
login` + `cloudflared tunnel route dns ...`) are not orchestrated by
OmniRoute. URLs are ephemeral and will change every restart.
- **Process supervision.** The cloudflared PID and resolved URL are persisted to
`cloudflared-state.json` so the dashboard can resume status across reloads.
### Enable / disable via REST
The endpoint uses an `{action: "enable" | "disable"}` body, not separate
`start`/`stop` paths. Management auth (admin session or admin API key) is
required.
```bash
# Enable
curl -X POST http://localhost:20128/api/tunnels/cloudflared \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=..." \
-d '{"action":"enable"}'
# Status
curl http://localhost:20128/api/tunnels/cloudflared \
-H "Cookie: auth_token=..."
# Disable
curl -X POST http://localhost:20128/api/tunnels/cloudflared \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=..." \
-d '{"action":"disable"}'
```
Or via dashboard: **Settings → Tunnels → Cloudflare**.
### Optional env vars
| Variable | Purpose |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `CLOUDFLARED_BIN` | Override the binary path. If set and valid, OmniRoute uses it instead of downloading. |
| `CLOUDFLARED_PROTOCOL` / `TUNNEL_TRANSPORT_PROTOCOL` | Transport protocol (default `http2`). |
## 2. ngrok
`src/lib/ngrokTunnel.ts` uses the **`@ngrok/ngrok` SDK** (in-process, no CLI
subprocess). The native module is imported lazily on first start so platforms
without prebuilt binaries do not break the app at boot.
### Prerequisites
1. Sign up at <https://ngrok.com>.
2. Copy your authtoken from the ngrok dashboard.
3. Provide it either via:
- `.env`: `NGROK_AUTHTOKEN=<token>`, or
- Dashboard: **Settings → Tunnels → ngrok**, or
- REST body (one-shot): `{"action":"enable","authToken":"<token>"}`.
If neither is configured, status returns `phase: "needs_auth"`.
### Enable / disable via REST
```bash
# Enable (uses NGROK_AUTHTOKEN from env)
curl -X POST http://localhost:20128/api/tunnels/ngrok \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=..." \
-d '{"action":"enable"}'
# Enable with inline token
curl -X POST http://localhost:20128/api/tunnels/ngrok \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=..." \
-d '{"action":"enable","authToken":"2abc..."}'
# Status
curl http://localhost:20128/api/tunnels/ngrok \
-H "Cookie: auth_token=..."
# Disable
curl -X POST http://localhost:20128/api/tunnels/ngrok \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=..." \
-d '{"action":"disable"}'
```
The response includes the assigned `publicUrl` (e.g.
`https://abcd-1234.ngrok-free.app`). Custom domains, regions, and policy rules
must be configured in the ngrok dashboard — OmniRoute itself only forwards the
local target URL to the SDK.
## 3. Tailscale Funnel
`src/lib/tailscaleTunnel.ts` orchestrates the system `tailscale` CLI to expose
the local API port via **Funnel** (Tailscale's public-internet egress for serve).
It supports the full lifecycle: install, login, daemon start, enable, disable.
The implementation invokes `tailscale funnel --bg <port>` (background mode). The
public URL has the shape `https://<machine>.<tailnet>.ts.net/`.
### Prerequisites
1. Install Tailscale (or let OmniRoute do it — see `install` endpoint below).
2. Sign in (`tailscale login` or via OmniRoute's `login` endpoint).
3. Enable Funnel for your tailnet in the Tailscale admin console:
<https://login.tailscale.com/admin/settings/features>.
On Linux and macOS the daemon (`tailscaled`) requires `sudo` to control. The
POST endpoints accept an optional `sudoPassword` field which is forwarded to
OmniRoute's MITM password cache (`getCachedPassword` / `setCachedPassword`) for
the duration of the call. Windows uses the default service install at
`C:\Program Files\Tailscale\tailscale.exe`.
### REST endpoints
Tailscale has a richer surface than the other backends because installation,
login, daemon, and tunnel are separate concerns.
| Endpoint | Method | Purpose |
| ------------------------------------- | ------ | --------------------------------------------------------------- |
| `/api/tunnels/tailscale` | `GET` | Aggregated tunnel status (`phase`, `tunnelUrl`, `apiUrl`, etc.) |
| `/api/tunnels/tailscale/check` | `GET` | Lower-level check: installed? logged in? daemon running? |
| `/api/tunnels/tailscale/install` | `POST` | Install Tailscale (SSE-streamed progress events) — Linux/macOS |
| `/api/tunnels/tailscale/start-daemon` | `POST` | Start `tailscaled` on Linux/macOS |
| `/api/tunnels/tailscale/login` | `POST` | Begin login flow; returns `authUrl` to open in a browser |
| `/api/tunnels/tailscale/enable` | `POST` | Start the Funnel for the API port |
| `/api/tunnels/tailscale/disable` | `POST` | Stop the Funnel |
All Tailscale endpoints require management auth (see `routeUtils.ts ::
requireTailscaleAuth`).
Example enable:
```bash
curl -X POST http://localhost:20128/api/tunnels/tailscale/enable \
-H "Content-Type: application/json" \
-H "Cookie: auth_token=..." \
-d '{"sudoPassword":"<linux-pwd>","port":20128}'
```
If Funnel is not enabled in the admin console, the response includes
`funnelNotEnabled: true` plus an `enableUrl` to open in a browser.
### Optional env vars
| Variable | Purpose |
| --------------- | ------------------------------------ |
| `TAILSCALE_BIN` | Override the `tailscale` binary path |
## Endpoint summary
| Endpoint | Method | Body | Auth |
| ------------------------------------- | ------ | ----------------------------------- | ---------- |
| `/api/tunnels/cloudflared` | `GET` | — | management |
| `/api/tunnels/cloudflared` | `POST` | `{action: "enable" \| "disable"}` | management |
| `/api/tunnels/ngrok` | `GET` | — | management |
| `/api/tunnels/ngrok` | `POST` | `{action, authToken?}` | management |
| `/api/tunnels/tailscale` | `GET` | — | management |
| `/api/tunnels/tailscale/check` | `GET` | — | management |
| `/api/tunnels/tailscale/install` | `POST` | `{sudoPassword?}` (SSE) | management |
| `/api/tunnels/tailscale/start-daemon` | `POST` | `{sudoPassword?}` | management |
| `/api/tunnels/tailscale/login` | `POST` | `{hostname?}` | management |
| `/api/tunnels/tailscale/enable` | `POST` | `{sudoPassword?, hostname?, port?}` | management |
| `/api/tunnels/tailscale/disable` | `POST` | `{sudoPassword?}` | management |
There is no central `/api/settings/tunnels` endpoint — each backend is
independent.
## OAuth callback considerations
When you expose OmniRoute through a tunnel, the dashboard and OAuth flows must
build callback URLs against the **public** hostname, not `localhost`. Otherwise
the OAuth provider redirects the user back to a URL its servers cannot reach,
and the handshake fails.
Dashboard edits and settings saves do not require pinning the tunnel hostname in
`NEXT_PUBLIC_BASE_URL`. The authenticated dashboard sends same-origin unsafe
requests with a session-bound CSRF token, so ephemeral Cloudflare Quick Tunnel
hosts can still be used for normal UI management after logging in.
Set:
```bash
NEXT_PUBLIC_BASE_URL=https://<your-tunnel-host>
```
and restart OmniRoute before initiating OAuth. For ephemeral Cloudflare Quick
Tunnels the URL changes after every restart, so prefer ngrok with a reserved
domain or Tailscale Funnel for production OAuth use.
## Health and monitoring
The dashboard surfaces tunnel state under **Settings → Tunnels**:
- Active backend(s) and current `phase` (`stopped`, `starting`, `running`,
`needs_auth`, `error`).
- The current public URL and the derived API URL (`<publicUrl>/v1`).
- The local target URL the tunnel is forwarding to.
- Last error message, if any.
For programmatic monitoring poll the per-backend `GET` endpoints. Running more
than one backend simultaneously is allowed; OmniRoute will track each
independently.
## Troubleshooting
### "cloudflared binary not found"
OmniRoute attempts to auto-install on first use. If the install is blocked
(restricted network, no GitHub access), download `cloudflared` manually from
<https://github.com/cloudflare/cloudflared/releases> and set
`CLOUDFLARED_BIN=/path/to/cloudflared`.
### "ngrok: authtoken required"
`phase: "needs_auth"` means no authtoken was found. Set `NGROK_AUTHTOKEN` in
`.env`, configure it via the dashboard, or pass `authToken` in the enable POST
body.
### "tailscale: funnel not enabled"
When the enable response includes `funnelNotEnabled: true`, Funnel is disabled
for your tailnet. Open the returned `enableUrl` (or the admin console feature
page) and toggle Funnel on.
### Tunnel URL changes break OAuth
Use ngrok with a reserved domain or Tailscale Funnel (both stable per-node).
Cloudflare Quick Tunnels are ephemeral by design and not recommended for
long-lived OAuth callbacks.
### Permission denied on Linux/macOS for Tailscale
`tailscaled` needs root. Provide `sudoPassword` to the relevant POST endpoint,
or run the daemon yourself (`sudo systemctl start tailscaled`).
## See also
- [PROXY_GUIDE.md](./PROXY_GUIDE.md) — outbound proxy (1proxy, SOCKS5, HTTP) for
egress traffic.
- [ENVIRONMENT.md](../reference/ENVIRONMENT.md) — full list of env vars including
`NEXT_PUBLIC_BASE_URL`.
- [FLY_IO_DEPLOYMENT_GUIDE.md](./FLY_IO_DEPLOYMENT_GUIDE.md),
[DOCKER_GUIDE.md](../guides/DOCKER_GUIDE.md) — alternatives to tunneling for stable
public hosting.
- Source: `src/lib/{cloudflaredTunnel,ngrokTunnel,tailscaleTunnel}.ts`,
`src/app/api/tunnels/`.
docs/ops/VM_DEPLOYMENT_GUIDE.md
+424
View File
@@ -0,0 +1,424 @@
---
title: "OmniRoute — Deployment Guide on VM with Cloudflare"
version: 3.8.40
lastUpdated: 2026-06-28
---
# OmniRoute — Deployment Guide on VM with Cloudflare
🌐 **Languages:** 🇺🇸 [English](./VM_DEPLOYMENT_GUIDE.md) | 🇧🇷 [Português (Brasil)](../i18n/pt-BR/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇪🇸 [Español](../i18n/es/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇫🇷 [Français](../i18n/fr/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇮🇹 [Italiano](../i18n/it/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇷🇺 [Русский](../i18n/ru/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇨🇳 [中文 (简体)](../i18n/zh-CN/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇩🇪 [Deutsch](../i18n/de/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇮🇳 [हिन्दी](../i18n/in/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇹🇭 [ไทย](../i18n/th/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇺🇦 [Українська](../i18n/uk-UA/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇸🇦 [العربية](../i18n/ar/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇯🇵 [日本語](../i18n/ja/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇻🇳 [Tiếng Việt](../i18n/vi/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇧🇬 [Български](../i18n/bg/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇩🇰 [Dansk](../i18n/da/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇫🇮 [Suomi](../i18n/fi/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇮🇱 [עברית](../i18n/he/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇭🇺 [Magyar](../i18n/hu/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇮🇩 [Bahasa Indonesia](../i18n/id/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇰🇷 [한국어](../i18n/ko/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇲🇾 [Bahasa Melayu](../i18n/ms/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇳🇱 [Nederlands](../i18n/nl/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇳🇴 [Norsk](../i18n/no/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇵🇹 [Português (Portugal)](../i18n/pt/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇷🇴 [Română](../i18n/ro/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇵🇱 [Polski](../i18n/pl/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇸🇰 [Slovenčina](../i18n/sk/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇸🇪 [Svenska](../i18n/sv/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇵🇭 [Filipino](../i18n/phi/docs/ops/VM_DEPLOYMENT_GUIDE.md) | 🇨🇿 [Čeština](../i18n/cs/docs/ops/VM_DEPLOYMENT_GUIDE.md)
Complete guide to install and configure OmniRoute on a VM (VPS) with domain managed via Cloudflare.
---
## Prerequisites
| Item | Minimum | Recommended |
| ---------- | ------------------------ | ---------------- |
| **CPU** | 1 vCPU | 2 vCPU |
| **RAM** | 1 GB | 2 GB |
| **Disk** | 10 GB SSD | 25 GB SSD |
| **OS** | Ubuntu 22.04 LTS | Ubuntu 24.04 LTS |
| **Domain** | Registered on Cloudflare | — |
| **Docker** | Docker Engine 24+ | Docker 27+ |
**Tested providers**: Akamai (Linode), DigitalOcean, Vultr, Hetzner, AWS Lightsail.
---
## 1. Configure the VM
### 1.1 Create the instance
On your preferred VPS provider:
- Choose Ubuntu 24.04 LTS
- Select the minimum plan (1 vCPU / 1 GB RAM)
- Set a strong root password or configure SSH key
- Note the **public IP** (e.g., `203.0.113.10`)
### 1.2 Connect via SSH
```bash
ssh root@203.0.113.10
```
### 1.3 Update the system
```bash
apt update && apt upgrade -y
```
### 1.4 Install Docker
```bash
# Install dependencies
apt install -y ca-certificates curl gnupg
# Add official Docker repository
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $ (. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```
### 1.5 Install nginx
```bash
apt install -y nginx
```
### 1.6 Configure Firewall (UFW)
```bash
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp # SSH
ufw allow 80/tcp # HTTP (redirect)
ufw allow 443/tcp # HTTPS
ufw enable
```
> **Tip**: For maximum security, restrict ports 80 and 443 to Cloudflare IPs only. See the [Advanced Security](#advanced-security) section.
---
## 2. Install OmniRoute
### 2.1 Create configuration directory
```bash
mkdir -p /opt/omniroute
```
### 2.2 Create environment variables file
```bash
cat > /opt/omniroute/.env << 'EOF'
# === Security ===
JWT_SECRET=CHANGE-TO-A-UNIQUE-64-CHAR-SECRET-KEY
INITIAL_PASSWORD=YourSecurePassword123!
API_KEY_SECRET=REPLACE-WITH-ANOTHER-SECRET-KEY
STORAGE_ENCRYPTION_KEY=REPLACE-WITH-THIRD-SECRET-KEY
STORAGE_ENCRYPTION_KEY_VERSION=v1
MACHINE_ID_SALT=CHANGE-TO-A-UNIQUE-SALT
OMNIROUTE_WS_BRIDGE_SECRET=REPLACE-WITH-WS-BRIDGE-SECRET # REQUIRED em produção: usado pelo Codex Responses WS bridge
# === App ===
PORT=20128
NODE_ENV=production
HOSTNAME=0.0.0.0
DATA_DIR=/app/data
APP_LOG_TO_FILE=true
AUTH_COOKIE_SECURE=true
REQUIRE_API_KEY=false
# === URLs (change to your domain) ===
# Internal server-to-server base URL for scheduled jobs / self-fetches.
BASE_URL=http://127.0.0.1:20128
# Browser-facing URL used for OAuth callbacks, dashboard links, and generated public URLs.
NEXT_PUBLIC_BASE_URL=https://llms.seudominio.com
# Optional explicit public origin override for generated public asset URLs.
# OMNIROUTE_PUBLIC_BASE_URL=https://llms.seudominio.com
# === Cloud Sync (optional) ===
# CLOUD_URL=https://cloud.omniroute.online
# NEXT_PUBLIC_CLOUD_URL=https://cloud.omniroute.online
EOF
```
> ⚠️ **IMPORTANT**: Generate unique secret keys! Use `openssl rand -hex 32` for each key.
### 2.3 Start the container
```bash
docker pull diegosouzapw/omniroute:latest
docker run -d \
--name omniroute \
--restart unless-stopped \
--env-file /opt/omniroute/.env \
-p 20128:20128 \
-v omniroute-data:/app/data \
diegosouzapw/omniroute:latest
```
### 2.4 Verify that it is running
```bash
docker ps | grep omniroute
docker logs omniroute --tail 20
```
It should display: `[DB] SQLite database ready` and `listening on port 20128`.
---
## 3. Configure nginx (Reverse Proxy)
### 3.1 Generate SSL certificate (Cloudflare Origin)
In the Cloudflare dashboard:
1. Go to **SSL/TLS → Origin Server**
2. Click **Create Certificate**
3. Keep the defaults (15 years, \*.yourdomain.com)
4. Copy the **Origin Certificate** and the **Private Key**
```bash
mkdir -p /etc/nginx/ssl
# Paste the certificate
nano /etc/nginx/ssl/origin.crt
# Paste the private key
nano /etc/nginx/ssl/origin.key
chmod 600 /etc/nginx/ssl/origin.key
```
### 3.2 Nginx Configuration
```bash
cat > /etc/nginx/sites-available/omniroute << 'NGINX'
# Default server — blocks direct access via IP
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
ssl_certificate /etc/nginx/ssl/origin.crt;
ssl_certificate_key /etc/nginx/ssl/origin.key;
server_name _;
return 444;
}
# OmniRoute — HTTPS
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name llms.yourdomain.com; # Change to your domain
ssl_certificate /etc/nginx/ssl/origin.crt;
ssl_certificate_key /etc/nginx/ssl/origin.key;
ssl_protocols TLSv1.2 TLSv1.3;
client_max_body_size 100M;
location / {
proxy_pass http://127.0.0.1:20128;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# SSE (Server-Sent Events) — streaming AI responses
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 600s;
proxy_send_timeout 600s;
}
}
# HTTP → HTTPS redirect
server {
listen 80;
listen [::]:80;
server_name llms.yourdomain.com;
return 301 https://$server_name$request_uri;
}
NGINX
```
Keep reverse-proxy stream timeouts aligned with your OmniRoute timeout env vars. If you raise
`FETCH_TIMEOUT_MS` / `STREAM_IDLE_TIMEOUT_MS`, raise `proxy_read_timeout` / `proxy_send_timeout`
above the same threshold.
OmniRoute uses `NEXT_PUBLIC_BASE_URL` as the canonical browser-facing origin for OAuth
callbacks and generated public links. Authenticated dashboard writes use same-origin requests
plus session-bound CSRF protection, so they do not require a static public base URL. The
`X-Forwarded-*` headers above are still useful routing metadata, but they are not a replacement
for setting the explicit public URL when OAuth or generated browser links need one. Only enable
`OMNIROUTE_TRUST_PROXY` if OmniRoute is not directly reachable by clients and your proxy
strips/rebuilds incoming forwarded headers.
### 3.3 Enable and Test
```bash
# Remove default configuration
rm -f /etc/nginx/sites-enabled/default
# Enable OmniRoute
ln -sf /etc/nginx/sites-available/omniroute /etc/nginx/sites-enabled/omniroute
# Test and reload
nginx -t && systemctl reload nginx
```
---
## 4. Configure Cloudflare DNS
### 4.1 Add DNS record
In the Cloudflare dashboard → DNS:
| Type | Name | Content | Proxy |
| ---- | ------ | ---------------------- | ---------- |
| A | `llms` | `203.0.113.10` (VM IP) | ✅ Proxied |
### 4.2 Configure SSL
Under **SSL/TLS → Overview**:
- Mode: **Full (Strict)**
Under **SSL/TLS → Edge Certificates**:
- Always Use HTTPS: ✅ On
- Minimum TLS Version: TLS 1.2
- Automatic HTTPS Rewrites: ✅ On
### 4.3 Testing
```bash
curl -sI https://llms.seudominio.com/health
# Should return HTTP/2 200
```
---
## 5. Operations and Maintenance
### Upgrade to a new version
```bash
docker pull diegosouzapw/omniroute:latest
docker stop omniroute && docker rm omniroute
docker run -d --name omniroute --restart unless-stopped \
--env-file /opt/omniroute/.env \
-p 20128:20128 \
-v omniroute-data:/app/data \
diegosouzapw/omniroute:latest
```
### View logs
```bash
docker logs -f omniroute # Real-time stream
docker logs omniroute --tail 50 # Last 50 lines
```
### Manual database backup
```bash
# Copy data from the volume to the host
docker cp omniroute:/app/data ./backup-$(date +%F)
# Or compress the entire volume
docker run --rm -v omniroute-data:/data -v $(pwd):/backup \
alpine tar czf /backup/omniroute-data-$(date +%F).tar.gz /data
```
### Restore from backup
```bash
docker stop omniroute
docker run --rm -v omniroute-data:/data -v $(pwd):/backup \
alpine sh -c "rm -rf /data/* && tar xzf /backup/omniroute-data-YYYY-MM-DD.tar.gz -C /"
docker start omniroute
```
---
## 6. Advanced Security
### Restrict nginx to Cloudflare IPs
```bash
cat > /etc/nginx/cloudflare-ips.conf << 'CF'
# Cloudflare IPv4 ranges — update periodically
# https://www.cloudflare.com/ips-v4/
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;
real_ip_header CF-Connecting-IP;
CF
```
Add the following to `nginx.conf` inside the `http {}` block:
```nginx
include /etc/nginx/cloudflare-ips.conf;
```
### Install fail2ban
```bash
apt install -y fail2ban
systemctl enable fail2ban
systemctl start fail2ban
# Check status
fail2ban-client status sshd
```
### Block direct access to the Docker port
```bash
# Prevent direct external access to port 20128
iptables -I DOCKER-USER -p tcp --dport 20128 -j DROP
iptables -I DOCKER-USER -i lo -p tcp --dport 20128 -j ACCEPT
# Persist the rules
apt install -y iptables-persistent
netfilter-persistent save
```
---
## 7. Deploy to Cloudflare Workers (Optional)
For remote access via Cloudflare Workers (without exposing the VM directly):
```bash
# In the local repository
cd omnirouteCloud
npm install
npx wrangler login
npx wrangler deploy
```
See also [TUNNELS_GUIDE.md](./TUNNELS_GUIDE.md) for the in-repo Cloudflare Tunnel walkthrough. The standalone `omnirouteCloud/` worker lives in a separate companion repo.
---
## Port Summary
| Port | Service | Access |
| ----- | ----------- | -------------------------- |
| 22 | SSH | Public (with fail2ban) |
| 80 | nginx HTTP | Redirect → HTTPS |
| 443 | nginx HTTPS | Via Cloudflare Proxy |
| 20128 | OmniRoute | Localhost only (via nginx) |
docs/ops/meta.json
+13
View File
@@ -0,0 +1,13 @@
{
"title": "Operations",
"pages": [
"RELEASE_CHECKLIST",
"RELEASE_GREEN",
"VM_DEPLOYMENT_GUIDE",
"FLY_IO_DEPLOYMENT_GUIDE",
"TUNNELS_GUIDE",
"PROXY_GUIDE",
"SQLITE_RUNTIME",
"COVERAGE_PLAN"
]
}