wehub-resource-sync
235 lines
15 KiB
Markdown
235 lines
15 KiB
Markdown
TODO: delete
|
|
|
|
# Model Sync Scripts
|
|
|
|
Model syncs are centralized in `packages/core/src/sync/index.ts`. The runner owns file IO, TOML formatting, validation, reporting, dry runs, and deletion behavior. `packages/core/script/sync-models.ts` is only the CLI wrapper for `bun models:sync`. Individual provider sync modules only fetch source data, parse it, and translate each source model into the catalog schema.
|
|
|
|
The grouped sync targets are available for local convenience, but CI syncs each provider separately so every provider gets its own reusable automation PR.
|
|
|
|
## Commands
|
|
|
|
- `bun models:sync aggregators` syncs every provider in the `aggregators` group.
|
|
- `bun models:sync openrouter` syncs only OpenRouter.
|
|
- `bun models:sync cloudflare-workers-ai` syncs only Cloudflare Workers AI.
|
|
- `bun models:sync cloudflare` syncs the Cloudflare sync group.
|
|
- `bun models:sync direct` syncs every provider in the `direct` group.
|
|
- `bun models:sync google` syncs only Google.
|
|
- `bun models:sync digitalocean` syncs only DigitalOcean.
|
|
- `bun models:sync xai` syncs only xAI.
|
|
- `bun models:sync kilo` syncs only Kilo.
|
|
- `bun models:sync openai` syncs only OpenAI catalog availability.
|
|
- `bun models:sync aggregators --dry-run` prints changes without writing model files.
|
|
- `bun models:sync aggregators --new-only` creates new model files but skips updates and removals.
|
|
- `bun validate` validates the generated catalog after a sync.
|
|
|
|
Sync runs also write `.sync/model-sync-report.md` for the automation workflow PR body. Do not commit that report from local runs.
|
|
|
|
## Runner Responsibilities
|
|
|
|
`packages/core/src/sync/index.ts` handles the shared behavior:
|
|
|
|
- Reads existing TOML files from the provider `modelsDir`.
|
|
- Parses existing files with `Bun.TOML.parse` and `AuthoredModelShape.partial()`.
|
|
- Resolves existing `base_model` / `base_model_omit` metadata before passing local metadata to provider modules.
|
|
- Calls the provider module to fetch, parse, and translate source models.
|
|
- Validates translated models with `AuthoredModel` before writing.
|
|
- Formats TOML consistently for all synced providers.
|
|
- Compares authored TOML shapes before writing so existing factored TOMLs stay factored instead of being expanded.
|
|
- Replaces symlinked files safely by removing the symlink before writing.
|
|
- Removes existing files that are no longer present in the desired synced set.
|
|
- Writes `.sync/model-sync-report.md` for GitHub Actions.
|
|
|
|
Because the runner removes files missing from the desired set, a provider module should only skip source models when deleting existing local files for those skipped IDs is intentional.
|
|
|
|
## Provider Modules
|
|
|
|
Provider modules live in `packages/core/src/sync/providers/`. A provider exports an object satisfying `SyncProvider<SourceModel>`:
|
|
|
|
```ts
|
|
export const provider = {
|
|
id: "provider-id",
|
|
name: "Provider Name",
|
|
modelsDir: "providers/provider-id/models",
|
|
async fetchModels() {
|
|
return fetch("https://example.com/models").then((response) => response.json());
|
|
},
|
|
parseModels(raw) {
|
|
return ProviderResponse.parse(raw).data;
|
|
},
|
|
translateModel(model, context) {
|
|
return {
|
|
id: model.id,
|
|
model: buildModel(model, context.existing(model.id)),
|
|
};
|
|
},
|
|
} satisfies SyncProvider<ProviderModel>;
|
|
```
|
|
|
|
Keep provider modules focused on provider-specific logic:
|
|
|
|
- Define Zod schemas for the provider API response.
|
|
- Fetch from the provider API, including auth headers when needed.
|
|
- Convert provider pricing units to per-1M-token catalog prices.
|
|
- Convert dates, modalities, limits, capabilities, and model IDs into catalog fields.
|
|
- Preserve existing hand-authored fields only when the provider API is not authoritative for that field.
|
|
- Preserve `base_model` and `base_model_omit` from existing TOMLs when updating a factored provider model.
|
|
- Return `undefined` from `translateModel` only when skipped source models should be treated as absent from the synced catalog.
|
|
|
|
Do not put TOML scanning, writing, deletion, reporting, or generic comparison logic in provider modules.
|
|
|
|
Provider sync code must use `base_model` and `base_model_omit`; do not write legacy `[extends]` tables. If a sync or generator updates a provider file that already uses `base_model`, it should keep that pointer and only write provider-specific overrides.
|
|
|
|
## Adding A Provider
|
|
|
|
1. Create `packages/core/src/sync/providers/<provider>.ts`.
|
|
2. Define strict-enough Zod schemas for the provider response.
|
|
3. Export a `SyncProvider` implementation with `fetchModels`, `parseModels`, and `translateModel`.
|
|
4. Add the provider to `providers` in `packages/core/src/sync/index.ts`.
|
|
5. Add the provider ID to an existing group or create a new group in `groups`.
|
|
6. Add any required API secrets to `.github/workflows/sync-models.yml` if the provider needs new credentials.
|
|
7. Run `bun models:sync <provider> --dry-run` to inspect the first diff.
|
|
8. Run `bun models:sync <provider>` to write files.
|
|
9. Run `bun models:sync <provider> --dry-run` again and expect a clean result.
|
|
10. Run `bun validate`.
|
|
|
|
Prefer small, provider-specific PRs when adding a provider. If the provider has ambiguous source data, keep it out of shared groups until the source-of-truth behavior is clear.
|
|
|
|
## Automation
|
|
|
|
`.github/workflows/sync-models.yml` runs on an hourly schedule and manually through `workflow_dispatch`.
|
|
|
|
The workflow:
|
|
|
|
- Checks out `dev`.
|
|
- Installs dependencies with Bun.
|
|
- Discovers sync providers with `bun models:sync --list-providers`.
|
|
- Runs one provider per matrix job with `bun models:sync ${{ matrix.provider }}`.
|
|
- Runs `bun validate`.
|
|
- Creates or updates a provider-specific sync PR only when `providers` changed.
|
|
- Uses `.sync/model-sync-report.md` as the PR body.
|
|
|
|
Each provider job checks out `dev` and writes to a fixed provider branch like `automation/sync-models-openrouter`. If that provider's sync PR is already open, later scheduled runs force-update the same branch and edit the existing PR instead of creating another one. Provider jobs do not share unmerged changes with each other; OpenRouter only uses `base_model` for model metadata entries already present on `dev`.
|
|
|
|
CI automatically picks up providers registered in `providers` in `packages/core/src/sync/index.ts`. Adding a new sync provider there is enough to get an hourly provider-specific sync job, branch, labels, title, and PR naming convention. The workflow only needs manual updates when a new provider requires new secrets or other environment variables.
|
|
|
|
Actions are pinned by commit SHA. Keep new workflow actions pinned the same way.
|
|
|
|
## OpenRouter Notes
|
|
|
|
OpenRouter is implemented in `packages/core/src/sync/providers/openrouter.ts`.
|
|
|
|
- Source endpoint: `https://openrouter.ai/api/v1/models`.
|
|
- Optional auth: `OPENROUTER_API_KEY`.
|
|
- Model IDs map directly to TOML paths under `providers/openrouter/models`.
|
|
- API prices are per-token strings and are converted to per-1M-token numbers.
|
|
- `structured_output` comes from `supported_parameters.includes("structured_outputs")` only.
|
|
- Existing `status`, `interleaved`, `knowledge`, `limit.input`, and `cost.tiers` may be preserved when OpenRouter is not authoritative enough for those fields.
|
|
- Canonical OpenRouter model IDs should emit `base_model` references to model metadata when a matching `models/` entry exists.
|
|
|
|
## Kilo Gateway Notes
|
|
|
|
Kilo Gateway is implemented in `packages/core/src/sync/providers/kilo.ts`.
|
|
|
|
- Source endpoint: `https://api.kilo.ai/api/gateway/models`.
|
|
- Optional auth: `KILO_API_KEY`.
|
|
- Model IDs map directly to TOML paths under `providers/kilo/models`.
|
|
- API prices are per-token strings and are converted to per-1M-token numbers.
|
|
- `structured_output` comes from `supported_parameters.includes("structured_outputs")` only.
|
|
- Existing `status`, `interleaved`, `knowledge`, `limit.input`, and `cost.tiers` may be preserved when Kilo is not authoritative enough for those fields.
|
|
- Canonical Kilo model IDs should emit `base_model` references to model metadata when a matching `models/` entry exists.
|
|
- `reasoning_options` is derived from `opencode.variants` when present.
|
|
|
|
## Cloudflare Workers AI Notes
|
|
|
|
Cloudflare Workers AI is implemented in `packages/core/src/sync/providers/cloudflare-workers-ai.ts`.
|
|
|
|
- Source endpoint: `https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_WORKERS_AI_SYNC_ACCOUNT_ID/ai/models/search?format=openrouter`.
|
|
- Required auth: `CLOUDFLARE_WORKERS_AI_SYNC_ACCOUNT_ID` and `CLOUDFLARE_WORKERS_AI_SYNC_API_TOKEN`.
|
|
- Use a dedicated token scoped to Workers AI read access so sync automation does not share deploy credentials.
|
|
- The endpoint is parsed as Cloudflare's OpenRouter-like Workers AI metadata.
|
|
- Model IDs map directly to TOML paths under `providers/cloudflare-workers-ai/models`.
|
|
- This sync target does not manage `providers/cloudflare-ai-gateway`, because the AI Gateway `/compat/models` endpoint does not support `format=openrouter` and does not provide enough model metadata for authoritative catalog sync.
|
|
|
|
## Google Notes
|
|
|
|
Google is implemented in `packages/core/src/sync/providers/google.ts`.
|
|
|
|
- Source endpoint: `https://generativelanguage.googleapis.com/v1beta/models`.
|
|
- Required auth: `GOOGLE_API_KEY`, `GEMINI_API_KEY`, or `GOOGLE_GENERATIVE_AI_API_KEY`.
|
|
- Model IDs are derived from the `models/{model}` resource names.
|
|
- The API is authoritative for display names, token limits, temperature metadata, and the `thinking` flag when present.
|
|
- Local Google models missing from the API response are removed.
|
|
- New Google API models are reported in `.sync/model-sync-report.md` but not created automatically because the API does not provide authoritative modalities, pricing, knowledge cutoff, release date, tool calling, or structured output metadata.
|
|
|
|
## xAI Notes
|
|
|
|
xAI is implemented in `packages/core/src/sync/providers/xai.ts`.
|
|
|
|
- Source endpoints: `https://api.x.ai/v1/language-models`, `https://api.x.ai/v1/image-generation-models`, and `https://api.x.ai/v1/video-generation-models`.
|
|
- Required auth: `XAI_API_KEY`.
|
|
- The richer typed endpoints provide model IDs, creation timestamps, modalities, pricing for language models, and prompt/input limits where available.
|
|
- Existing xAI models are updated from API-authoritative fields while local metadata is preserved for fields the API does not expose, especially output token limits and some feature/capability flags.
|
|
- New xAI API models are reported in `.sync/model-sync-report.md` but not created automatically because the API does not provide enough authoritative metadata for complete catalog entries.
|
|
|
|
## OpenAI Notes
|
|
|
|
- OpenAI is implemented in `packages/core/src/sync/providers/openai.ts`.
|
|
- Source endpoint: `https://api.openai.com/v1/models`.
|
|
- Required auth: `OPENAI_API_KEY` from an automation account with access to the full first-party catalog.
|
|
- The endpoint is used only to monitor catalog availability. Existing TOMLs are preserved byte-for-byte, including models absent from the response, because model access can be scoped to the API project.
|
|
- Fine-tuned and customer-owned models are excluded. Unknown first-party models are reported for manual review without changing the catalog.
|
|
|
|
## OVHcloud Notes
|
|
|
|
OVHcloud AI Endpoints is implemented in `packages/core/src/sync/providers/ovhcloud.ts`.
|
|
|
|
- Source endpoint: `https://catalog.endpoints.ai.ovh.net/rest/v2/openrouter`.
|
|
- No auth required: the catalog is public.
|
|
- Model IDs are lowercased from the catalog `id` to match the existing TOML paths under `providers/ovhcloud/models`.
|
|
- API prices are per-token strings and are converted to per-1M-token numbers; free models (price `0`) get no `[cost]` section.
|
|
- `reasoning`, `tool_call`, and `structured_output` come from `supported_features`; `temperature` comes from `supported_sampling_parameters`.
|
|
- Authored `reasoning_options` are preserved for reasoning models. `Qwen3-32B` supports toggling reasoning through OVHcloud's documented `/no_think` prompt control. Both gpt-oss models support `low`, `medium`, and `high` reasoning effort. The Qwen3.5 models support `none`, `low`, `medium`, and `high`; Qwen3.6-27B additionally supports `minimal`.
|
|
- `attachment` is derived from non-text `input_modalities`, and `open_weights` from the presence of `hugging_face_id`.
|
|
- `release_date`/`last_updated` default to the catalog `created` timestamp but preserve any existing hand-authored dates; `knowledge`, `family`, `status`, `interleaved`, and `limit.input` are preserved when present.
|
|
|
|
## DigitalOcean Notes
|
|
|
|
- DigitalOcean is implemented in `packages/core/src/sync/providers/digitalocean.ts`.
|
|
- Source endpoints: `https://api.digitalocean.com/v2/gen-ai/models` for lifecycle and reasoning metadata, and the public `https://api.digitalocean.com/v2/gen-ai/models/catalog` for availability, modalities, limits, and pricing.
|
|
- Required auth: `DIGITALOCEAN_API_TOKEN` or `DIGITALOCEAN_ACCESS_TOKEN` for the control-plane model endpoint; the model catalog is public.
|
|
- The sync manages serverless text-output models. Other model types, dedicated-only models, and local models absent from the API are retained for manual lifecycle review.
|
|
- Catalog pricing updates standard, cache-read, cache-write, and extended-context rates while preserving authored reasoning and audio prices.
|
|
|
|
## Vercel Status
|
|
|
|
Vercel is intentionally not wired into `bun models:sync` right now. Keep using the existing `vercel:generate` script until Vercel sync behavior is redesigned and reviewed separately.
|
|
|
|
Do not add Vercel model changes to OpenRouter sync PRs.
|
|
|
|
## Chutes Notes
|
|
|
|
Chutes is implemented in `packages/core/src/sync/providers/chutes.ts`.
|
|
|
|
- Run it with `bun models:sync chutes` or `bun chutes:sync`.
|
|
- Source endpoint: `https://llm.chutes.ai/v1/models`; no auth required (the model list is public).
|
|
- Model IDs map directly to TOML paths under `providers/chutes/models`.
|
|
- `reasoning`, `tool_call`, and `structured_output` come from `supported_features`; `temperature` comes from `supported_sampling_parameters`.
|
|
- `reasoning_options` is always an empty array: the API advertises a `reasoning` capability but exposes no toggle or effort parameter, so there is no provider evidence for a reasoning option.
|
|
- TEE model IDs emit `base_model` references to matching `models/` metadata; checkpoints without a canonical entry (e.g. `Qwen3-235B-A22B-Thinking-2507`, `DeepSeek-V3.2`) are written inline.
|
|
- `attachment` is derived from non-text `input_modalities`, and all models are `open_weights`.
|
|
- `release_date`/`last_updated` default to the API `created` timestamp but preserve existing hand-authored dates; `knowledge`, `family`, `status`, `interleaved`, and `limit.input` are preserved when present.
|
|
|
|
## Venice Notes
|
|
|
|
Venice is implemented in `packages/core/src/sync/providers/venice.ts`.
|
|
|
|
- Run it with `bun models:sync venice` or `bun venice:sync`.
|
|
- `VENICE_API_KEY` is optional locally and includes models visible to that account when set.
|
|
- Models missing from the API response are removed from the Venice catalog.
|
|
- Every Venice model uses `base_model`; flattened IDs are matched to provider-agnostic metadata before provider-specific overrides are written.
|
|
- Every Venice model declares `reasoning_options`; models without API-provided effort levels use an empty array.
|
|
|
|
## Standalone Generators
|
|
|
|
Some provider scripts in `packages/core/script/generate-*.ts` are not wired into `bun models:sync`. When updating those scripts, preserve existing `base_model` and `base_model_omit` fields for generated TOMLs that already use model metadata inheritance. New inheritance-aware output should use `base_model`; do not reintroduce legacy `[extends]` syntax.
|