wehub-resource-sync
a789495a98
FreeBSD Smoke / FreeBSD Smoke (x86_64) (push) Has been cancelled
CI / Quality Guardrails (push) Has been cancelled
CI / Build & Test (macos-latest) (push) Has been cancelled
CI / Build & Test (ubuntu-latest) (push) Has been cancelled
CI / Build & Test (windows-latest) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / PowerShell Syntax (push) Has been cancelled
CI / Windows Cross-Target Check (Linux) (push) Has been cancelled
167 lines
8.1 KiB
Markdown
167 lines
8.1 KiB
Markdown
# Provider Doctor
|
|
|
|
`jcode provider-doctor` is a user-facing diagnostic that answers one question:
|
|
|
|
> Why isn't my provider/model (or the model picker) working?
|
|
|
|
It walks the same strict end-to-end checkpoints that the live coverage ledger
|
|
tracks (`jcode provider-test-coverage`), but as an interactive command you can run
|
|
yourself, with clear pass/fail output and a "what to try next" hint on the first
|
|
failure.
|
|
|
|
It works with **OpenAI-compatible providers** (cerebras, fpt, nvidia-nim,
|
|
comtegra, deepseek, groq, openrouter, and other `openai-compatible` profiles).
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
# Validate jcode's own wiring for a provider, no API key, no spend:
|
|
jcode provider-doctor cerebras --tier offline
|
|
|
|
# Validate the key + live model catalog (needs a key, negligible spend):
|
|
jcode provider-doctor cerebras --tier catalog
|
|
|
|
# Full readiness, including real chat, streaming, and tool calls (spends balance):
|
|
jcode provider-doctor cerebras --tier full
|
|
|
|
# Pin a specific model and emit JSON for scripting/CI:
|
|
jcode provider-doctor cerebras --model gpt-oss-120b --tier full --json
|
|
```
|
|
|
|
The model defaults to the provider's default model (or the first live catalog
|
|
model). Use the global `--model` flag to pin a specific one.
|
|
|
|
## Tiers
|
|
|
|
Pick how much to exercise. Each tier validates as much as is possible given its
|
|
constraints, so you can debug cheaply and escalate only when needed.
|
|
|
|
| Tier | Needs key? | Spends balance? | What it adds | Catches |
|
|
| --- | --- | --- | --- | --- |
|
|
| `offline` | no | no | jcode-side wiring against a synthetic catalog | catalog reload, picker rendering, fallback labeling, and model-switch routing bugs for this provider |
|
|
| `catalog` (default) | yes | ~none | live `GET /models` | bad/missing key, dead endpoint, model not in the live catalog |
|
|
| `full` | yes | yes | non-streaming chat, streaming, tool-call loop | the model actually chats, streams, and supports tool-calling |
|
|
|
|
Only the `full` tier can earn strict ("READY") coverage. The lighter tiers
|
|
intentionally record the API-dependent checkpoints as skipped, so nothing is
|
|
over-credited in the coverage ledger.
|
|
|
|
## Checkpoints
|
|
|
|
Every run reports these strict checkpoints in order. A pair is fully ready only
|
|
when all of them pass on the `full` tier.
|
|
|
|
1. `auth_credential_loaded` - a credential was found for the provider
|
|
2. `model_catalog_live_endpoint` - the live `/models` endpoint returned models
|
|
3. `catalog_hot_reload_current_session` - the catalog reloaded into the session
|
|
4. `picker_live_models` - the picker shows the live models, including the selected one
|
|
5. `picker_fallback_labeling` - routes are live-catalog backed, not static fallback
|
|
6. `model_switch_route` - switching models produces a provider-explicit route
|
|
7. `non_streaming_chat_completion` - a basic chat reply came back (full tier)
|
|
8. `streaming_chat_completion` - a streamed reply came back (full tier)
|
|
9. `tool_call_parse` - the model emitted a parseable tool call (full tier)
|
|
10. `tool_execution_loop` - the tool-call loop ran (full tier)
|
|
11. `tool_result_followup` - the tool result was fed back (full tier)
|
|
12. `real_jcode_tool_smoke` - an end-to-end tool smoke passed (full tier)
|
|
|
|
(Checkpoints 1-2 plus the auth-lifecycle stages are pre-flight; 7-12 are the
|
|
API-dependent ones gated behind `--tier full`.)
|
|
|
|
## Reading the output
|
|
|
|
```
|
|
Provider doctor: Cerebras / gpt-oss-120b
|
|
Tier: catalog (API key, ~no spend: adds live catalog fetch)
|
|
...
|
|
[ PASS] Credential loaded Loaded credential from CEREBRAS_API_KEY
|
|
[ PASS] Live model catalog endpoint 2 live model(s) returned
|
|
[ PASS] Catalog hot reload in current session 2 catalog route(s) reloaded
|
|
[ PASS] Picker shows live models 2 model(s) in picker, selected `gpt-oss-120b`
|
|
[ PASS] Picker fallback labeling all routes backed by live catalog (no static fallback)
|
|
[ PASS] Model switch route switch request `cerebras:...` routed via `openai-compatible:cerebras`
|
|
[ skip] Non-streaming chat completion catalog tier: requires --tier full (spends balance)
|
|
...
|
|
Verdict: tier `catalog` passed. Run `--tier full` to confirm full readiness (spends balance).
|
|
```
|
|
|
|
- `PASS` / `FAIL` - the checkpoint ran and passed/failed.
|
|
- `skip` - the current tier does not run this checkpoint (use `--tier full`).
|
|
- The verdict line tells you whether the tier passed, fully passed (`READY`), or
|
|
failed, and on failure points at the first failing checkpoint with a next step.
|
|
|
|
The command exits non-zero when the chosen tier did not fully pass, so it can be
|
|
used as a CI/scripting gate.
|
|
|
|
## Spend tracking (how much does a run cost?)
|
|
|
|
Balance-spending tiers (`catalog` makes a catalog call, `full` makes several
|
|
chat/stream/tool calls) report exactly what they consumed so you can budget:
|
|
|
|
```
|
|
Spend this run: 3 billable API calls, 554 tokens (289 in + 265 out), cost not reported by provider
|
|
```
|
|
|
|
- **billable API calls** - how many requests actually hit the provider.
|
|
- **tokens** - prompt + completion totals summed across those calls, when the
|
|
provider returns a `usage` block. Streaming probes request
|
|
`stream_options.include_usage` so streamed calls are counted too.
|
|
- **cost** - shown as a USD figure only when the provider reports a `cost`
|
|
field; many providers (e.g. cerebras) only return tokens, so you'll see
|
|
"cost not reported by provider" and can multiply tokens by your plan's rate.
|
|
A full cerebras run is roughly 550-620 tokens (about $0.0003).
|
|
|
|
`--json` includes the same data under a `spend` object
|
|
(`billable_calls`, `prompt_tokens`, `completion_tokens`, `total_tokens`,
|
|
`has_token_data`, `reported_cost_usd`).
|
|
|
|
This spend is **persisted** into the coverage ledger alongside the run, so
|
|
`jcode provider-test-coverage` shows a cumulative "Recorded spend" footer
|
|
summing the latest run per pair. That gives you a durable, at-a-glance answer to
|
|
"how much has exercising this coverage cost me so far?"
|
|
|
|
## Typical debugging flow
|
|
|
|
1. **"My picker is broken / shows the wrong models."**
|
|
Run `--tier offline`. If `picker_live_models`, `picker_fallback_labeling`, or
|
|
`model_switch_route` fail, it's a jcode-side routing bug for that provider:
|
|
capture the output and file an issue.
|
|
|
|
2. **"It won't connect / says auth failed."**
|
|
Run `--tier catalog`. If `auth_credential_loaded` or
|
|
`model_catalog_live_endpoint` fail, the key/endpoint is the problem. Run
|
|
`jcode login --provider <provider>`.
|
|
|
|
3. **"It connects but the model behaves badly."**
|
|
Run `--tier full`. If `non_streaming_chat_completion` /
|
|
`streaming_chat_completion` / the `tool_*` checkpoints fail, the model itself
|
|
is the issue; try another model from the live catalog.
|
|
|
|
## Relationship to coverage
|
|
|
|
Every doctor run records a live-verification event into the coverage ledger,
|
|
tagged with the tier (`doctor_tier`). A `full`-tier pass that clears all 11
|
|
strict checkpoints flips the pair to strict ("READY") in
|
|
`jcode provider-test-coverage`. Lighter tiers record the API-dependent
|
|
checkpoints as skipped, so they never over-credit a pair.
|
|
|
|
`jcode provider-test-coverage` renders the same 11 checkpoints as an 11-stage
|
|
pipeline. Each observed pair gets one compact line: a status token (`READY`, or
|
|
`N/11` = how many stages it cleared) followed by `provider / model`, and then,
|
|
for any pair that is not yet READY, the first blocker plus the exact
|
|
`provider-doctor` command to push it past that blocker. So the two commands are
|
|
two views of one pipeline: the coverage report shows where every pair is stuck
|
|
and hands you the doctor command to advance it.
|
|
|
|
Each line ends with a freshness note, e.g.:
|
|
|
|
```
|
|
READY cerebras / gpt-oss-120b last tested 9 minutes ago (2026-05-30) by developer (dev build)
|
|
6/11 nvidia-nim / gemma-4-31b failed at `streaming reply`; run `jcode provider-doctor nvidia-nim --model gemma-4-31b --tier full`; last tested 2 days ago ...
|
|
```
|
|
|
|
- **how long ago** the most recent run was, in plain English plus the absolute
|
|
date, so you can tell at a glance whether the evidence is stale.
|
|
- **who ran it**: a clean release build is labeled `user (release build)` (real
|
|
user evidence), a dirty/dev build is `developer (dev build)`. This is derived
|
|
durably from the build flag recorded with each run, not guessed.
|