promptfoo--promptfoo/docs/plans/eng-1770.md

Plan: promptfoo eval -c <uuid> Cloud Config Support

Context

Currently, promptfoo redteam run -c <uuid> supports loading configs from Promptfoo Cloud by UUID, but promptfoo eval -c only accepts local file paths. This feature extends the same cloud config loading pattern to the eval command, allowing users to run promptfoo eval -c <cloud-uuid> to fetch and execute a config stored in Promptfoo Cloud.

The ticket (ENG-1770) has two parts:

1. Open Source (this PR): Make the CLI accept a cloud UUID for eval -c
2. Cloud: Show the promptfoo eval -c <uuid> command in the Cloud run modal (separate repo)

Changes

1. Add getEvalConfigFromCloud() to src/util/cloud.ts

Create a new function modeled after the existing getConfigFromCloud() (line 88-114) but hitting a different endpoint for eval configs:

export async function getEvalConfigFromCloud(id: string): Promise<UnifiedConfig> {
  // Same pattern as getConfigFromCloud but using `configs/${id}` endpoint
}

• Endpoint: GET /api/v1/configs/${id} (eval configs, not redteam-specific)
• Reuse existing makeRequest() helper (line 25) and cloudConfig.isEnabled() check pattern
• Same error handling pattern as getConfigFromCloud

2. Add UUID detection to src/commands/eval.ts

In doEval() (line 107), add UUID detection before the existing config path processing at line 142. This mirrors the pattern in src/redteam/commands/run.ts:62-79.

// Before the existing config path processing (line 142)
const UUID_REGEX = /^[A-Fa-f0-9]{8}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{12}$/;

if (cmdObj.config?.length === 1 && UUID_REGEX.test(cmdObj.config[0])) {
  const cloudConfigObj = await getEvalConfigFromCloud(cmdObj.config[0]);
  defaultConfig = cloudConfigObj;
  cmdObj.config = undefined;
}

Key behaviors:

• Only trigger when exactly one config path is provided and it's a UUID
• Fetch the config from cloud and use it as defaultConfig
• Clear cmdObj.config so resolveConfigs uses defaultConfig instead of trying to read a file
• Import getEvalConfigFromCloud from ../../util/cloud

3. Update eval command description

Update the -c option description at line 903-906 to mention cloud UUID support:

.option(
  '-c, --config <paths...>',
  'Path to configuration file or cloud config UUID. Automatically loads promptfooconfig.yaml',
)

4. Add tests

Create test cases in a new test file or add to existing eval command tests, following the pattern from test/redteam/commands/run.test.ts:

• Test UUID detection triggers cloud fetch
• Test local file paths bypass UUID detection
• Test error when cloud is not enabled
• Test multiple config paths with UUID (should not trigger UUID detection)

Files to Modify

File	Change
src/util/cloud.ts	Add getEvalConfigFromCloud() function
src/commands/eval.ts	Add UUID detection + cloud fetch in doEval(), update -c description
test/commands/eval.test.ts or new test file	Add tests for UUID cloud config

Existing Code to Reuse

• makeRequest() from src/util/cloud.ts:25 - authenticated HTTP helper
• cloudConfig.isEnabled() from src/globalConfig/cloud.ts - auth check
• UUID regex pattern from src/redteam/commands/run.ts:17
• resolveConfigs() from src/util/config/load.ts:481 - existing config resolution (no changes needed)

Verification

1. Build: npm run build should succeed
2. Lint: npm run l && npm run f
3. Unit tests: Run existing + new tests with npx vitest src/commands/eval and npx vitest src/util/cloud
4. Manual test (if cloud access available):
   • npm run local -- eval -c <valid-uuid> --env-file .env should fetch config from cloud
   • npm run local -- eval -c path/to/config.yaml should still work as before
   • npm run local -- eval -c <invalid-uuid-format> should fall through to file resolution

Decisions (Reconciled)

1. Use the OSS endpoint contract: GET /api/v1/configs/:id returning an envelope ({ config: ... }), not a raw payload.
2. Support persisted config shape with providers and tests; loading must not require additional requests.
3. Provider references in the config should use promptfoo://provider/<uuid>.
4. Prompts should be emitted as plain strings.
5. tests defaults to [] when missing.
6. description falls back to config.name when missing.
7. --watch is not supported when -c <uuid> is used. CLI should fail fast with a clear error.
8. If -c <value> matches UUID format but cloud fetch fails (404/auth disabled), hard-fail.
9. If multiple -c values are supplied and any value is a UUID, fail with an explicit error stating only one -c value is allowed for cloud UUID mode.
10. Add tests in both places:
    • test/commands/eval.test.ts for UUID detection/CLI behavior
    • test/util/cloud.test.ts for getEvalConfigFromCloud() contract/error handling
11. Scope is eval only (not redteam eval).
12. No temporary UI note or minimum-version gating is required.
13. In UUID mode, clear/ignore defaultConfigPath for the run to prevent accidental local reload/fallback behavior.
14. Read-time schema normalization should normalize legacy fields (providerIds/testCases) into canonical fields (providers/tests).

CLI Error Messages (exact draft text)

1. Multiple -c values with UUID mode:
   • Cloud config UUID mode supports exactly one -c value. Use: promptfoo eval -c <cloud-config-uuid>
2. UUID mode with --watch:
   • --watch is not supported when using a cloud config UUID with -c. Use a local config file path for watch mode.
3. UUID-shaped value with failed cloud fetch:
   • Failed to load cloud eval config "<uuid>". <reason>. Cloud UUID inputs do not fall back to local file paths. Check authentication and that the UUID exists.

Remaining Follow-ups

None.