tracer-cloud--opensre/core/llm/AGENTS.md

Hosted LLM Runtime

This package owns hosted LLM provider clients and runtime helpers used by the agent loop. Subprocess-backed LLM CLIs live under integrations/llm_cli/.

Where provider wiring lives

File	Role
config/config.py	Declares LLMProvider, provider env vars, defaults, and validation requirements.
config/llm_auth/provider_catalog.py	Canonical ProviderSpec metadata shared by wizard, auth, and runtime checks.
core/llm/factory.py	Single routing entrypoint: resolve_llm_route(), get_llm(role), reset_llm_clients().
core/llm/client_builders.py	Construct the client for a resolved route: build_agent_client(), build_reasoning_client().
core/llm/providers/provider_registry.py	FIRST_PARTY_PROVIDERS table (models, max_tokens, LiteLLM prefix, api-key env) the builders read.
core/llm/transport_mode.py	OPENSRE_LLM_TRANSPORT (sdk vs litellm) and use_litellm_for_provider().
core/llm/internal/client_cache_key.py	Singleton cache invalidation key (transport, runtime_provider).
core/llm/providers/openai_compat_providers.py	OpenAI-compatible provider catalog and model/base-URL resolution.
core/llm/providers/azure_openai.py	Azure OpenAI helpers: endpoint normalization, deployment selection, LiteLLM kwargs.
core/llm/transports/litellm/routing.py	Per-provider LiteLLM client construction (model prefix, api_base, api_version).
core/llm/transports/litellm/clients.py	LiteLLMAgentClient / LiteLLMLLMClient wrappers around litellm.completion.
core/llm/transports/sdk/agent_clients.py	Native SDK tool-calling clients (Anthropic, OpenAI, Bedrock, CLI-backed).
core/llm/transports/sdk/llm_clients.py	Native SDK non-agent clients.
core/llm/shared/tool_schema_normalize.py	JSON Schema normalization shared by strict tool-calling adapters.
surfaces/cli/wizard/config.py	Onboarding metadata (SUPPORTED_PROVIDERS) and model choices.
surfaces/cli/wizard/env_sync.py	.env synchronization when provider/model choices change.

User-facing setup and env var tables: docs/llm-providers.mdx.

Transport: native SDK vs LiteLLM

Default path is native vendor SDKs (OPENSRE_LLM_TRANSPORT unset or sdk).

LiteLLM path (OPENSRE_LLM_TRANSPORT=litellm): routes hosted API providers through core/llm/transports/litellm/routing.py instead of core/llm/transports/sdk/*.

Azure OpenAI (LLM_PROVIDER=azure-openai) always uses LiteLLM — even when OPENSRE_LLM_TRANSPORT is unset. Onboarding writes OPENSRE_LLM_TRANSPORT=litellm to .env; switching away from Azure removes that key so other providers return to SDK routing.

Dispatch entrypoints — all routing lives in one place, core/llm/factory.py; construction lives in core/llm/client_builders.py:

get_llm(role)  # role ∈ {AGENT, REASONING, CLASSIFICATION, TOOLCALL}   # factory.py
  → resolve_llm_route()               # the single provider/transport decision  # factory.py
  → client_builders.build_agent_client(route) / build_reasoning_client(route, model_type)
      cli_provider_registration?  → CLI-backed subprocess client
      use_litellm_for_provider? → build_litellm_*_client(settings, provider)   # transports/litellm/routing.py
      else      → native SDK client in transports/sdk/agent_clients.py or transports/sdk/llm_clients.py

When changing routing, edit only resolve_llm_route in factory.py; when changing how a provider's client is built, edit the builders in client_builders.py — there is no second copy to keep in sync.

One cache in factory.py keyed by (role, transport, runtime_provider), invalidated together on (transport, runtime_provider) change (not transport alone). REPL /model and wizard env sync call reset_llm_clients() directly.

Adding a Hosted API Provider

1. Add the provider literal to LLMProvider and normalization/validation paths in config/config.py.
2. Add ProviderSpec in config/llm_auth/provider_catalog.py and matching ProviderOption in surfaces/cli/wizard/config.py (model env vars, defaults, endpoint_env if needed).
3. Add runtime construction (routing itself stays in core/llm/factory.py; clients are built in core/llm/client_builders.py):
   • First-party provider (its own SDK models + a LiteLLM prefix): add one row to FIRST_PARTY_PROVIDERS in core/llm/providers/provider_registry.py — the SDK and LiteLLM builders read the table, so no per-provider branch is needed unless the client class is new.
   • SDK path: add the client class in core/llm/transports/sdk/llm_clients.py and/or core/llm/transports/sdk/agent_clients.py; the builders in client_builders.py select it.
   • LiteLLM path (optional or required): covered by the registry row; only add a branch in core/llm/transports/litellm/routing.py for a non-standard case (e.g. Azure).
   • OpenAI-compatible: register in providers/openai_compat_providers.py (SDK compat path) and/or transports/litellm/routing.py (LiteLLM path).
4. Update surfaces/cli/wizard/env_sync.py if you introduce new non-secret env keys; keep endpoint keys in active_non_secret when the provider needs persisted URL/version settings.
5. Add or update tests under tests/core/runtime/llm/ and wizard tests if onboarding changes.

Azure OpenAI (azure-openai)

Azure uses deployment names (not public OpenAI model IDs) and a resource base URL:

• AZURE_OPENAI_BASE_URL, AZURE_OPENAI_API_KEY, AZURE_OPENAI_API_VERSION (default applied when unset)
• AZURE_OPENAI_*_MODEL env vars hold deployment names in the user's Azure resource
• LiteLLM model string: azure/<deployment> via azure_openai_litellm_model()

Do not add a separate Azure client class — extend transports/litellm/routing.py and helpers in providers/azure_openai.py.

For investigation tool calling details, see docs/investigation-tool-calling.md.