omnigent-ai--omnigent/tests/codex_parity

Codex Parity Tests

This suite verifies Omnigent's Codex integration by running the real boundary we care about:

Omnigent CodexExecutor
  -> real codex app-server process
  -> mock OpenAI Responses API

The important choice is that the tests do not mock the Omnigent-to-Codex API. They start a real Codex CLI and only replace the upstream model endpoint. That means the test covers Codex app-server JSON-RPC behavior, Codex request serialization, retry notifications, streaming notifications, and dynamic tool round trips.

Architecture

pytest
  |
  | starts
  v
Rust sidecar: tests/codex_parity/sidecar
  |
  | uses upstream Codex test helper crate
  v
core_test_support::responses / WireMock
  ^
  | /v1/responses
  |
real codex app-server
  ^
  | JSON-RPC app-server protocol
  |
Omnigent CodexExecutor

The Rust sidecar exists because Codex's mock Responses API helpers are Rust test-support code in the public Codex repository. Rather than reimplementing that mock in Python, the sidecar pulls the upstream test-support crate directly through Cargo:

core_test_support = { git = "https://github.com/openai/codex.git", rev = "..." }

That keeps the fake Responses wire format aligned with Codex upstream. Pytest still owns the test scenarios and assertions; the sidecar only starts WireMock, serves queued SSE fixtures, and reports captured requests.

The revision is pinned in tests/codex_parity/sidecar/Cargo.toml so the parity harness is reproducible without requiring a checked-in Codex submodule. Updating the upstream fixture implementation is a normal Cargo dependency bump: change the Codex rev, refresh Cargo.lock, and run the parity tests.

Fixture Flow

Each test passes a list of model responses to the sidecar:

sidecar = codex_responses_sidecar(
    [
        [
            ev_response_created("resp-1"),
            ev_assistant_message("msg-1", "hello"),
            ev_completed("resp-1"),
        ]
    ]
)

Each inner list becomes one SSE response body. Codex consumes one body per POST /v1/responses request. Multi-turn scenarios enqueue multiple inner lists, for example a dynamic tool call followed by the assistant's final answer after Omnigent returns the tool result.

The sidecar prints one JSON ready line with a base_url. The pytest fixture passes that URL into CodexExecutor using the existing gateway override path, so Codex sends model traffic to the sidecar instead of OpenAI.

After a turn, pytest asks the sidecar for captured requests over a small JSONL stdin/stdout protocol:

{"op": "requests", "min": 1, "timeout_ms": 5000}

The response includes stable fields that are useful for parity assertions: request path, selected headers, and JSON body.

Coverage

test_codex_executor_parity.py covers executor-observable turn behavior:

• sdk/python/tests/test_app_server_run.py
  • mock Responses request path/model/input
  • explicit token usage crossing the app-server boundary
  • last unknown-phase message selection
  • final-answer phase preference
  • commentary-only output not becoming the final response
  • failed Responses events surfacing as turn errors
• sdk/python/tests/test_app_server_streaming.py
  • text delta routing and completed-turn response
• selected request-routing behavior from codex-rs/core/tests/suite/*
  • dynamic tool call/result round trip through real Codex app-server

test_codex_goal.py covers the Codex goal contract Omnigent relies on:

• upstream app-server goal operations
  • thread/goal/set + thread/goal/get + thread/goal/clear round trip
  • pause/resume through thread/goal/set status-only updates
  • explicit tokenBudget: null preservation
  • idempotent thread/goal/clear
  • budgetLimited preservation when setting the same objective
  • persisted blocked and usageLimited goal statuses
• Omnigent AP goal routes
  • PUT /v1/sessions/{id}/codex_goal forwards objective, budget, and mode
  • PATCH /v1/sessions/{id}/codex_goal/status forwards pause/resume
  • Codex-owned terminal statuses are rejected as user-writeable inputs
  • Codex-owned terminal statuses returned by the runner are preserved
  • API-shaped misses return JSON 404s instead of the SPA shell

That is comprehensive for the goal surface Omnigent owns because it exercises both sides of the integration: real Codex app-server JSON-RPC for every goal state transition we depend on, and Omnigent's public HTTP route mapping for every browser control we expose. It intentionally does not copy Codex TUI slash-menu/status rendering tests; Omnigent does not embed that TUI path. It also does not duplicate Codex's internal goal-extension accounting tests except where the app-server result is part of Omnigent's public contract.

Not yet represented here: upstream SDK-only app-server tests for lifecycle, login, approvals, steer/interrupt, local/remote image input, and skill input. Those APIs do not have a direct Omnigent CodexExecutor surface yet, so they need either executor-facing analogs or a separate SDK compatibility harness before they can be one-for-one parity tests.

Updating From Codex Upstream

The upstream Codex fixture dependency is pinned in tests/codex_parity/sidecar/Cargo.toml:

core_test_support = { git = "https://github.com/openai/codex.git", rev = "..." }

To refresh the harness:

1. Update that rev to the Codex commit you want to validate against.
2. Refresh tests/codex_parity/sidecar/Cargo.lock by building or testing the sidecar.
3. Inspect the pinned Codex checkout under Cargo's git cache, usually ~/.cargo/git/checkouts/codex-*/<rev>/.
4. Compare these upstream files against the local parity files:
   • sdk/python/tests/test_app_server_run.py
   • sdk/python/tests/test_app_server_streaming.py
   • sdk/python/tests/test_app_server_goal_operations.py
   • sdk/python/tests/test_client_rpc_methods.py
   • codex-rs/app-server/tests/suite/v2/thread_resume.rs
   • codex-rs/ext/goal/tests/goal_extension_backend.rs
   • codex-rs/prompts/src/goals_tests.rs
5. Port new app-server public-contract goal cases into tests/codex_parity/test_codex_goal.py. Keep TUI-only cases classified as intentionally excluded unless Omnigent starts exposing that path.
6. Run the focused goal file, then the full parity suite:

pytest tests/codex_parity/test_codex_goal.py \
  --codex-parity \
  --codex-bin "$(which codex)" \
  -q

pytest tests/codex_parity \
  --codex-parity \
  --codex-bin "$(which codex)" \
  -q

Running

Run against the Codex CLI on PATH:

pytest tests/codex_parity --codex-parity -v

Run against one explicit binary:

pytest tests/codex_parity --codex-parity --codex-bin "$(which codex)" -v

Compare multiple Codex versions:

pytest tests/codex_parity \
  --codex-parity \
  --codex-bin /path/to/codex-old \
  --codex-bin /path/to/codex-new \
  -v

You can also set CODEX_TEST_BINS to an os.pathsep-separated list.

At Databricks, use the internal PyPI proxy when syncing the Python test environment:

uv --no-config run --frozen \
  --default-index https://pypi-proxy.cloud.databricks.com/simple/ \
  --extra dev \
  pytest tests/codex_parity --codex-parity --codex-bin "$(which codex)" -q

Why This Shape

Mocking the Omnigent-to-Codex API would test our assumptions about Codex's app-server protocol. This suite instead lets Codex define that contract by running the actual CLI/app-server implementation. Only the final network hop is mocked, which gives us stable, deterministic tests while still catching protocol drift between Omnigent and Codex.