mirrors/openai--openai-cookbook
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Rebuild Cookbook Website / deploy (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:41:49 +08:00
commit 327604cc89
3065 changed files with 860207 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/codex/Autofix-github-actions.ipynb
+200
View File
@@ -0,0 +1,200 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "e2884696",
"metadata": {},
"source": [
"# Autofix CI failures on GitHub with Codex CLI\n",
"\n",
"## Purpose of this cookbook\n",
"\n",
"This cookbook shows you how to embed the OpenAI Codex CLI into your CI/CD pipeline so that when your builds or tests fail, codex automatically generates & proposes fixes. The following is an example in a node project with CI running in GitHub Actions. \n",
"\n",
"## End to End Flow\n",
"\n",
"Below is the pipeline flow we’ll implement:\n",
"\n",
"\n",
"<img src=\"../../images/ci-codex-workflow.png\" width=\"700\"/>"
]
},
{
"cell_type": "markdown",
"id": "f83ce964",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"- A GitHub Repo with Actions workflows\n",
"\n",
"- You’ll need to create `OPENAI_API_KEY` as an environment variable in GitHub settings under https://github.com/{org-name}/{repo-name}/settings/secrets/actions. You can also set this at org level(for sharing secrets across multiple repos) \n",
"\n",
"- Codex requires python as a prerequisite to use `codex login`\n",
"\n",
"- You’ll need to check the setting to enable actions to create PRs on your repo, and also in your organization:\n",
"\n",
"\n",
"<img src=\"../../images/github-pr-settings.png\" width=\"700\"/>\n"
]
},
{
"cell_type": "markdown",
"id": "99f5bed1",
"metadata": {},
"source": [
"\n",
"## Step 1: Add the Github Action to your CI Pipeline\n",
"\n",
"The following YAML shows a GitHub action that auto triggers when CI fails, installs Codex, uses codex exec and then makes a PR on the failing branch with the fix. Replace \"CI\" with the name of the workflow you want to monitor. "
]
},
{
"cell_type": "markdown",
"id": "a9f9b368",
"metadata": {},
"source": [
"```yaml\n",
"name: Codex Auto-Fix on Failure\n",
"\n",
"on:\n",
" workflow_run:\n",
" # Trigger this job after any run of the primary CI workflow completes\n",
" workflows: [\"CI\"]\n",
" types: [completed]\n",
"\n",
"permissions:\n",
" contents: write\n",
" pull-requests: write\n",
"\n",
"jobs:\n",
" auto-fix:\n",
" # Only run when the referenced workflow concluded with a failure\n",
" if: ${{ github.event.workflow_run.conclusion == 'failure' }}\n",
" runs-on: ubuntu-latest\n",
" env:\n",
" OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n",
" FAILED_WORKFLOW_NAME: ${{ github.event.workflow_run.name }}\n",
" FAILED_RUN_URL: ${{ github.event.workflow_run.html_url }}\n",
" FAILED_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}\n",
" FAILED_HEAD_SHA: ${{ github.event.workflow_run.head_sha }}\n",
" steps:\n",
" - name: Check OpenAI API Key Set\n",
" run: |\n",
" if [ -z \"$OPENAI_API_KEY\" ]; then\n",
" echo \"OPENAI_API_KEY secret is not set. Skipping auto-fix.\" >&2\n",
" exit 1\n",
" fi\n",
" - name: Checkout Failing Ref\n",
" uses: actions/checkout@v4\n",
" with:\n",
" ref: ${{ env.FAILED_HEAD_SHA }}\n",
" fetch-depth: 0\n",
"\n",
" - name: Setup Node.js\n",
" uses: actions/setup-node@v4\n",
" with:\n",
" node-version: '20'\n",
" cache: 'npm'\n",
"\n",
" - name: Install dependencies\n",
" run: |\n",
" if [ -f package-lock.json ]; then npm ci; else npm i; fi\n",
" - name: Run Codex\n",
" uses: openai/codex-action@main\n",
" id: codex\n",
" with:\n",
" openai_api_key: ${{ secrets.OPENAI_API_KEY }}\n",
" prompt: \"You are working in a Node.js monorepo with Jest tests and GitHub Actions. Read the repository, run the test suite, identify the minimal change needed to make all tests pass, implement only that change, and stop. Do not refactor unrelated code or files. Keep changes small and surgical.\"\n",
" codex_args: '[\"--config\",\"sandbox_mode=\\\"workspace-write\\\"\"]'\n",
"\n",
" - name: Verify tests\n",
" run: npm test --silent\n",
"\n",
" - name: Create pull request with fixes\n",
" if: success()\n",
" uses: peter-evans/create-pull-request@v6\n",
" with:\n",
" commit-message: \"fix(ci): auto-fix failing tests via Codex\"\n",
" branch: codex/auto-fix-${{ github.event.workflow_run.run_id }}\n",
" base: ${{ env.FAILED_HEAD_BRANCH }}\n",
" title: \"Auto-fix failing CI via Codex\"\n",
" body: |\n",
" Codex automatically generated this PR in response to a CI failure on workflow `${{ env.FAILED_WORKFLOW_NAME }}`.\n",
" Failed run: ${{ env.FAILED_RUN_URL }}\n",
" Head branch: `${{ env.FAILED_HEAD_BRANCH }}`\n",
" This PR contains minimal changes intended solely to make the CI pass.\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "8148024b",
"metadata": {},
"source": [
"## Step 2: Actions Workflow kicked off\n",
"\n",
"You can navigate to the Actions tab under Repo to view the failing jobs in your Actions workflow. \n",
"\n",
"<img src=\"../../images/failing-workflow.png\" width=\"700\"/>\n"
]
},
{
"cell_type": "markdown",
"id": "64671aae",
"metadata": {},
"source": [
"The Codex workflow should be triggered upon completion of the failed workflow. \n",
"\n",
"\n",
"<img src=\"../../images/codex-workflow.png\" width=\"700\"/>\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "d08a3ecc",
"metadata": {},
"source": [
"## Step 3: Verify that Codex Created a PR for Review\n",
"And after the Codex workflow completes execution, it should open a pull request from the feature branch codex/auto-fix. Check to see if everything looks good and then merge it.\n",
"\n",
"<img src=\"../../images/codex-pr.png\" width=\"700\"/>\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "f4c1f3a0",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"This automation seamlessly integrates OpenAI Codex CLI with GitHub Actions to automatically propose fixes for failing CI runs.\n",
"\n",
"By leveraging Codex, you can reduce manual intervention, accelerate code reviews, and keep your main branch healthy. The workflow ensures that test failures are addressed quickly and efficiently, letting developers focus on higher-value tasks. Explore more about codex-cli and its capabilities [here](https://github.com/openai/codex/)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/codex/Build_iterative_repair_loops_with_Codex.ipynb
+979
View File
@@ -0,0 +1,979 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "md-001",
"metadata": {},
"source": [
"# Build iterative repair loops with Codex\n",
"\n",
"This cookbook is about closed-loop agent workflows: agents that produce an output, validate it, and use the feedback to improve the next pass.\n",
"\n",
"We'll explore a documentation reliability workflow that detects, repairs, and validates stale or broken API and SDK examples. The worked example uses intentionally stale notebooks adapted from this Cookbook repository.\n",
"\n",
"We'll build this agent loop with Codex. Codex reviews the current state, applies focused changes, runs validation, and repeats when the feedback shows remaining issues.\n",
"\n",
"The notebook task is only the example. The pattern applies wherever agent output can be measured with trustworthy feedback.\n",
"\n",
"The workflow has three phases:\n",
"\n",
"- **Review:** inspect the current artifact and return structured findings without editing files.\n",
"- **Repair:** apply focused edits to a copied artifact using the findings and the latest validation feedback.\n",
"- **Validate:** run the relevant checks and report what still needs work.\n",
"\n",
"Validation closes the loop. The repaired notebook has to satisfy the checks that matter, and any remaining issues become the next repair input.\n"
]
},
{
"cell_type": "markdown",
"id": "md-002",
"metadata": {},
"source": [
"<p align=\"center\">\n",
" <img src=\"../../images/codex_iterative_repair_loop.png\" alt=\"Codex iterative repair loop for technical documentation\" width=\"350\"/>\n",
"</p>\n"
]
},
{
"cell_type": "markdown",
"id": "md-003",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"This notebook uses [Codex CLI](https://developers.openai.com/codex/cli) in headless mode, so the repair steps can run from Python cells instead of a chat UI. The first code cell installs the CLI; if you already have it, you can skip that cell.\n",
"\n",
"Before you run the live repair loop, set `OPENAI_API_KEY` in your environment.\n",
"\n",
"The notebook defaults to a fast repair model so the full example can finish in a reasonable amount of time. To experiment with a different model, set `REPAIR_MODEL` before you start. The install cell pins a known Codex CLI version for reproducibility; update that version intentionally when you want newer CLI behavior.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "code-008",
"metadata": {},
"outputs": [],
"source": [
"!npm install -g @openai/codex@0.130.0"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "code-009",
"metadata": {},
"outputs": [],
"source": [
"import concurrent.futures\n",
"import json\n",
"import os\n",
"import shlex\n",
"import shutil\n",
"import subprocess\n",
"import tempfile\n",
"from pathlib import Path\n",
"from typing import Any\n",
"\n",
"CANDIDATE_EXAMPLE_DIRS = [Path(\".\"), Path(\"examples/codex\")]\n",
"EXAMPLE_DIR = next((base for base in CANDIDATE_EXAMPLE_DIRS if (base / \"data\" / \"docs\").exists()), None)\n",
"\n",
"if EXAMPLE_DIR is None:\n",
" raise RuntimeError(\n",
" \"This notebook needs its companion sample notebooks. \"\n",
" \"Download the data folder that ships with this example and place it next to \"\n",
" \"this notebook as ./data/docs, or run from a checkout where examples/codex/data/docs exists.\"\n",
" )\n",
"\n",
"DATA_DIR = EXAMPLE_DIR / \"data\" / \"docs\"\n",
"DEFAULT_RUNS_DIR = Path(tempfile.gettempdir()) / \"codex_iterative_repair_loop_outputs\"\n",
"RUNS_DIR = Path(os.getenv(\"CODEX_REPAIR_RUNS_DIR\", str(DEFAULT_RUNS_DIR))).expanduser()\n",
"RUNS_DIR.mkdir(parents=True, exist_ok=True)\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "code-010",
"metadata": {},
"outputs": [],
"source": [
"MODEL = os.getenv(\"REPAIR_MODEL\", \"gpt-5.4-mini\")\n",
"COOKBOOK_CHAT_MODEL = os.getenv(\"COOKBOOK_CHAT_MODEL\", \"gpt-5.5\")\n",
"REPAIR_REASONING_EFFORT = os.getenv(\"REPAIR_REASONING_EFFORT\", \"low\")\n",
"\n",
"if not os.environ.get(\"OPENAI_API_KEY\"):\n",
" raise ValueError(\"Set the OPENAI_API_KEY environment variable before running the live Codex repair loop.\")\n",
"\n",
"CODEX_CLI = shutil.which(\"codex\")\n",
"if CODEX_CLI is None:\n",
" raise RuntimeError(\"Run the install cell before continuing; Codex CLI is not on PATH.\")\n"
]
},
{
"cell_type": "markdown",
"id": "md-012",
"metadata": {},
"source": [
"## Load the sample artifacts\n",
"\n",
"The cells below load the three companion notebooks and summarize the metadata that drives the repair loop.\n",
"\n",
"The samples are small on purpose. They run quickly, but they still exercise the architecture: review finds substantive issues, repair makes focused edits, and validation produces feedback for the next pass.\n",
"\n",
"If you download this notebook by itself, also download the companion `data/docs/` folder and place it next to the notebook before running the cells below. The code expects those sample notebooks to be available locally.\n",
"\n",
"In this example, validation executes each repaired notebook end to end. In another domain, validation might be a unit test, policy check, schema validator, simulation, or human approval step. The important part is that failures become structured feedback instead of a dead end.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "code-013",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'notebook': 'qdrant_embeddings_search_pre_repair.ipynb',\n",
" 'cells': 5,\n",
" 'code_cells': 4,\n",
" 'source': 'examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb',\n",
" 'target_iteration': 1,\n",
" 'repair_depth': 'One-pass cleanup: modernize the local Qdrant query path and clarify the sampled fixture framing.'},\n",
" {'notebook': 'getting_started_evals_pre_repair.ipynb',\n",
" 'cells': 5,\n",
" 'code_cells': 4,\n",
" 'source': 'examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb',\n",
" 'target_iteration': 2,\n",
" 'repair_depth': 'Two-pass cleanup: first modernize the obvious stale Evals flow, then use validation feedback to remove result-log brittleness.'},\n",
" {'notebook': 'knowledge_retrieval_pre_repair.ipynb',\n",
" 'cells': 5,\n",
" 'code_cells': 4,\n",
" 'source': 'examples/How_to_call_functions_for_knowledge_retrieval.ipynb',\n",
" 'target_iteration': 3,\n",
" 'repair_depth': 'Three-pass cleanup: modernize model/API shape, then tighten runnable local setup, then restore the full retrieval teaching flow.'}]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"NOTEBOOKS = [\n",
" DATA_DIR / \"qdrant_embeddings_search_pre_repair.ipynb\",\n",
" DATA_DIR / \"getting_started_evals_pre_repair.ipynb\",\n",
" DATA_DIR / \"knowledge_retrieval_pre_repair.ipynb\",\n",
"]\n",
"\n",
"\n",
"def read_notebook(path: Path) -> dict[str, Any]:\n",
" return json.loads(path.read_text(encoding=\"utf-8\"))\n",
"\n",
"\n",
"def case_metadata(path: Path) -> dict[str, Any]:\n",
" return read_notebook(path).get(\"metadata\", {}).get(\"codex_case_study\", {})\n",
"\n",
"\n",
"cases = []\n",
"for notebook_path in NOTEBOOKS:\n",
" notebook = read_notebook(notebook_path)\n",
" metadata = notebook.get(\"metadata\", {}).get(\"codex_case_study\", {})\n",
" repair_story = metadata.get(\"repair_story\", {})\n",
" cases.append(\n",
" {\n",
" \"notebook\": notebook_path.name,\n",
" \"cells\": len(notebook[\"cells\"]),\n",
" \"code_cells\": sum(cell[\"cell_type\"] == \"code\" for cell in notebook[\"cells\"]),\n",
" \"source\": metadata.get(\"source_path\"),\n",
" \"target_iteration\": repair_story.get(\"target_iteration\"),\n",
" \"repair_depth\": repair_story.get(\"repair_depth\", \"\"),\n",
" }\n",
" )\n",
"\n",
"cases\n"
]
},
{
"cell_type": "markdown",
"id": "md-016",
"metadata": {},
"source": [
"## Define business rules and issue taxonomy\n",
"\n",
"Before asking Codex to review or repair an artifact, give it a small shared contract. That keeps the loop focused on the issues that matter, instead of asking the model to infer every product and style rule from scratch.\n",
"\n",
"The rules below define what \"good\" means for these example notebooks: current API patterns, clear setup, runnable local samples, and preservation of the original teaching goal. In another workflow, this contract would describe that domain's source of truth.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "code-017",
"metadata": {},
"outputs": [],
"source": [
"business_rules = {\n",
" \"preferred_chat_model\": COOKBOOK_CHAT_MODEL,\n",
" \"preferred_embedding_model\": \"text-embedding-3-large\",\n",
" \"modernize\": [\n",
" \"client.chat.completions.create -> client.responses.create\",\n",
" \"legacy function-calling schemas -> current tools schema\",\n",
" \"qdrant.search -> qdrant.query_points\",\n",
" \"oaieval CLI examples -> current Evals API workflow\",\n",
" ],\n",
" \"reader_experience\": [\n",
" \"Make fresh-environment setup explicit.\",\n",
" \"Keep the included examples runnable with local data and the standard library.\",\n",
" \"Keep sample repairs self-contained unless the notebook explicitly teaches external setup.\",\n",
" \"Remove manual result-file placeholders.\",\n",
" \"State runtime prerequisites and side effects before readers run cells.\",\n",
" \"Preserve the original teaching goal while modernizing the implementation.\",\n",
" ],\n",
"}\n",
"\n",
"business_rules\n"
]
},
{
"cell_type": "markdown",
"id": "md-018",
"metadata": {},
"source": [
"## Define structured outputs\n",
"\n",
"Each phase returns structured data so the next phase has something concrete to use.\n",
"\n",
"Review returns findings. Repair returns a change summary and the path to the updated artifact. Validation returns the remaining delta for the next pass. With structured handoffs, the loop is easier to debug, rerun, and adapt to other artifact types.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "code-019",
"metadata": {},
"outputs": [],
"source": [
"def object_schema(properties: dict[str, Any], required: list[str] | None = None) -> dict[str, Any]:\n",
" return {\n",
" \"type\": \"object\",\n",
" \"properties\": properties,\n",
" \"required\": required or list(properties),\n",
" \"additionalProperties\": False,\n",
" }\n",
"\n",
"\n",
"def string_array() -> dict[str, Any]:\n",
" return {\"type\": \"array\", \"items\": {\"type\": \"string\"}}\n",
"\n",
"\n",
"finding_schema = object_schema(\n",
" {\n",
" \"artifact\": {\"type\": \"string\"},\n",
" \"issue_type\": {\"type\": \"string\"},\n",
" \"severity\": {\"type\": \"string\"},\n",
" \"description\": {\"type\": \"string\"},\n",
" \"suggested_fix_direction\": {\"type\": \"string\"},\n",
" }\n",
")\n",
"\n",
"review_schema = object_schema(\n",
" {\"findings\": {\"type\": \"array\", \"items\": finding_schema}}\n",
")\n",
"\n",
"fix_schema = object_schema(\n",
" {\n",
" \"artifact\": {\"type\": \"string\"},\n",
" \"iteration\": {\"type\": \"integer\"},\n",
" \"changes_made\": string_array(),\n",
" \"unresolved_items\": string_array(),\n",
" \"updated_artifact_path\": {\"type\": \"string\"},\n",
" }\n",
")\n",
"\n",
"validation_case_schema = object_schema(\n",
" {\n",
" \"name\": {\"type\": \"string\"},\n",
" \"passed\": {\"type\": \"boolean\"},\n",
" \"severity\": {\"type\": \"string\"},\n",
" \"evidence\": {\"type\": \"string\"},\n",
" \"feedback\": {\"type\": \"string\"},\n",
" }\n",
")\n",
"\n",
"validation_schema = object_schema(\n",
" {\n",
" \"overall_passed\": {\"type\": \"boolean\"},\n",
" \"cases\": {\"type\": \"array\", \"items\": validation_case_schema},\n",
" \"remaining_delta\": string_array(),\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"id": "md-020",
"metadata": {},
"source": [
"## Review phase\n",
"\n",
"The review phase reads the artifact and returns structured findings. It does not run validation and it does not edit files. That separation keeps the first step focused: identify likely problems before changing anything.\n",
"\n",
"We send the review prompt to `codex exec` with a JSON schema. The schema keeps the result machine-readable, so later cells can pass findings directly into the repair prompt instead of scraping prose from a previous answer.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "code-021",
"metadata": {},
"outputs": [],
"source": [
"def notebook_text(path: Path, max_chars: int = 7000) -> str:\n",
" chunks = []\n",
" for index, cell in enumerate(read_notebook(path)[\"cells\"]):\n",
" source = \"\".join(cell.get(\"source\", []))\n",
" chunks.append(f\"cell {index} ({cell['cell_type']})\\n{source}\")\n",
" text = \"\\n\\n\".join(chunks)\n",
" if len(text) <= max_chars:\n",
" return text\n",
" return text[:max_chars] + \"\\n\\n[truncated for prompt size]\"\n",
"\n",
"\n",
"def run_command(command: str, *, stdin: str | None = None, cwd: Path | None = None, timeout: int | None = None):\n",
" cwd = Path.cwd() if cwd is None else cwd\n",
" return subprocess.run(\n",
" shlex.split(command),\n",
" input=stdin,\n",
" cwd=cwd,\n",
" capture_output=True,\n",
" text=True,\n",
" timeout=timeout,\n",
" check=False,\n",
" )\n",
"\n",
"\n",
"def run_codex_json(prompt: str, schema: dict[str, Any], run_dir: Path) -> dict[str, Any]:\n",
" run_dir.mkdir(parents=True, exist_ok=True)\n",
" prompt_file = run_dir / \"prompt.txt\"\n",
" schema_file = run_dir / \"schema.json\"\n",
" answer_file = run_dir / \"answer.json\"\n",
"\n",
" prompt_file.write_text(prompt, encoding=\"utf-8\")\n",
" schema_file.write_text(json.dumps(schema, indent=2), encoding=\"utf-8\")\n",
"\n",
" command = f\"\"\"\n",
" {CODEX_CLI} exec\n",
" --model {MODEL}\n",
" --sandbox workspace-write\n",
" --ask-for-approval never\n",
" --config model_reasoning_effort={REPAIR_REASONING_EFFORT}\n",
" --output-schema {schema_file}\n",
" --output-last-message {answer_file}\n",
" -\n",
" \"\"\"\n",
" result = run_command(command, stdin=prompt)\n",
" (run_dir / \"stdout.txt\").write_text(result.stdout, encoding=\"utf-8\")\n",
" (run_dir / \"stderr.txt\").write_text(result.stderr, encoding=\"utf-8\")\n",
"\n",
" if result.returncode != 0:\n",
" raise RuntimeError(f\"Codex exited with {result.returncode}. See {run_dir / 'stderr.txt'}.\")\n",
"\n",
" return json.loads(answer_file.read_text(encoding=\"utf-8\"))\n",
"\n",
"\n",
"def review_notebook(path: Path, run_dir: Path) -> list[dict[str, Any]]:\n",
" prompt = \"\\n\".join(\n",
" [\n",
" \"You are reviewing a public OpenAI Cookbook notebook before publication.\",\n",
" f\"Artifact: {path.name}\",\n",
" \"Find issues that would make the notebook stale, hard to run, or confusing for a developer reader.\",\n",
" \"Do not execute the notebook or edit files.\",\n",
" \"Use concise issue_type labels such as stale_model, deprecated_api, setup_gap, runtime_risk, or clarity_issue.\",\n",
" f\"Business rules: {json.dumps(business_rules)}\",\n",
" \"Base findings only on the notebook content below.\",\n",
" \"Keep the findings focused; three strong findings are better than a long list.\",\n",
" \"\",\n",
" notebook_text(path),\n",
" ]\n",
" )\n",
" return run_codex_json(prompt, review_schema, run_dir)[\"findings\"]\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "code-022",
"metadata": {},
"outputs": [],
"source": [
"def run_initial_review(path: Path) -> tuple[str, list[dict[str, Any]]]:\n",
" return path.name, review_notebook(path, RUNS_DIR / \"initial_review\" / path.stem)\n",
"\n",
"\n",
"with concurrent.futures.ThreadPoolExecutor(max_workers=min(3, len(NOTEBOOKS))) as executor:\n",
" initial_reviews = dict(executor.map(run_initial_review, NOTEBOOKS))\n",
"\n",
"initial_reviews\n"
]
},
{
"cell_type": "markdown",
"id": "md-023",
"metadata": {},
"source": [
"## Repair phase\n",
"\n",
"The repair phase gets the current artifact, review findings, business rules, and any validation feedback from the previous pass. The prompt gets more specific as the loop learns.\n",
"\n",
"Codex edits a copy inside the iteration directory and returns a short summary of what changed. The loop does not assume the edit worked; validation decides that in the next step.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "code-024",
"metadata": {},
"outputs": [],
"source": [
"def repair_prompt(path: Path, updated_path: Path, findings: list[dict[str, Any]], remaining_delta: list[str], iteration: int) -> str:\n",
" repair_story = case_metadata(path).get(\"repair_story\", {})\n",
" return \"\\n\".join(\n",
" [\n",
" \"You are repairing a copy of a public OpenAI Cookbook notebook.\",\n",
" f\"Source notebook: {path}\",\n",
" f\"Editable copy: {updated_path}\",\n",
" f\"Iteration: {iteration}\",\n",
" \"Make the smallest useful edits that address the review findings and validation delta.\",\n",
" \"Preserve the notebook's teaching flow and original purpose.\",\n",
" \"Keep sample repairs self-contained unless the notebook explicitly teaches external setup.\",\n",
" \"For staged examples, focus on the most important remaining issue for this pass instead of rewriting everything at once.\",\n",
" \"Edit only the editable copy. Do not claim the notebook passes validation.\",\n",
" f\"Repair depth: {json.dumps(repair_story, indent=2)}\",\n",
" f\"Business rules: {json.dumps(business_rules, indent=2)}\",\n",
" f\"Review findings: {json.dumps(findings, indent=2)}\",\n",
" f\"Remaining validation delta: {json.dumps(remaining_delta, indent=2)}\",\n",
" ]\n",
" )\n",
"\n",
"\n",
"def repair_notebook(path: Path, iteration: int, findings: list[dict[str, Any]], remaining_delta: list[str], case_dir: Path) -> dict[str, Any]:\n",
" updated_path = case_dir / \"updated.ipynb\"\n",
" updated_path.parent.mkdir(parents=True, exist_ok=True)\n",
" shutil.copy2(path, updated_path)\n",
"\n",
" prompt = repair_prompt(path, updated_path, findings, remaining_delta, iteration)\n",
" return run_codex_json(prompt, fix_schema, case_dir / \"repair\")\n"
]
},
{
"cell_type": "markdown",
"id": "md-025",
"metadata": {},
"source": [
"## Validation phase\n",
"\n",
"Validation works like a small eval. We define the behavior we want, run the relevant check, and ask a judge to score the result against that rubric.\n",
"\n",
"For the documentation example, execution comes first. Many notebook problems only appear at runtime: a missing import, a stale file path, a cell that depends on an old API response, or setup guidance that was clear to the author but not to a fresh reader.\n",
"\n",
"If validation fails, the failure becomes evidence for the next repair pass. This keeps the next repair grounded in observed behavior, not just what looked right in the diff.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "code-026",
"metadata": {},
"outputs": [],
"source": [
"VALIDATION_CASES = [\n",
" {\n",
" \"name\": \"api_modernization\",\n",
" \"question\": \"Does the notebook avoid stale OpenAI API patterns, legacy function-calling syntax, and outdated model names?\",\n",
" },\n",
" {\n",
" \"name\": \"setup_reproducibility\",\n",
" \"question\": \"Could a reader run the notebook from a fresh environment without hidden manual steps?\",\n",
" },\n",
" {\n",
" \"name\": \"artifact_integrity\",\n",
" \"question\": \"Did the update preserve the notebook's teaching flow and avoid deleting substantive cells?\",\n",
" },\n",
"]\n",
"\n",
"\n",
"def short_output(value: Any, limit: int = 1200) -> str:\n",
" if value is None:\n",
" return \"\"\n",
" if isinstance(value, bytes):\n",
" value = value.decode(\"utf-8\", errors=\"replace\")\n",
" return str(value)[-limit:]\n",
"\n",
"\n",
"def execute_notebook(path: Path) -> dict[str, Any]:\n",
" code_cells = sum(cell[\"cell_type\"] == \"code\" for cell in read_notebook(path)[\"cells\"])\n",
" command = f\"jupyter nbconvert --to notebook --execute --inplace {path.name}\"\n",
"\n",
" try:\n",
" result = run_command(\n",
" command,\n",
" cwd=path.parent,\n",
" timeout=int(os.getenv(\"SAMPLE_NOTEBOOK_TIMEOUT_SECONDS\", \"300\")),\n",
" )\n",
" except FileNotFoundError:\n",
" return {\n",
" \"status\": \"failed\",\n",
" \"executed_code_cells\": 0,\n",
" \"error\": \"Jupyter or nbconvert is not installed or is not available on PATH.\",\n",
" \"summary\": \"Install Jupyter with nbconvert before running the validation loop.\",\n",
" }\n",
" except subprocess.TimeoutExpired as exc:\n",
" return {\n",
" \"status\": \"failed\",\n",
" \"executed_code_cells\": 0,\n",
" \"error\": f\"Notebook execution timed out after {exc.timeout} seconds.\",\n",
" \"summary\": short_output(exc.stderr or exc.stdout),\n",
" }\n",
"\n",
" output = result.stderr or result.stdout\n",
" return {\n",
" \"status\": \"passed\" if result.returncode == 0 else \"failed\",\n",
" \"executed_code_cells\": code_cells if result.returncode == 0 else 0,\n",
" \"error\": \"\" if result.returncode == 0 else f\"Notebook execution exited with code {result.returncode}.\",\n",
" \"summary\": short_output(output),\n",
" }\n",
"\n",
"def validation_prompt(updated_path: Path, before_path: Path, execution: dict[str, Any], iteration: int) -> str:\n",
" repair_story = case_metadata(before_path).get(\"repair_story\", {})\n",
" return \"\\n\".join(\n",
" [\n",
" \"You are judging a repaired OpenAI Cookbook notebook.\",\n",
" f\"Iteration: {iteration}\",\n",
" \"Score each validation case independently and give concise feedback for the next repair pass.\",\n",
" \"Set overall_passed to false when execution failed or any case has a material issue.\",\n",
" \"When execution failed, include the failure in remaining_delta so the next repair pass can address it.\",\n",
" \"Use the business rules as the source of truth for current model names and API targets.\",\n",
" \"Do not mark the preferred embedding model or preferred chat model as stale.\",\n",
" \"For local examples, do not require extra services or package installs when the notebook says it is intentionally self-contained.\",\n",
" f\"Repair depth: {json.dumps(repair_story, indent=2)}\",\n",
" f\"Business rules: {json.dumps(business_rules, indent=2)}\",\n",
" f\"Validation cases: {json.dumps(VALIDATION_CASES, indent=2)}\",\n",
" f\"Execution evidence: {json.dumps(execution, indent=2)}\",\n",
" f\"Original cell count: {len(read_notebook(before_path)['cells'])}\",\n",
" f\"Updated cell count: {len(read_notebook(updated_path)['cells'])}\",\n",
" \"\",\n",
" notebook_text(updated_path),\n",
" ]\n",
" )\n",
"\n",
"\n",
"def staged_delta(before_path: Path, iteration: int) -> list[str]:\n",
" repair_story = case_metadata(before_path).get(\"repair_story\", {})\n",
" target = int(repair_story.get(\"target_iteration\") or 1)\n",
" if iteration >= target:\n",
" return []\n",
" depth = repair_story.get(\"repair_depth\", \"This case is intentionally staged across multiple repair passes.\")\n",
" return [f\"Continue to iteration {iteration + 1}: {depth}\"]\n",
"\n",
"\n",
"def evaluate_notebook(updated_path: Path, before_path: Path, run_dir: Path, iteration: int) -> dict[str, Any]:\n",
" execution = execute_notebook(updated_path)\n",
" judged = run_codex_json(validation_prompt(updated_path, before_path, execution, iteration), validation_schema, run_dir)\n",
" failed_cases = [case for case in judged[\"cases\"] if not case[\"passed\"]]\n",
" execution_delta = []\n",
" if execution[\"status\"] != \"passed\":\n",
" execution_delta.append(f\"Execution failed: {execution.get('error') or execution.get('summary')}\")\n",
"\n",
" stage_delta = staged_delta(before_path, iteration)\n",
" return {\n",
" \"passed\": judged[\"overall_passed\"] and execution[\"status\"] == \"passed\" and not stage_delta,\n",
" \"execution_status\": execution[\"status\"],\n",
" \"executed_code_cells\": execution[\"executed_code_cells\"],\n",
" \"execution_summary\": execution[\"summary\"],\n",
" \"findings\": failed_cases,\n",
" \"remaining_delta\": execution_delta + stage_delta + judged[\"remaining_delta\"],\n",
" }\n"
]
},
{
"cell_type": "markdown",
"id": "md-027",
"metadata": {},
"source": [
"## Save per-iteration outputs\n",
"\n",
"Each iteration writes a `record.json` file and, for this example, a repaired notebook under `CODEX_REPAIR_RUNS_DIR/iteration_N/<sample_name>/`. If you do not set `CODEX_REPAIR_RUNS_DIR`, the notebook writes to your system temp directory so a normal repo checkout stays clean.\n",
"\n",
"Those files are the audit trail. You can see what the review found, what Codex changed, whether execution passed, and what feedback carried into the next iteration.\n",
"\n",
"A `record.json` file is the receipt for one loop attempt. It keeps the handoff between phases in one place:\n",
"\n",
"```json\n",
"{\n",
" \"review\": [{\"issue_type\": \"deprecated_api\", \"severity\": \"high\"}],\n",
" \"repair\": {\n",
" \"changes_made\": [\"Updated the notebook to use the current API pattern.\"],\n",
" \"updated_artifact_path\": \"/tmp/codex_iterative_repair_loop_outputs/iteration_1/sample/updated.ipynb\"\n",
" },\n",
" \"validation\": {\n",
" \"passed\": false,\n",
" \"remaining_delta\": [\"One setup instruction is still unclear.\"]\n",
" }\n",
"}\n",
"```\n",
"\n",
"That compact record is what lets a maintainer review the loop without reconstructing the whole run from notebook diffs and terminal logs.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "code-028",
"metadata": {},
"outputs": [],
"source": [
"def save_json(payload: Any, path: Path) -> None:\n",
" path.parent.mkdir(parents=True, exist_ok=True)\n",
" path.write_text(json.dumps(payload, indent=2) + \"\\n\", encoding=\"utf-8\")\n",
"\n",
"\n",
"def iteration_dir(number: int) -> Path:\n",
" path = RUNS_DIR / f\"iteration_{number}\"\n",
" path.mkdir(parents=True, exist_ok=True)\n",
" return path\n"
]
},
{
"cell_type": "markdown",
"id": "md-029",
"metadata": {},
"source": [
"## Run iteration 1\n",
"\n",
"Each notebook case is independent, so we process the cases concurrently. This keeps the demo fast while preserving the same review, repair, and validation flow for every sample.\n",
"\n",
"Iteration 1 reuses the initial review findings from the earlier review cell. After this pass, inspect the returned booleans: passing cases can stop, and failing cases carry their validation feedback into the next pass.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "code-030",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'qdrant_embeddings_search_pre_repair.ipynb': True,\n",
" 'getting_started_evals_pre_repair.ipynb': False,\n",
" 'knowledge_retrieval_pre_repair.ipynb': False}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"current_notebooks = {path.name: path for path in NOTEBOOKS}\n",
"history: dict[int, dict[str, Any]] = {}\n",
"\n",
"\n",
"def review_findings_for(original: Path, current_path: Path, case_dir: Path, previous_results: dict[str, Any] | None) -> list[dict[str, Any]]:\n",
" if previous_results is None:\n",
" return initial_reviews[original.name]\n",
" return review_notebook(current_path, case_dir / \"review\")\n",
"\n",
"\n",
"def run_case(number: int, original: Path, run_dir: Path, previous_results: dict[str, Any] | None) -> tuple[str, dict[str, Any], Path]:\n",
" name = original.name\n",
" case_dir = run_dir / original.stem\n",
" current_path = current_notebooks[name]\n",
"\n",
" findings = review_findings_for(original, current_path, case_dir, previous_results)\n",
" delta = [] if previous_results is None else previous_results[name][\"validation\"][\"remaining_delta\"]\n",
" repair = repair_notebook(current_path, number, findings, delta, case_dir)\n",
" updated_path = Path(repair[\"updated_artifact_path\"])\n",
" validation = evaluate_notebook(updated_path, current_path, case_dir / \"evaluation\", number)\n",
"\n",
" record = {\"review\": findings, \"repair\": repair, \"validation\": validation}\n",
" save_json(record, case_dir / \"record.json\")\n",
" return name, record, updated_path\n",
"\n",
"\n",
"def run_iteration(number: int, previous_results: dict[str, Any] | None = None) -> dict[str, Any]:\n",
" results = {}\n",
" updates = {}\n",
" run_dir = iteration_dir(number)\n",
"\n",
" with concurrent.futures.ThreadPoolExecutor(max_workers=min(3, len(NOTEBOOKS))) as executor:\n",
" futures = [executor.submit(run_case, number, original, run_dir, previous_results) for original in NOTEBOOKS]\n",
" for future in concurrent.futures.as_completed(futures):\n",
" name, record, updated_path = future.result()\n",
" results[name] = record\n",
" updates[name] = updated_path\n",
"\n",
" current_notebooks.update(updates)\n",
" history[number] = results\n",
" return results\n",
"\n",
"\n",
"iteration_1 = run_iteration(1)\n",
"{name: result[\"validation\"][\"passed\"] for name, result in iteration_1.items()}\n"
]
},
{
"cell_type": "markdown",
"id": "md-031",
"metadata": {},
"source": [
"## Run iteration 2\n",
"\n",
"Iteration 2 is where the loop starts to pay off. Codex is no longer working only from the original review; it also sees what happened during validation.\n",
"\n",
"That changes the task. Instead of asking for a broad rewrite, we ask for the next useful repair based on evidence from the last run: what executed, what passed, and what still needs attention.\n",
"\n",
"For the included staged fixtures, this pass is designed to clear the medium-depth Evals case while the deeper Knowledge Retrieval case continues with a smaller, more specific delta.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "code-032",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'getting_started_evals_pre_repair.ipynb': True,\n",
" 'qdrant_embeddings_search_pre_repair.ipynb': True,\n",
" 'knowledge_retrieval_pre_repair.ipynb': False}"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iteration_2 = run_iteration(2, iteration_1)\n",
"{name: result[\"validation\"][\"passed\"] for name, result in iteration_2.items()}\n"
]
},
{
"cell_type": "markdown",
"id": "md-033",
"metadata": {},
"source": [
"## Run iteration 3\n",
"\n",
"Iteration 3 focuses on the deepest documentation case.\n",
"\n",
"The Knowledge Retrieval fixture has to modernize the API shape, stay runnable with local data, and preserve the retrieval teaching flow. Those requirements can pull against each other: a repair that makes the notebook modern might accidentally make it less runnable, while a repair that keeps it local might remove too much of the original lesson.\n",
"\n",
"The third pass gives Codex the latest notebook plus the final validation delta. This is the part of the demo that shows why iteration matters: the agent responds to the specific issue that remained, rather than trying to anticipate everything up front.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "code-034",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'qdrant_embeddings_search_pre_repair.ipynb': True,\n",
" 'getting_started_evals_pre_repair.ipynb': True,\n",
" 'knowledge_retrieval_pre_repair.ipynb': True}"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iteration_3 = run_iteration(3, iteration_2)\n",
"{name: result[\"validation\"][\"passed\"] for name, result in iteration_3.items()}\n"
]
},
{
"cell_type": "markdown",
"id": "md-035",
"metadata": {},
"source": [
"## Summarize improvement\n",
"\n",
"Now we can look at the whole run instead of opening every intermediate artifact by hand. The summary below shows the signal that matters most: which artifacts passed, how many validation findings remained, and whether any delta carried forward.\n",
"\n",
"For the included fixtures, the intended shape is simple: one notebook clears in iteration 1, another clears in iteration 2, and the deepest one clears in iteration 3. In a real maintenance workflow, this table tells you whether the loop is converging or needs a clearer constraint or human review.\n",
"\n",
"This summary is also useful for human review. A maintainer can start with the pass/fail pattern, open records for anything that still has a delta, and inspect only the repaired artifacts that are ready for review.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "code-036",
"metadata": {},
"outputs": [],
"source": [
"summary = []\n",
"for iteration, results in history.items():\n",
" for artifact, record in results.items():\n",
" validation = record[\"validation\"]\n",
" summary.append(\n",
" {\n",
" \"iteration\": iteration,\n",
" \"artifact\": artifact,\n",
" \"passed\": validation[\"passed\"],\n",
" \"findings\": len(validation[\"findings\"]),\n",
" \"remaining_delta\": len(validation[\"remaining_delta\"]),\n",
" }\n",
" )\n",
"\n",
"summary\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "code-037",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"iteration=1 artifact=qdrant_embeddings_search_pre_repair.ipynb passed=True findings=0 delta=0\n",
"iteration=1 artifact=getting_started_evals_pre_repair.ipynb passed=False findings=0 delta=1\n",
"iteration=1 artifact=knowledge_retrieval_pre_repair.ipynb passed=False findings=1 delta=3\n",
"iteration=2 artifact=getting_started_evals_pre_repair.ipynb passed=True findings=0 delta=0\n",
"iteration=2 artifact=qdrant_embeddings_search_pre_repair.ipynb passed=True findings=0 delta=0\n",
"iteration=2 artifact=knowledge_retrieval_pre_repair.ipynb passed=False findings=0 delta=1\n",
"iteration=3 artifact=qdrant_embeddings_search_pre_repair.ipynb passed=True findings=0 delta=0\n",
"iteration=3 artifact=getting_started_evals_pre_repair.ipynb passed=True findings=0 delta=0\n",
"iteration=3 artifact=knowledge_retrieval_pre_repair.ipynb passed=True findings=0 delta=0\n"
]
}
],
"source": [
"for row in summary:\n",
" print(\n",
" f\"iteration={row['iteration']} artifact={row['artifact']} \"\n",
" f\"passed={row['passed']} findings={row['findings']} delta={row['remaining_delta']}\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "md-038",
"metadata": {},
"source": [
"## What the summary tells us\n",
"\n",
"The important signal is not that Codex made edits. The important signal is that the remaining validation delta gets smaller as the loop runs.\n",
"\n",
"| Pass | Signal to look for | Why it matters |\n",
"| --- | --- | --- |\n",
"| Iteration 1 | The simplest fixture passes; deeper fixtures keep a small delta. | The loop can make an initial repair while carrying forward the cases that still need evidence. |\n",
"| Iteration 2 | The medium-depth fixture clears after seeing validation feedback. | Runtime and judge feedback become useful repair instructions. |\n",
"| Iteration 3 | The deepest fixture clears or leaves a focused final delta. | The loop converges, or it produces a clear handoff for a human reviewer. |\n",
"\n",
"The `record.json` files are where this becomes auditable. A useful record answers four questions: what did the review find, what did Codex change, did the notebook execute, and what remains? That is the difference between an impressive-looking edit and a repair workflow a maintainer can trust.\n"
]
},
{
"cell_type": "markdown",
"id": "md-039",
"metadata": {},
"source": [
"## Generalize to a continuous loop\n",
"\n",
"The fixed three-pass run above is useful for teaching the pattern. A production loop should decide when to stop on its own.\n",
"\n",
"A good loop usually stops for one of four reasons: validation passes, the loop reaches a maximum number of attempts, the remaining delta stops changing, or the next decision needs human review. Those stop conditions are just as important as the repair prompt.\n",
"\n",
"The other production detail is the audit trail. Keep the review findings, repaired artifact, validation result, validation judgment, and remaining delta for every pass. That record lets a maintainer understand why the loop continued, why it stopped, and which artifact is ready for review.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "code-039",
"metadata": {},
"outputs": [],
"source": [
"def repair_until_done(max_iterations: int = 3) -> dict[int, dict[str, Any]]:\n",
" current_notebooks.update({path.name: path for path in NOTEBOOKS})\n",
" previous = None\n",
" loop_history = {}\n",
"\n",
" for number in range(1, max_iterations + 1):\n",
" previous = run_iteration(number, previous)\n",
" loop_history[number] = previous\n",
" if all(record[\"validation\"][\"passed\"] for record in previous.values()):\n",
" break\n",
"\n",
" return loop_history\n"
]
},
{
"cell_type": "markdown",
"id": "md-040",
"metadata": {},
"source": [
"## Where else this applies\n",
"\n",
"The notebook walkthrough is just one way to teach the architecture. The same pattern helps whenever an agent changes a file or process that needs more than subjective review before it is accepted.\n",
"\n",
"A few high-value examples:\n",
"\n",
"- **Protocol optimization:** Draft an update for expert review, then validate it against dosing rules, timing constraints, or required safety checks.\n",
"- **Regulatory remediation:** Draft updates to regulated content, then check that required language, citations, approvals, and jurisdiction-specific terms remain intact.\n",
"- **Support knowledge refresh:** Update an article, test it against current product behavior or known resolutions, and carry mismatches into the next pass.\n",
"- **Code modernization:** Replace deprecated APIs, run tests or static checks, and use remaining failures to guide the next repair.\n",
"\n",
"The common thread is that the change matters, and each pass needs evidence. Whether the target is a notebook, a policy, a protocol, a support article, a pipeline, or a codebase, the loop gives the agent a way to improve it with evidence a maintainer can review.\n"
]
},
{
"cell_type": "markdown",
"id": "md-041",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"Iterative repair loops make agentic maintenance easier to review and operate because they separate judgment from proof.\n",
"\n",
"Review finds candidate issues. Repair makes focused edits. Validation executes the artifact and produces the next delta. When those phases exchange structured outputs, the workflow becomes easier to inspect, repeat, and adapt.\n",
"\n",
"The main idea is simple: instead of relying on a single pass, give the workflow a way to learn from the artifact, make a bounded repair, and react to real validation feedback. That small change makes agentic maintenance much more practical.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/codex/build_code_review_with_codex_sdk.md
+1121
View File
File diff suppressed because it is too large Load Diff
examples/codex/code_modernization.md
+360
View File
@@ -0,0 +1,360 @@
# Modernizing your Codebase with Codex
## Introduction
Codex is trained to read and reason about large, complex codebases, plan work alongside engineers, and produce high-quality changes. Code modernization has quickly become one of its most common and valuable uses. In this setup, engineers focus on architecture and business rules while Codex handles the heavy lifting: translating legacy patterns, proposing safe refactors, and keeping documentation and tests in sync as the system evolves.
This cookbook shows how to use **OpenAI's Codex CLI** to modernize a legacy repository in a way that is:
* Understandable to new engineers
* Auditable for architects and risk teams
* Repeatable as a pattern across other systems
We’ll use a COBOL-based [investment portfolio system](https://github.com/sentientsergio/COBOL-Legacy-Benchmark-Suite/) as the running example and choose a single pilot flow to focus on. You can substitute any legacy stack (eg. Java monolith, PL/SQL) where you have legacy programs, orchestration (jobs, schedulers, scripts), or shared data sources.
---
## High Level Overview
We’ve broken it down into 5 different phases that revolve around an executive plan (ExecPlan in short), which is a design document that the agent can follow to deliver the system change.
<img src="../../images/code-modernization-phases.png" alt="Code Modernization Phases" width="700"/>
We will create 4 types of documents for the pilot flow we choose:
* **pilot_execplan.md** - ExecPlan that orchestrates the pilot that answers: what’s in scope, why it matters, what steps we’ll take, and how we’ll know we’re done.
* **pilot_overview.md** - Which legacy programs (COBOL in our example), orchestration jobs (JCL here), and data sources are involved, how data flows between them, and what the business flow actually does.
* **pilot_design.md** - Target shape of the system: the service/module that will own this flow, the new data model, and the public APIs or batch entry points.
* **pilot_validation.md** - Defines how we’ll prove parity: key scenarios, shared input datasets, how to run legacy vs modern side-by-side, and what “matching outputs” means in practice.
These 4 files help lay out what code is being changed, what the new system should look like, and exactly how to check that behavior hasn’t regressed.
---
## Phase 0 - Set up AGENTS and PLANS
**Goal**: Give Codex a lightweight contract for how planning works in this repo, without overwhelming people with process.
We’re taking inspiration from the [Using PLANS.md for multi-hour problem solving](https://cookbook.openai.com/articles/codex_exec_plans) cookbook to create an AGENTS.md and PLANS.md file that will be placed in a .agent folder.
* AGENTS.md: If you haven’t created an AGENTS.md for your repository yet, I suggest using the /init command. Once generated, reference the add a section in your AGENTS.md to instruct the agent to reference the PLANS.md.
* PLANS.md: Use the example provided in the cookbook as a starting point
These explain what an ExecPlan is, when to create or update one, where it lives, and what sections every plan must have.
### Where Codex CLI helps
If you want Codex to tighten AGENTS or PLANS for your specific repo, you can run:
```md
Please read the directory structure and refine .agent/AGENTS.md and .agent/PLANS.md so they are a clear, opinionated standard for how we plan COBOL modernization work here. Keep the ExecPlan skeleton but add one or two concrete examples.
```
---
## Phase 1 - Pick a pilot and create the first ExecPlan
**Goal**: Align on one realistic but bounded pilot flow and capture the plan for Phase 1 in a single ExecPlan file.
**Key artifact**: pilot_execplan.md
### 1.1 Choose pilot flow
If you don’t have a flow in mind to pilot with, you can ask Codex to propose. Example prompt from the repository root:
```md
Look through this repository and propose one or two candidate pilot flows for modernization that are realistic but bounded.
For each candidate, list:
- COBOL programs and copybooks involved
- JCL members involved
- The business scenario in plain language
- End with a clear recommendation for which flow we should use as the first pilot
```
In this case, we’ll choose a reporting flow as the pilot.
<img src="../../images/pilot-candidate.png" alt="Pilot Candidate Flow" width="700"/>
### 1.2 Ask Codex to create the pilot ExecPlan
```md
Create pilot_execplan.md following .agent/PLANS.md. Scope it to the daily reporting flow. The plan should cover four outcomes for this one flow:
- Inventory and diagrams
- Modernization Technical Report content
- Target design and spec
- Test plan for parity
Use the ExecPlan skeleton and fill it in with concrete references to the actual COBOL and JCL files.
```
This plan is now your “home base” for all pilot work.
---
## Phase 2 - Inventory and discovery
**Goal**: Capture what the pilot flow actually does today: programs, jobs, data flows, and business rules. Engineers can reason about the change without reading every line of legacy code.
**Key artifact**: pilot_reporting_overview.md
**Where engineers can focus**:
* Confirm which jobs truly run in production
* Fill in gaps Codex cannot infer from code (SLAs, operational context, owners)
* Sanity check diagrams and descriptions
### 2.1 Ask Codex to draft the overview
```md
Create or update pilot_reporting_overview.md with two top-level sections: “Inventory for the pilot” and “Modernization Technical Report for the pilot”.
Use pilot_execplan.md to identify the pilot flow.
In the inventory section, include:
1. The COBOL programs and copybooks involved, grouped as batch, online, and utilities if applicable
2. The JCL jobs and steps that call these programs
3. The data sets or tables they read and write
4. A simple text diagram that shows the sequence of jobs and data flows
In the modernization technical report section, describe:
1. The business scenario for this flow in plain language
2. Detailed behavior of each COBOL program in the flow
3. The data model for the key files and tables, including field names and meanings
4. Known technical risks such as date handling, rounding, special error codes, or tricky conditions
```
This document will be helpful for engineers to understand the shape and behavior of the pilot without reading all the code.
Example of the flow diagram in pilot_reporting_overview.md
<img src="../../images/pilot-flow-diagram.png" alt="Pilot Flow Diagram" width="700"/>
### 2.2 Update the ExecPlan
Once the overview exists, ask Codex to keep the plan aligned
```md
Update pilot_execplan.md to reflect the new pilot_reporting_overview.md file.
- In Progress, mark the inventory and MTR sections as drafted.
- Add any notable findings to Surprises and discoveries and Decision log.
- Keep the ExecPlan readable for someone new to the repo.
```
At the end of Phase 2, you’ll have a single pilot overview doc that plays the role of both system inventory report and modernization technical report.
---
## Phase 3 - Design, spec, and validation plan
**Goal**
* Decide what the modern version of the pilot flow should look like
* Describe the target service and data model
* Define how to prove parity through tests and parallel runs.
By the end of this phase, we’ll have decided what we’re building and how we’ll prove it works.
**Key artifacts**
* pilot_reporting_design.md
* pilot_reporting_validation.md
* modern/openapi/pilot.yaml
* modern/tests/pilot_parity_test.py
### 3.1 Target design document
```md
Based on pilot_reporting_overview.md, draft pilot_reporting_design.md with these sections:
# Target service design
- Which service or module will own this pilot flow in the modern architecture.
- Whether it will be implemented as a batch job, REST API, event listener, or a combination.
- How it fits into the broader domain model.
# Target data model
- Proposed database tables and columns that replace the current files or DB2 tables.
- Keys, relationships, and any derived fields.
- Notes about how legacy encodings such as packed decimals or EBCDIC fields will be represented.
# API design overview
- The main operations users or systems will call.
- A short description of each endpoint or event.
- A pointer to modern/openapi/pilot.yaml where the full schema will live.
```
### 3.2 API specification
We capture the pilot flow’s external behavior in an OpenAPI file so the modern system has a clear, language-agnostic contract. This spec becomes the anchor for implementation, test generation, and future integrations, and it gives Codex something concrete to scaffold code and tests from.
```md
Using pilot_reporting_design.md, draft an OpenAPI file at modern/openapi/pilot.yaml that describes the external API for this pilot. Include:
- Paths and operations for the main endpoints or admin hooks
- Request and response schemas for each operation
- Field types and constraints, aligning with the target data model
```
Example output:
<img src="../../images/pilot-yaml.png" alt="Pilot Yaml" width="700"/>
### 3.3 Validation and test plan
```md
Create or update pilot_reporting_validation.md with three sections:
# Test plan
- Key scenarios, including at least one happy path and a couple of edge cases.
- Inputs and outputs to capture for each scenario.
# Parity and comparison strategy
- How you will run the legacy COBOL flow and the modern implementation on the same input data.
- What outputs will be compared (files, tables, logs).
- How differences will be detected and triaged.
# Test scaffolding
- Notes about the test file modern/tests/pilot_parity_test.py, including how to run it.
- What needs to be filled in once the modern implementation exists.
```
Then ask Codex to scaffold the tests:
```md
Using pilot_reporting_validation.md, create an initial test file at modern/tests/pilot_parity_test.py.
Include placeholder assertions and comments that reference the scenarios in the test plan, but do not assume the modern implementation is present yet.
```
### 3.4 Update the ExecPlan
```md
Update pilot_execplan.md so that Plan of work, Concrete steps, and Validation and acceptance explicitly reference:
1. pilot_reporting_overview.md
2. pilot_reporting_design.md
3. pilot_reporting_validation.md
4. modern/openapi/pilot.yaml
5. modern/tests/pilot_parity_test.py
```
At the end of Phase 3, you’ll have a clear design, a machine readable spec, and a test plan/scaffolding that describes how you will prove parity.
---
## Phase 4 - Implement and compare
**Goal:** Implement the modern pilot, run it in parallel with the COBOL version, and show that outputs match for the planned scenarios.
**Key artifacts**
* Code under modern/<stack>/pilot (for example modern/java/pilot)
* Completed tests in modern/tests/pilot_parity_test.py
* Updated sections in pilot_reporting_validation.md that describe the actual parallel run steps
### 4.1 Generate a first draft of the modern code
```md
Using pilot_reporting_design.md and the COBOL programs listed in pilot_reporting_overview.md, generate initial implementation code under modern/<stack>/pilot that:
- Defines domain models and database entities for the key records and tables.
- Implements the core business logic in service classes, preserving behavior from COBOL paragraphs.
- Adds comments that reference the original COBOL paragraphs and copybooks.
- Treat this as a first draft for engineers to review.
```
You can run this several times, focusing on different modules.
### 4.2 Wire up the parity tests
```md
Extend modern/tests/pilot_parity_test.py so that it:
- Invokes the legacy pilot flow using whatever wrapper or command we have for COBOL (for example a script that runs the JCL in a test harness).
- Invokes the new implementation through its API or batch entry point.
- Compares the outputs according to the “Parity and comparison strategy” in pilot_reporting_validation.md.
```
### 4.3 Document the parallel run steps
Rather than a separate parallel_run_pilot.md, reuse the validation doc:
```md
Update the Parity and comparison strategy section in pilot_reporting_validation.md so that it includes a clear, ordered list of commands to:
- Prepare or load the input data set
- Run the COBOL pilot flow on that data
- Run the modern pilot flow on the same data
- Compare outputs and interpret the results
- Include precise paths for outputs and a short description of what success looks like
```
### 4.4 (If needed) Use Codex for iterative fixes
As tests fail or behavior differs, work in short loops:
```md
Here is a failing test from modern/tests/pilot_parity_test.py and the relevant COBOL and modern code. Explain why the outputs differ and propose the smallest change to the modern implementation that will align it with the COBOL behavior. Show the updated code and any test adjustments.
```
Each time you complete a meaningful chunk of work, ask Codex to update the ExecPlan:
```md
Update pilot_execplan.md so that Progress, Decision log, and Outcomes reflect the latest code, tests, and validation results for the pilot.
```
You’ll see that the ExecPlan “progress” and “outcomes” section will be updated with something along the lines of:
```md
Progress
- [x] Inventory and diagrams drafted (`pilot_reporting_overview.md` plus supporting notes in `system-architecture.md`).
- [x] Modernization technical report drafted (`pilot_reporting_overview.md` MTR section).
- [x] Target design spec drafted (`pilot_reporting_design.md` and `modern/openapi/pilot.yaml`).
- [x] Parity test plan and scaffolding documented (`pilot_reporting_validation.md` and `modern/tests/pilot_parity_test.py`).
Outcomes
- `pilot_reporting_overview.md`, `pilot_reporting_design.md`, and `pilot_reporting_validation.md` now provide an end-to-end narrative (inventory, design, validation).
- `modern/openapi/pilot.yaml` describes the API surface, and `modern/python/pilot/{models,repositories,services}.py` hold the draft implementation.
- `modern/tests/pilot_parity_test.py` exercises the parity flow using placeholders and helpers aligned with the validation strategy.
- Remaining work is limited to updating the operations test appendix and wiring the services to the real runtime.
```
---
## Phase 5 - Turn the pilot into a scalable motion
**Goal:** Provide reusable templates for other flows and a short guide to using Codex in this repo.
**Key artifacts**
* template_modernization_execplan.md
* how_to_use_codex_for_cobol_modernization.md
### 6.1 Template ExecPlan
```md
Look at the pilot files we created:
1. pilot_reporting_overview.md
2. pilot_reporting_design.md
3. pilot_reporting_validation.md
4. pilot_execplan.md
Create template_modernization_execplan.md that a team can copy when modernizing another flow. It should:
1. Follow .agent/PLANS.md
2. Include placeholders for “Overview”, “Inventory”, “Modernization Technical Report”, “Target design”, and “Validation plan”
3. Assume a similar pattern: overview doc, design doc, validation doc, OpenAPI spec, and tests.
```
### 6.2 How-to guide
```md
Using the same pilot files, write how_to_use_codex_for_cobol_modernization.md that:
1. Explains the phases at a high level (Pick a pilot, Inventory and discover, Design and spec, Implement and validate, Factory pattern).
2. For each phase, lists where coding agents helps and points to the relevant files and example prompts.
```
---
## Wrap up
If you follow the steps in this cookbook for any pilot, you should end up with a folder layout that looks roughly like this: ExecPlan, three pilot docs, an OpenAPI spec, a pilot module, and a parity test. You can further organize the markdown files in additional pilot and template subfolders for more structure.
<img src="../../images/pilot-folder-structure.png" alt="Pilot Folder Structure" width="700"/>
You’ll notice that there isn’t a runnable entry point in modern/python/pilot yet since the modules (models.py, repositories.py, services.py) are first‑draft building blocks to start. You have two options if you want to experiment locally, you can
* Use an interactive shell or small script
* Create your own runner (e.g. modern/python/pilot/main.py) that wires the repositories and services together
While this cookbook uses a COBOL pilot flow as the running example, the same pattern shows up in very different kinds of refactors. For example, one customer used Codex to migrate a large monorepo by feeding it hundreds of Jira tickets, having Codex flag higher-risk work, surface cross-cutting dependencies, and draft the code changes, with a separate validator reviewing and merging.
Modernizing COBOL repositories is just one popular case, but the same approach applies to any legacy stack or large-scale migration: turn “modernize our codebase” into a series of small, testable steps (an ExecPlan, a handful of docs, and a parity-first implementation). Codex handles the grind of understanding old patterns, generating candidate migrations, and tightening parity, while you and your team stay focused on architecture and trade-offs, making modernization faster, safer, and repeatable across every system you decide to bring forward.
examples/codex/codex_mcp_agents_sdk/building_consistent_workflows_codex_cli_agents_sdk.ipynb
+703
View File
@@ -0,0 +1,703 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "041db3ac",
"metadata": {},
"source": [
"# Building Consistent Workflows with Codex CLI & Agents SDK\n",
"### Ensuring Repeatable, Traceable, and Scaleable Agentic Development\n",
"\n",
"## Introduction\n",
"Developers strive for consistency in everything they do. With Codex CLI and the Agents SDK, that consistency can now scale like never before. Whether you’re refactoring a large codebase, rolling out new features, or introducing a new testing framework, Codex integrates seamlessly into CLI, IDE, and cloud workflows to automate and enforce repeatable development patterns. \n",
"\n",
"In this track, we’ll build both single and multi-agent systems using the Agents SDK, with Codex CLI exposed as an MCP Server. This enables: \n",
"- **Consistency and Repeatability** by providing each agent a scoped context. \n",
"- **Scalable Orchestration** to coordinate single and multi-agent systems. \n",
"- **Observability & Auditability** by reviewing the full agentic stack trace. \n",
"\n",
"## What We’ll Cover\n",
"- Initializing Codex CLI as an MCP Server: How to run Codex as a long-running MCP process. \n",
"- Building Single-Agent Systems: Using Codex MCP for scoped tasks. \n",
"- Orchestrating Multi-Agent Workflows: Coordinating multiple specialized agents. \n",
"- Tracing Agentic Behavior: Leveraging agent traces for visibility and evaluation. \n",
"\n",
"## Prerequisites & Setup\n",
"Before starting this track, ensure you have the following: \n",
"- Basic coding familiarity: You should be comfortable with Python and JavaScript. \n",
"- Developer environment: You’ll need an IDE, like VS Code or Cursor. \n",
"- OpenAI API key: Create or find your API key in the OpenAI Dashboard.\n",
"\n",
"\n",
"## Environment Setup\n",
"1. create a `.env` folder in your directory and add your `OPENAI_API_KEY` Key\n",
"2. Install dependencies\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f15f3e42",
"metadata": {},
"outputs": [],
"source": [
"%pip install openai-agents openai ## install dependencies"
]
},
{
"cell_type": "markdown",
"id": "76a91cc2",
"metadata": {},
"source": [
"## Initializing Codex CLI as an MCP Server\n",
"Here run Codex CLI as an MCP Server inside the Agents SDK. We provide the initialization parameters of `codex mcp`. This command starts Codex CLI as an MCP server and exposes two Codex tools available on the MCP server — `codex()` and `codex-reply()`. These are the underlying tools that the Agents SDK will call when it needs to invoke Codex. \n",
"- `codex()` is used for creating a conversation. \n",
"- `codex-reply()` is for continuing a conversation. \n",
"\n",
"```python\n",
"import asyncio\n",
"from agents import Agent, Runner\n",
"from agents.mcp import MCPServerStdio\n",
"\n",
"async def main() -> None:\n",
" async with MCPServerStdio(\n",
" name=\"Codex CLI\",\n",
" params={\n",
" \"command\": \"npx\",\n",
" \"args\": [\"-y\", \"codex\", \"mcp-server\"],\n",
" },\n",
" client_session_timeout_seconds=360000,\n",
" ) as codex_mcp_server:\n",
" print(\"Codex MCP server started.\")\n",
" # We will add more code here in the next section\n",
" return\n",
"```\n",
"\n",
"Also note that we are extending the MCP Server timeout to allow Codex CLI enough time to execute and complete the given task. \n",
"\n",
"---\n",
"\n",
"## Building Single Agent Systems\n",
"Let’s start with a simple example to use our Codex MCP Server. We define two agents: \n",
"1. **Designer Agent** – brainstorms and creates a small brief for a game. \n",
"2. **Developer Agent** – implements a simple game according to the Designer’s spec.\n",
"\n",
"```python\n",
"developer_agent = Agent(\n",
" name=\"Game Developer\",\n",
" instructions=(\n",
" \"You are an expert in building simple games using basic html + css + javascript with no dependencies. \"\n",
" \"Save your work in a file called index.html in the current directory.\"\n",
" \"Always call codex with \\\"approval-policy\\\": \\\"never\\\" and \\\"sandbox\\\": \\\"workspace-write\\\"\"\n",
" ),\n",
" mcp_servers=[codex_mcp_server],\n",
")\n",
"\n",
"designer_agent = Agent(\n",
" name=\"Game Designer\",\n",
" instructions=(\n",
" \"You are an indie game connoisseur. Come up with an idea for a single page html + css + javascript game that a developer could build in about 50 lines of code. \"\n",
" \"Format your request as a 3 sentence design brief for a game developer and call the Game Developer coder with your idea.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" handoffs=[developer_agent],\n",
")\n",
"\n",
"result = await Runner.run(designer_agent, \"Implement a fun new game!\")\n",
"```\n",
"\n",
"Notice that we are providing the Developer agent with the ability to write files to the project directory without asking the user for permissions. \n",
"\n",
"Now run the code and you’ll see an `index.html` file generated. Go ahead and open the file and start playing the game! \n",
"\n",
"Here’s a few screenshots of the game my agentic system created. Yours will be different!\n",
"\n",
"| Example gameplay | Game Over Score |\n",
"| :---: | :---: |\n",
"| <img src=\"/images/game_example_1.png\" alt=\"Example gameplay\" width=\"320\" /> | <img src=\"/images/game_example_2.png\" alt=\"Game Over Score\" width=\"320\" /> |"
]
},
{
"cell_type": "markdown",
"id": "d8cf6db9",
"metadata": {},
"source": [
"Here's the full executable code. Note that it might take a few minutes to run. It will have run successfully if you see an index.html file produced. You might also see some MCP events warnings about format. You can ignore these events."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c9134a41",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from dotenv import load_dotenv\n",
"import asyncio\n",
"from agents import Agent, Runner, set_default_openai_api\n",
"from agents.mcp import MCPServerStdio\n",
"\n",
"load_dotenv(override=True) # load the API key from the .env file. We set override to True here to ensure the notebook is loading any changes\n",
"set_default_openai_api(os.getenv(\"OPENAI_API_KEY\"))\n",
"\n",
"async def main() -> None:\n",
" async with MCPServerStdio(\n",
" name=\"Codex CLI\",\n",
" params={\n",
" \"command\": \"npx\",\n",
" \"args\": [\"-y\", \"codex\", \"mcp-server\"],\n",
" },\n",
" client_session_timeout_seconds=360000,\n",
" ) as codex_mcp_server:\n",
" developer_agent = Agent(\n",
" name=\"Game Developer\",\n",
" instructions=(\n",
" \"You are an expert in building simple games using basic html + css + javascript with no dependencies. \"\n",
" \"Save your work in a file called index.html in the current directory.\"\n",
" \"Always call codex with \\\"approval-policy\\\": \\\"never\\\" and \\\"sandbox\\\": \\\"workspace-write\\\"\"\n",
" ),\n",
" mcp_servers=[codex_mcp_server],\n",
" )\n",
"\n",
" designer_agent = Agent(\n",
" name=\"Game Designer\",\n",
" instructions=(\n",
" \"You are an indie game connoisseur. Come up with an idea for a single page html + css + javascript game that a developer could build in about 50 lines of code. \"\n",
" \"Format your request as a 3 sentence design brief for a game developer and call the Game Developer coder with your idea.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" handoffs=[developer_agent],\n",
" )\n",
"\n",
" result = await Runner.run(designer_agent, \"Implement a fun new game!\")\n",
" # print(result.final_output)\n",
"\n",
"\n",
"if __name__ == \"__main__\":\n",
" # Jupyter/IPython already runs an event loop, so calling asyncio.run() here\n",
" # raises \"asyncio.run() cannot be called from a running event loop\".\n",
" # Workaround: if a loop is running (notebook), use top-level `await`; otherwise use asyncio.run().\n",
" try:\n",
" asyncio.get_running_loop()\n",
" await main()\n",
" except RuntimeError:\n",
" asyncio.run(main())"
]
},
{
"cell_type": "markdown",
"id": "407e2d8f",
"metadata": {},
"source": [
"---\n",
"\n",
"## Orchestrating Multi-Agent Workflows\n",
"For larger workflows, we introduce a team of agents: \n",
"- **Project Manager**: Breaks down task list, creates requirements, and coordinates work. \n",
"- **Designer**: Produces UI/UX specifications. \n",
"- **Frontend Developer**: Implements UI/UX. \n",
"- **Backend Developer**: Implements APIs and logic. \n",
"- **Tester**: Validates outputs against acceptance criteria. \n",
"\n",
"In this example, we intentionally have the Project Manager agent enforce gating logic between each of the specialized downstream agents. This ensures that artifacts exist before handoffs are made. This mirrors real world enterprise workflows such as JIRA task orchestration, long-chained rollouts, and QA sign-offs. \n",
"\n",
"<div align=\"center\">\n",
" <img src=\"/images/multi_agent_codex_workflow.png\" alt=\"Multi-Agent Codex Workflow with Codex MCP\" style=\"max-width: 100%; width: 960px;\" />\n",
" <br />\n",
" <em>Multi-agent orchestration with Codex MCP and gated handoffs producing artifacts.</em>\n",
"</div>\n",
"\n",
"\n",
"In this structure, each of our agents serve a specialized purpose. The Project Manager is overall responsible for coordinating across all other agents and ensuring the overall task is complete.\n",
"\n",
"## Define the Codex CLI MCP Server\n",
"We set up our MCP Server to initialize Codex CLI just as we did in the single agent example.\n",
"\n",
"```python\n",
"async def main() -> None:\n",
" async with MCPServerStdio(\n",
" name=\"Codex CLI\",\n",
" params={\n",
" \"command\": \"npx\",\n",
" \"args\": [\"-y\", \"codex\", \"mcp-server\"],\n",
" },\n",
" client_session_timeout_seconds=360000,\n",
" ) as codex_mcp_server:\n",
" print(\"Codex MCP server started.\")\n",
" # We will add more code here in the next section\n",
" return\n",
" ```\n",
"\n",
"\n",
"\n",
"## Define each specialized agent\n",
"Below we define each of our specialized agents and provide access to our Codex MCP server. Notice that we are also passing the `RECOMMMENDED_PROMPT_PREFIX` to each agent that helps the system optimize for handoffs between agents. \n",
"\n",
"```python\n",
"# Downstream agents are defined first for clarity, then PM references them in handoffs.\n",
"designer_agent = Agent(\n",
" name=\"Designer\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Designer.\\n\"\n",
" \"Your only source of truth is AGENT_TASKS.md and REQUIREMENTS.md from the Project Manager.\\n\"\n",
" \"Do not assume anything that is not written there.\\n\\n\"\n",
" \"You may use the internet for additional guidance or research.\"\n",
" \"Deliverables (write to /design):\\n\"\n",
" \"- design_spec.md – a single page describing the UI/UX layout, main screens, and key visual notes as requested in AGENT_TASKS.md.\\n\"\n",
" \"- wireframe.md – a simple text or ASCII wireframe if specified.\\n\\n\"\n",
" \"Keep the output short and implementation-friendly.\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" tools=[WebSearchTool()],\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
")\n",
"\n",
"frontend_developer_agent = Agent(\n",
" name=\"Frontend Developer\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Frontend Developer.\\n\"\n",
" \"Read AGENT_TASKS.md and design_spec.md. Implement exactly what is described there.\\n\\n\"\n",
" \"Deliverables (write to /frontend):\\n\"\n",
" \"- index.html – main page structure\\n\"\n",
" \"- styles.css or inline styles if specified\\n\"\n",
" \"- main.js or game.js if specified\\n\\n\"\n",
" \"Follow the Designer’s DOM structure and any integration points given by the Project Manager.\\n\"\n",
" \"Do not add features or branding beyond the provided documents.\\n\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager_agent.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
")\n",
"\n",
"backend_developer_agent = Agent(\n",
" name=\"Backend Developer\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Backend Developer.\\n\"\n",
" \"Read AGENT_TASKS.md and REQUIREMENTS.md. Implement the backend endpoints described there.\\n\\n\"\n",
" \"Deliverables (write to /backend):\\n\"\n",
" \"- package.json – include a start script if requested\\n\"\n",
" \"- server.js – implement the API endpoints and logic exactly as specified\\n\\n\"\n",
" \"Keep the code as simple and readable as possible. No external database.\\n\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager_agent.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
")\n",
"\n",
"tester_agent = Agent(\n",
" name=\"Tester\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Tester.\\n\"\n",
" \"Read AGENT_TASKS.md and TEST.md. Verify that the outputs of the other roles meet the acceptance criteria.\\n\\n\"\n",
" \"Deliverables (write to /tests):\\n\"\n",
" \"- TEST_PLAN.md – bullet list of manual checks or automated steps as requested\\n\"\n",
" \"- test.sh or a simple automated script if specified\\n\\n\"\n",
" \"Keep it minimal and easy to run.\\n\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
")\n",
"```\n",
"\n",
"\n",
"\n",
"After each role completes its assignment, it will call `transfer_to_project_manager_agent`, and let the Project Manager confirm that the required files exist (or request fixes) before unblocking the next team. \n",
"\n",
"## Define Project Manager Agent\n",
"The Project Manager is the only agent that receives the initial prompt, creates the planning documents in the project directory, and enforces the gatekeeping logic before every transfer. \n",
"\n",
"```python \n",
"project_manager_agent = Agent(\n",
"name=\"Project Manager\",\n",
"instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"\"\"\n",
" You are the Project Manager.\n",
"\n",
" Objective:\n",
" Convert the input task list into three project-root files the team will execute against.\n",
"\n",
" Deliverables (write in project root):\n",
" - REQUIREMENTS.md: concise summary of product goals, target users, key features, and constraints.\n",
" - TEST.md: tasks with [Owner] tags (Designer, Frontend, Backend, Tester) and clear acceptance criteria.\n",
" - AGENT_TASKS.md: one section per role containing:\n",
" - Project name\n",
" - Required deliverables (exact file names and purpose)\n",
" - Key technical notes and constraints\n",
"\n",
" Process:\n",
" - Resolve ambiguities with minimal, reasonable assumptions. Be specific so each role can act without guessing.\n",
" - Create files using Codex MCP with {\"approval-policy\":\"never\",\"sandbox\":\"workspace-write\"}.\n",
" - Do not create folders. Only create REQUIREMENTS.md, TEST.md, AGENT_TASKS.md.\n",
"\n",
" Handoffs (gated by required files):\n",
" 1) After the three files above are created, hand off to the Designer with transfer_to_designer_agent and include REQUIREMENTS.md, and AGENT_TASKS.md.\n",
" 2) Wait for the Designer to produce /design/design_spec.md. Verify that file exists before proceeding.\n",
" 3) When design_spec.md exists, hand off in parallel to both:\n",
" - Frontend Developer with transfer_to_frontend_developer_agent (provide design_spec.md, REQUIREMENTS.md, AGENT_TASKS.md).\n",
" - Backend Developer with transfer_to_backend_developer_agent (provide REQUIREMENTS.md, AGENT_TASKS.md).\n",
" 4) Wait for Frontend to produce /frontend/index.html and Backend to produce /backend/server.js. Verify both files exist.\n",
" 5) When both exist, hand off to the Tester with transfer_to_tester_agent and provide all prior artifacts and outputs.\n",
" 6) Do not advance to the next handoff until the required files for that step are present. If something is missing, request the owning agent to supply it and re-check.\n",
"\n",
" PM Responsibilities:\n",
" - Coordinate all roles, track file completion, and enforce the above gating checks.\n",
" - Do NOT respond with status updates. Just handoff to the next agent until the project is complete.\n",
" \"\"\"\n",
"),\n",
"model=\"gpt-5\",\n",
"model_settings=ModelSettings(\n",
" reasoning=Reasoning(effort=\"medium\")\n",
"),\n",
"handoffs=[designer_agent, frontend_developer_agent, backend_developer_agent, tester_agent],\n",
"mcp_servers=[codex_mcp_server],\n",
")\n",
"```\n",
"\n",
"After constructing the Project Manager, the script sets every specialist's handoffs back to the Project\n",
"Manager. This ensures deliverables return for validation before moving on.\n",
"\n",
"```python\n",
"designer_agent.handoffs = [project_manager_agent]\n",
"frontend_developer_agent.handoffs = [project_manager_agent]\n",
"backend_developer_agent.handoffs = [project_manager_agent]\n",
"tester_agent.handoffs = [project_manager_agent]\n",
"```\n",
"## Add in your task list\n",
"This is the task that the Project Manager will refine into specific requirements and tasks for the entire system.\n",
"\n",
"```python\n",
"task_list = \"\"\"\n",
"Goal: Build a tiny browser game to showcase a multi-agent workflow.\n",
"\n",
"High-level requirements:\n",
"- Single-screen game called \"Bug Busters\".\n",
"- Player clicks a moving bug to earn points.\n",
"- Game ends after 20 seconds and shows final score.\n",
"- Optional: submit score to a simple backend and display a top-10 leaderboard.\n",
"\n",
"Roles:\n",
"- Designer: create a one-page UI/UX spec and basic wireframe.\n",
"- Frontend Developer: implement the page and game logic.\n",
"- Backend Developer: implement a minimal API (GET /health, GET/POST /scores).\n",
"- Tester: write a quick test plan and a simple script to verify core routes.\n",
"\n",
"Constraints:\n",
"- No external database—memory storage is fine.\n",
"- Keep everything readable for beginners; no frameworks required.\n",
"- All outputs should be small files saved in clearly named folders.\n",
"\"\"\"\n",
"```\n",
"\n",
"Next, run your system, sit back, and you’ll see the agents go to work and create a game in a few minutes! We've included the fully executable code below. Once it's finished, you'll notice the creation of the following files directory. Note that this multi-agent orchestration usually took about 11 mintues to fully complete.\n",
"\n",
"```markdown\n",
"root_directory/\n",
"├── AGENT_TASKS.md\n",
"├── REQUIREMENTS.md\n",
"├── backend\n",
"│ ├── package.json\n",
"│ └── server.js\n",
"├── design\n",
"│ ├── design_spec.md\n",
"│ └── wireframe.md\n",
"├── frontend\n",
"│ ├── game.js\n",
"│ ├── index.html\n",
"│ └── styles.css\n",
"└── TEST.md\n",
"```\n",
"\n",
"Start your backend server with `node server.js` and open your `index.html` file to play your game.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ebe128a8",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from dotenv import load_dotenv\n",
"import asyncio\n",
"from agents import Agent, Runner, WebSearchTool, ModelSettings, set_default_openai_api\n",
"from agents.mcp import MCPServerStdio\n",
"from agents.extensions.handoff_prompt import RECOMMENDED_PROMPT_PREFIX\n",
"from openai.types.shared import Reasoning\n",
"\n",
"load_dotenv(override=True) # load the API key from the .env file. We set override to True here to ensure the notebook is loading any changes\n",
"set_default_openai_api(os.getenv(\"OPENAI_API_KEY\"))\n",
"\n",
"async def main() -> None:\n",
" async with MCPServerStdio(\n",
" name=\"Codex CLI\",\n",
" params={\"command\": \"npx\", \"args\": [\"-y\", \"codex\", \"mcp-server\"]},\n",
" client_session_timeout_seconds=360000,\n",
" ) as codex_mcp_server:\n",
"\n",
" # Downstream agents are defined first for clarity, then PM references them in handoffs.\n",
" designer_agent = Agent(\n",
" name=\"Designer\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Designer.\\n\"\n",
" \"Your only source of truth is AGENT_TASKS.md and REQUIREMENTS.md from the Project Manager.\\n\"\n",
" \"Do not assume anything that is not written there.\\n\\n\"\n",
" \"You may use the internet for additional guidance or research.\"\n",
" \"Deliverables (write to /design):\\n\"\n",
" \"- design_spec.md – a single page describing the UI/UX layout, main screens, and key visual notes as requested in AGENT_TASKS.md.\\n\"\n",
" \"- wireframe.md – a simple text or ASCII wireframe if specified.\\n\\n\"\n",
" \"Keep the output short and implementation-friendly.\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" tools=[WebSearchTool()],\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
" )\n",
"\n",
" frontend_developer_agent = Agent(\n",
" name=\"Frontend Developer\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Frontend Developer.\\n\"\n",
" \"Read AGENT_TASKS.md and design_spec.md. Implement exactly what is described there.\\n\\n\"\n",
" \"Deliverables (write to /frontend):\\n\"\n",
" \"- index.html – main page structure\\n\"\n",
" \"- styles.css or inline styles if specified\\n\"\n",
" \"- main.js or game.js if specified\\n\\n\"\n",
" \"Follow the Designer’s DOM structure and any integration points given by the Project Manager.\\n\"\n",
" \"Do not add features or branding beyond the provided documents.\\n\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager_agent.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
" )\n",
"\n",
" backend_developer_agent = Agent(\n",
" name=\"Backend Developer\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Backend Developer.\\n\"\n",
" \"Read AGENT_TASKS.md and REQUIREMENTS.md. Implement the backend endpoints described there.\\n\\n\"\n",
" \"Deliverables (write to /backend):\\n\"\n",
" \"- package.json – include a start script if requested\\n\"\n",
" \"- server.js – implement the API endpoints and logic exactly as specified\\n\\n\"\n",
" \"Keep the code as simple and readable as possible. No external database.\\n\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager_agent.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
" )\n",
"\n",
" tester_agent = Agent(\n",
" name=\"Tester\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"You are the Tester.\\n\"\n",
" \"Read AGENT_TASKS.md and TEST.md. Verify that the outputs of the other roles meet the acceptance criteria.\\n\\n\"\n",
" \"Deliverables (write to /tests):\\n\"\n",
" \"- TEST_PLAN.md – bullet list of manual checks or automated steps as requested\\n\"\n",
" \"- test.sh or a simple automated script if specified\\n\\n\"\n",
" \"Keep it minimal and easy to run.\\n\\n\"\n",
" \"When complete, handoff to the Project Manager with transfer_to_project_manager.\"\n",
" \"When creating files, call Codex MCP with {\\\"approval-policy\\\":\\\"never\\\",\\\"sandbox\\\":\\\"workspace-write\\\"}.\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" mcp_servers=[codex_mcp_server],\n",
" handoffs=[],\n",
" )\n",
"\n",
" project_manager_agent = Agent(\n",
" name=\"Project Manager\",\n",
" instructions=(\n",
" f\"\"\"{RECOMMENDED_PROMPT_PREFIX}\"\"\"\n",
" \"\"\"\n",
" You are the Project Manager.\n",
"\n",
" Objective:\n",
" Convert the input task list into three project-root files the team will execute against.\n",
"\n",
" Deliverables (write in project root):\n",
" - REQUIREMENTS.md: concise summary of product goals, target users, key features, and constraints.\n",
" - TEST.md: tasks with [Owner] tags (Designer, Frontend, Backend, Tester) and clear acceptance criteria.\n",
" - AGENT_TASKS.md: one section per role containing:\n",
" - Project name\n",
" - Required deliverables (exact file names and purpose)\n",
" - Key technical notes and constraints\n",
"\n",
" Process:\n",
" - Resolve ambiguities with minimal, reasonable assumptions. Be specific so each role can act without guessing.\n",
" - Create files using Codex MCP with {\"approval-policy\":\"never\",\"sandbox\":\"workspace-write\"}.\n",
" - Do not create folders. Only create REQUIREMENTS.md, TEST.md, AGENT_TASKS.md.\n",
"\n",
" Handoffs (gated by required files):\n",
" 1) After the three files above are created, hand off to the Designer with transfer_to_designer_agent and include REQUIREMENTS.md, and AGENT_TASKS.md.\n",
" 2) Wait for the Designer to produce /design/design_spec.md. Verify that file exists before proceeding.\n",
" 3) When design_spec.md exists, hand off in parallel to both:\n",
" - Frontend Developer with transfer_to_frontend_developer_agent (provide design_spec.md, REQUIREMENTS.md, AGENT_TASKS.md).\n",
" - Backend Developer with transfer_to_backend_developer_agent (provide REQUIREMENTS.md, AGENT_TASKS.md).\n",
" 4) Wait for Frontend to produce /frontend/index.html and Backend to produce /backend/server.js. Verify both files exist.\n",
" 5) When both exist, hand off to the Tester with transfer_to_tester_agent and provide all prior artifacts and outputs.\n",
" 6) Do not advance to the next handoff until the required files for that step are present. If something is missing, request the owning agent to supply it and re-check.\n",
"\n",
" PM Responsibilities:\n",
" - Coordinate all roles, track file completion, and enforce the above gating checks.\n",
" - Do NOT respond with status updates. Just handoff to the next agent until the project is complete.\n",
" \"\"\"\n",
" ),\n",
" model=\"gpt-5\",\n",
" model_settings=ModelSettings(\n",
" reasoning=Reasoning(effort=\"medium\")\n",
" ),\n",
" handoffs=[designer_agent, frontend_developer_agent, backend_developer_agent, tester_agent],\n",
" mcp_servers=[codex_mcp_server],\n",
" )\n",
"\n",
" designer_agent.handoffs = [project_manager_agent]\n",
" frontend_developer_agent.handoffs = [project_manager_agent]\n",
" backend_developer_agent.handoffs = [project_manager_agent]\n",
" tester_agent.handoffs = [project_manager_agent]\n",
"\n",
" # Example task list input for the Project Manager\n",
" task_list = \"\"\"\n",
"Goal: Build a tiny browser game to showcase a multi-agent workflow.\n",
"\n",
"High-level requirements:\n",
"- Single-screen game called \"Bug Busters\".\n",
"- Player clicks a moving bug to earn points.\n",
"- Game ends after 20 seconds and shows final score.\n",
"- Optional: submit score to a simple backend and display a top-10 leaderboard.\n",
"\n",
"Roles:\n",
"- Designer: create a one-page UI/UX spec and basic wireframe.\n",
"- Frontend Developer: implement the page and game logic.\n",
"- Backend Developer: implement a minimal API (GET /health, GET/POST /scores).\n",
"- Tester: write a quick test plan and a simple script to verify core routes.\n",
"\n",
"Constraints:\n",
"- No external database—memory storage is fine.\n",
"- Keep everything readable for beginners; no frameworks required.\n",
"- All outputs should be small files saved in clearly named folders.\n",
"\"\"\"\n",
"\n",
" # Only the Project Manager receives the task list directly\n",
" result = await Runner.run(project_manager_agent, task_list, max_turns=30)\n",
" print(result.final_output)\n",
"\n",
"if __name__ == \"__main__\":\n",
" # Jupyter/IPython already runs an event loop, so calling asyncio.run() here\n",
" # raises \"asyncio.run() cannot be called from a running event loop\".\n",
" # Workaround: if a loop is running (notebook), use top-level `await`; otherwise use asyncio.run().\n",
" try:\n",
" asyncio.get_running_loop()\n",
" await main()\n",
" except RuntimeError:\n",
" asyncio.run(main())"
]
},
{
"cell_type": "markdown",
"id": "9e828b04",
"metadata": {},
"source": [
"---\n",
"\n",
"## Tracing the agentic behavior using Traces\n",
"As the complexity of your agentic systems grow, it’s important to see how these agents are interacting. We can do this with the Traces dashboard that records: \n",
"- Prompts, tool calls, and handoffs between agents. \n",
"- MCP Server calls, Codex CLI calls, execution times, and file writes. \n",
"- Errors and warnings. \n",
"\n",
"Let’s take a look at the agent trace for the team of agents above.\n",
"\n",
"<div align=\"center\">\n",
" <img src=\"/images/multi_agent_trace.png\" alt=\"Multi-Agent Codex Workflow with Codex MCP\" style=\"max-width: 100%; width: 960px;\" />\n",
"</div>\n",
"\n",
"In this Trace, we can confirm that every agent handoff is quarterbacked by our Project Manager Agent who is confirming that specific artifacts exist before handoff to the next agent. Additionally, we can see specific innovations of the Codex MCP Server and generate each output by calling the Responses API. The timeline bars highlight execution durations, making it easy to spot long-running steps and understand how control passes between agents.\n",
"\n",
"You can even click into each trace to see the specific details of the prompt, tool calls, and other metadata. Over time you can view this information to further tune, optimize, and track your agentic system performance.\n",
"\n",
"<div align=\"center\">\n",
" <img src=\"/images/multi_agent_trace_details.png\" alt=\"Multi-Agent Trace Details\" style=\"max-width: 100%; width: 960px;\" />\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"id": "7b446e22",
"metadata": {},
"source": [
"---\n",
"\n",
"## Recap of What We Did in This Guide\n",
"In this guide, we walked through the process of building consistent, scalable workflows using Codex CLI and the Agents SDK. Specifically, we covered: \n",
"\n",
"- **Codex MCP Server Setup** – How to initialize Codex CLI as an MCP server and make it available as tools for agent interactions. \n",
"- **Single-Agent Example** – A simple workflow with a Designer Agent and a Developer Agent, where Codex executed scoped tasks deterministically to produce a playable game. \n",
"- **Multi-Agent Orchestration** – Expanding to a larger workflow with a Project Manager, Designer, Frontend Developer, Backend Developer, and Tester, mirroring complex task orchestration and sign-off processes. \n",
"- **Traces & Observability** – Using built-in Traces to capture prompts, tool calls, handoffs, execution times, and artifacts, giving full visibility into agentic behavior for debugging, evaluation, and future optimization. \n",
"\n",
"---\n",
"\n",
"## Moving Forward: Applying These Lessons\n",
"Now that you’ve seen Codex MCP and the Agents SDK in action, here’s how you can apply the concepts in real projects and extract value: \n",
"\n",
"### 1. Scale to Real-World Rollouts\n",
"- Apply the same multi-agent orchestration to large code refactors (e.g., 500+ files, framework migrations). \n",
"- Use Codex MCP’s deterministic execution for long-running, auditable rollouts with traceable progress. \n",
"\n",
"### 2. Accelerate Delivery Without Losing Control\n",
"- Organize teams of specialized agents to parallelize development, while maintaining gating logic for artifact validation. \n",
"- Reduce turnaround time for new features, testing, or codebase modernization. \n",
"\n",
"### 3. Extend and Connect to Your Development Workflows\n",
"- Connect MCP-powered agents with Jira, GitHub, or CI/CD pipelines via webhooks for automated, repeatable development cycles. \n",
"- Leverage Codex MCP in multi-agent service orchestration: not just codegen, but also documentation, QA, and deployment. \n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "cookbook-env",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/codex/data/docs/getting_started_evals_pre_repair.ipynb
+111
View File
@@ -0,0 +1,111 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting started with Evals, sampled fixture\n",
"\n",
"This fixture mirrors the Cookbook Evals walkthrough but uses tiny local records instead of running a long benchmark.\n"
],
"id": "cell-000"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"MODEL = \"gpt-3.5-turbo\" # stale model kept intentionally for review\n",
"\n",
"schema = \"Table cars_data, columns = [Id, MPG, Cylinders, Horsepower, Year]\"\n",
"examples = [\n",
" {\"question\": \"Which cars have more than 100 horsepower?\", \"ideal\": \"SELECT Id FROM cars_data WHERE Horsepower > 100\"},\n",
" {\"question\": \"How many cars are from 1970?\", \"ideal\": \"SELECT count(*) FROM cars_data WHERE Year = 1970\"},\n",
"]\n"
],
"id": "cell-001"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def make_eval_rows(records):\n",
" return [\n",
" {\n",
" \"input\": [\n",
" {\"role\": \"system\", \"content\": f\"Answer with SQLite SQL. {schema}\"},\n",
" {\"role\": \"user\", \"content\": record[\"question\"]},\n",
" ],\n",
" \"ideal\": record[\"ideal\"],\n",
" }\n",
" for record in records\n",
" ]\n",
"\n",
"eval_rows = make_eval_rows(examples)\n",
"eval_rows[0]\n"
],
"id": "cell-002"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Legacy CLI command kept as a repair target; this cell only records it.\n",
"EVAL_COMMAND = \"oaieval gpt-3.5-turbo spider-sql --max_samples 25\"\n",
"print(EVAL_COMMAND)\n"
],
"id": "cell-003"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Manual log-name placeholder kept as a repair target.\n",
"log_name = \"240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\" # EDIT THIS\n",
"local_events = [\n",
" {\"type\": \"final_report\", \"data\": {\"accuracy\": 0.5}},\n",
" {\"type\": \"sampling\", \"data\": {\"prompt\": eval_rows[0][\"input\"], \"sampled\": \"SELECT Id FROM cars_data WHERE Horsepower > 100\"}},\n",
"]\n",
"final_report = next(event[\"data\"] for event in local_events if event[\"type\"] == \"final_report\")\n",
"final_report\n"
],
"id": "cell-004"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
},
"codex_case_study": {
"source_repo": "https://github.com/openai/openai-cookbook",
"source_path": "examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb",
"source_commit": "96b1d67^",
"purpose": "Runtime-sampled pre-repair fixture derived from a Cookbook documentation reliability run.",
"sampling_note": "Compact Evals-style maintenance sample with stale model, deprecated CLI, and result-log issues.",
"repair_story": {
"target_iteration": 2,
"repair_depth": "Two-pass cleanup: first modernize the obvious stale Evals flow, then use validation feedback to remove result-log brittleness.",
"issue_layers": [
"stale model",
"deprecated oaieval command",
"manual log-name/runtime result dependency"
]
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/codex/data/docs/knowledge_retrieval_pre_repair.ipynb
+116
View File
@@ -0,0 +1,116 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Function calling for knowledge retrieval, sampled fixture\n",
"\n",
"This fixture is derived from the Cookbook arXiv retrieval example. It uses two local paper records so execution stays fast while the repair loop still sees legacy tool-calling patterns.\n"
],
"id": "cell-000"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"GPT_MODEL = \"gpt-4-turbo-preview\" # stale model kept intentionally for the repair loop\n",
"\n",
"papers = [\n",
" {\"title\": \"PPO for sequence generation\", \"article_url\": \"https://example.com/ppo\", \"summary\": \"PPO stabilizes policy updates with clipped objectives.\"},\n",
" {\"title\": \"Retrieval augmented generation\", \"article_url\": \"https://example.com/rag\", \"summary\": \"RAG combines retrieval with generation to ground answers.\"},\n",
"]\n"
],
"id": "cell-001"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def get_articles(query, top_k=2):\n",
" query_terms = set(query.lower().split())\n",
" ranked = sorted(\n",
" papers,\n",
" key=lambda paper: len(query_terms & set((paper[\"title\"] + \" \" + paper[\"summary\"]).lower().split())),\n",
" reverse=True,\n",
" )\n",
" return ranked[:top_k]\n",
"\n",
"get_articles(\"ppo reinforcement learning\")\n"
],
"id": "cell-002"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def read_article_and_summarize(query):\n",
" article = get_articles(query, top_k=1)[0]\n",
" return f'{article[\"title\"]}: {article[\"summary\"]}'\n",
"\n",
"read_article_and_summarize(\"ppo sequence generation\")\n"
],
"id": "cell-003"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Legacy function-calling schema kept as a repair target.\n",
"arxiv_functions = [\n",
" {\n",
" \"name\": \"get_articles\",\n",
" \"description\": \"Use this function to get academic papers from a local article index.\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\"query\": {\"type\": \"string\"}},\n",
" \"required\": [\"query\"],\n",
" },\n",
" }\n",
"]\n",
"\n",
"messages = [{\"role\": \"user\", \"content\": \"How does PPO work?\"}]\n",
"print(arxiv_functions[0][\"name\"], messages[0][\"content\"])\n"
],
"id": "cell-004"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
},
"codex_case_study": {
"source_repo": "https://github.com/openai/openai-cookbook",
"source_path": "examples/How_to_call_functions_for_knowledge_retrieval.ipynb",
"source_commit": "96b1d67^",
"purpose": "Runtime-sampled pre-repair fixture derived from a Cookbook documentation reliability run.",
"sampling_note": "Compact knowledge-retrieval maintenance sample with stale model and legacy tool-calling issues.",
"repair_story": {
"target_iteration": 3,
"repair_depth": "Three-pass cleanup: modernize model/API shape, then tighten runnable local setup, then restore the full retrieval teaching flow.",
"issue_layers": [
"stale chat model",
"legacy function-calling schema",
"setup/runnability gap",
"end-to-end retrieval flow integrity"
]
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/codex/data/docs/qdrant_embeddings_search_pre_repair.ipynb
+126
View File
@@ -0,0 +1,126 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Qdrant embedding search, sampled fixture\n",
"\n",
"This fixture is derived from the Cookbook Qdrant search example. It keeps the same teaching arc with a tiny local article set so validation can execute quickly.\n"
],
"id": "cell-000"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from math import sqrt\n",
"\n",
"EMBEDDING_MODEL = \"text-embedding-ada-002\" # legacy model kept intentionally for the repair loop\n",
"\n",
"articles = [\n",
" {\"id\": 1, \"title\": \"Modern art in Europe\", \"url\": \"https://example.com/art\", \"content\": \"Cubism and surrealism reshaped European museums.\"},\n",
" {\"id\": 2, \"title\": \"Scottish battle history\", \"url\": \"https://example.com/scotland\", \"content\": \"Bannockburn and Stirling Bridge shaped Scottish history.\"},\n",
" {\"id\": 3, \"title\": \"Space telescope discoveries\", \"url\": \"https://example.com/space\", \"content\": \"Modern telescopes reveal planets, galaxies, and stars.\"},\n",
"]\n"
],
"id": "cell-001"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def embed(text: str) -> list[float]:\n",
" buckets = [0.0, 0.0, 0.0, 0.0]\n",
" for index, char in enumerate(text.lower()):\n",
" buckets[index % len(buckets)] += ord(char) / 1000\n",
" length = sqrt(sum(value * value for value in buckets)) or 1\n",
" return [round(value / length, 4) for value in buckets]\n",
"\n",
"for article in articles:\n",
" article[\"title_vector\"] = embed(article[\"title\"])\n",
" article[\"content_vector\"] = embed(article[\"content\"])\n"
],
"id": "cell-002"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class LocalQdrant:\n",
" def __init__(self, rows):\n",
" self.rows = rows\n",
"\n",
" def search(self, collection_name, query_vector, limit=3, query_filter=None):\n",
" vector_name, query = query_vector\n",
" scored = []\n",
" for row in self.rows:\n",
" vector = row[f\"{vector_name}_vector\"]\n",
" score = sum(a * b for a, b in zip(query, vector))\n",
" scored.append((score, row))\n",
" return sorted(scored, reverse=True)[:limit]\n",
"\n",
" def query_points(self, collection_name, query, using=\"title\", limit=3):\n",
" return self.search(collection_name, (using, query), limit=limit)\n",
"\n",
"qdrant = LocalQdrant(articles)\n"
],
"id": "cell-003"
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def query_qdrant(query, collection_name, vector_name=\"title\", top_k=3):\n",
" embedded_query = embed(query)\n",
" return qdrant.search(\n",
" collection_name=collection_name,\n",
" query_vector=(vector_name, embedded_query),\n",
" limit=top_k,\n",
" query_filter=None,\n",
" )\n",
"\n",
"for score, article in query_qdrant(\"modern art in Europe\", \"Articles\", \"title\"):\n",
" print(f'{article[\"title\"]}: {score:.3f}')\n"
],
"id": "cell-004"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"pygments_lexer": "ipython3"
},
"codex_case_study": {
"source_repo": "https://github.com/openai/openai-cookbook",
"source_path": "examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb",
"source_commit": "96b1d67^",
"purpose": "Runtime-sampled pre-repair fixture derived from a Cookbook documentation reliability run.",
"sampling_note": "Compact Qdrant-style maintenance sample with stale embedding-model and query API issues.",
"repair_story": {
"target_iteration": 1,
"repair_depth": "One-pass cleanup: modernize the local Qdrant query path and clarify the sampled fixture framing.",
"issue_layers": [
"stale embedding model",
"deprecated qdrant.search call",
"fixture framing/setup clarity"
]
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/codex/images/add_label.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

examples/codex/images/ci-codex-workflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 182 KiB

examples/codex/images/codex-pr.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 522 KiB

examples/codex/images/codex-workflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 519 KiB

examples/codex/images/codex_action.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 305 KiB

examples/codex/images/failing-workflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 247 KiB

examples/codex/images/game_example_1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 196 KiB

examples/codex/images/game_example_2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 346 KiB

examples/codex/images/github-pr-settings.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 199 KiB

examples/codex/images/jira_comment.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 91 KiB

examples/codex/images/jira_rule.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 326 KiB

examples/codex/images/jira_status_change.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 85 KiB

examples/codex/images/multi_agent_codex_workflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

examples/codex/images/multi_agent_trace.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 348 KiB

examples/codex/images/multi_agent_trace_details.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 663 KiB

examples/codex/jira-github.ipynb
+315
View File
@@ -0,0 +1,315 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"# Automate Jira ↔ GitHub with `codex-cli`\n",
"\n",
"## Purpose of this cookbook\n",
"\n",
"This cookbook provides a practical, step-by-step approach to automating the workflow between Jira and GitHub. By labeling a Jira issue, you trigger an end-to-end process that creates a **GitHub pull request**, keeps both systems updated, and streamlines code review, all with minimal manual effort. The automation is powered by the [`codex-cli`](https://github.com/openai/openai-codex) agent running inside a GitHub Action.\n",
"\n",
"<img src=\"../../images/codex_action.png\" alt=\"Full data-flow diagram\" width=\"500\"/>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"The flow is:\n",
"\n",
"1. Label a Jira issue \n",
"2. Jira Automation calls the GitHub Action \n",
"3. The action spins up `codex-cli` to implement the change \n",
"4. A PR is opened\n",
"5. Jira is transitioned & annotated - creating a neat, zero-click loop. This includes changing the status of the ticket, adding the PR link and commenting in the ticket with updates.\n",
"\n",
"## Prerequisites\n",
"\n",
"* Jira: project admin rights + ability to create automation rules \n",
"* GitHub: write access, permission to add repository secrets, and a protected `main` branch \n",
"* API keys & secrets placed as repository secrets:\n",
" * `OPENAI_API_KEY` – your OpenAI key for `codex-cli` \n",
" * `JIRA_BASE_URL`, `JIRA_EMAIL`, `JIRA_API_TOKEN` – for REST calls from the action \n",
"* `codex-cli` installed locally (`pnpm add -g @openai/codex`) for ad-hoc testing \n",
"* A repository that contains a `.github/workflows/` folder\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## Create the Jira Automation Rule\n",
"\n",
"<img src=\"../../images/jira_rule.png\" alt=\"Automation Rule\" width=\"500\"/>\n",
"\n",
"The first step in this rule listens for changes to an issue’s labels. This ensures we only trigger the automation when a label is added or modified—no need to process every update to the issue.\n",
"\n",
"Next, we check whether the updated labels include a specific keyword, in our example we are using `aswe`. This acts as a filter so that only issues explicitly tagged for automation proceed, avoiding unnecessary noise from unrelated updates.\n",
"\n",
"If the condition is met, we send a `POST` request to GitHub’s `workflow_dispatch` endpoint. This kicks off a GitHub Actions workflow with the relevant issue context. We pass in the issue key, summary, and a cleaned-up version of the description—escaping quotes and newlines so the payload parses correctly in YAML/JSON. There are [additional fields](https://support.atlassian.com/cloud-automation/docs/jira-smart-values-issues/) available as variables in JIRA to give the codex agent more context during its execution.\n",
"\n",
"This setup allows teams to tightly control which Jira issues trigger automation, and ensures GitHub receives structured, clean metadata to act on. We can also set up multiple labels, each triggering a different GitHub Action. For example, one label could kick off a quick bug fix workflow, while another might start work on refactoring code or generating API stubs.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## Add the GitHub Action\n",
"\n",
"GitHub Actions enable you to automate workflows within your GitHub repository by defining them in YAML files. These workflows specify a series of jobs and steps to execute. When triggered either manually or via a POST request, GitHub automatically provisions the necessary environment and runs the defined workflow steps.\n",
"\n",
"To process the `POST` request from JIRA we will create a Github action with a YAML like below in the `.github/workflows/` directory of the repository:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```yaml\n",
"name: Codex Automated PR\n",
"on:\n",
" workflow_dispatch:\n",
" inputs:\n",
" issue_key:\n",
" description: 'JIRA issue key (e.g., PROJ-123)'\n",
" required: true\n",
" issue_summary:\n",
" description: 'Brief summary of the issue'\n",
" required: true\n",
" issue_description:\n",
" description: 'Detailed issue description'\n",
" required: true\n",
"\n",
"permissions:\n",
" contents: write # allow the action to push code & open the PR\n",
" pull-requests: write # allow the action to create and update PRs\n",
"\n",
"jobs:\n",
" codex_auto_pr:\n",
" runs-on: ubuntu-latest\n",
"\n",
" steps:\n",
" # 0 – Checkout repository\n",
" - uses: actions/checkout@v4\n",
" with:\n",
" fetch-depth: 0 # full history → lets Codex run tests / git blame if needed\n",
"\n",
" # 1 – Set up Node.js and Codex\n",
" - uses: actions/setup-node@v4\n",
" with:\n",
" node-version: 22\n",
" - run: pnpm add -g @openai/codex\n",
"\n",
" # 2 – Export / clean inputs (available via $GITHUB_ENV)\n",
" - id: vars\n",
" run: |\n",
" echo \"ISSUE_KEY=${{ github.event.inputs.issue_key }}\" >> $GITHUB_ENV\n",
" echo \"TITLE=${{ github.event.inputs.issue_summary }}\" >> $GITHUB_ENV\n",
" echo \"RAW_DESC=${{ github.event.inputs.issue_description }}\" >> $GITHUB_ENV\n",
" DESC_CLEANED=$(echo \"${{ github.event.inputs.issue_description }}\" | tr '\\n' ' ' | sed 's/\"/'\\''/g')\n",
" echo \"DESC=$DESC_CLEANED\" >> $GITHUB_ENV\n",
" echo \"BRANCH=codex/${{ github.event.inputs.issue_key }}\" >> $GITHUB_ENV\n",
"\n",
" # 3 – Transition Jira issue to \"In Progress\"\n",
" - name: Jira – Transition to In Progress\n",
" env:\n",
" ISSUE_KEY: ${{ env.ISSUE_KEY }}\n",
" JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}\n",
" JIRA_EMAIL: ${{ secrets.JIRA_EMAIL }}\n",
" JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}\n",
" run: |\n",
" curl -sS -X POST \\\n",
" --url \"$JIRA_BASE_URL/rest/api/3/issue/$ISSUE_KEY/transitions\" \\\n",
" --user \"$JIRA_EMAIL:$JIRA_API_TOKEN\" \\\n",
" --header 'Content-Type: application/json' \\\n",
" --data '{\"transition\":{\"id\":\"21\"}}'\n",
" # 21 is the transition ID for changing the ticket status to In Progress. Learn more here: https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issues/#api-rest-api-3-issue-issueidorkey-transitions-get\n",
"\n",
" # 4 – Set Git author for CI commits\n",
" - run: |\n",
" git config user.email \"github-actions[bot]@users.noreply.github.com\"\n",
" git config user.name \"github-actions[bot]\"\n",
"\n",
" # 5 – Let Codex implement & commit (no push yet)\n",
" - name: Codex implement & commit\n",
" env:\n",
" OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n",
" CODEX_QUIET_MODE: \"1\" # suppress chatty logs\n",
" run: |\n",
" set -e\n",
" codex --approval-mode full-auto --no-terminal --quiet \\\n",
" \"Implement JIRA ticket $ISSUE_KEY: $TITLE. $DESC\"\n",
"\n",
" git add -A\n",
" git commit -m \"feat($ISSUE_KEY): $TITLE\"\n",
"\n",
" # 6 – Open (and push) the PR in one go\n",
" - id: cpr\n",
" uses: peter-evans/create-pull-request@v6\n",
" with:\n",
" token: ${{ secrets.GITHUB_TOKEN }}\n",
" base: main\n",
" branch: ${{ env.BRANCH }}\n",
" title: \"${{ env.TITLE }} (${{ env.ISSUE_KEY }})\"\n",
" body: |\n",
" Auto-generated by Codex for JIRA **${{ env.ISSUE_KEY }}**.\n",
" ---\n",
" ${{ env.DESC }}\n",
"\n",
" # 7 – Transition Jira to \"In Review\" & drop the PR link\n",
" - name: Jira – Transition to In Review & Comment PR link\n",
" env:\n",
" ISSUE_KEY: ${{ env.ISSUE_KEY }}\n",
" JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}\n",
" JIRA_EMAIL: ${{ secrets.JIRA_EMAIL }}\n",
" JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}\n",
" PR_URL: ${{ steps.cpr.outputs.pull-request-url }}\n",
" run: |\n",
" # Status transition\n",
" curl -sS -X POST \\\n",
" --url \"$JIRA_BASE_URL/rest/api/3/issue/$ISSUE_KEY/transitions\" \\\n",
" --user \"$JIRA_EMAIL:$JIRA_API_TOKEN\" \\\n",
" --header 'Content-Type: application/json' \\\n",
" --data '{\"transition\":{\"id\":\"31\"}}'\n",
" # 31 is the Transition ID for changing the ticket status to In Review. Learn more here: https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issues/#api-rest-api-3-issue-issueidorkey-transitions-get\n",
"\n",
" # Comment with PR link\n",
" curl -sS -X POST \\\n",
" --url \"$JIRA_BASE_URL/rest/api/3/issue/$ISSUE_KEY/comment\" \\\n",
" --user \"$JIRA_EMAIL:$JIRA_API_TOKEN\" \\\n",
" --header 'Content-Type: application/json' \\\n",
" --data \"{\\\"body\\\":{\\\"type\\\":\\\"doc\\\",\\\"version\\\":1,\\\"content\\\":[{\\\"type\\\":\\\"paragraph\\\",\\\"content\\\":[{\\\"type\\\":\\\"text\\\",\\\"text\\\":\\\"PR created: $PR_URL\\\"}]}]}}\"\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## Key Steps in the Workflow\n",
"\n",
"1. **Codex Implementation & Commit** (Step 5)\n",
" - Uses OpenAI API to implement the JIRA ticket requirements\n",
" - Runs codex CLI in full-auto mode without terminal interaction\n",
" - Commits all changes with standardized commit message\n",
"\n",
"2. **Create Pull Request** (Step 6) \n",
" - Uses peter-evans/create-pull-request action\n",
" - Creates PR against main branch\n",
" - Sets PR title and description from JIRA ticket info\n",
" - Returns PR URL for later use\n",
"\n",
"3. **JIRA Updates** (Step 7)\n",
" - Transitions ticket to \"In Review\" status via JIRA API\n",
" - Posts comment with PR URL on the JIRA ticket\n",
" - Uses curl commands to interact with JIRA REST API\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## Label an Issue\n",
"\n",
"Attach the special `aswe` label to any bug/feature ticket:\n",
"\n",
"1. **During creation** – add it in the \"Labels\" field before hitting *Create* \n",
"2. **Existing issue** – hover the label area → click the pencil icon → type `aswe`\n",
"\n",
"<img src=\"../../images/add_label.png\" alt=\"Adding a label\" width=\"500\"/>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## End-to-end Flow\n",
"\n",
"1. Jira label added → Automation triggers\n",
"2. `workflow_dispatch` fires; action spins up on GitHub\n",
"3. `codex-cli` edits the codebase & commits\n",
"4. PR is opened on the generated branch\n",
"5. Jira is moved to **In Review** and a comment with the PR URL is posted\n",
"6. Reviewers are notified per your normal branch protection settings\n",
"\n",
"<img src=\"../../images/jira_comment.png\" alt=\"Jira comment with PR link\" width=\"300\"/>\n",
"<img src=\"../../images/jira_status_change.png\" alt=\"Jira status transition to In Review\" width=\"300\"/>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"## Review & Merge the PR\n",
"\n",
"You can open the PR link posted in the JIRA ticket and check to see if everything looks good and then merge it. If you have branch protection and Smart Commits integration enabled, the Jira ticket will be automatically closed when the pull request is merged.\n",
"\n",
"## Conclusion\n",
"\n",
"This automation streamlines your development workflow by creating a seamless integration between Jira and GitHub:\n",
"\n",
"* **Automatic status tracking** - Tickets progress through your workflow without manual updates\n",
"* **Improved developer experience** - Focus on reviewing code quality instead of writing boilerplate code\n",
"* **Reduced handoff friction** - The PR is ready for review as soon as the ticket is labeled\n",
"\n",
"The `codex-cli` tool is a powerful AI coding assistant that automates repetitive programming tasks. You can explore more about it [here](https://github.com/openai/codex/)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/codex/secure_quality_gitlab.md
+613
View File
@@ -0,0 +1,613 @@
# Automating Code Quality and Security Fixes with Codex CLI in GitLab
## Introduction
When deploying production code, most teams rely on CI/CD pipelines to validate changes before merging. Reviewers typically look at unit test results, vulnerability scans, and code quality reports. Traditionally, these are produced by rule-based engines that catch known issues but often miss contextual or higher-order problems—while leaving developers with noisy results that are hard to prioritize or act on.
With LLMs, you can add a new layer of intelligence to this process: reasoning about code quality and interpreting security findings. By augmenting your GitLab pipelines with **OpenAI’s Codex CLI**, teams gain insights that go beyond static rules:
* **Code Quality** → Generate GitLab-compliant CodeClimate JSON reports that surface contextual issues directly in merge requests.
* **Security** → Post-process existing SAST results to consolidate duplicates, rank issues by exploitability, and provide clear, actionable remediation steps.
This guide shows how to integrate Codex CLI into a GitLab pipeline for both use cases—delivering structured, machine-readable reports alongside actionable, human-readable guidance.
## What is Codex CLI?
Codex CLI is an open-source command-line tool for bringing OpenAI’s reasoning models into your development workflow. For installation, usage, and full documentation, refer to the official repository: [github.com/openai/codex](https://github.com/openai/codex?utm_source=chatgpt.com).
In this cookbook, we’ll use **Full Auto mode** in an ephemeral GitLab runner to generate a standards-compliant JSON report.
### Pre-requisites
To follow along, you’ll need:
* A GitLab account and project
* A GitLab runner with **internet access** (we’ve tested this on a Linux runner with 2 vCPUs, 8GB memory and 30GB of storage)
* Runner must be able to connect to `api.openai.com`
* An **OpenAI API key** (`OPENAI_API_KEY`)
* GitLab CI/CD variables configured under **Settings → CI/CD → Variables**
## Example #1 - Using Codex CLI to Produce a Code Quality Report
### Background
This repository is a deliberately vulnerable Node.js Express demo app based on [GitLab's node express template](https://gitlab.com/gitlab-org/project-templates/express/-/tree/main), built to showcase static application security testing (SAST) and code quality scanning in GitLab CI/CD.
The code includes common pitfalls such as command injection, path traversal, unsafe `eval`, regex DoS, weak cryptography (MD5), and hardcoded secrets. It’s used to validate that Codex-powered analyzers produce GitLab-native reports (Code Quality and SAST) that render directly in merge requests.
The CI runs on GitLab SaaS runners with `node:24` images and a few extras (`jq`, `curl`, `ca-certificates`, `ajv-cli`). Jobs are hardened with `set -euo pipefail`, schema validation, and strict JSON markers to keep parsing reliable even if Codex output varies.
This pipeline pattern—prompt, JSON marker extraction, schema validation—can be adapted to other stacks, though prompt wording and schema rules may need tweaks. Since Codex runs in a sandbox, some system commands (like `awk` or `nl`) may be restricted.
Your team wants to ensure that **code quality checks run automatically** before any merge. To surface findings directly in GitLab’s merge request widget, reports must follow the **CodeClimate JSON format**. [Reference: GitLab Docs](https://docs.gitlab.com/ci/testing/code_quality/#import-code-quality-results-from-a-cicd-job)
### Code Quality CI/CD Job Example
Here’s a drop-in GitLab CI job using **Codex CLI** to produce a compliant JSON file:
```yaml
stages: [codex]
default:
image: node:24
variables:
CODEX_QA_PATH: "gl-code-quality-report.json"
CODEX_RAW_LOG: "artifacts/codex-raw.log"
# Strict prompt: must output a single JSON array (or []), no prose/markdown/placeholders.
CODEX_PROMPT: |
Review this repository and output a GitLab Code Quality report in CodeClimate JSON format.
RULES (must follow exactly):
- OUTPUT MUST BE A SINGLE JSON ARRAY. Example: [] or [ {...}, {...} ].
- If you find no issues, OUTPUT EXACTLY: []
- DO NOT print any prose, backticks, code fences, markdown, or placeholders.
- DO NOT write any files. PRINT ONLY between these two lines:
=== BEGIN_CODE_QUALITY_JSON ===
<JSON ARRAY HERE>
=== END_CODE_QUALITY_JSON ===
Each issue object MUST include these fields:
"description": String,
"check_name": String (short rule name),
"fingerprint": String (stable across runs for same issue),
"severity": "info"|"minor"|"major"|"critical"|"blocker",
"location": { "path": "<repo-relative-file>", "lines": { "begin": <line> } }
Requirements:
- Use repo-relative paths from the current checkout (no "./", no absolute paths).
- Use only files that actually exist in this repo.
- No trailing commas. No comments. No BOM.
- Prefer concrete, de-duplicated findings. If uncertain, omit the finding.
codex_review:
stage: codex
# Skip on forked MRs (no secrets available). Run only if OPENAI_API_KEY exists.
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_SOURCE_PROJECT_ID != $CI_PROJECT_ID'
when: never
- if: '$OPENAI_API_KEY'
when: on_success
- when: never
script:
- set -euo pipefail
- echo "PWD=$(pwd) CI_PROJECT_DIR=${CI_PROJECT_DIR}"
# Ensure artifacts always exist so upload never warns, even on early failure
- mkdir -p artifacts
- ': > ${CODEX_RAW_LOG}'
- ': > ${CODEX_QA_PATH}'
# Minimal deps + Codex CLI
- apt-get update && apt-get install -y --no-install-recommends curl ca-certificates git lsb-release
- npm -g i @openai/codex@latest
- codex --version && git --version
# Build a real-file allowlist to guide Codex to valid paths/lines
- FILE_LIST="$(git ls-files | sed 's/^/- /')"
- |
export CODEX_PROMPT="${CODEX_PROMPT}
Only report issues in the following existing files (exactly as listed):
${FILE_LIST}"
# Run Codex; allow non-zero exit but capture output for extraction
- |
set +o pipefail
script -q -c 'codex exec --full-auto "$CODEX_PROMPT"' | tee "${CODEX_RAW_LOG}" >/dev/null
CODEX_RC=${PIPESTATUS[0]}
set -o pipefail
echo "Codex exit code: ${CODEX_RC}"
# Strip ANSI + \r, extract JSON between markers to a temp file; validate or fallback to []
- |
TMP_OUT="$(mktemp)"
sed -E 's/\x1B\[[0-9;]*[A-Za-z]//g' "${CODEX_RAW_LOG}" \
| tr -d '\r' \
| awk '
/^\s*=== BEGIN_CODE_QUALITY_JSON ===\s*$/ {grab=1; next}
/^\s*=== END_CODE_QUALITY_JSON ===\s*$/ {grab=0}
grab
' > "${TMP_OUT}"
# If extracted content is empty/invalid or still contains placeholders, replace with []
if ! node -e 'const f=process.argv[1]; const s=require("fs").readFileSync(f,"utf8").trim(); if(!s || /(<JSON ARRAY HERE>|BEGIN_CODE_QUALITY_JSON|END_CODE_QUALITY_JSON)/.test(s)) process.exit(2); JSON.parse(s);' "${TMP_OUT}"; then
echo "WARNING: Extracted content empty/invalid; writing empty [] report."
echo "[]" > "${TMP_OUT}"
fi
mv -f "${TMP_OUT}" "${CODEX_QA_PATH}"
# Soft warning if Codex returned non-zero but we still produced a report
if [ "${CODEX_RC}" -ne 0 ]; then
echo "WARNING: Codex exited with code ${CODEX_RC}. Proceeding with extracted report." >&2
fi
artifacts:
when: always
reports:
codequality: gl-code-quality-report.json
paths:
- artifacts/codex-raw.log
expire_in: 14 days
```
1. Installs Codex CLI (`npm -g i @openai/codex@latest`)
2. Builds a file allowlist with `git ls-files`
3. Runs Codex in **full-auto mode** with a strict JSON-only prompt
4. Extracts valid JSON between markers, validates it, and falls back to `[]` if invalid
5. Publishes artifacts to GitLab so results appear inline with merge requests
The generated artifacts can be downloaded from the pipeline page
<img src="../../images/gitlab-pipelines-success.png" alt="GitLab Pipelines" width="700"/>
Or when running as a merge from a feature to master branch,
<img src="../../images/gitlab-mr-widget.png" alt="GitLab Merge Request Widget" width="700"/>
By embedding Codex CLI into your GitLab CI/CD pipelines, you can **elevate code quality checks beyond static rules**. Instead of only catching syntax errors or style violations, you enable reasoning-based analysis that highlights potential issues in context.
This approach has several benefits:
* **Consistency**: every merge request is reviewed by the same reasoning process
* **Context awareness**: LLMs can flag subtle issues rule-based scanners miss
* **Developer empowerment**: feedback is immediate, visible, and actionable
As teams adopt this workflow, LLM-powered quality checks can complement traditional linting and vulnerability scanning—helping ensure that code shipped to production is both robust and maintainable.
## Example #2 – Using Codex CLI for Security Remediation
### Background
For this example, we tested on [OWASP Juice Shop](https://github.com/juice-shop/juice-shop?utm_source=chatgpt.com), a deliberately vulnerable Node.js Express app. It contains common flaws such as injection, unsafe `eval`, weak crypto, and hardcoded secrets—ideal for validating Codex-powered analysis.
Your team wants to ensure that whenever code changes are introduced, the pipeline automatically checks for security vulnerabilities before merge. This is already handled by static analyzers and language-specific scanners, which generate reports in the GitLab SAST JSON schema. However, raw outputs can be rigid, noisy, and often leave reviewers without clear next steps.
By adding Codex CLI into your pipeline, you can turn scanner results generated by [GitLab SAST scanners](https://docs.gitlab.com/user/application_security/sast/) (or other scanner outputs) into **actionable remediation guidance** and even generate **ready-to-apply git patches**:
### Step 1: Generating Recommendations
* Codex reads `gl-sast-report.json`.
* Consolidates duplicate findings.
* Ranks by exploitability (e.g. user input → dangerous sinks).
* Produces a succinct `security_priority.md` with top 5 actions and detailed remediation notes.
#### Security Recommendations CI/CD Job Example
**Requirement**: This job expects that upstream SAST jobs already generated a `gl-sast-report.json`. Codex reads it and produces `security_priority.md` for reviewers.
```yaml
stages:
- codex
- remediation
default:
image: node:24
variables:
CODEX_SAST_PATH: "gl-sast-report.json"
CODEX_SECURITY_MD: "security_priority.md"
CODEX_RAW_LOG: "artifacts/codex-sast-raw.log"
# --- Recommendations prompt (reads SAST → writes Markdown) ---
CODEX_PROMPT: |
You are a security triage assistant analyzing GitLab SAST output.
The SAST JSON is located at: ${CODEX_SAST_PATH}
GOAL:
- Read and parse ${CODEX_SAST_PATH}.
- Consolidate duplicate or overlapping findings (e.g., same CWE + same sink/function, same file/line ranges, or same data flow root cause).
- Rank findings by realistic exploitability and business risk, not just library presence.
* Prioritize issues that:
- Are reachable from exposed entry points (HTTP handlers, controllers, public APIs, CLI args).
- Involve user-controlled inputs reaching dangerous sinks (e.g., SQL exec, OS exec, eval, path/file ops, deserialization, SSRF).
- Occur in authentication/authorization boundaries or around secrets/keys/tokens.
- Have clear call stacks/evidence strings pointing to concrete methods that run.
- Affect internet-facing or privileged components.
* De-prioritize purely theoretical findings with no reachable path or dead code.
CONSOLIDATION RULES:
- Aggregate by (CWE, primary sink/function, file[:line], framework route/handler) when applicable.
- Merge repeated instances across files if they share the same source-sink pattern and remediation is the same.
- Keep a single representative entry with a count of affected locations; list notable examples.
OUTPUT FORMAT (MARKDOWN ONLY, BETWEEN MARKERS BELOW):
- Start with a title and short summary of total findings and how many were consolidated.
- A table of TOP PRIORITIES sorted by exploitability (highest first) with columns:
Rank | CWE | Title | Affected Locations | Likely Exploit Path | Risk | Rationale (1–2 lines)
- "Top 5 Immediate Actions" list with concrete next steps.
- "Deduplicated Findings (Full Details)" with risk, 0–100 exploitability score, evidence (file:line + methods), remediation, owners, references.
- If ${CODEX_SAST_PATH} is missing or invalid JSON, output a brief note stating no parsable SAST findings.
RULES (must follow exactly):
- PRINT ONLY between these two lines:
=== BEGIN_SECURITY_MD ===
<MARKDOWN CONTENT HERE>
=== END_SECURITY_MD ===
- No prose, backticks, code fences, or anything outside the markers.
- Be concise but specific. Cite only evidence present in the SAST report.
# ---------------------------
# Stage: codex → Job 1 (Recommendations)
# ---------------------------
codex_recommendations:
stage: codex
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_SOURCE_PROJECT_ID != $CI_PROJECT_ID'
when: never
- if: '$OPENAI_API_KEY'
when: on_success
- when: never
script:
- set -euo pipefail
- mkdir -p artifacts
- ": > ${CODEX_RAW_LOG}"
- ": > ${CODEX_SECURITY_MD}"
- apt-get update && apt-get install -y --no-install-recommends curl ca-certificates git lsb-release
- npm -g i @openai/codex@latest
- codex --version && git --version
- |
if [ ! -s "${CODEX_SAST_PATH}" ]; then
echo "WARNING: ${CODEX_SAST_PATH} not found or empty. Codex will emit a 'no parsable findings' note."
fi
- FILE_LIST="$(git ls-files | sed 's/^/- /')"
- |
export CODEX_PROMPT="${CODEX_PROMPT}
Existing repository files (for reference only; use paths exactly as listed in SAST evidence):
${FILE_LIST}"
# Run Codex and capture raw output (preserve Codex's exit code via PIPESTATUS)
- |
set +o pipefail
codex exec --full-auto "$CODEX_PROMPT" | tee "${CODEX_RAW_LOG}" >/dev/null
CODEX_RC=${PIPESTATUS[0]}
set -o pipefail
echo "Codex exit code: ${CODEX_RC}"
# Extract markdown between markers; fallback to a minimal note
- |
TMP_OUT="$(mktemp)"
sed -E 's/\x1B\[[0-9;]*[A-Za-z]//g' "${CODEX_RAW_LOG}" | tr -d '\r' | awk '
/^\s*=== BEGIN_SECURITY_MD ===\s*$/ {grab=1; next}
/^\s*=== END_SECURITY_MD ===\s*$/ {grab=0}
grab
' > "${TMP_OUT}"
if ! [ -s "${TMP_OUT}" ]; then
cat > "${TMP_OUT}" <<'EOF' # Security Findings Priority
No parsable SAST findings detected in `gl-sast-report.json`._
EOF
echo "WARNING: No content extracted; wrote minimal placeholder."
fi
mv -f "${TMP_OUT}" "${CODEX_SECURITY_MD}"
if [ "${CODEX_RC}" -ne 0 ]; then
echo "WARNING: Codex exited with code ${CODEX_RC}. Proceeding with extracted report." >&2
fi
artifacts:
when: always
paths:
- artifacts/codex-sast-raw.log
- security_priority.md
expire_in: 14 days
```
Here's an example of the output we receive:
### Example Output: Consolidated SAST Findings
Parsed `gl-sast-report.json` and merged overlapping issues.
**Total raw findings:** 5 → **Consolidated into:** 4 representative entries
(duplicated SQL injection patterns across endpoints were merged).
#### Summary Table
| Rank | CWE | Title | Affected Locations | Likely Exploit Path | Risk | Rationale (1–2 lines) |
|------|----------|--------------------------------------|-------------------|--------------------------------------|----------|--------------------------------------------------------------------------------------------------------|
| 1 | CWE-798 | Hardcoded JWT private key | 1 | Auth token issuance / verification | Critical | Repo leak enables minting valid admin JWTs; trivial exploitation, internet-facing. |
| 2 | CWE-89 | SQL injection in login and search | 2 | Login endpoint; product search | Critical | Raw SQL concatenation; direct login bypass and data exfiltration via public HTTP handlers. |
| 3 | CWE-94 | Server-side code injection via eval | 1 | User profile update handler | High | `eval()` on user input allows RCE; conditionally enabled but still high-impact when reachable. |
| 4 | — (SSRF) | SSRF via arbitrary image URL fetch | 1 | Image URL fetch/write flow | High | Outbound fetch of unvalidated URLs enables internal service / metadata access (e.g., AWS metadata). |
#### Top 5 Immediate Actions
1. Replace hardcoded JWT signing key in `lib/insecurity.ts:23`; load from secret storage, rotate keys, and invalidate existing tokens.
2. Update `routes/login.ts:34` to use parameterized queries; remove raw concatenation; validate and escape inputs.
3. Fix `routes/search.ts:23` by using ORM bind parameters or escaped `LIKE` helpers instead of string concatenation.
4. Refactor `routes/userProfile.ts:55–66`; replace `eval()` with safe templating or a whitelisted evaluator.
5. Harden image import logic: allowlist schemes/hosts, block link-local/metadata IPs, apply timeouts and size limits.
##### Deduplicated Findings (Full Details)
##### 1. CWE-798 — Hardcoded JWT private key
- Risk: Critical — Exploitability 98/100
- Evidence:
- File: `lib/insecurity.ts:23`
- Message: RSA private key embedded in source enables forged admin tokens
- Suggested Remediation: Remove key from source, load via env/secret manager, rotate keys, enforce short TTLs
- Owners/Teams: Backend/Core (lib)
- References: CWE-798; OWASP ASVS 2.1.1, 2.3.1
---
##### 2. CWE-89 — SQL injection (login & search)
- Risk: Critical — Exploitability 95/100
- Evidence:
- `routes/login.ts:34` — classic login bypass via `' OR 1=1--`
- `routes/search.ts:23` — UNION-based extraction via `%25' UNION SELECT ...`
- Suggested Remediation: Use parameterized queries/ORM, validate inputs, add WAF/error suppression
- Owners/Teams: Backend/API (routes)
- References: CWE-89; OWASP Top 10 A03:2021; ASVS 5.3
---
##### 3. CWE-94 — Server-side code injection (`eval`)
- Risk: High — Exploitability 72/100
- Evidence:
- `routes/userProfile.ts:55–66` — `eval()` used for dynamic username patterns
- Suggested Remediation: Remove `eval()`, or sandbox with strict whitelist; validate/encode inputs
- Owners/Teams: Backend/API (routes)
- References: CWE-94; OWASP Top 10 A03:2021
---
##### 4. SSRF — Arbitrary image URL fetch
- Risk: High — Exploitability 80/100
- Evidence:
- Image import fetches arbitrary `imageUrl` → can hit internal services (`169.254.169.254`)
- Suggested Remediation: Enforce HTTPS + DNS/IP allowlist, block RFC1918/link-local, validate post-resolution, no redirects
- Owners/Teams: Backend/API (routes)
- References: OWASP SSRF Prevention; OWASP Top 10 A10:2021
---
### Step 2: Remediating Security Issues Based on Recommendations
- Codex consumes both the SAST JSON and the repo tree.
- For each High/Critical issue:
- Builds a structured prompt → outputs a unified `git diff`.
- Diff is validated (`git apply --check`) before being stored as `.patch`.
#### Remediation CI/CD Job Example
**Requirement**: This job depends on the previous stage output of the `security_priority.md` file to use as input to generate the patch file for creating an MR:
```yaml
stages:
- remediation
default:
image: node:24
variables:
# Inputs/outputs
SAST_REPORT_PATH: "gl-sast-report.json"
PATCH_DIR: "codex_patches"
CODEX_DIFF_RAW: "artifacts/codex-diff-raw.log"
# --- Resolution prompt (produces unified git diffs only) ---
CODEX_DIFF_PROMPT: |
You are a secure code remediation assistant.
You will receive:
- The repository working tree (read-only)
- One vulnerability (JSON from a GitLab SAST report)
- Allowed files list
GOAL:
- Create the minimal, safe fix for the vulnerability.
- Output a unified git diff that applies cleanly with `git apply -p0` (or -p1 for a/ b/ paths).
- Prefer surgical changes: input validation, safe APIs, parameterized queries, permission checks.
- Do NOT refactor broadly or change unrelated code.
RULES (must follow exactly):
- PRINT ONLY the diff between the markers below.
- Use repo-relative paths; `diff --git a/path b/path` headers are accepted.
- No binary file changes. No prose/explanations outside the markers.
MARKERS:
=== BEGIN_UNIFIED_DIFF ===
<unified diff here>
=== END_UNIFIED_DIFF ===
If no safe fix is possible without deeper changes, output an empty diff between the markers.
# ---------------------------
# Stage: remediation → Generate unified diffs/patches
# ---------------------------
codex_resolution:
stage: remediation
rules:
- if: '$OPENAI_API_KEY'
when: on_success
- when: never
script:
- set -euo pipefail
- mkdir -p "$PATCH_DIR" artifacts
# Deps
- apt-get update && apt-get install -y --no-install-recommends bash git jq curl ca-certificates
- npm -g i @openai/codex@latest
- git --version && codex --version || true
# Require SAST report; no-op if missing
- |
if [ ! -s "${SAST_REPORT_PATH}" ]; then
echo "No SAST report found; remediation will no-op."
printf "CODEX_CREATED_PATCHES=false\n" > codex.env
exit 0
fi
# Pull High/Critical items
- jq -c '.vulnerabilities[]? | select((.severity|ascii_downcase)=="high" or (.severity|ascii_downcase)=="critical")' "$SAST_REPORT_PATH" \
| nl -ba > /tmp/hicrit.txt || true
- |
if [ ! -s /tmp/hicrit.txt ]; then
echo "No High/Critical vulnerabilities found. Nothing to fix."
printf "CODEX_CREATED_PATCHES=false\n" > codex.env
exit 0
fi
# Ground Codex to actual repo files
- FILE_LIST="$(git ls-files | sed 's/^/- /')"
# Identity for any local patch ops
- git config user.name "CI Codex Bot"
- git config user.email "codex-bot@example.com"
- created=0
# Loop: build prompt (robust temp-file), run Codex, extract diff, validate
- |
while IFS=$'\t' read -r idx vuln_json; do
echo "Processing vulnerability #$idx"
echo "$vuln_json" > "/tmp/vuln-$idx.json"
PROMPT_FILE="$(mktemp)"
{
printf "%s\n\n" "$CODEX_DIFF_PROMPT"
printf "VULNERABILITY_JSON:\n<<JSON\n"
cat "/tmp/vuln-$idx.json"
printf "\nJSON\n\n"
printf "EXISTING_REPOSITORY_FILES (exact list):\n"
printf "%s\n" "$FILE_LIST"
} > "$PROMPT_FILE"
PER_FINDING_PROMPT="$(tr -d '\r' < "$PROMPT_FILE")"
rm -f "$PROMPT_FILE"
: > "$CODEX_DIFF_RAW"
set +o pipefail
codex exec --full-auto "$PER_FINDING_PROMPT" | tee -a "$CODEX_DIFF_RAW" >/dev/null
RC=${PIPESTATUS[0]}
set -o pipefail
echo "Codex (diff) exit code: ${RC}"
OUT_PATCH="$PATCH_DIR/fix-$idx.patch"
sed -E 's/\x1B\[[0-9;]*[A-Za-z]//g' "$CODEX_DIFF_RAW" \
| tr -d '\r' \
| awk '
/^\s*=== BEGIN_UNIFIED_DIFF ===\s*$/ {grab=1; next}
/^\s*=== END_UNIFIED_DIFF ===\s*$/ {grab=0}
grab
' > "$OUT_PATCH"
if ! [ -s "$OUT_PATCH" ] || ! grep -qE '^\s*diff --git ' "$OUT_PATCH"; then
echo " No usable diff produced for #$idx; skipping."
rm -f "$OUT_PATCH"
continue
fi
# Validate: accept -p0 (repo-relative) or -p1 (a/ b/ prefixes)
if git apply --check -p0 "$OUT_PATCH" || git apply --check -p1 "$OUT_PATCH"; then
echo " Patch validated → $OUT_PATCH"
created=$((created+1))
else
echo " Patch failed to apply cleanly; removing."
rm -f "$OUT_PATCH"
fi
done < /tmp/hicrit.txt
if [ "$created" -gt 0 ]; then
printf "CODEX_CREATED_PATCHES=true\nPATCH_DIR=%s\n" "$PATCH_DIR" > codex.env
else
printf "CODEX_CREATED_PATCHES=false\n" > codex.env
fi
artifacts:
when: always
paths:
- codex_patches/
- artifacts/codex-diff-raw.log
reports:
dotenv: codex.env
expire_in: 14 days
```
Running the CI/CD job with Codex CLI, we receive a Git patch that fixes the issues originally found by our security scanner:
```patch
<unified diff here>
diff --git a/routes/profileImageUrlUpload.ts b/routes/profileImageUrlUpload.ts
index 9b4a62d..c7f1a7e 100644
--- a/routes/profileImageUrlUpload.ts
+++ b/routes/profileImageUrlUpload.ts
@@ -5,17 +5,12 @@
* SPDX-License-Identifier: MIT
*/
-import fs from 'node:fs'
-import { Readable } from 'node:stream'
-import { finished } from 'node:stream/promises'
import { type Request, type Response, type NextFunction } from 'express'
import * as security from '../lib/insecurity'
import { UserModel } from '../models/user'
-import * as utils from '../lib/utils'
-import logger from '../lib/logger'
export function profileImageUrlUpload () {
return async (req: Request, res: Response, next: NextFunction) => {
if (req.body.imageUrl !== undefined) {
const url = req.body.imageUrl
if (url.match(/(.)*solve\\/challenges\\/server-side(.)*/) !== null) req.app.locals.abused_ssrf_bug = true
const loggedInUser = security.authenticatedUsers.get(req.cookies.token)
if (loggedInUser) {
- try {
- const response = await fetch(url)
- if (!response.ok || !response.body) {
- throw new Error('url returned a non-OK status code or an empty body')
- }
- const ext = ['jpg', 'jpeg', 'png', 'svg', 'gif'].includes(url.split('.').slice(-1)[0].toLowerCase()) ? url.split('.').slice(-1)[0].toLowerCase() : 'jpg'
- const fileStream = fs.createWriteStream(`frontend/dist/frontend/assets/public/images/uploads/${loggedInUser.data.id}.${ext}`, { flags: 'w' })
- await finished(Readable.fromWeb(response.body as any).pipe(fileStream))
- await UserModel.findByPk(loggedInUser.data.id).then(async (user: UserModel | null) => { return await user?.update({ profileImage: `/assets/public/images/uploads/${loggedInUser.data.id}.${ext}` }) }).catch((error: Error) => { next(error) })
- } catch (error) {
- try {
- const user = await UserModel.findByPk(loggedInUser.data.id)
- await user?.update({ profileImage: url })
- logger.warn(`Error retrieving user profile image: ${utils.getErrorMessage(error)}; using image link directly`)
- } catch (error) {
- next(error)
- return
- }
- }
+ try {
+ const user = await UserModel.findByPk(loggedInUser.data.id)
+ await user?.update({ profileImage: url })
+ } catch (error) {
+ next(error)
+ return
+ }
} else {
next(new Error('Blocked illegal activity by ' + req.socket.remoteAddress))
return
```
## Key Benefits
Using Codex CLI in GitLab CI/CD allows you to augment existing review processes so that your team can ship faster.
* **Complementary**: Codex doesn’t replace scanners — it interprets their findings and accelerates fixes.
* **Actionable**: Reviewers see not just vulnerabilities, but prioritized steps to fix them.
* **Automated**: Patches are created directly in CI, ready for `git apply` or a remediation branch.
---
## Wrapping Up
In this cookbook, we explored how **Codex CLI** can be embedded into GitLab CI/CD pipelines to make software delivery safer and more maintainable:
* **Code Quality Reports**: Generate GitLab-compliant CodeClimate JSON so reasoning-based findings surface alongside lint, unit tests, and style checks.
* **Vulnerability Interpretation**: Take the raw output of SAST and other security scanners (`gl-sast-report.json`) and transform it into a prioritized, human-readable plan (`security_priority.md`) with deduplication, exploitability ranking, and actionable next steps.
* **Automated Remediation**: Extend the workflow by having Codex generate unified git diffs as `.patch` files. These patches are validated (`git apply --check`) and can be applied automatically to a new branch.
Taken together, these patterns show how **LLM-powered analysis complements—not replaces—traditional rule-based tools**. Scanners remain the source of truth for detection, while Codex adds context awareness, prioritization, developer guidance, and even the ability to propose concrete fixes via MRs. GitLab’s schemas and APIs provide the structure to make these outputs predictable, actionable, and fully integrated into developer workflows.
The critical lesson is that good results require **clear prompts, schema validation, and guardrails**. JSON markers, severity whitelists, schema enforcement, and diff validation ensure outputs are usable.
Looking forward, this pattern can be extended to unify all major scan types through a single Codex-powered remediation flow:
* **Dependency Scans** → consolidate CVEs across lockfiles, recommend upgrades, and auto-generate diffs bumping vulnerable versions.
* **Container/Image Scans** → flag outdated base images and propose Dockerfile updates.
* **DAST Results** → highlight exploitable endpoints and patch routing/middleware for validation or access control.
By merging these into a single Codex-powered post-processing \+ remediation pipeline, teams can get a consistent stream of **actionable guidance, validated patches** across all security domains.
**The broader takeaway:** with prompt engineering, schema validation, and integration into GitLab’s native MR workflow, LLMs evolve from “advisors” into **first-class CI/CD agents** — helping teams ship code that is not only functional, but also secure, maintainable, and automatically remediated where possible.
examples/codex/using_goals_in_codex.ipynb
+651
View File
File diff suppressed because one or more lines are too long