chore: import upstream snapshot with attribution
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,17 @@
|
||||
# GitHub Actions (maintainer reference)
|
||||
|
||||
Internal notes for repository automation under `.github/workflows/`. Not published on the docs site.
|
||||
|
||||
## Workflows
|
||||
|
||||
| Workflow | Purpose |
|
||||
| -------- | ------- |
|
||||
| [`ci.yml`](ci.yml) | PR/push quality gates and sharded pytest |
|
||||
| [`ci-labels-windows.yml`](ci-labels-windows.yml) | Optional Windows CI (`ci:windows` label) |
|
||||
| [`codeql.yml`](codeql.yml) | CodeQL security analysis |
|
||||
| [`greptile-pr-reminder.yml`](greptile-pr-reminder.yml) | Greptile review nudge on PR open |
|
||||
| [`celebrate-merged-pr.yml`](celebrate-merged-pr.yml) | Post-merge celebration comment |
|
||||
| [`good-first-issue-assign.yml`](good-first-issue-assign.yml) | Auto-assign good first issues |
|
||||
| [`release.yml`](release.yml) | Release builds and artifacts |
|
||||
|
||||
See [CI.md](../../CI.md) for local parity commands before push.
|
||||
@@ -0,0 +1,176 @@
|
||||
name: Benchmark image — build + push to ECR (any adapter)
|
||||
|
||||
# Adapter-agnostic image build. The bench image carries the full
|
||||
# ``tests/benchmarks/`` tree, so every registered adapter ships in the
|
||||
# same image. ``benchmark-run.yml`` selects which adapter actually runs
|
||||
# via its ``config`` input. See ``tests/benchmarks/_framework/registry.py``
|
||||
# for the registration contract.
|
||||
|
||||
# Builds tests/benchmarks/cloudopsbench/infra/Dockerfile.bench and pushes the resulting image to the
|
||||
# opensre-bench ECR repository. The bench container is what
|
||||
# `Benchmark run (manual)` invokes on AWS Fargate, so this workflow's
|
||||
# output is the input to that one.
|
||||
#
|
||||
# Triggered automatically on changes to the bench code (or the Dockerfile
|
||||
# itself) and manually via workflow_dispatch.
|
||||
#
|
||||
# After the image is pushed, update the task definition by re-applying the
|
||||
# Terraform with the new tag:
|
||||
#
|
||||
# cd tests/benchmarks/cloudopsbench/infra
|
||||
# terraform apply -var="image_tag=<TAG>"
|
||||
#
|
||||
# (Or wire a follow-up step into this workflow to update task definition
|
||||
# automatically — out of scope for v1.)
|
||||
#
|
||||
# Tag format: short git SHA (`git rev-parse --short HEAD`). Stable, unique
|
||||
# per commit, recognizable in `aws ecr describe-images` output. ECR is
|
||||
# IMMUTABLE — a tag pushed once cannot be overwritten, so re-running the
|
||||
# workflow on the same commit just re-tags (no-op layer push).
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tag:
|
||||
description: Image tag to push (default = short git SHA)
|
||||
required: false
|
||||
type: string
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- "tests/benchmarks/cloudopsbench/infra/Dockerfile.bench"
|
||||
- "tests/benchmarks/cloudopsbench/infra/Dockerfile.bench.dockerignore"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "surfaces/**"
|
||||
- "config/**"
|
||||
- "core/**"
|
||||
- "platform/deployment/**"
|
||||
- "integrations/**"
|
||||
- "platform/**"
|
||||
- "tools/**"
|
||||
- "tests/benchmarks/**"
|
||||
- ".github/workflows/benchmark-image.yml"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # required for AWS OIDC role assumption
|
||||
|
||||
concurrency:
|
||||
# Allow one in-flight build per ref so two pushes in quick succession
|
||||
# don't race ECR. The newer push cancels the older.
|
||||
group: bench-image-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
AWS_REGION: us-east-1
|
||||
# Account ID is repo-level configuration, not a secret. Set as a GitHub
|
||||
# repository Variable (Settings > Secrets and variables > Actions >
|
||||
# Variables) so it doesn't need to be edited in every workflow file
|
||||
# when the bench moves accounts.
|
||||
ACCOUNT_ID: ${{ vars.AWS_ACCOUNT_ID }}
|
||||
ECR_REPOSITORY: opensre-bench
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
name: build + push
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v5
|
||||
with:
|
||||
# Fetch enough history that `git rev-parse --short HEAD` is stable
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Resolve image tag
|
||||
id: tag
|
||||
env:
|
||||
# Route `inputs.tag` through env so the shell treats it as DATA,
|
||||
# not code. Without this, a workflow_dispatch caller could inject
|
||||
# shell commands via a crafted `tag` value containing $() or
|
||||
# backticks. See:
|
||||
# https://securitylab.github.com/research/github-actions-untrusted-input/
|
||||
INPUT_TAG: ${{ inputs.tag }}
|
||||
run: |
|
||||
if [ -n "$INPUT_TAG" ]; then
|
||||
TAG="$INPUT_TAG"
|
||||
else
|
||||
TAG="$(git rev-parse --short HEAD)"
|
||||
fi
|
||||
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
|
||||
echo "Will push: $ECR_REPOSITORY:$TAG"
|
||||
|
||||
- name: Configure AWS credentials (OIDC role assumption)
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/opensre-bench-github-actions
|
||||
role-session-name: github-actions-bench-image-${{ github.run_id }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Login to Amazon ECR
|
||||
id: ecr-login
|
||||
uses: aws-actions/amazon-ecr-login@v2
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
# Buildx adds multi-platform support + better cache control. Even
|
||||
# for single-platform builds, it's the modern default.
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Build and push
|
||||
id: build
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: tests/benchmarks/cloudopsbench/infra/Dockerfile.bench
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
tags: |
|
||||
${{ steps.ecr-login.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ steps.tag.outputs.tag }}
|
||||
# Stamp the COMMIT SHA into the image so the runtime can read it
|
||||
# via the OPENSRE_SHA env var. We use github.sha (the full 40-char
|
||||
# commit being built), NOT steps.tag.outputs.tag (which resolves
|
||||
# to the user-supplied inputs.tag like ``hotfix-june`` on
|
||||
# workflow_dispatch and would stamp an unverifiable string as if
|
||||
# it were a SHA). The image tag (for ECR naming) and OPENSRE_SHA
|
||||
# (for provenance) are intentionally decoupled — the tag is for
|
||||
# humans/operators; the SHA is for reproducibility. The runtime
|
||||
# gate also validates SHA shape (7-40 lowercase hex chars), so
|
||||
# even a manually-built image with a bad OPENSRE_SHA fails loudly.
|
||||
build-args: |
|
||||
OPENSRE_SHA=${{ github.sha }}
|
||||
# GitHub Actions cache for Docker layers - speeds up rebuilds
|
||||
# when only the source changes (deps stay in the cached layer).
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
provenance: false # smaller manifest; matches ECR's tolerance for OCI
|
||||
|
||||
- name: Summarise
|
||||
# Route step outputs through env so the shell sees them as DATA,
|
||||
# not code. `steps.tag.outputs.tag` is derived verbatim from the
|
||||
# `inputs.tag` user input — without env scoping, a workflow_dispatch
|
||||
# caller could inject shell commands here even though the
|
||||
# "Resolve image tag" step is hardened. Same risk applies to any
|
||||
# tag-derived chain.
|
||||
env:
|
||||
REGISTRY: ${{ steps.ecr-login.outputs.registry }}
|
||||
TAG: ${{ steps.tag.outputs.tag }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
run: |
|
||||
echo "## Image pushed" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| field | value |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| --- | --- |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Registry | \`$REGISTRY\` |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Repository | \`$ECR_REPOSITORY\` |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Tag | \`$TAG\` |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Digest | \`$DIGEST\` |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "To deploy this image:" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "\`\`\`bash" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "cd tests/benchmarks/cloudopsbench/infra" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "terraform apply -var=\"image_tag=$TAG\"" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "\`\`\`" >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,135 @@
|
||||
name: Benchmark image — promote tag to task definition (any adapter)
|
||||
|
||||
# Adapter-agnostic image promotion. Rebinds the ECS task definition to
|
||||
# a specific image tag. The image carries every registered adapter; the
|
||||
# config supplied to ``benchmark-run.yml`` picks which one runs.
|
||||
#
|
||||
# Manually-triggered workflow that runs `terraform apply -var=image_tag=<TAG>`
|
||||
# in tests/benchmarks/cloudopsbench/infra/ to register a new ECS task definition revision pointing at
|
||||
# the chosen ECR image. This is the privileged "deploy" step that comes
|
||||
# between an image push (automatic) and a bench run (manual).
|
||||
#
|
||||
# Why not auto-promote on every image push? An image build is a code-change
|
||||
# event. A task-def update is a deploy event. Decoupling them lets you
|
||||
# stage many images and choose deliberately which one production runs.
|
||||
#
|
||||
# Trigger from the GitHub UI:
|
||||
# Actions → "Benchmark image — promote tag to task definition" → Run
|
||||
#
|
||||
# Pre-reqs (one-time):
|
||||
# - tests/benchmarks/cloudopsbench/infra/ Terraform applied at least once. The opensre-bench-github-actions
|
||||
# OIDC role's permissions (ecs:RegisterTaskDefinition, state-bucket read/write,
|
||||
# lock-table read/write, iam:PassRole) are granted by the github_actions_run_bench
|
||||
# inline policy in tests/benchmarks/cloudopsbench/infra/iam_oidc.tf — any apply of that module attaches them.
|
||||
# - Repo secrets seeded into AWS Secrets Manager
|
||||
# - Repo vars set (AWS_ACCOUNT_ID etc., see tests/benchmarks/cloudopsbench/infra/AWS_BENCH_SETUP.md step 4)
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
image_tag:
|
||||
description: 'ECR image tag to promote (e.g. 3792493)'
|
||||
required: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # required for AWS OIDC role assumption
|
||||
|
||||
concurrency:
|
||||
group: benchmark-promote-image
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
promote:
|
||||
name: terraform apply image_tag=${{ inputs.image_tag }}
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
env:
|
||||
AWS_REGION: us-east-1
|
||||
|
||||
steps:
|
||||
- name: Verify required repo variables
|
||||
env:
|
||||
AWS_ACCOUNT_ID: ${{ vars.AWS_ACCOUNT_ID }}
|
||||
run: |
|
||||
if [ -z "${AWS_ACCOUNT_ID:-}" ]; then
|
||||
echo "::error::Missing repo variable AWS_ACCOUNT_ID. See tests/benchmarks/cloudopsbench/infra/AWS_BENCH_SETUP.md."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Configure AWS credentials (OIDC role assumption)
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/opensre-bench-github-actions
|
||||
role-session-name: bench-promote-${{ github.run_id }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Verify image tag exists in ECR
|
||||
# Fail loudly if the operator typos the tag, before Terraform tries
|
||||
# to register a task definition pointing at a missing image.
|
||||
env:
|
||||
IMAGE_TAG: ${{ inputs.image_tag }}
|
||||
run: |
|
||||
if ! aws ecr describe-images \
|
||||
--repository-name opensre-bench \
|
||||
--image-ids imageTag="$IMAGE_TAG" \
|
||||
>/dev/null 2>&1; then
|
||||
echo "::error::Image tag $IMAGE_TAG not found in ECR repo opensre-bench."
|
||||
echo "::error::Push it first via 'Benchmark image — build + push to ECR'."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: hashicorp/setup-terraform@v3
|
||||
with:
|
||||
terraform_version: 1.7.5
|
||||
|
||||
- name: Terraform init
|
||||
working-directory: tests/benchmarks/cloudopsbench/infra
|
||||
run: terraform init -input=false
|
||||
|
||||
- name: Terraform apply
|
||||
# Plan is captured in the workflow log; review it in the run page.
|
||||
# -auto-approve is intentional — this workflow IS the human approval
|
||||
# (the operator triggered it manually with a specific tag).
|
||||
#
|
||||
# -target=aws_ecs_task_definition.bench scopes apply to the task
|
||||
# definition (and its data-source / role dependencies) only. Without
|
||||
# this, every dispatch tries to reconcile every resource in the
|
||||
# module — IAM, S3, ECR, etc. — against whatever ref the workflow
|
||||
# was dispatched from. Out-of-band local applies cause that
|
||||
# reconciliation to attempt rollbacks the workflow role isn't
|
||||
# permitted to perform (iam:DetachRolePolicy, iam:PutRolePolicy),
|
||||
# failing the run even when the task-def update itself succeeded.
|
||||
working-directory: tests/benchmarks/cloudopsbench/infra
|
||||
env:
|
||||
IMAGE_TAG: ${{ inputs.image_tag }}
|
||||
run: |
|
||||
terraform apply -input=false -auto-approve \
|
||||
-target=aws_ecs_task_definition.bench \
|
||||
-var="image_tag=$IMAGE_TAG"
|
||||
|
||||
- name: Surface the new task definition revision in the job summary
|
||||
working-directory: tests/benchmarks/cloudopsbench/infra
|
||||
env:
|
||||
IMAGE_TAG: ${{ inputs.image_tag }}
|
||||
run: |
|
||||
TASK_DEF_ARN=$(terraform output -raw task_definition_arn)
|
||||
IMAGE_URI=$(aws ecs describe-task-definition \
|
||||
--task-definition "$TASK_DEF_ARN" \
|
||||
--query 'taskDefinition.containerDefinitions[0].image' \
|
||||
--output text)
|
||||
{
|
||||
echo "## Image promoted"
|
||||
echo ""
|
||||
echo "- Promoted tag: \`$IMAGE_TAG\`"
|
||||
echo "- New task definition ARN: \`$TASK_DEF_ARN\`"
|
||||
echo "- Image now in task definition: \`$IMAGE_URI\`"
|
||||
echo ""
|
||||
echo "### Next step"
|
||||
echo ""
|
||||
echo "Trigger **Benchmark — run on Fargate** to launch a run against this image."
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,45 @@
|
||||
name: Update Benchmark README Section
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- "docs/benchmarks/results.md"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
update-benchmark-readme:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Update README benchmark section
|
||||
run: uv run python -m tests.benchmarks.toolcall_model_benchmark.readme_updater
|
||||
|
||||
- name: Commit changes
|
||||
run: |
|
||||
git config user.name "benchmark-readme-action"
|
||||
git config user.email "benchmark-readme-action@noreply.github.com"
|
||||
git add README.md
|
||||
git diff --staged --quiet || git commit -m "docs(benchmark): update README with latest benchmark results"
|
||||
git pull --rebase origin main
|
||||
git push
|
||||
@@ -0,0 +1,191 @@
|
||||
name: Benchmark — run on Fargate (any adapter)
|
||||
|
||||
# Manually-triggered benchmark run on AWS Fargate.
|
||||
#
|
||||
# Adapter-agnostic. This workflow does NOT know or care which benchmark
|
||||
# is running. The ``config`` input names a YAML file; the YAML names an
|
||||
# adapter (``benchmark: <name>``); the framework's registry resolves it
|
||||
# to a registered ``BenchmarkAdapter`` subclass. The same workflow runs
|
||||
# any adapter packaged into the bench image — CloudOpsBench today,
|
||||
# ToolCallBench / future adapters tomorrow, with zero changes
|
||||
# here. See ``tests/benchmarks/_framework/registry.py``.
|
||||
#
|
||||
# Why ECS RunTask instead of a GitHub-hosted ubuntu runner: a full bench
|
||||
# grid runs for hours and writes hundreds of MB of artifacts — Fargate
|
||||
# handles it cleanly, has AWS Secrets Manager wired in via tests/benchmarks/cloudopsbench/infra/,
|
||||
# and writes to the per-run S3 bucket. The workflow's job is just to
|
||||
# launch the task and print where to watch.
|
||||
#
|
||||
# Trigger from the GitHub UI:
|
||||
# Actions → "Benchmark run (manual)" → Run workflow → fill inputs
|
||||
#
|
||||
# Pre-reqs (one-time, see tests/benchmarks/cloudopsbench/infra/README.md):
|
||||
# - tests/benchmarks/cloudopsbench/infra/ Terraform applied
|
||||
# - Bench image pushed to ECR via benchmark-image.yml
|
||||
# - Repo secrets seeded into AWS Secrets Manager via benchmark-seed-secret.yml
|
||||
# - Repo variables set:
|
||||
# AWS_ACCOUNT_ID, BENCH_ECS_CLUSTER, BENCH_TASK_DEFINITION_FAMILY,
|
||||
# BENCH_SUBNET_IDS (comma-separated), BENCH_SECURITY_GROUP_ID
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
config:
|
||||
# No adapter-specific default — the operator must point at a
|
||||
# specific config. Leaving the field empty surfaces the choice
|
||||
# rather than silently dispatching the CloudOpsBench smoke. The
|
||||
# path is relative to the container's repo root. Examples:
|
||||
# tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml
|
||||
# tests/benchmarks/<adapter>/configs/<config>.yml
|
||||
description: 'Path to YAML config inside the container (e.g. tests/benchmarks/<adapter>/configs/<config>.yml)'
|
||||
required: true
|
||||
dev_mode:
|
||||
description: 'Dev mode (skip integrity gates, no pre-reg needed)'
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
# Note: there is intentionally no `image_tag` input here. ECS RunTask
|
||||
# container overrides do NOT support overriding the image URI — that lives
|
||||
# on the task definition itself. The image actually pulled is whatever
|
||||
# was registered the last time `terraform apply -var=image_tag=<tag>` ran
|
||||
# in tests/benchmarks/cloudopsbench/infra/. Adding a workflow input here would silently mislead
|
||||
# operators into thinking they control the image per run. To change the
|
||||
# image, push it via `Benchmark image — build + push to ECR`, then re-apply
|
||||
# Terraform with the new tag.
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # required for AWS OIDC role assumption
|
||||
|
||||
concurrency:
|
||||
group: benchmark-run
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
launch:
|
||||
name: launch ECS task
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15 # we only LAUNCH the task; we don't wait for it
|
||||
|
||||
env:
|
||||
AWS_REGION: us-east-1
|
||||
|
||||
steps:
|
||||
- name: Verify required repo variables
|
||||
# Fail loudly BEFORE the AWS auth step if any required repo variable
|
||||
# is missing. Without this, an unset var would surface downstream as
|
||||
# an opaque error like "Task Definition can not be blank" from
|
||||
# describe-task-definition. See tests/benchmarks/cloudopsbench/infra/AWS_BENCH_SETUP.md step 4 for how
|
||||
# to set these.
|
||||
env:
|
||||
AWS_ACCOUNT_ID: ${{ vars.AWS_ACCOUNT_ID }}
|
||||
BENCH_ECS_CLUSTER: ${{ vars.BENCH_ECS_CLUSTER }}
|
||||
BENCH_TASK_DEFINITION_FAMILY: ${{ vars.BENCH_TASK_DEFINITION_FAMILY }}
|
||||
BENCH_SUBNET_IDS: ${{ vars.BENCH_SUBNET_IDS }}
|
||||
BENCH_SECURITY_GROUP_ID: ${{ vars.BENCH_SECURITY_GROUP_ID }}
|
||||
run: |
|
||||
missing=()
|
||||
for var in AWS_ACCOUNT_ID BENCH_ECS_CLUSTER BENCH_TASK_DEFINITION_FAMILY \
|
||||
BENCH_SUBNET_IDS BENCH_SECURITY_GROUP_ID; do
|
||||
if [ -z "${!var:-}" ]; then
|
||||
missing+=("$var")
|
||||
fi
|
||||
done
|
||||
if [ "${#missing[@]}" -gt 0 ]; then
|
||||
echo "::error::Missing repo variable(s): ${missing[*]}"
|
||||
echo "::error::Set them under Settings → Secrets and variables → Actions → Variables."
|
||||
echo "::error::Values come from \`cd tests/benchmarks/cloudopsbench/infra && terraform output\` — see tests/benchmarks/cloudopsbench/infra/AWS_BENCH_SETUP.md step 4."
|
||||
exit 1
|
||||
fi
|
||||
echo "All 5 required repo variables are set."
|
||||
|
||||
- name: Configure AWS credentials (OIDC role assumption)
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/opensre-bench-github-actions
|
||||
role-session-name: bench-run-${{ github.run_id }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Launch ECS RunTask
|
||||
# Inputs bound to env vars so the shell does the interpolation —
|
||||
# GitHub Actions template syntax (${{ ... }}) is expanded before
|
||||
# the shell sees it, which would let a crafted input inject shell
|
||||
# metacharacters. Env vars are quoted at use-site.
|
||||
env:
|
||||
BENCH_CONFIG: ${{ inputs.config }}
|
||||
BENCH_DEV_FLAG: ${{ inputs.dev_mode == true && '--dev' || '' }}
|
||||
CLUSTER: ${{ vars.BENCH_ECS_CLUSTER }}
|
||||
TASK_FAMILY: ${{ vars.BENCH_TASK_DEFINITION_FAMILY }}
|
||||
SUBNETS: ${{ vars.BENCH_SUBNET_IDS }}
|
||||
SECURITY_GROUP: ${{ vars.BENCH_SECURITY_GROUP_ID }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# Surface the image the task definition will actually pull, so
|
||||
# operators see the truth (not a workflow input that ECS ignores).
|
||||
IMAGE_URI=$(aws ecs describe-task-definition \
|
||||
--task-definition "$TASK_FAMILY" \
|
||||
--query 'taskDefinition.containerDefinitions[0].image' \
|
||||
--output text)
|
||||
|
||||
# Overrides JSON: container env vars passed at runtime. The
|
||||
# container's entrypoint reads BENCH_CONFIG + BENCH_DEV_FLAG
|
||||
# and invokes:
|
||||
# uv run python -m tests.benchmarks._framework.cli run \
|
||||
# "$BENCH_CONFIG" $BENCH_DEV_FLAG
|
||||
OVERRIDES=$(jq -nc \
|
||||
--arg config "$BENCH_CONFIG" \
|
||||
--arg dev_flag "$BENCH_DEV_FLAG" \
|
||||
'{
|
||||
containerOverrides: [{
|
||||
name: "bench",
|
||||
environment: [
|
||||
{name: "BENCH_CONFIG", value: $config},
|
||||
{name: "BENCH_DEV_FLAG", value: $dev_flag}
|
||||
]
|
||||
}]
|
||||
}')
|
||||
|
||||
TASK_ARN=$(aws ecs run-task \
|
||||
--cluster "$CLUSTER" \
|
||||
--task-definition "$TASK_FAMILY" \
|
||||
--launch-type FARGATE \
|
||||
--network-configuration "awsvpcConfiguration={subnets=[$SUBNETS],securityGroups=[$SECURITY_GROUP],assignPublicIp=ENABLED}" \
|
||||
--overrides "$OVERRIDES" \
|
||||
--query 'tasks[0].taskArn' \
|
||||
--output text)
|
||||
|
||||
if [ -z "$TASK_ARN" ] || [ "$TASK_ARN" = "None" ]; then
|
||||
echo "::error::run-task returned no taskArn"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TASK_ID="${TASK_ARN##*/}"
|
||||
echo "task_arn=$TASK_ARN" >> "$GITHUB_OUTPUT"
|
||||
echo "task_id=$TASK_ID" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Job summary — visible in the workflow run page
|
||||
{
|
||||
echo "## Bench run launched"
|
||||
echo ""
|
||||
echo "- Image (from task definition): \`$IMAGE_URI\`"
|
||||
echo "- Config: \`$BENCH_CONFIG\`"
|
||||
echo "- Dev mode: \`${BENCH_DEV_FLAG:-(off)}\`"
|
||||
echo "- Task ARN: \`$TASK_ARN\`"
|
||||
echo ""
|
||||
echo "### Watch it"
|
||||
echo ""
|
||||
echo "Stream logs locally:"
|
||||
echo ""
|
||||
echo '```bash'
|
||||
echo "aws logs tail /ecs/opensre-bench --follow"
|
||||
echo '```'
|
||||
echo ""
|
||||
echo "Or via AWS Console:"
|
||||
echo "https://us-east-1.console.aws.amazon.com/ecs/v2/clusters/$CLUSTER/tasks/$TASK_ID"
|
||||
echo ""
|
||||
echo "Artifacts land in the bench S3 bucket under \`runs/\` when the task completes."
|
||||
echo ""
|
||||
echo "_To change the running image: push a new tag via \`Benchmark image — build + push to ECR\`, then \`cd tests/benchmarks/cloudopsbench/infra && terraform apply -var=image_tag=<tag>\`._"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -0,0 +1,102 @@
|
||||
name: Benchmark secret — seed GitHub repo secret into AWS Secrets Manager
|
||||
|
||||
# Manually-triggered workflow that copies a GitHub repo secret into AWS
|
||||
# Secrets Manager at opensre-bench/llm/<secret>. The bench container reads
|
||||
# its LLM API keys from Secrets Manager at runtime; this workflow is how
|
||||
# you put them there without touching a developer laptop.
|
||||
#
|
||||
# The `secret` dropdown enforces which target is valid — must match one
|
||||
# of the four IAM-granted secret ARNs in tests/benchmarks/cloudopsbench/infra/iam_oidc.tf
|
||||
# (anthropic / openai / deepseek / hf_token).
|
||||
#
|
||||
# Why a workflow instead of a developer running `aws secretsmanager
|
||||
# put-secret-value` locally: keeps keys off developer laptops + out of
|
||||
# shell history, and centralizes rotation — rotate the value in the GH
|
||||
# repo secret and re-run this workflow to propagate.
|
||||
#
|
||||
# Pre-reqs:
|
||||
# - tests/benchmarks/cloudopsbench/infra/ Terraform has been applied (the secret resource exists)
|
||||
# - Corresponding GitHub repo secret is set:
|
||||
# ANTHROPIC_API_KEY / OPENAI_API_KEY / DEEPSEEK_API_KEY / HF_TOKEN
|
||||
#
|
||||
# Order of operations:
|
||||
# 1. terraform-bench.yml (plan) on PR
|
||||
# 2. terraform apply locally
|
||||
# 3. this workflow (workflow_dispatch) — pick the secret to seed
|
||||
# from the dropdown, repeat once per LLM provider key
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
secret:
|
||||
description: Which secret to seed (must match the AWS resource name)
|
||||
required: true
|
||||
type: choice
|
||||
options:
|
||||
- anthropic_api_key
|
||||
- openai_api_key
|
||||
- deepseek_api_key
|
||||
- hf_token
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # required for AWS OIDC role assumption
|
||||
|
||||
concurrency:
|
||||
group: bench-seed-${{ inputs.secret }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
seed:
|
||||
name: seed ${{ inputs.secret }}
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
env:
|
||||
AWS_REGION: us-east-1
|
||||
SECRET_ID: opensre-bench/llm/${{ inputs.secret }}
|
||||
|
||||
steps:
|
||||
- name: Configure AWS credentials (OIDC role assumption)
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/opensre-bench-github-actions
|
||||
role-session-name: github-actions-seed-${{ inputs.secret }}-${{ github.run_id }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Verify secret resource exists
|
||||
run: |
|
||||
if ! aws secretsmanager describe-secret --secret-id "$SECRET_ID" >/dev/null 2>&1; then
|
||||
echo "::error::Secret $SECRET_ID not found. Run \`terraform apply\` in tests/benchmarks/cloudopsbench/infra/ first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Put secret value
|
||||
# All four candidate values are bound at workflow level. The case
|
||||
# statement picks the one matching the chosen target — unused
|
||||
# values stay as masked env vars (GH redacts secret refs in logs).
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
TARGET: ${{ inputs.secret }}
|
||||
run: |
|
||||
case "$TARGET" in
|
||||
anthropic_api_key) V="$ANTHROPIC_API_KEY" ;;
|
||||
openai_api_key) V="$OPENAI_API_KEY" ;;
|
||||
deepseek_api_key) V="$DEEPSEEK_API_KEY" ;;
|
||||
hf_token) V="$HF_TOKEN" ;;
|
||||
*)
|
||||
echo "::error::Unknown target: $TARGET (dropdown should have prevented this)"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
if [ -z "$V" ]; then
|
||||
echo "::error::GitHub repo secret for $TARGET is unset. Configure it under Settings > Secrets and variables > Actions."
|
||||
exit 1
|
||||
fi
|
||||
aws secretsmanager put-secret-value \
|
||||
--secret-id "$SECRET_ID" \
|
||||
--secret-string "$V" >/dev/null
|
||||
echo "Seeded $SECRET_ID (length=${#V})."
|
||||
@@ -0,0 +1,37 @@
|
||||
name: Celebrate merged pull requests
|
||||
|
||||
# Runs after a PR lands (merged into the target branch). Uses pull_request_target so the
|
||||
# token can comment on merged PRs from forks; checkout is base/default only — no PR head.
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [closed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
celebrate-merge:
|
||||
name: Post-merge celebration comment
|
||||
runs-on: ubuntu-latest
|
||||
if: >-
|
||||
github.event.pull_request.merged == true &&
|
||||
github.event.pull_request.user.type != 'Bot' &&
|
||||
github.event.pull_request.user.login != 'dependabot[bot]'
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Build celebration comment
|
||||
id: celebrate
|
||||
env:
|
||||
DISCORD_INVITE_URL: https://discord.com/invite/opensre
|
||||
CONTRIBUTOR_LOGIN: ${{ github.event.pull_request.user.login }}
|
||||
run: python3 .github/scripts/merged_pr_celebration_message.py
|
||||
|
||||
- name: Comment on merged PR
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: >
|
||||
gh pr comment "${{ github.event.pull_request.number }}"
|
||||
--repo "${{ github.repository }}"
|
||||
--body-file comment.md
|
||||
@@ -0,0 +1,109 @@
|
||||
# Optional Windows CI when PR label `ci:windows` is present.
|
||||
# Separate workflow + concurrency group so Windows label runs are isolated.
|
||||
|
||||
name: CI Labels (Windows)
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
types: [opened, synchronize, reopened, labeled, unlabeled]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
# Keep in sync with ci.yml — single source of truth for lint/typecheck/coverage targets.
|
||||
PYTHON_SOURCE_PATHS: "config core integrations platform surfaces tools"
|
||||
|
||||
concurrency:
|
||||
group: ci-labels-windows-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
windows-quality:
|
||||
if: contains(github.event.pull_request.labels.*.name, 'ci:windows')
|
||||
name: windows quality
|
||||
runs-on: windows-latest
|
||||
timeout-minutes: 15
|
||||
continue-on-error: true
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Lint
|
||||
run: uv run python -m ruff check ${{ env.PYTHON_SOURCE_PATHS }} tests/
|
||||
|
||||
- name: Format check
|
||||
run: uv run python -m ruff format --check ${{ env.PYTHON_SOURCE_PATHS }} tests/
|
||||
|
||||
- name: Typecheck
|
||||
run: uv run python -m mypy ${{ env.PYTHON_SOURCE_PATHS }}
|
||||
|
||||
windows-test:
|
||||
if: contains(github.event.pull_request.labels.*.name, 'ci:windows')
|
||||
name: windows test
|
||||
needs: [windows-quality]
|
||||
runs-on: windows-latest
|
||||
timeout-minutes: 25
|
||||
continue-on-error: true
|
||||
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
JWT_TOKEN: ${{ secrets.JWT_TOKEN }}
|
||||
TRACER_ORG_ID: ${{ secrets.TRACER_ORG_ID }}
|
||||
TRACER_WEB_APP_URL: ${{ secrets.TRACER_WEB_APP_URL }}
|
||||
GRAFANA_READ_TOKEN: ${{ secrets.GRAFANA_READ_TOKEN }}
|
||||
GCLOUD_HOSTED_METRICS_ID: ${{ secrets.GCLOUD_HOSTED_METRICS_ID }}
|
||||
GCLOUD_HOSTED_METRICS_URL: ${{ secrets.GCLOUD_HOSTED_METRICS_URL }}
|
||||
GCLOUD_HOSTED_LOGS_ID: ${{ secrets.GCLOUD_HOSTED_LOGS_ID }}
|
||||
GCLOUD_HOSTED_LOGS_URL: ${{ secrets.GCLOUD_HOSTED_LOGS_URL }}
|
||||
GCLOUD_RW_API_KEY: ${{ secrets.GCLOUD_RW_API_KEY }}
|
||||
GCLOUD_OTLP_ENDPOINT: ${{ secrets.GCLOUD_OTLP_ENDPOINT }}
|
||||
GCLOUD_OTLP_AUTH_HEADER: ${{ secrets.GCLOUD_OTLP_AUTH_HEADER }}
|
||||
PYTHONUTF8: "1"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Run full tests
|
||||
run: >-
|
||||
uv run python -m pytest -n auto -v
|
||||
--cov=config --cov=core --cov=platform.deployment.aws --cov=platform.deployment --cov=integrations --cov=platform --cov=surfaces --cov=tools
|
||||
--cov-report=term-missing
|
||||
--cov-report=html
|
||||
--ignore=tests/e2e/kubernetes_local_alert_simulation
|
||||
--ignore=tests/synthetic
|
||||
-m "not synthetic and not integration"
|
||||
@@ -0,0 +1,468 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "surfaces/**"
|
||||
- "config/**"
|
||||
- "core/**"
|
||||
- "platform/deployment/**"
|
||||
- "integrations/**"
|
||||
- "platform/**"
|
||||
- "tools/**"
|
||||
- "tests/**"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "pytest.ini"
|
||||
- "ruff.toml"
|
||||
- "mypy.ini"
|
||||
- "Makefile"
|
||||
- ".importlinter"
|
||||
- ".importlinter.strict"
|
||||
- ".github/ci/**"
|
||||
- ".github/workflows/ci.yml"
|
||||
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "surfaces/**"
|
||||
- "config/**"
|
||||
- "core/**"
|
||||
- "platform/deployment/**"
|
||||
- "integrations/**"
|
||||
- "platform/**"
|
||||
- "tools/**"
|
||||
- "tests/**"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "pytest.ini"
|
||||
- "ruff.toml"
|
||||
- "mypy.ini"
|
||||
- "Makefile"
|
||||
- ".importlinter"
|
||||
- ".importlinter.strict"
|
||||
- ".github/ci/**"
|
||||
- ".github/workflows/ci.yml"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
PYTHON_SOURCE_PATHS: "config core integrations platform surfaces tools"
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
quality:
|
||||
name: quality (${{ matrix.os }})
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest]
|
||||
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 15
|
||||
continue-on-error: ${{ matrix.os == 'windows-latest' }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Lint
|
||||
run: uv run python -m ruff check $PYTHON_SOURCE_PATHS tests/
|
||||
|
||||
- name: Format check
|
||||
run: uv run python -m ruff format --check $PYTHON_SOURCE_PATHS tests/
|
||||
|
||||
- name: Typecheck
|
||||
run: uv run python -m mypy $PYTHON_SOURCE_PATHS
|
||||
|
||||
- name: Import graph
|
||||
run: uv run python .github/ci/check_imports.py --strict
|
||||
|
||||
# Full layered contracts (.importlinter.strict), not the minimal default
|
||||
# .importlinter config used by ``make check-imports``.
|
||||
- name: Import boundaries
|
||||
run: uv run lint-imports --config .importlinter.strict
|
||||
|
||||
test:
|
||||
name: test (${{ matrix.shard }})
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 5
|
||||
matrix:
|
||||
include:
|
||||
- os: ubuntu-latest
|
||||
shard: tools-runtime
|
||||
# ``tests/core/agent`` is excluded here; it runs in cli-runtime,
|
||||
# which pins openai for its live action-planning oracles.
|
||||
extra_pytest_args: >-
|
||||
--ignore=tests/core/agent
|
||||
pytest_paths: >-
|
||||
tests/tools
|
||||
tests/core
|
||||
tests/platform
|
||||
tests/utils
|
||||
tests/masking
|
||||
tests/watch_dog
|
||||
- os: ubuntu-latest
|
||||
shard: cli-runtime
|
||||
# Live shell action-agent contracts are validated on openai,
|
||||
# matching interactive-shell-live.yml. The
|
||||
# default anthropic tool-call model (haiku) cannot reliably perform
|
||||
# compound action planning, so pin this shard to openai.
|
||||
# ``tests/core/agent`` runs here because it contains the live
|
||||
# action-planning oracles (``test_live_action_planning`` and
|
||||
# ``test_live_turn_execution_oracle``) that need the openai pin —
|
||||
# plus the static ``test_import_boundaries`` guard the surfaces
|
||||
# restructure (#3299) added.
|
||||
llm_provider: openai
|
||||
pytest_paths: >-
|
||||
tests/cli
|
||||
tests/interactive_shell
|
||||
tests/agent
|
||||
tests/core/agent
|
||||
tests/fleet_monitoring
|
||||
tests/analytics
|
||||
tests/tools/investigation
|
||||
tests/delivery
|
||||
tests/sandbox
|
||||
tests/hermes
|
||||
- os: ubuntu-latest
|
||||
shard: integrations-and-misc
|
||||
pytest_paths: >-
|
||||
tests/integrations
|
||||
tests/deployment
|
||||
tests/benchmarks
|
||||
tests/chaos_engineering
|
||||
tests/shared
|
||||
tests/config
|
||||
tests/github_ci
|
||||
tests/packaging
|
||||
tests/scheduler
|
||||
gateway/tests
|
||||
- os: ubuntu-latest
|
||||
shard: e2e-general
|
||||
pytest_paths: >-
|
||||
tests/e2e
|
||||
extra_pytest_args: >-
|
||||
--ignore=tests/e2e/kubernetes
|
||||
--ignore=tests/e2e/openclaw
|
||||
--ignore=tests/e2e/upstream_lambda
|
||||
--ignore=tests/e2e/upstream_prefect_ecs_fargate
|
||||
--ignore=tests/e2e/upstream_apache_flink_ecs
|
||||
- os: ubuntu-latest
|
||||
shard: e2e-provider-and-openclaw
|
||||
pytest_paths: >-
|
||||
tests/e2e/openclaw
|
||||
tests/e2e/upstream_lambda
|
||||
tests/e2e/upstream_prefect_ecs_fargate
|
||||
tests/e2e/upstream_apache_flink_ecs
|
||||
tests/e2e/kubernetes
|
||||
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 30
|
||||
|
||||
env:
|
||||
# Per-shard LLM provider for live_llm contracts; defaults to anthropic.
|
||||
LLM_PROVIDER: ${{ matrix.llm_provider || 'anthropic' }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
JWT_TOKEN: ${{ secrets.JWT_TOKEN }}
|
||||
GRAFANA_READ_TOKEN: ${{ secrets.GRAFANA_READ_TOKEN }}
|
||||
GCLOUD_HOSTED_METRICS_ID: ${{ secrets.GCLOUD_HOSTED_METRICS_ID }}
|
||||
GCLOUD_HOSTED_METRICS_URL: ${{ secrets.GCLOUD_HOSTED_METRICS_URL }}
|
||||
GCLOUD_HOSTED_LOGS_ID: ${{ secrets.GCLOUD_HOSTED_LOGS_ID }}
|
||||
GCLOUD_HOSTED_LOGS_URL: ${{ secrets.GCLOUD_HOSTED_LOGS_URL }}
|
||||
GCLOUD_RW_API_KEY: ${{ secrets.GCLOUD_RW_API_KEY }}
|
||||
GCLOUD_OTLP_ENDPOINT: ${{ secrets.GCLOUD_OTLP_ENDPOINT }}
|
||||
GCLOUD_OTLP_AUTH_HEADER: ${{ secrets.GCLOUD_OTLP_AUTH_HEADER }}
|
||||
PYTHONUTF8: "1"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Validate pytest marker
|
||||
env:
|
||||
# Fork PRs do not receive repository secrets; skip live_llm contracts
|
||||
# (see interactive-shell-live.yml). Index a JSON string array so the
|
||||
# value is always a marker string — never a bare boolean from
|
||||
# ``a && 'x' || 'y'`` (boolean false → ``-m false`` → xdist
|
||||
# ``N workers [0 items]``, which can still exit 0).
|
||||
PYTEST_MARKER_EXPR: ${{ fromJSON('["not synthetic", "not (synthetic or live_llm)"]')[github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == true] }}
|
||||
run: |
|
||||
echo "PYTEST_MARKER_EXPR=${PYTEST_MARKER_EXPR}"
|
||||
case "${PYTEST_MARKER_EXPR}" in
|
||||
"not synthetic"|"not (synthetic or live_llm)") ;;
|
||||
*)
|
||||
echo "Refusing unexpected PYTEST_MARKER_EXPR: '${PYTEST_MARKER_EXPR}'" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
- name: Run tests
|
||||
env:
|
||||
COVERAGE_FILE: .coverage.${{ matrix.shard }}
|
||||
PYTEST_MARKER_EXPR: ${{ fromJSON('["not synthetic", "not (synthetic or live_llm)"]')[github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == true] }}
|
||||
# Keep ``run: >`` (folded) so ``matrix.pytest_paths`` stays one argv list.
|
||||
# A ``run: |`` + ``\`` rewrite can drop/split those paths and yield
|
||||
# ``N workers [0 items]`` even when the marker is correct.
|
||||
run: >
|
||||
uv run python -m pytest -n auto -v
|
||||
${{ matrix.pytest_paths }}
|
||||
--cov=config
|
||||
--cov=core
|
||||
--cov=platform.deployment.aws --cov=platform.deployment
|
||||
--cov=integrations
|
||||
--cov=platform
|
||||
--cov=surfaces
|
||||
--cov=tools
|
||||
--cov-report=
|
||||
--ignore=tests/e2e/kubernetes_local_alert_simulation
|
||||
--ignore=tests/synthetic
|
||||
${{ matrix.extra_pytest_args }}
|
||||
-m "${PYTEST_MARKER_EXPR}"
|
||||
|
||||
- name: Upload shard coverage data
|
||||
if: >-
|
||||
always() &&
|
||||
github.event_name == 'push' &&
|
||||
github.ref == 'refs/heads/main'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: coverage-data-${{ matrix.shard }}
|
||||
path: .coverage.${{ matrix.shard }}*
|
||||
if-no-files-found: error
|
||||
include-hidden-files: true
|
||||
retention-days: 7
|
||||
|
||||
coverage-report:
|
||||
name: coverage-report
|
||||
runs-on: ubuntu-latest
|
||||
needs: [test]
|
||||
if: >-
|
||||
always() &&
|
||||
github.event_name == 'push' &&
|
||||
github.ref == 'refs/heads/main'
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Download shard coverage artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: coverage-data-*
|
||||
path: ./
|
||||
merge-multiple: true
|
||||
|
||||
- name: Combine coverage and build report
|
||||
run: |
|
||||
uv run python -m coverage combine .
|
||||
uv run python -m coverage html
|
||||
|
||||
- name: Upload combined coverage
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: coverage-report-ubuntu-latest
|
||||
path: |
|
||||
htmlcov/
|
||||
.coverage
|
||||
retention-days: 7
|
||||
|
||||
test-kubernetes:
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
timeout-minutes: 15
|
||||
|
||||
if: >-
|
||||
github.repository == 'Tracer-Cloud/opensre' &&
|
||||
(github.event_name == 'push' ||
|
||||
(
|
||||
github.event_name == 'pull_request' &&
|
||||
(
|
||||
contains(github.event.pull_request.title, 'k8s') ||
|
||||
contains(github.event.pull_request.title, 'kubernetes')
|
||||
)
|
||||
))
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Run Kubernetes tests
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: us-east-1
|
||||
run: uv run python -m tests.e2e.kubernetes.test_local
|
||||
|
||||
should-run-thorough:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'Tracer-Cloud/opensre'
|
||||
outputs:
|
||||
thorough: ${{ steps.filter.outputs.thorough }}
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: dorny/paths-filter@v4
|
||||
id: filter
|
||||
with:
|
||||
filters: |
|
||||
thorough:
|
||||
- ".github/workflows/ci.yml"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "platform/deployment/**"
|
||||
- "integrations/**"
|
||||
- "tools/investigation/**"
|
||||
- "tools/**"
|
||||
- "tests/e2e/cloudwatch_demo/**"
|
||||
- "tests/e2e/upstream_lambda/**"
|
||||
- "tests/e2e/upstream_prefect_ecs_fargate/**"
|
||||
- "tests/e2e/upstream_apache_flink_ecs/**"
|
||||
|
||||
test-thorough:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [test, should-run-thorough]
|
||||
if: >-
|
||||
github.event_name == 'push' &&
|
||||
github.ref == 'refs/heads/main' &&
|
||||
github.repository == 'Tracer-Cloud/opensre' &&
|
||||
needs.should-run-thorough.outputs.thorough == 'true'
|
||||
timeout-minutes: 30
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- name: cloudwatch-demo
|
||||
command: python3 -m tests.e2e.cloudwatch_demo.test_aws
|
||||
- name: upstream-lambda
|
||||
command: python3 -m tests.e2e.upstream_lambda.test_agent_e2e
|
||||
- name: prefect-ecs-fargate
|
||||
command: python3 -m tests.e2e.upstream_prefect_ecs_fargate.test_agent_e2e
|
||||
- name: flink-ecs
|
||||
command: python3 -m tests.e2e.upstream_apache_flink_ecs.test_agent_e2e
|
||||
|
||||
name: test-thorough (${{ matrix.name }})
|
||||
|
||||
env:
|
||||
# Thorough e2e suites make live LLM calls; run them on openai because the
|
||||
# default anthropic account is credit-exhausted. Requires a valid, funded
|
||||
# OPENAI_API_KEY secret.
|
||||
LLM_PROVIDER: openai
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
JWT_TOKEN: ${{ secrets.JWT_TOKEN }}
|
||||
TRACER_API_URL: ${{ secrets.TRACER_API_URL }}
|
||||
TRACER_INGEST_TOKEN: ${{ secrets.TRACER_INGEST_TOKEN }}
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
AWS_REGION: us-east-1
|
||||
AWS_DEFAULT_REGION: us-east-1
|
||||
GRAFANA_READ_TOKEN: ${{ secrets.GRAFANA_READ_TOKEN }}
|
||||
GCLOUD_HOSTED_METRICS_ID: ${{ secrets.GCLOUD_HOSTED_METRICS_ID }}
|
||||
GCLOUD_HOSTED_METRICS_URL: ${{ secrets.GCLOUD_HOSTED_METRICS_URL }}
|
||||
GCLOUD_HOSTED_LOGS_ID: ${{ secrets.GCLOUD_HOSTED_LOGS_ID }}
|
||||
GCLOUD_HOSTED_LOGS_URL: ${{ secrets.GCLOUD_HOSTED_LOGS_URL }}
|
||||
GCLOUD_RW_API_KEY: ${{ secrets.GCLOUD_RW_API_KEY }}
|
||||
CLOUDWATCH_VERIFY_LOGS: ${{ matrix.name == 'cloudwatch-demo' && '0' || '1' }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Install tracer CLI
|
||||
run: |
|
||||
curl -sSL https://install.tracer.cloud | sh -s user_36fbN6K6FwEgJFHsQv1pUByo8K6
|
||||
|
||||
- name: Initialize tracer (optional)
|
||||
run: |
|
||||
sudo tracer init --token ${{ secrets.JWT_TOKEN }}
|
||||
continue-on-error: true
|
||||
timeout-minutes: 2
|
||||
|
||||
- name: Run test
|
||||
run: uv run ${{ matrix.command }}
|
||||
timeout-minutes: 30
|
||||
@@ -0,0 +1,118 @@
|
||||
name: Closed-loop learning (weekly miss triage)
|
||||
|
||||
# Weekly reminder workflow for the closed-loop learning process documented
|
||||
# in docs/closed-loop-learning.mdx. Runs at 09:00 UTC every Monday and
|
||||
# also supports manual triggering.
|
||||
#
|
||||
# The workflow does not consume the per-user ``~/.opensre/misses.jsonl``
|
||||
# store directly (that lives on engineer machines, not on the runner).
|
||||
# Its purpose is to:
|
||||
# 1) Open or refresh a tracking issue that nudges the on-call engineer
|
||||
# to run ``opensre misses stats --since 7d`` and then
|
||||
# ``opensre misses export --since 7d --top 10 --out tests/benchmarks/production_misses/``.
|
||||
# 2) Verify the ``misses_command`` CLI surface is still wired and exits
|
||||
# cleanly, so the weekly process is not blocked by a CLI regression.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 9 * * 1"
|
||||
workflow_dispatch: {}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
smoke:
|
||||
name: Verify opensre misses CLI
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Smoke test misses CLI
|
||||
run: |
|
||||
uv run opensre misses --help
|
||||
uv run opensre misses list --json
|
||||
uv run opensre misses stats --json
|
||||
|
||||
remind:
|
||||
name: Open weekly triage reminder
|
||||
needs: smoke
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Open weekly tracking issue (close last week's first)
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
// One issue per week. Previous week's tracker is closed before
|
||||
// opening this week's so triage history is per-week (GitHub's
|
||||
// natural unit) rather than one ever-growing comment thread.
|
||||
// Uses the existing "pending triage" label so we do not
|
||||
// auto-create a new label on first run.
|
||||
const week = new Date().toISOString().slice(0, 10);
|
||||
const titlePrefix = "Weekly closed-loop learning triage";
|
||||
const title = `${titlePrefix} — week of ${week}`;
|
||||
const body = [
|
||||
"Weekly nudge per docs/closed-loop-learning.mdx.",
|
||||
"",
|
||||
"Steps for this week's on-call:",
|
||||
"- [ ] `opensre misses stats --since 7d`",
|
||||
"- [ ] Review recurring `(alert, taxonomy)` pairs",
|
||||
"- [ ] `opensre misses export --since 7d --top 10 --out tests/benchmarks/production_misses/`",
|
||||
"- [ ] Open a PR labelled `benchmark` with the new scenarios",
|
||||
"- [ ] Trigger the benchmark workflow on the PR branch",
|
||||
].join("\n");
|
||||
|
||||
const { data: openIssues } = await github.rest.issues.listForRepo({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
state: "open",
|
||||
labels: "pending triage",
|
||||
per_page: 100,
|
||||
});
|
||||
|
||||
const stale = openIssues.filter(
|
||||
(i) => i.title.startsWith(titlePrefix) && i.title !== title,
|
||||
);
|
||||
for (const issue of stale) {
|
||||
await github.rest.issues.update({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: issue.number,
|
||||
state: "closed",
|
||||
state_reason: "completed",
|
||||
});
|
||||
}
|
||||
|
||||
const existing = openIssues.find((i) => i.title === title);
|
||||
if (existing) {
|
||||
return;
|
||||
}
|
||||
|
||||
await github.rest.issues.create({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
title,
|
||||
body,
|
||||
labels: ["pending triage"],
|
||||
});
|
||||
@@ -0,0 +1,43 @@
|
||||
name: CodeQL
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
# Skip CodeQL for docs-only PRs (Markdown/MDX and the docs site tree) to
|
||||
# save CI minutes. Not a required check, so skipping cannot block merge.
|
||||
# `push` to main and the weekly schedule keep full coverage regardless.
|
||||
paths-ignore:
|
||||
- '**/*.md'
|
||||
- '**/*.mdx'
|
||||
- 'docs/**'
|
||||
schedule:
|
||||
- cron: "0 6 * * 1"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
actions: read
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
name: Analyze (python)
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@v4
|
||||
with:
|
||||
languages: python
|
||||
build-mode: none
|
||||
config-file: .github/codeql/codeql-config.yml
|
||||
queries: security-and-quality
|
||||
|
||||
- name: Perform CodeQL Analysis
|
||||
uses: github/codeql-action/analyze@v4
|
||||
with:
|
||||
category: /language:python
|
||||
@@ -0,0 +1,148 @@
|
||||
name: Docker Publish
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [published]
|
||||
schedule:
|
||||
- cron: "30 2 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tag:
|
||||
description: "Release tag to build (e.g. v0.1.2026.7.8). Defaults to the latest release."
|
||||
required: false
|
||||
type: string
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
concurrency:
|
||||
group: docker-publish
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
steps:
|
||||
- name: Log in to GHCR
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Resolve release tag and image tags
|
||||
id: meta
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
|
||||
DISPATCH_TAG: ${{ inputs.tag }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
image="ghcr.io/${GITHUB_REPOSITORY,,}"
|
||||
latest_tag="$(gh api "repos/${GITHUB_REPOSITORY}/releases/latest" --jq .tag_name)"
|
||||
|
||||
case "$EVENT_NAME" in
|
||||
release)
|
||||
if [ "$RELEASE_PRERELEASE" = "true" ]; then
|
||||
echo "Skipping prerelease $RELEASE_TAG (rolling main builds are not published to GHCR)."
|
||||
echo "build=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
tag="$RELEASE_TAG"
|
||||
;;
|
||||
workflow_dispatch)
|
||||
tag="${DISPATCH_TAG:-$latest_tag}"
|
||||
gh release view "$tag" --repo "$GITHUB_REPOSITORY" >/dev/null
|
||||
;;
|
||||
*)
|
||||
tag="$latest_tag"
|
||||
;;
|
||||
esac
|
||||
|
||||
version="${tag#v}"
|
||||
|
||||
# The scheduled run rebuilds the newest release; skip it when that
|
||||
# version is already in the registry (e.g. the release event or a
|
||||
# dispatch already published it).
|
||||
if [ "$EVENT_NAME" = "schedule" ] \
|
||||
&& docker manifest inspect "${image}:${version}" >/dev/null 2>&1; then
|
||||
echo "Image ${image}:${version} already published; nothing to do."
|
||||
echo "build=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
tags="${image}:${version}"
|
||||
if [ "$tag" = "$latest_tag" ]; then
|
||||
tags="${tags}"$'\n'"${image}:latest"
|
||||
else
|
||||
echo "Tag $tag is older than latest release $latest_tag; not moving :latest."
|
||||
fi
|
||||
|
||||
{
|
||||
echo "build=true"
|
||||
echo "tag=${tag}"
|
||||
echo "version=${version}"
|
||||
echo "image=${image}"
|
||||
echo "tags<<EOF"
|
||||
echo "$tags"
|
||||
echo "EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: actions/checkout@v5
|
||||
if: steps.meta.outputs.build == 'true'
|
||||
with:
|
||||
ref: ${{ steps.meta.outputs.tag }}
|
||||
|
||||
- name: Sync release version into pyproject.toml
|
||||
if: steps.meta.outputs.build == 'true'
|
||||
shell: bash
|
||||
run: python3 platform/packaging/sync_release_version.py --tag "${{ steps.meta.outputs.tag }}"
|
||||
|
||||
- name: Set up QEMU
|
||||
if: steps.meta.outputs.build == 'true'
|
||||
uses: docker/setup-qemu-action@v3
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.meta.outputs.build == 'true'
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Build and push image
|
||||
if: steps.meta.outputs.build == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: |
|
||||
org.opencontainers.image.title=opensre
|
||||
org.opencontainers.image.description=OpenSRE unified image (MODE=web FastAPI health app, MODE=gateway Telegram gateway)
|
||||
org.opencontainers.image.source=https://github.com/${{ github.repository }}
|
||||
org.opencontainers.image.version=${{ steps.meta.outputs.version }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Smoke test published image
|
||||
if: steps.meta.outputs.build == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
image="${{ steps.meta.outputs.image }}:${{ steps.meta.outputs.version }}"
|
||||
docker pull "$image"
|
||||
version_output="$(docker run --rm --entrypoint opensre "$image" --version)"
|
||||
printf '%s\n' "$version_output"
|
||||
case "$version_output" in
|
||||
*"${{ steps.meta.outputs.version }}"*) ;;
|
||||
*)
|
||||
echo "Image version mismatch: expected ${{ steps.meta.outputs.version }}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,186 @@
|
||||
# OpenClaw end-to-end CI.
|
||||
#
|
||||
# Triggers, in order of intent:
|
||||
# 1. PRs that touch OpenClaw-relevant paths (auto, primary signal)
|
||||
# 2. Pushes to main that touch the same paths (post-merge regression catch)
|
||||
# 3. Daily schedule at 06:00 UTC (catches environment drift —
|
||||
# OpenClaw version bumps, MCP SDK
|
||||
# breaks, npm install regressions)
|
||||
# 4. Adding the `ci:openclaw` label to a PR (manual re-run on matching PRs)
|
||||
# 5. Workflow_dispatch from the Actions UI (manual force-run on any ref,
|
||||
# no PR needed)
|
||||
#
|
||||
# The path filter at the trigger level is the primary gate, so non-OpenClaw PRs
|
||||
# pay no CI cost. The label is only useful as a re-trigger on PRs that already
|
||||
# matched paths; for force-running against an unrelated branch, use the manual
|
||||
# dispatch button in the Actions UI.
|
||||
|
||||
name: CI (OpenClaw E2E)
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
types: [opened, synchronize, reopened, labeled, unlabeled]
|
||||
paths:
|
||||
- "integrations/openclaw.py"
|
||||
- "tools/OpenClawMCPTool/**"
|
||||
- "tests/e2e/openclaw/**"
|
||||
- "tests/utils/alert_factory/**"
|
||||
- ".github/workflows/e2e-openclaw.yml"
|
||||
- "docs/openclaw.mdx"
|
||||
- ".tool-versions"
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "integrations/openclaw.py"
|
||||
- "tools/OpenClawMCPTool/**"
|
||||
- "tests/e2e/openclaw/**"
|
||||
- "tests/utils/alert_factory/**"
|
||||
- ".github/workflows/e2e-openclaw.yml"
|
||||
- "docs/openclaw.mdx"
|
||||
- ".tool-versions"
|
||||
schedule:
|
||||
- cron: "0 6 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
concurrency:
|
||||
group: ci-openclaw-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# Quality gates (lint / format-check / mypy) are run by the main
|
||||
# ``ci.yml`` workflow for every PR. We don't duplicate them here —
|
||||
# this workflow's sole job is the OpenClaw e2e suite. If you reach
|
||||
# this workflow, your PR has already cleared the quality gates.
|
||||
openclaw-test:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
name: openclaw test
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
|
||||
# No ``env:`` block by design. CI runs the boot + use-case
|
||||
# scenarios only — the full-RCA sub-tests call an LLM, cost API
|
||||
# credits, and take ~25s each, so we deliberately don't wire any
|
||||
# LLM secret in. They auto-skip via
|
||||
# ``LLM_CREDENTIAL_SKIP_REASON`` when no key is present in env.
|
||||
# Contributors with ANTHROPIC_API_KEY / OPENAI_API_KEY /
|
||||
# GEMINI_API_KEY set locally still run them via
|
||||
# ``make test-openclaw``. Lang Smith env vars stay scoped to the
|
||||
# local-RCA path for the same reason — no value tracing the
|
||||
# use-case-only CI runs.
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Set up Node (OpenClaw requires >=22.12; version pinned in .tool-versions)
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version-file: ".tool-versions"
|
||||
|
||||
- name: Cache npm downloads
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.npm
|
||||
key: ${{ runner.os }}-openclaw-npm-${{ hashFiles('.tool-versions') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-openclaw-npm-
|
||||
|
||||
- name: Install OpenClaw CLI
|
||||
# ``continue-on-error: true`` so an npm-registry hiccup, a package-name
|
||||
# rename on OpenClaw's side, or a network blip does not kill the whole
|
||||
# workflow. The "Probe OpenClaw availability" step below catches the
|
||||
# failure and downgrades the run to "tests skipped" with a clear
|
||||
# explanation rather than crashing.
|
||||
continue-on-error: true
|
||||
id: install_openclaw
|
||||
run: npm install -g openclaw --prefer-offline --no-audit
|
||||
|
||||
- name: Probe OpenClaw availability
|
||||
# Whether or not ``npm install`` succeeded, check if ``openclaw`` is
|
||||
# actually runnable. Failure modes we surface explicitly:
|
||||
# - CLI not installed (npm step failed silently)
|
||||
# - CLI installed but Node version mismatch (older runner image)
|
||||
# - CLI installed but the bundled JS errored on first run
|
||||
# ``openclaw_runnable`` env var feeds the next step's gating.
|
||||
run: |
|
||||
set +e
|
||||
if ! command -v openclaw >/dev/null 2>&1; then
|
||||
echo "::warning::openclaw CLI is not on PATH (npm install may have failed); the e2e suite will be skipped this run."
|
||||
echo "openclaw_runnable=false" >> "$GITHUB_ENV"
|
||||
exit 0
|
||||
fi
|
||||
version_output="$(openclaw --version 2>&1)"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "::warning::openclaw is on PATH but errors when run:"
|
||||
echo "$version_output"
|
||||
echo "::warning::Skipping e2e suite this run."
|
||||
echo "openclaw_runnable=false" >> "$GITHUB_ENV"
|
||||
exit 0
|
||||
fi
|
||||
echo "✓ OpenClaw verified: $version_output"
|
||||
echo "openclaw_runnable=true" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Note LLM credential policy
|
||||
# CI deliberately does not wire any LLM key, so the full-RCA
|
||||
# sub-tests skip every run. This step is informational — it
|
||||
# makes the policy obvious in the run log instead of leaving a
|
||||
# silent skip that a reviewer has to dig for. Run RCA locally
|
||||
# with ``make test-openclaw`` if you have a key configured.
|
||||
run: |
|
||||
echo "ℹ︎ CI policy: full-RCA sub-tests are skipped (LLM credentials intentionally not wired)."
|
||||
echo " Boot + use-case sub-tests still run and gate the merge."
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Run OpenClaw E2E suite
|
||||
# Skips with a clear message if the OpenClaw probe failed earlier,
|
||||
# so the workflow finishes green with a documented "couldn't run"
|
||||
# status rather than a confusing red on the install step.
|
||||
run: |
|
||||
if [ "$openclaw_runnable" != "true" ]; then
|
||||
echo "::warning::Skipping pytest invocation — openclaw CLI unavailable in this environment. See earlier warnings."
|
||||
exit 0
|
||||
fi
|
||||
uv run python -m pytest -m e2e -v tests/e2e/openclaw/
|
||||
|
||||
- name: Upload gateway logs on failure
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: openclaw-gateway-logs
|
||||
path: /tmp/openclaw-e2e-logs/
|
||||
if-no-files-found: ignore
|
||||
retention-days: 7
|
||||
|
||||
- name: Summary
|
||||
# Always runs, regardless of earlier step outcomes. The probe
|
||||
# step above downgrades a missing-CLI / wrong-Node environment
|
||||
# to "tests skipped" rather than failing the workflow, so this
|
||||
# summary is the one place a reviewer sees why fewer tests ran
|
||||
# than expected. Real pytest failures still fail the job.
|
||||
if: always()
|
||||
run: |
|
||||
echo "### OpenClaw E2E run summary"
|
||||
echo " OpenClaw runnable: ${openclaw_runnable:-unknown}"
|
||||
echo " Full-RCA sub-tests: skipped by CI policy (no LLM key wired)"
|
||||
echo " Run RCA locally with: make test-openclaw"
|
||||
@@ -0,0 +1,52 @@
|
||||
# When someone with zero merged PRs in this repo (and not OWNER/MEMBER/COLLABORATOR) comments
|
||||
# on an open issue labeled `good first issue`, assign them and reply — only if the issue has no
|
||||
# assignees yet (first eligible commenter wins; includes when a human pre-assigned someone).
|
||||
# issue_comment also fires for PR review threads; those payloads include issue.pull_request.
|
||||
name: Good first issue — auto-assign
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
concurrency:
|
||||
group: good-first-issue-assign-${{ github.repository }}-${{ github.event.issue.number }}
|
||||
cancel-in-progress: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
pull-requests: read
|
||||
|
||||
jobs:
|
||||
assign:
|
||||
name: Assign new contributor on good first issue
|
||||
runs-on: ubuntu-latest
|
||||
if: >-
|
||||
github.event.issue.state == 'open' &&
|
||||
github.event.comment.user.type != 'Bot' &&
|
||||
!github.event.issue.pull_request
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Decide assign + notice body
|
||||
id: decide
|
||||
env:
|
||||
GITHUB_EVENT_PATH: ${{ github.event_path }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: python3 .github/scripts/good_first_issue_assign.py
|
||||
|
||||
- name: Add assignee and notify
|
||||
if: steps.decide.outputs.should_assign == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
ISSUE_NUMBER: ${{ github.event.issue.number }}
|
||||
REPO_FULL: ${{ github.repository }}
|
||||
COMMENTER_LOGIN: ${{ github.event.comment.user.login }}
|
||||
run: |
|
||||
gh issue edit "${ISSUE_NUMBER}" \
|
||||
--repo "${REPO_FULL}" \
|
||||
--add-assignee "${COMMENTER_LOGIN}"
|
||||
gh issue comment "${ISSUE_NUMBER}" \
|
||||
--repo "${REPO_FULL}" \
|
||||
--body-file assign_comment.md
|
||||
@@ -0,0 +1,46 @@
|
||||
name: Greptile PR reminder
|
||||
|
||||
# One-time nudge when a PR is opened or reopened. Uses pull_request_target so the token can
|
||||
# comment on PRs from forks. Does not checkout the PR head — only posts a static reminder (see
|
||||
# SECURITY: pull_request_target guidance in GitHub Actions docs).
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened, reopened]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
remind:
|
||||
name: Post Greptile review reminder
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event.pull_request.user.type != 'Bot'
|
||||
steps:
|
||||
- name: Comment with Greptile guidance
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
BASE_REF: ${{ github.event.pull_request.base.ref }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
base_ref="${BASE_REF}"
|
||||
contributing_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/blob/${base_ref}/CONTRIBUTING.md#greptile-code-review"
|
||||
pr="${PR_NUMBER}"
|
||||
repo="${GITHUB_REPOSITORY}"
|
||||
{
|
||||
echo '### Greptile code review'
|
||||
echo ''
|
||||
echo "This repo uses **Greptile** for automated review. Before merge, aim for **Confidence Score: 5/5** with **zero unresolved** review threads — see [CONTRIBUTING.md](${contributing_url})."
|
||||
echo ''
|
||||
echo '**Run a review** — add a PR comment with:'
|
||||
echo ''
|
||||
echo '```'
|
||||
echo '@greptile review'
|
||||
echo '```'
|
||||
echo ''
|
||||
echo 'Give it **~5-10 minutes** (sometimes longer) for results, then fix feedback and re-trigger until you reach **Confidence Score: 5/5**.'
|
||||
echo ''
|
||||
echo '**Optional:** automate with the [greploop skill](https://skills.sh/greptileai/skills/greploop).'
|
||||
} > comment.md
|
||||
gh pr comment "$pr" --repo "$repo" --body-file comment.md
|
||||
@@ -0,0 +1,50 @@
|
||||
name: Hermes Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "tests/synthetic/hermes_rca/**"
|
||||
- "tests/e2e/hermes/**"
|
||||
- "tests/synthetic/mock_hermes_backend/**"
|
||||
- "tools/HermesSessionEvidenceTool/**"
|
||||
- ".github/workflows/hermes-tests.yml"
|
||||
- "Makefile"
|
||||
- "pytest.ini"
|
||||
schedule:
|
||||
- cron: "17 3 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
synthetic:
|
||||
name: Hermes synthetic offline
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: astral-sh/setup-uv@v5
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Run Hermes synthetic tests
|
||||
run: uv run pytest tests/synthetic/hermes_rca -q
|
||||
|
||||
- name: Run Hermes offline suite
|
||||
run: uv run python -m tests.synthetic.hermes_rca.run_suite --offline-only
|
||||
|
||||
e2e:
|
||||
name: Hermes e2e/meta
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: astral-sh/setup-uv@v5
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Run Hermes e2e/meta tests
|
||||
run: uv run pytest tests/e2e/hermes -q
|
||||
@@ -0,0 +1,160 @@
|
||||
name: Interactive Shell Live (PR + post-merge)
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "core/agent/**"
|
||||
- "core/agent_harness/**"
|
||||
- "tests/core/agent/**"
|
||||
- "surfaces/interactive_shell/**"
|
||||
- "core/runtime/**"
|
||||
- "tools/**"
|
||||
- "integrations/**"
|
||||
- "pytest.ini"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- ".github/workflows/interactive-shell-live.yml"
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "core/agent/**"
|
||||
- "core/agent_harness/**"
|
||||
- "tests/core/agent/**"
|
||||
- "surfaces/interactive_shell/**"
|
||||
- "core/runtime/**"
|
||||
- "tools/**"
|
||||
- "integrations/**"
|
||||
- "pytest.ini"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- ".github/workflows/interactive-shell-live.yml"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
shard_indexes:
|
||||
description: "Comma-separated shard indexes to run (0-7)"
|
||||
required: false
|
||||
default: "0,1,2,3,4,5,6,7"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: interactive-shell-live-${{ github.event.pull_request.number || github.sha }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
turn-checks:
|
||||
name: turn-checks (no-LLM)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
env:
|
||||
OPENSRE_DISABLE_KEYRING: "1"
|
||||
PYTHONUTF8: "1"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Deterministic turn + fixture integrity
|
||||
run: >-
|
||||
uv run python -m pytest -n auto -v
|
||||
tests/core/agent/
|
||||
-m "not live_llm"
|
||||
|
||||
turn-live:
|
||||
# Run on push to main, manual dispatch, and same-repo PRs. Fork PRs have no
|
||||
# secrets, so they skip the live job (the no-LLM turn-checks job above
|
||||
# still gates them) instead of failing the credential-validation step.
|
||||
if: >-
|
||||
github.repository == 'Tracer-Cloud/opensre' &&
|
||||
(github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false)
|
||||
name: turn-live shard ${{ matrix.shard_index }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard_index: ${{ fromJSON(github.event_name == 'workflow_dispatch' && format('[{0}]', github.event.inputs.shard_indexes) || '[0, 1, 2, 3, 4, 5, 6, 7]') }}
|
||||
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
LLM_PROVIDER: openai
|
||||
OPENSRE_DISABLE_KEYRING: "1"
|
||||
TURN_SHARD_TOTAL: "8"
|
||||
TURN_SHARD_INDEX: ${{ matrix.shard_index }}
|
||||
PYTHONUTF8: "1"
|
||||
# @live gather scenarios (316, 333–335, 337) require these secrets in CI.
|
||||
# Missing values fail the job in preflight and via skip_or_fail in tests.
|
||||
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
SENTRY_ORG_SLUG: tracer-30
|
||||
SENTRY_PROJECT_SLUG: python
|
||||
SENTRY_URL: https://sentry.io
|
||||
DD_API_KEY: ${{ secrets.DD_API_KEY }}
|
||||
DD_APP_KEY: ${{ secrets.DD_APP_KEY }}
|
||||
DD_SITE: datadoghq.com
|
||||
GRAFANA_READ_TOKEN: ${{ secrets.GRAFANA_READ_TOKEN }}
|
||||
GRAFANA_INSTANCE_URL: ${{ secrets.GRAFANA_INSTANCE_URL }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Validate live turn credentials
|
||||
run: |
|
||||
missing=()
|
||||
if [ -z "${OPENAI_API_KEY}" ]; then
|
||||
missing+=(OPENAI_API_KEY)
|
||||
fi
|
||||
if [ -z "${SENTRY_AUTH_TOKEN}" ]; then
|
||||
missing+=(SENTRY_AUTH_TOKEN)
|
||||
fi
|
||||
if [ -z "${DD_API_KEY}" ]; then
|
||||
missing+=(DD_API_KEY)
|
||||
fi
|
||||
if [ -z "${DD_APP_KEY}" ]; then
|
||||
missing+=(DD_APP_KEY)
|
||||
fi
|
||||
if [ ${#missing[@]} -gt 0 ]; then
|
||||
echo "::error::Missing secrets for live turn tests: ${missing[*]}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run sharded live turn suite
|
||||
run: >-
|
||||
uv run python -m pytest -n auto -v
|
||||
-m live_llm
|
||||
tests/core/agent/test_turn_scenarios.py
|
||||
-k "test_live_turn_execution_oracle or test_live_action_planning"
|
||||
@@ -0,0 +1,678 @@
|
||||
name: Release
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths-ignore:
|
||||
- "docs/**"
|
||||
- "tests/benchmarks/cloudopsbench/infra/**"
|
||||
- "platform/deployment/install-proxy/**"
|
||||
- "tests/e2e/kubernetes/helm/scripts/**"
|
||||
- "tests/**"
|
||||
- "**/*.md"
|
||||
- "**/*.mdx"
|
||||
- ".claude/**"
|
||||
schedule:
|
||||
- cron: "30 0 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
channel:
|
||||
description: "Release channel to publish."
|
||||
required: false
|
||||
default: release
|
||||
type: choice
|
||||
options:
|
||||
- release
|
||||
- main
|
||||
tag:
|
||||
description: "Optional tag to release (e.g. v0.1.2026.6.26 or v0.1). Ignored for the main channel."
|
||||
required: false
|
||||
type: string
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
actions: read
|
||||
models: read
|
||||
|
||||
concurrency:
|
||||
group: release-${{ github.event_name == 'push' && 'main' || (github.event_name == 'workflow_dispatch' && inputs.channel == 'main' && 'main') || 'stable' }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
channel: ${{ steps.meta.outputs.channel }}
|
||||
tag_name: ${{ steps.meta.outputs.tag_name }}
|
||||
version_name: ${{ steps.meta.outputs.version_name }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Resolve release metadata
|
||||
id: meta
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
DISPATCH_CHANNEL: ${{ inputs.channel }}
|
||||
DISPATCH_TAG: ${{ inputs.tag }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
channel="release"
|
||||
|
||||
if [ "$EVENT_NAME" = "push" ]; then
|
||||
channel="main"
|
||||
fi
|
||||
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] && [ "$DISPATCH_CHANNEL" = "main" ]; then
|
||||
channel="main"
|
||||
fi
|
||||
|
||||
if [ "$channel" = "main" ]; then
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] && [ -n "$DISPATCH_TAG" ]; then
|
||||
echo "The main channel does not accept a custom tag." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
year="$(date -u +%Y)"
|
||||
month="$(date -u +%-m)"
|
||||
day="$(date -u +%-d)"
|
||||
short_sha="${GITHUB_SHA:0:7}"
|
||||
main_version="0.1.${year}.${month}.${day}+main.${short_sha}"
|
||||
|
||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||
echo "tag_name=main-build" >> "$GITHUB_OUTPUT"
|
||||
echo "version_name=${main_version}" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] && [ -n "$DISPATCH_TAG" ]; then
|
||||
tag_name="$DISPATCH_TAG"
|
||||
else
|
||||
year="$(date -u +%Y)"
|
||||
month="$(date -u +%-m)"
|
||||
day="$(date -u +%-d)"
|
||||
# Keep the v0.1 prefix so the tag makes clear the product is still
|
||||
# v0.1, while the date stays useful (e.g. v0.1.2026.6.26).
|
||||
tag_name="v0.1.${year}.${month}.${day}"
|
||||
fi
|
||||
|
||||
echo "channel=release" >> "$GITHUB_OUTPUT"
|
||||
echo "tag_name=${tag_name}" >> "$GITHUB_OUTPUT"
|
||||
echo "version_name=${tag_name#v}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
verify:
|
||||
runs-on: ubuntu-latest
|
||||
needs: prepare
|
||||
if: github.event_name != 'push'
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Lint
|
||||
run: make lint
|
||||
|
||||
- name: Type check
|
||||
run: make typecheck
|
||||
|
||||
- name: CLI smoke tests
|
||||
run: make test-cli-smoke
|
||||
|
||||
build-python-dist:
|
||||
if: needs.prepare.outputs.channel == 'release' && needs.verify.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
needs: [verify, prepare]
|
||||
env:
|
||||
TAG_NAME: ${{ needs.prepare.outputs.tag_name }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Sync release version
|
||||
shell: bash
|
||||
run: python platform/packaging/sync_release_version.py --tag "$TAG_NAME"
|
||||
|
||||
- name: Build Python distributions
|
||||
run: |
|
||||
uv sync --frozen --extra release-dist
|
||||
uv run python -m build
|
||||
uv run twine check dist/*
|
||||
|
||||
- name: Verify Python distribution version
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
VERSION="${TAG_NAME#v}"
|
||||
test -f "dist/opensre-${VERSION}.tar.gz"
|
||||
test -f "dist/opensre-${VERSION}-py3-none-any.whl"
|
||||
|
||||
- name: Upload Python distributions
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: release-python-dist
|
||||
path: dist/*
|
||||
if-no-files-found: error
|
||||
|
||||
build-binaries:
|
||||
if: always() && (needs.prepare.outputs.channel == 'main' || needs.verify.result == 'success')
|
||||
needs: [verify, prepare]
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ubuntu-22.04
|
||||
target: linux-x64
|
||||
binary_name: opensre
|
||||
archive_ext: tar.gz
|
||||
pyinstaller_mode: onedir
|
||||
- runner: ubuntu-22.04-arm
|
||||
target: linux-arm64
|
||||
binary_name: opensre
|
||||
archive_ext: tar.gz
|
||||
pyinstaller_mode: onedir
|
||||
- runner: macos-15-intel
|
||||
target: darwin-x64
|
||||
binary_name: opensre
|
||||
archive_ext: tar.gz
|
||||
pyinstaller_mode: onedir
|
||||
- runner: macos-latest
|
||||
target: darwin-arm64
|
||||
binary_name: opensre
|
||||
archive_ext: tar.gz
|
||||
pyinstaller_mode: onedir
|
||||
- runner: windows-latest
|
||||
target: windows-x64
|
||||
binary_name: opensre.exe
|
||||
archive_ext: zip
|
||||
pyinstaller_mode: onefile
|
||||
# windows-arm64 is currently excluded from the default release matrix:
|
||||
# cryptography does not publish win_arm64 wheels, so dependency install
|
||||
# falls back to a source build that requires an OpenSSL toolchain on the
|
||||
# GitHub-hosted runner.
|
||||
runs-on: ${{ matrix.runner }}
|
||||
env:
|
||||
RELEASE_CHANNEL: ${{ needs.prepare.outputs.channel }}
|
||||
TAG_NAME: ${{ needs.prepare.outputs.tag_name }}
|
||||
VERSION_NAME: ${{ needs.prepare.outputs.version_name }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Sync binary version
|
||||
shell: bash
|
||||
run: python platform/packaging/sync_release_version.py --version "$VERSION_NAME"
|
||||
|
||||
- name: Install binary build dependencies
|
||||
shell: bash
|
||||
run: uv sync --frozen --extra release-binary
|
||||
|
||||
- name: Stage stdlib platform module for bundling
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The first-party ``platform`` package shadows the stdlib ``platform``
|
||||
# module. PyInstaller does not lay out the stdlib as loose ``.py`` files,
|
||||
# so bundle a copy of the genuine module that platform/__init__.py can
|
||||
# load from ``sys._MEIPASS`` at runtime (see _FROZEN_STDLIB_DIR).
|
||||
rm -rf .stdlib_vendor
|
||||
mkdir -p .stdlib_vendor
|
||||
uv run python -c "import os, shutil, sysconfig; src = os.path.join(sysconfig.get_path('stdlib'), 'platform.py'); shutil.copy(src, os.path.join('.stdlib_vendor', 'platform.py')); print('Staged stdlib platform from ' + src)"
|
||||
|
||||
- name: Build binary
|
||||
run: >-
|
||||
uv run pyinstaller surfaces/cli/__main__.py
|
||||
--name opensre
|
||||
--${{ matrix.pyinstaller_mode }}
|
||||
--clean
|
||||
--noconfirm
|
||||
--collect-data surfaces.cli
|
||||
--collect-data config
|
||||
--copy-metadata opensre
|
||||
--collect-data litellm
|
||||
--hidden-import tiktoken_ext
|
||||
--hidden-import tiktoken_ext.openai_public
|
||||
--collect-submodules integrations
|
||||
--collect-submodules surfaces.interactive_shell
|
||||
--collect-submodules tools
|
||||
--add-data "platform:platform"
|
||||
--add-data ".stdlib_vendor:_opensre_stdlib_platform"
|
||||
|
||||
- name: Smoke test binary (Unix)
|
||||
if: runner.os != 'Windows'
|
||||
shell: bash
|
||||
run: |
|
||||
set -uo pipefail
|
||||
# Capture the exit status explicitly so a crashing binary still prints
|
||||
# its stdout/stderr (under `set -e` the failed substitution aborted the
|
||||
# step before the output was ever shown, hiding the real error).
|
||||
# onefile builds place the executable at ./dist/<name>; onedir builds
|
||||
# produce a ./dist/opensre/ directory containing the executable. A bare
|
||||
# `-x` test matches the onedir directory too (dirs carry the search
|
||||
# bit), so require a regular file before treating it as the binary.
|
||||
if [ -f "./dist/${{ matrix.binary_name }}" ] && [ -x "./dist/${{ matrix.binary_name }}" ]; then
|
||||
BINARY_PATH="./dist/${{ matrix.binary_name }}"
|
||||
else
|
||||
BINARY_PATH="./dist/opensre/${{ matrix.binary_name }}"
|
||||
fi
|
||||
set +e
|
||||
VERSION_OUTPUT="$("$BINARY_PATH" --version 2>&1)"
|
||||
VERSION_STATUS=$?
|
||||
set -e
|
||||
printf '%s\n' "$VERSION_OUTPUT"
|
||||
if [ "$VERSION_STATUS" -ne 0 ]; then
|
||||
printf '::error::%s --version exited with status %s\n' "${{ matrix.binary_name }}" "$VERSION_STATUS" >&2
|
||||
exit "$VERSION_STATUS"
|
||||
fi
|
||||
case "$VERSION_OUTPUT" in
|
||||
*"$VERSION_NAME"*) ;;
|
||||
*)
|
||||
printf 'Binary version mismatch: expected %s but saw %s\n' "$VERSION_NAME" "$VERSION_OUTPUT" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
"$BINARY_PATH" -h >/dev/null
|
||||
LITELLM_DATA="./dist/opensre/_internal/litellm/model_prices_and_context_window_backup.json"
|
||||
if [ ! -f "$LITELLM_DATA" ]; then
|
||||
echo "LiteLLM package data missing from Unix onedir bundle: ${LITELLM_DATA}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Check Linux binary glibc compatibility
|
||||
if: runner.os == 'Linux'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
max_glibc="$(
|
||||
find dist/opensre -type f \( -name opensre -o -name '*.so' -o -name '*.so.*' \) -print0 \
|
||||
| xargs -0 strings 2>/dev/null \
|
||||
| grep -Eo 'GLIBC_[0-9]+\.[0-9]+(\.[0-9]+)?' \
|
||||
| sort -Vu \
|
||||
| tail -n 1 \
|
||||
|| true
|
||||
)"
|
||||
echo "Max required glibc symbol: ${max_glibc:-none}"
|
||||
if [ -n "$max_glibc" ] && [ "$(printf '%s\n' "$max_glibc" GLIBC_2.35 | sort -V | tail -n 1)" != "GLIBC_2.35" ]; then
|
||||
echo "Linux binary requires ${max_glibc}; pin the Linux runner back to Ubuntu 22.04 or lower dependency wheel requirements." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Smoke test binary (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
shell: pwsh
|
||||
run: |
|
||||
$versionOutput = & ".\dist\${{ matrix.binary_name }}" --version 2>&1 | Out-String
|
||||
$versionText = $versionOutput.Trim()
|
||||
Write-Host $versionText
|
||||
$expectedVersion = $env:VERSION_NAME
|
||||
if ($versionText -notmatch [regex]::Escape($expectedVersion)) {
|
||||
throw "Binary version mismatch. Expected '$expectedVersion' but saw '$versionText'."
|
||||
}
|
||||
& ".\dist\${{ matrix.binary_name }}" -h | Out-Null
|
||||
|
||||
- name: Package binary archive (Unix)
|
||||
if: runner.os != 'Windows'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$RELEASE_CHANNEL" = "release" ]; then
|
||||
ASSET_BASENAME="opensre_${TAG_NAME#v}_${{ matrix.target }}"
|
||||
else
|
||||
ASSET_BASENAME="opensre_main_${{ matrix.target }}"
|
||||
fi
|
||||
tar -C dist -czf "${ASSET_BASENAME}.tar.gz" "${{ matrix.binary_name }}"
|
||||
shasum -a 256 "${ASSET_BASENAME}.tar.gz" > "${ASSET_BASENAME}.tar.gz.sha256"
|
||||
|
||||
- name: Package binary archive (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
shell: pwsh
|
||||
run: |
|
||||
if ($env:RELEASE_CHANNEL -eq "release") {
|
||||
$assetBaseName = "opensre_$($env:TAG_NAME.TrimStart('v'))_${{ matrix.target }}"
|
||||
} else {
|
||||
$assetBaseName = "opensre_main_${{ matrix.target }}"
|
||||
}
|
||||
Compress-Archive -Path "dist\${{ matrix.binary_name }}" -DestinationPath "${assetBaseName}.zip"
|
||||
$hash = (Get-FileHash -Algorithm SHA256 "${assetBaseName}.zip").Hash.ToLowerInvariant()
|
||||
Set-Content -Path "${assetBaseName}.zip.sha256" -Value "$hash ${assetBaseName}.zip"
|
||||
|
||||
- name: Upload binary archive
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ env.RELEASE_CHANNEL == 'main' && 'main-' || '' }}release-${{ matrix.target }}
|
||||
path: |
|
||||
opensre_*_${{ matrix.target }}.${{ matrix.archive_ext }}
|
||||
opensre_*_${{ matrix.target }}.${{ matrix.archive_ext }}.sha256
|
||||
if-no-files-found: error
|
||||
|
||||
publish-release:
|
||||
if: needs.prepare.outputs.channel == 'release'
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
- prepare
|
||||
- build-python-dist
|
||||
- build-binaries
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
with:
|
||||
ref: ${{ github.event.repository.default_branch }}
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Download release artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: release-*
|
||||
path: release-assets
|
||||
merge-multiple: true
|
||||
|
||||
- name: Resolve release context
|
||||
id: release_ctx
|
||||
env:
|
||||
TAG_NAME: ${{ needs.prepare.outputs.tag_name }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
default_branch="${{ github.event.repository.default_branch }}"
|
||||
git fetch origin "$default_branch" --tags --force
|
||||
target_sha="$(git rev-parse "origin/$default_branch")"
|
||||
|
||||
previous_tag="$(
|
||||
git tag --list 'v0.1.[0-9][0-9][0-9][0-9].[0-9]*.[0-9]*' --sort=-v:refname \
|
||||
| grep -v -x "$TAG_NAME" \
|
||||
| head -n 1 \
|
||||
|| true
|
||||
)"
|
||||
|
||||
range_spec="$target_sha"
|
||||
if [ -n "$previous_tag" ]; then
|
||||
range_spec="${previous_tag}..${target_sha}"
|
||||
fi
|
||||
|
||||
{
|
||||
printf 'target_sha=%s\n' "$target_sha"
|
||||
printf 'previous_tag=%s\n' "$previous_tag"
|
||||
printf 'range_spec=%s\n' "$range_spec"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create release notes
|
||||
env:
|
||||
TAG_NAME: ${{ needs.prepare.outputs.tag_name }}
|
||||
RANGE_SPEC: ${{ steps.release_ctx.outputs.range_spec }}
|
||||
PREVIOUS_TAG: ${{ steps.release_ctx.outputs.previous_tag }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
{
|
||||
echo "## Changelog"
|
||||
echo
|
||||
if [ -n "$PREVIOUS_TAG" ]; then
|
||||
echo "_Changes since ${PREVIOUS_TAG}_"
|
||||
else
|
||||
echo "_Changes up to ${TAG_NAME}_"
|
||||
fi
|
||||
echo
|
||||
git log "$RANGE_SPEC" --no-merges -n 100 --pretty='- %s (%h) — %an'
|
||||
echo
|
||||
} > GENERATED_CHANGELOG.md
|
||||
|
||||
# Summarize commits into prose via GitHub Models (gpt-4o-mini).
|
||||
# Falls back to raw changelog if the API is unavailable.
|
||||
raw_commits="$(git log "$RANGE_SPEC" --no-merges --pretty='%s' | head -40 || true)"
|
||||
if [ -n "$raw_commits" ]; then
|
||||
api_body="$(jq -n --arg commits "$raw_commits" '{
|
||||
model: "gpt-4o-mini",
|
||||
messages: [
|
||||
{
|
||||
role: "system",
|
||||
content: "You are writing a Discord release announcement for an open-source SRE CLI tool called opensre. Summarize the git commits into 2-4 short, punchy prose sentences — no bullet points, no headers, no markdown. Write in present tense, active voice. Focus on what users will notice: new features, fixes, performance, integrations. Be specific but concise."
|
||||
},
|
||||
{role: "user", content: ("Commits:\n" + $commits)}
|
||||
],
|
||||
max_tokens: 300,
|
||||
temperature: 0.4
|
||||
}')"
|
||||
curl -sS --max-time 20 \
|
||||
-H "Authorization: Bearer $GITHUB_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://models.inference.ai.azure.com/chat/completions" \
|
||||
-d "$api_body" 2>/dev/null \
|
||||
| jq -r '.choices[0].message.content // empty' 2>/dev/null \
|
||||
| tr -d '\r' > DISCORD_NARRATIVE.md || true
|
||||
|
||||
if [ -s DISCORD_NARRATIVE.md ]; then
|
||||
echo "LLM narrative generated ($(wc -c < DISCORD_NARRATIVE.md) bytes)."
|
||||
else
|
||||
echo "LLM summary unavailable; Discord will use raw changelog."
|
||||
rm -f DISCORD_NARRATIVE.md
|
||||
fi
|
||||
fi
|
||||
|
||||
CB='```'
|
||||
{
|
||||
printf '## Install\n\n'
|
||||
printf '### cURL (macOS / Linux)\n\n%sbash\ncurl -fsSL https://install.opensre.com | bash -s -- --release --version %s\n%s\n\n' "$CB" "${TAG_NAME#v}" "$CB"
|
||||
printf '### Homebrew (macOS / Linux)\n\n%sbash\nbrew tap tracer-cloud/tap\nbrew install tracer-cloud/tap/opensre\n%s\n\n' "$CB" "$CB"
|
||||
printf '### PowerShell (Windows)\n\n%spowershell\n$env:OPENSRE_INSTALL_CHANNEL="release"; $env:OPENSRE_VERSION="%s"; irm https://install.opensre.com | iex\n%s\n\n' "$CB" "${TAG_NAME#v}" "$CB"
|
||||
printf '### Python\n\n%sbash\npipx install opensre\n%s\n\n' "$CB" "$CB"
|
||||
} > RELEASE_NOTES.md
|
||||
cat GENERATED_CHANGELOG.md >> RELEASE_NOTES.md
|
||||
|
||||
- name: Create GitHub release
|
||||
id: github_release
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAG_NAME: ${{ needs.prepare.outputs.tag_name }}
|
||||
TARGET_SHA: ${{ steps.release_ctx.outputs.target_sha }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if gh release view "$TAG_NAME" --repo "${{ github.repository }}" >/dev/null 2>&1; then
|
||||
echo "created=false" >> "$GITHUB_OUTPUT"
|
||||
echo "Release $TAG_NAME already exists; nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
VERSION="${TAG_NAME#v}"
|
||||
|
||||
# Every release-channel publish is the newest stable build at this
|
||||
# point, so mark it as GitHub's Latest. The title keeps the full
|
||||
# version (e.g. "OpenSRE 0.1.2026.6.26") so the v0.1 line is clear.
|
||||
gh release create "$TAG_NAME" \
|
||||
release-assets/* \
|
||||
--repo "${{ github.repository }}" \
|
||||
--target "$TARGET_SHA" \
|
||||
--title "OpenSRE ${VERSION}" \
|
||||
--notes-file RELEASE_NOTES.md \
|
||||
--latest
|
||||
|
||||
echo "created=true" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Announce on Discord
|
||||
if: steps.github_release.outputs.created == 'true' && !github.event.repository.fork && !github.event.repository.private
|
||||
continue-on-error: true
|
||||
env:
|
||||
DISCORD_WEBHOOK_URL_UPDATE: ${{ secrets.DISCORD_WEBHOOK_URL_UPDATE }}
|
||||
DISCORD_RELEASES_ROLE_ID: ${{ secrets.DISCORD_RELEASES_ROLE_ID }}
|
||||
DISCORD_RELEASE_LOGO_EMOJI: ${{ secrets.DISCORD_RELEASE_LOGO_EMOJI }}
|
||||
DISCORD_RELEASE_LOGO_URL: ${{ secrets.DISCORD_RELEASE_LOGO_URL }}
|
||||
RELEASE_TAG: ${{ needs.prepare.outputs.tag_name }}
|
||||
RELEASE_URL: ${{ github.server_url }}/${{ github.repository }}/releases/tag/${{ needs.prepare.outputs.tag_name }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if [ -z "${DISCORD_WEBHOOK_URL_UPDATE:-}" ]; then
|
||||
echo "DISCORD_WEBHOOK_URL_UPDATE is not set; skipping Discord announcement."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Prefer LLM-generated narrative; fall back to raw changelog.
|
||||
if [ -f DISCORD_NARRATIVE.md ] && [ -s DISCORD_NARRATIVE.md ]; then
|
||||
changelog_file="DISCORD_NARRATIVE.md"
|
||||
elif [ -f GENERATED_CHANGELOG.md ]; then
|
||||
changelog_file="GENERATED_CHANGELOG.md"
|
||||
else
|
||||
echo "No changelog available." > /tmp/discord_fallback.md
|
||||
changelog_file="/tmp/discord_fallback.md"
|
||||
fi
|
||||
|
||||
# Use an external Python script to build the JSON payload.
|
||||
# This avoids jq issues with multiline LLM output containing control characters.
|
||||
payload="$(CHANGELOG_FILE="$changelog_file" python3 .github/scripts/build-discord-payload.py)"
|
||||
|
||||
curl --fail -sS --max-time 30 -X POST -H "Content-Type: application/json" \
|
||||
-d "$payload" "$DISCORD_WEBHOOK_URL_UPDATE"
|
||||
|
||||
- name: Sync Homebrew tap formula
|
||||
if: steps.github_release.outputs.created == 'true'
|
||||
continue-on-error: true
|
||||
env:
|
||||
HOMEBREW_TAP_GITHUB_TOKEN: ${{ secrets.HOMEBREW_TAP_GITHUB_TOKEN }}
|
||||
VERSION: ${{ needs.prepare.outputs.tag_name }}
|
||||
ASSET_DIR: release-assets
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${HOMEBREW_TAP_GITHUB_TOKEN:-}" ]; then
|
||||
echo "HOMEBREW_TAP_GITHUB_TOKEN is not set; skipping Homebrew tap sync."
|
||||
exit 0
|
||||
fi
|
||||
export VERSION="${VERSION#v}"
|
||||
bash .github/scripts/sync-homebrew-tap-formula.sh
|
||||
|
||||
publish-main-release:
|
||||
if: always() && needs.prepare.outputs.channel == 'main' && needs.build-binaries.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
- prepare
|
||||
- build-binaries
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
# The rolling `main-build` tag is force-moved to each new main commit
|
||||
# below. Whenever main has advanced past a commit that touches
|
||||
# .github/workflows/**, the default GITHUB_TOKEN (the
|
||||
# github-actions[bot] GitHub App) is refused with "refusing to allow a
|
||||
# GitHub App to create or update workflow ... without `workflows`
|
||||
# permission" — and that scope cannot be granted via the permissions
|
||||
# block. MAIN_BUILD_TAG_TOKEN must be a token that carries workflow
|
||||
# write access (classic PAT with `repo` + `workflow`, a fine-grained
|
||||
# PAT with Contents + Workflows: write, or a GitHub App token). It is
|
||||
# persisted so the tag push below authenticates with it. Falls back to
|
||||
# GITHUB_TOKEN so the rest of the job still runs if the secret is unset
|
||||
# (the tag push will then fail as before until the secret is added).
|
||||
token: ${{ secrets.MAIN_BUILD_TAG_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
persist-credentials: true
|
||||
|
||||
- name: Download main release artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: main-release-*
|
||||
path: main-release-assets
|
||||
merge-multiple: true
|
||||
|
||||
- name: Create release notes
|
||||
env:
|
||||
VERSION_NAME: ${{ needs.prepare.outputs.version_name }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
short_sha="$(printf '%s' "$GITHUB_SHA" | cut -c1-7)"
|
||||
built_at="$(date -u +"%Y-%m-%d %H:%M UTC")"
|
||||
|
||||
CB='```'
|
||||
{
|
||||
printf '## Main build\n\nRolling binary build from `main`.\n\n'
|
||||
printf -- '- Version: `%s`\n- Commit: `%s`\n- Built: %s\n\n' "$VERSION_NAME" "$short_sha" "$built_at"
|
||||
printf '### Install\n\nmacOS / Linux:\n\n%sbash\ncurl -fsSL https://install.opensre.com | bash\n%s\n\n' "$CB" "$CB"
|
||||
printf 'Equivalent explicit main channel:\n\n%sbash\ncurl -fsSL https://install.opensre.com | bash -s -- --main\n%s\n\n' "$CB" "$CB"
|
||||
printf 'Windows:\n\n%spowershell\nirm https://install.opensre.com | iex\n%s\n' "$CB" "$CB"
|
||||
} > MAIN_RELEASE_NOTES.md
|
||||
|
||||
- name: Move main build tag to the latest commit
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git tag -f "${{ needs.prepare.outputs.tag_name }}" "$GITHUB_SHA"
|
||||
git push origin "refs/tags/${{ needs.prepare.outputs.tag_name }}" --force
|
||||
|
||||
- name: Publish rolling main release
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
tag_name="${{ needs.prepare.outputs.tag_name }}"
|
||||
|
||||
if gh release view "$tag_name" --repo "${{ github.repository }}" >/dev/null 2>&1; then
|
||||
gh release upload "$tag_name" main-release-assets/* --repo "${{ github.repository }}" --clobber
|
||||
gh release edit "$tag_name" \
|
||||
--repo "${{ github.repository }}" \
|
||||
--title "Main" \
|
||||
--notes-file MAIN_RELEASE_NOTES.md
|
||||
exit 0
|
||||
fi
|
||||
|
||||
gh release create "$tag_name" \
|
||||
main-release-assets/* \
|
||||
--repo "${{ github.repository }}" \
|
||||
--target "$GITHUB_SHA" \
|
||||
--title "Main" \
|
||||
--notes-file MAIN_RELEASE_NOTES.md \
|
||||
--prerelease
|
||||
@@ -0,0 +1,84 @@
|
||||
name: Synthetic Deterministic Tests
|
||||
|
||||
# Regression guardrail: run every offline (no-LLM) synthetic test on every
|
||||
# PR and merge so pipeline refactors can't silently break investigations.
|
||||
#
|
||||
# Tests that need a live LLM are excluded via the marker filter below.
|
||||
# Those suites live in interactive-shell-live.yml and hermes-tests.yml instead.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "surfaces/**"
|
||||
- "config/**"
|
||||
- "core/**"
|
||||
- "integrations/**"
|
||||
- "platform/**"
|
||||
- "tools/**"
|
||||
- "tests/synthetic/**"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "pytest.ini"
|
||||
- ".github/workflows/synthetic-deterministic.yml"
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "surfaces/**"
|
||||
- "config/**"
|
||||
- "core/**"
|
||||
- "integrations/**"
|
||||
- "platform/**"
|
||||
- "tools/**"
|
||||
- "tests/synthetic/**"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "pytest.ini"
|
||||
- ".github/workflows/synthetic-deterministic.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: synthetic-det-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
offline:
|
||||
name: Synthetic offline (deterministic)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Run deterministic synthetic tests
|
||||
# Excludes markers that require a live LLM:
|
||||
# synthetic — full RCA scenario suites (EKS, RDS, Hermes, Dagster …)
|
||||
# axis2 — adversarial scenario suites (also call the LLM)
|
||||
# live_llm — explicit live-credential gate
|
||||
# e2e — requires live infrastructure
|
||||
run: |
|
||||
uv run pytest tests/synthetic \
|
||||
-m "not synthetic and not live_llm and not e2e and not axis2" \
|
||||
-q
|
||||
|
||||
- name: Run hermes_rca offline scenario suite
|
||||
run: uv run python -m tests.synthetic.hermes_rca.run_suite --offline-only
|
||||
@@ -0,0 +1,282 @@
|
||||
name: terraform-bench (plan)
|
||||
|
||||
# Runs terraform fmt + init + validate + plan against tests/benchmarks/cloudopsbench/infra/ on every PR
|
||||
# that touches the bench infra. Posts the plan output as a sticky PR comment so
|
||||
# reviewers can see what would change before approving.
|
||||
#
|
||||
# Never applies — apply is developer-local in v1 (see tests/benchmarks/cloudopsbench/infra/README.md).
|
||||
# When apply-from-CI is desirable later, add a separate workflow that runs on
|
||||
# merge-to-main with a different (higher-privileged) AWS role.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "tests/benchmarks/cloudopsbench/infra/**"
|
||||
- ".github/workflows/terraform-bench.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
id-token: write # required for AWS OIDC role assumption
|
||||
|
||||
concurrency:
|
||||
group: terraform-bench-${{ github.workflow }}-${{ github.head_ref || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
plan:
|
||||
name: terraform plan
|
||||
if: github.repository == 'Tracer-Cloud/opensre' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
defaults:
|
||||
run:
|
||||
working-directory: tests/benchmarks/cloudopsbench/infra
|
||||
|
||||
env:
|
||||
AWS_REGION: us-east-1
|
||||
# tfplan + plan.txt artifacts are not uploaded; plan output goes into the
|
||||
# PR comment. If you want a downloadable plan binary, add an
|
||||
# actions/upload-artifact step after `terraform show`.
|
||||
TF_IN_AUTOMATION: "true"
|
||||
TF_INPUT: "0"
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Setup Terraform
|
||||
uses: hashicorp/setup-terraform@v3
|
||||
with:
|
||||
terraform_version: "1.7.5"
|
||||
# Wrapper MUST stay enabled (default) — it's what populates
|
||||
# `steps.<id>.outputs.stdout` for the `terraform show` step that
|
||||
# feeds the sticky PR comment. Disabling it makes the comment
|
||||
# render with a blank plan body. See greptile review on initial
|
||||
# PR for details.
|
||||
|
||||
- name: Configure AWS credentials (OIDC role assumption)
|
||||
# No long-lived AWS keys. The role is provisioned by tests/benchmarks/cloudopsbench/infra/
|
||||
# Terraform; ARN is hardcoded here because Terraform outputs aren't
|
||||
# available before the workflow can run. Account ID is the
|
||||
# tracer-cloud account.
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/opensre-bench-terraform-plan
|
||||
role-session-name: github-actions-terraform-plan-${{ github.run_id }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: terraform fmt
|
||||
id: fmt
|
||||
run: terraform fmt -check -recursive -no-color
|
||||
continue-on-error: true
|
||||
|
||||
- name: terraform init
|
||||
id: init
|
||||
run: terraform init -no-color
|
||||
|
||||
- name: terraform validate
|
||||
id: validate
|
||||
run: terraform validate -no-color
|
||||
|
||||
- name: terraform plan
|
||||
id: plan
|
||||
run: terraform plan -no-color -input=false -lock=false -out=tfplan
|
||||
continue-on-error: true
|
||||
|
||||
- name: terraform show
|
||||
id: show
|
||||
if: steps.plan.outcome == 'success'
|
||||
run: terraform show -no-color tfplan
|
||||
|
||||
- name: Post plan to PR
|
||||
if: github.event_name == 'pull_request' && always()
|
||||
uses: actions/github-script@v7
|
||||
env:
|
||||
PLAN_STDOUT: ${{ steps.show.outputs.stdout }}
|
||||
FMT_OUTCOME: ${{ steps.fmt.outcome }}
|
||||
INIT_OUTCOME: ${{ steps.init.outcome }}
|
||||
VALIDATE_OUTCOME: ${{ steps.validate.outcome }}
|
||||
PLAN_OUTCOME: ${{ steps.plan.outcome }}
|
||||
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const MARKER = '<!-- terraform-bench-plan-comment -->';
|
||||
|
||||
const fmt = process.env.FMT_OUTCOME;
|
||||
const init = process.env.INIT_OUTCOME;
|
||||
const validate = process.env.VALIDATE_OUTCOME;
|
||||
const plan = process.env.PLAN_OUTCOME;
|
||||
const url = process.env.WORKFLOW_URL;
|
||||
|
||||
const icon = (outcome) => outcome === 'success' ? '✅' :
|
||||
outcome === 'failure' ? '❌' :
|
||||
outcome === 'skipped' ? '⏭️' : '⚠️';
|
||||
|
||||
let planBody = process.env.PLAN_STDOUT || '';
|
||||
// GitHub PR comments cap at 65536 chars. Reserve ~1000 for surrounding text.
|
||||
const MAX = 60000;
|
||||
let truncated = false;
|
||||
if (planBody.length > MAX) {
|
||||
planBody = planBody.slice(-MAX);
|
||||
truncated = true;
|
||||
}
|
||||
|
||||
const body = [
|
||||
MARKER,
|
||||
'### terraform-bench plan',
|
||||
'',
|
||||
'| step | outcome |',
|
||||
'| --- | --- |',
|
||||
`| fmt | ${icon(fmt)} \`${fmt}\` |`,
|
||||
`| init | ${icon(init)} \`${init}\` |`,
|
||||
`| validate | ${icon(validate)} \`${validate}\` |`,
|
||||
`| plan | ${icon(plan)} \`${plan}\` |`,
|
||||
'',
|
||||
fmt !== 'success' ? '> Run `terraform fmt -recursive` in `tests/benchmarks/cloudopsbench/infra/` to fix formatting.' : '',
|
||||
'',
|
||||
plan === 'success'
|
||||
? `<details><summary>Plan output${truncated ? ' (truncated — see <a href="' + url + '">workflow logs</a> for full output)' : ''}</summary>\n\n\`\`\`hcl\n${planBody}\n\`\`\`\n\n</details>`
|
||||
: `Plan did not run successfully — see [workflow logs](${url}) for details.`,
|
||||
'',
|
||||
`_Updated by [\`terraform-bench.yml\`](${url})._`,
|
||||
].join('\n');
|
||||
|
||||
// Find an existing comment to update, else create a new one.
|
||||
const { data: comments } = await github.rest.issues.listComments({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
});
|
||||
const existing = comments.find(c => c.body && c.body.includes(MARKER));
|
||||
|
||||
if (existing) {
|
||||
await github.rest.issues.updateComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: existing.id,
|
||||
body,
|
||||
});
|
||||
} else {
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
body,
|
||||
});
|
||||
}
|
||||
|
||||
- name: Fail if any step errored
|
||||
if: |
|
||||
steps.fmt.outcome == 'failure' ||
|
||||
steps.init.outcome == 'failure' ||
|
||||
steps.validate.outcome == 'failure' ||
|
||||
steps.plan.outcome == 'failure'
|
||||
run: |
|
||||
echo "::error::One or more terraform steps failed. See PR comment + workflow logs."
|
||||
exit 1
|
||||
|
||||
# ---------------------------------------------------------------------- #
|
||||
# Security + lint scanners — run in parallel with plan. #
|
||||
# #
|
||||
# Suppression policy (intentional skips) is defined per-tool: #
|
||||
# - checkov: tests/benchmarks/cloudopsbench/infra/.checkov.yaml #
|
||||
# - tfsec: tests/benchmarks/cloudopsbench/infra/.tfsec/config.yml #
|
||||
# - tflint: tests/benchmarks/cloudopsbench/infra/.tflint.hcl #
|
||||
# Each suppression includes a justification at the source. Day-one #
|
||||
# findings should be REAL findings — change configs, not the workflow, #
|
||||
# if a rule needs to be skipped. #
|
||||
# #
|
||||
# checkov + tfsec hard-fail (security blocks PRs). tflint soft-fails #
|
||||
# because its rules cover code quality / deprecations, not security — #
|
||||
# we want visibility without blocking. #
|
||||
# ---------------------------------------------------------------------- #
|
||||
|
||||
tflint:
|
||||
name: tflint
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: tests/benchmarks/cloudopsbench/infra
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Setup tflint
|
||||
uses: terraform-linters/setup-tflint@v4
|
||||
with:
|
||||
tflint_version: latest
|
||||
|
||||
- name: tflint --init (download plugins)
|
||||
run: tflint --init
|
||||
|
||||
- name: tflint
|
||||
id: tflint
|
||||
run: tflint --recursive --format compact
|
||||
continue-on-error: true
|
||||
|
||||
- name: Surface tflint findings (non-blocking)
|
||||
if: steps.tflint.outcome == 'failure'
|
||||
run: |
|
||||
echo "::warning::tflint reported findings. Code-quality only — does not block PR. Review in workflow logs."
|
||||
|
||||
tfsec:
|
||||
name: tfsec
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v5
|
||||
|
||||
# tfsec via Docker, bypassing aquasecurity/tfsec-action.
|
||||
#
|
||||
# Why direct Docker run: the action shells out to api.github.com to
|
||||
# resolve the latest release binary, and the aquasecurity org has an
|
||||
# IP allow list that blocks GitHub Actions runner IPs — making the
|
||||
# action effectively unusable from CI. The aquasec/tfsec image on
|
||||
# Docker Hub has no such restriction.
|
||||
#
|
||||
# Migration path (v2): tfsec is in maintenance mode; rule set lives on
|
||||
# in Trivy. Replace with `aquasec/trivy:latest config /src` when
|
||||
# ready — same .tfsec/config.yml is recognized.
|
||||
#
|
||||
# Suppressions: tests/benchmarks/cloudopsbench/infra/.tfsec/config.yml. tfsec exits 1 on any
|
||||
# finding by default, which fails the step — exactly the hard-fail
|
||||
# behavior we want. Add a justified exclude to the config rather
|
||||
# than disabling the step.
|
||||
- name: tfsec scan (via Docker)
|
||||
run: |
|
||||
docker run --rm \
|
||||
-v "${{ github.workspace }}/tests/benchmarks/cloudopsbench/infra:/src" \
|
||||
aquasec/tfsec:latest \
|
||||
/src \
|
||||
--config-file=/src/.tfsec/config.yml \
|
||||
--no-color
|
||||
|
||||
checkov:
|
||||
name: checkov
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
# Suppressions live in tests/benchmarks/cloudopsbench/infra/.checkov.yaml. soft_fail is
|
||||
# FALSE — real findings block the PR. Add a justified skip-check
|
||||
# to the config rather than disabling here.
|
||||
- name: checkov scan
|
||||
uses: bridgecrewio/checkov-action@v12
|
||||
with:
|
||||
directory: tests/benchmarks/cloudopsbench/infra
|
||||
framework: terraform
|
||||
config_file: tests/benchmarks/cloudopsbench/infra/.checkov.yaml
|
||||
quiet: true
|
||||
soft_fail: false
|
||||
# Re-enable SARIF upload when CodeQL alerts are wanted:
|
||||
# output_format: cli,sarif
|
||||
# output_file_path: console,checkov-results.sarif
|
||||
@@ -0,0 +1,76 @@
|
||||
name: Tracer Demo – Prefect ECS
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
tracer-demo-prefect-ecs:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
JWT_TOKEN: ${{ secrets.JWT_TOKEN }}
|
||||
TRACER_ORG_ID: ${{ secrets.TRACER_ORG_ID }}
|
||||
TRACER_WEB_APP_URL: ${{ secrets.TRACER_WEB_APP_URL }}
|
||||
TRACER_API_URL: ${{ secrets.TRACER_API_URL }}
|
||||
TRACER_INGEST_TOKEN: ${{ secrets.TRACER_INGEST_TOKEN }}
|
||||
|
||||
AWS_REGION: us-east-1
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: uv sync --frozen --extra dev
|
||||
|
||||
- name: Configure AWS Credentials
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Install tracer CLI
|
||||
run: |
|
||||
curl -sSL https://install.tracer.cloud | CLI_BRANCH=dev sh -s user_36fbN6K6FwEgJFHsQv1pUByo8K6
|
||||
|
||||
- name: Initialize tracer
|
||||
run: |
|
||||
sudo tracer init --token ${{ secrets.JWT_TOKEN }}
|
||||
|
||||
- name: Export Tracer run id
|
||||
shell: bash
|
||||
run: |
|
||||
RUN_ID=$(tracer info --json | python3 -c "import sys,json; print(json.load(sys.stdin)['run']['id'])")
|
||||
PIPELINE_NAME=$(tracer info --json | python3 -c "import sys,json; print(json.load(sys.stdin)['pipeline']['name'])")
|
||||
ORG_SLUG=$(tracer info --json | python3 -c "import sys,json; print(json.load(sys.stdin)['pipeline']['organization'])")
|
||||
|
||||
echo "TRACER_RUN_ID=$RUN_ID" >> $GITHUB_ENV
|
||||
echo "TRACER_PIPELINE_NAME=$PIPELINE_NAME" >> $GITHUB_ENV
|
||||
echo "TRACER_ORG_SLUG=$ORG_SLUG" >> $GITHUB_ENV
|
||||
|
||||
# Optional but helpful: stable trace id for ingest rows
|
||||
echo "TRACER_TRACE_ID=trace_$RUN_ID" >> $GITHUB_ENV
|
||||
|
||||
echo "Exported TRACER_RUN_ID=$RUN_ID"
|
||||
|
||||
- name: Run Prefect ECS Demo
|
||||
continue-on-error: true
|
||||
run: |
|
||||
uv run python -m tests.e2e.upstream_prefect_ecs_fargate.test_agent_e2e
|
||||
@@ -0,0 +1,85 @@
|
||||
name: Trigger K8s Alert
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
verify:
|
||||
description: "Verify logs in Datadog + post to Slack"
|
||||
type: boolean
|
||||
default: true
|
||||
schedule:
|
||||
- cron: "0 9 * * 1" # every Monday 9am UTC
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
trigger-alert:
|
||||
if: github.repository == 'Tracer-Cloud/opensre'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
AWS_REGION: us-east-1
|
||||
DD_API_KEY: ${{ secrets.DD_API_KEY }}
|
||||
DD_APP_KEY: ${{ secrets.DD_APP_KEY }}
|
||||
DD_SITE: datadoghq.com
|
||||
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
SLACK_DEVS_ALERTS_CHANNEL_ID: C09S7GDG60J
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Configure AWS credentials
|
||||
uses: aws-actions/configure-aws-credentials@v4
|
||||
with:
|
||||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
aws-region: ${{ env.AWS_REGION }}
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v8.1.0
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: |
|
||||
pyproject.toml
|
||||
uv.lock
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install project
|
||||
run: uv sync --frozen
|
||||
|
||||
- name: Trigger pipeline failure via centralized config (~10s)
|
||||
id: trigger
|
||||
run: |
|
||||
echo "trigger_epoch=$(date +%s)" >> "$GITHUB_OUTPUT"
|
||||
set +e
|
||||
uv run python -m tests.e2e.kubernetes.trigger_alert | tee /tmp/trigger.log
|
||||
trigger_exit=${PIPESTATUS[0]}
|
||||
set -e
|
||||
echo "trigger_completed_epoch=$(date +%s)" >> "$GITHUB_OUTPUT"
|
||||
if grep -q '"status": "accepted_timeout"' /tmp/trigger.log; then
|
||||
echo "trigger_accepted_timeout=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "trigger_accepted_timeout=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
exit "$trigger_exit"
|
||||
|
||||
- name: Report trigger result
|
||||
run: |
|
||||
echo "Trigger command completed successfully"
|
||||
|
||||
- name: Verify Datadog + Slack (~5 min)
|
||||
if: ${{ inputs.verify == true || github.event_name == 'schedule' }}
|
||||
run: |
|
||||
POST_TRIGGER_WAIT=""
|
||||
if [ "${{ steps.trigger.outputs.trigger_accepted_timeout }}" = "true" ]; then
|
||||
POST_TRIGGER_WAIT="--post-trigger-wait 90"
|
||||
fi
|
||||
uv run python -m tests.e2e.kubernetes.trigger_alert \
|
||||
--verify-only \
|
||||
--since-epoch ${{ steps.trigger.outputs.trigger_completed_epoch }} \
|
||||
--dd-max-wait 300 \
|
||||
$POST_TRIGGER_WAIT
|
||||
Reference in New Issue
Block a user