wehub-resource-sync
426e9eeabd
Benchmark Bridge Tests / benchmark (bunx @biomejs/biome check packages/lifeops-bench/src, benchmark-lint) (push) Waiting to run
Benchmark Bridge Tests / benchmark (bunx vitest run --config packages/lifeops-bench/vitest.config.ts --root packages/lifeops-bench --passWithNoTests, benchmark-tests) (push) Waiting to run
Build Agent Image / build-and-push (push) Waiting to run
Chat shell gestures / Chat shell gesture + parity e2e (push) Waiting to run
ci / test (push) Waiting to run
ci / lint-and-format (push) Waiting to run
ci / build (push) Waiting to run
ci / dev-startup (push) Waiting to run
Cloud Gateway Discord / Test (push) Waiting to run
Cloud Gateway Webhook / Test (push) Waiting to run
Cloud Tests / lint-and-types (push) Waiting to run
Cloud Tests / unit-tests (push) Waiting to run
Cloud Tests / integration-tests (push) Waiting to run
Cloud Tests / e2e-tests (push) Blocked by required conditions
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
Deploy Apps Worker (Product 2) / Determine environment (push) Waiting to run
Deploy Apps Worker (Product 2) / Deploy apps worker to apps-control host (${{ needs.determine-env.outputs.environment }}) (push) Blocked by required conditions
Deploy Eliza Provisioning Worker / Determine environment (push) Waiting to run
Deploy Eliza Provisioning Worker / Deploy worker to Hetzner host (${{ needs.determine-env.outputs.environment }} @ ${{ needs.determine-env.outputs.deployment_sha }}) (push) Blocked by required conditions
Dev Smoke / Classify changed paths (push) Waiting to run
Dev Smoke / bun run dev onboarding chat (push) Blocked by required conditions
Dev Smoke / Vite HMR dependency-level smoke (push) Blocked by required conditions
Electrobun Submodule Guard / electrobun gitlink is fetchable (push) Waiting to run
gitleaks / gitleaks (push) Waiting to run
Markdown Links / Relative Markdown Links (push) Waiting to run
Publish @elizaos/example-code / check_npm (push) Waiting to run
Publish @elizaos/example-code / publish_npm (push) Blocked by required conditions
Publish @elizaos/plugin-elizacloud / verify_version (push) Waiting to run
Publish @elizaos/plugin-elizacloud / publish_npm (push) Blocked by required conditions
Quality (Extended) / Homepage Build (PR smoke) (push) Waiting to run
Quality (Extended) / Comment-only diff guard (push) Waiting to run
Quality (Extended) / Format + Type Safety Ratchet (push) Waiting to run
Quality (Extended) / Develop Gate (secret scan + UI determinism) (push) Waiting to run
Quality (Extended) / Develop Gate (lint) (push) Waiting to run
Sandbox Live Smoke / Sandbox live smoke (push) Waiting to run
Snap Build & Test / Build Snap (amd64) (push) Waiting to run
Snap Build & Test / Build Snap (arm64) (push) Waiting to run
supply-chain / sbom (push) Waiting to run
supply-chain / vulnerability-scan (push) Waiting to run
Build, Push & Deploy to Phala Cloud / build-and-push (push) Waiting to run
Test Packaging / Validate Packaging Configs (push) Waiting to run
Test Packaging / PyPI on Python ${{ matrix.python }} (push) Waiting to run
Test Packaging / Pack & Test JS Tarballs (push) Waiting to run
Test Packaging / elizaos CLI global-install smoke (node + bun) (push) Waiting to run
UI Fixture E2E / ui-fixture-e2e (push) Waiting to run
UI Fixture E2E / fixture-e2e (push) Waiting to run
UI Story Gate / story-gate (push) Waiting to run
vault-ci / test (macos-latest) (push) Waiting to run
vault-ci / test (ubuntu-latest) (push) Waiting to run
vault-ci / test (windows-latest) (push) Waiting to run
vault-ci / app-core wiring tests (push) Waiting to run
verify-patches / verify patches/CHECKSUMS.sha256 (push) Waiting to run
Voice Benchmark Smoke / voice-emotion fixture smoke (push) Waiting to run
Voice Benchmark Smoke / voiceagentbench fixture smoke (push) Waiting to run
Voice Benchmark Smoke / voicebench-quality unit smoke (push) Waiting to run
Voice Benchmark Smoke / voicebench TypeScript unit (no audio) (push) Waiting to run
Voice Benchmark Smoke / voice bench smoke summary (push) Blocked by required conditions
Windows CI / windows ([bun run --cwd packages/app-core test bun run --cwd packages/elizaos test bun run --cwd packages/cloud/shared test], app-and-cli) (push) Waiting to run
Windows CI / windows ([bun run --cwd packages/scenario-runner test bun run --cwd packages/vault test bun run --cwd packages/security test bun run --cwd plugins/plugin-coding-tools test], framework-packages) (push) Waiting to run
Windows CI / windows ([bun run --cwd plugins/plugin-elizacloud test bun run --cwd plugins/plugin-discord test bun run --cwd plugins/plugin-anthropic test bun run --cwd plugins/plugin-openai test bun run --cwd plugins/plugin-app-control test bun run --cwd plugins/pl… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4 node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh node packages/scripts/run… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4 bun run --cwd packages/core test bun run --cwd packages/shared test], core-runtime, 75) (push) Waiting to run
Test Packaging / Build & Test PyPI Package (push) Waiting to run
Voice Workbench / headless workbench (mocked backends) (push) Has been cancelled
Voice Workbench / real acoustic lane (nightly, provisioned only) (push) Has been cancelled
74 lines
1.8 KiB
Python
74 lines
1.8 KiB
Python
"""
|
|
REALM-Bench: Real-World Planning Benchmark for LLMs and Multi-Agent Systems.
|
|
|
|
Faithful implementation of the 11 canonical scenarios (P1..P11) from:
|
|
|
|
Geng et al., arXiv:2502.18836
|
|
https://github.com/genglongling/REALM-Bench
|
|
|
|
Upstream task definitions, instance datasets, and JSSP benchmarks are
|
|
vendored under ``packages/benchmarks/realm/upstream/`` (see
|
|
``upstream/ATTRIBUTION.md``).
|
|
|
|
The agent loop targets the eliza TS bridge via
|
|
``eliza_adapter.realm.ElizaREALMAgent``. A deterministic
|
|
``_MockREALMAgent`` is exposed for smoke tests and CI.
|
|
"""
|
|
|
|
from benchmarks.realm.dataset import REALMDataset
|
|
from benchmarks.realm.evaluator import MetricsCalculator, REALMEvaluator
|
|
from benchmarks.realm.runner import REALMRunner
|
|
from benchmarks.realm.types import (
|
|
LEADERBOARD_NOTE,
|
|
LEADERBOARD_SCORES,
|
|
MULTI_AGENT_PROBLEMS,
|
|
PROBLEM_DESCRIPTIONS,
|
|
PROBLEM_TO_FAMILY,
|
|
PROBLEMS_WITH_DISRUPTIONS,
|
|
ExecutionModel,
|
|
OracleFamily,
|
|
PlanningAction,
|
|
PlanningStep,
|
|
PlanningTrajectory,
|
|
PlanStatus,
|
|
REALMCategory, # back-compat alias for RealmProblem
|
|
REALMConfig,
|
|
REALMMetrics,
|
|
REALMReport,
|
|
REALMResult,
|
|
REALMResultDetails,
|
|
REALMResultMetrics,
|
|
REALMTask,
|
|
REALMTestCase,
|
|
RealmProblem,
|
|
)
|
|
|
|
__all__ = [
|
|
"RealmProblem",
|
|
"REALMCategory",
|
|
"OracleFamily",
|
|
"PROBLEM_TO_FAMILY",
|
|
"PROBLEMS_WITH_DISRUPTIONS",
|
|
"MULTI_AGENT_PROBLEMS",
|
|
"PROBLEM_DESCRIPTIONS",
|
|
"REALMConfig",
|
|
"REALMMetrics",
|
|
"REALMReport",
|
|
"REALMResult",
|
|
"REALMResultMetrics",
|
|
"REALMResultDetails",
|
|
"REALMTask",
|
|
"REALMTestCase",
|
|
"PlanningAction",
|
|
"PlanningStep",
|
|
"PlanningTrajectory",
|
|
"PlanStatus",
|
|
"ExecutionModel",
|
|
"LEADERBOARD_SCORES",
|
|
"LEADERBOARD_NOTE",
|
|
"REALMDataset",
|
|
"REALMEvaluator",
|
|
"MetricsCalculator",
|
|
"REALMRunner",
|
|
]
|