hkuds--cli-anything/macrocli/agent-harness/MACROCLI.md

MacroCLI — Agent Harness SOP

What Is This?

MacroCLI is a layered CLI that turns valuable GUI workflows into parameterized, agent-callable macros. The agent sends one command:

cli-anything-macrocli macro run export_png --param output=/tmp/out.png --json

The system handles everything else: parameter validation, precondition checks, backend selection, step execution, postcondition verification, and structured result output. The agent never touches the GUI directly.

Architecture

Agent
  └─▶  cli-anything-macrocli macro run <name> --param k=v --json   (L6: CLI)
             │
        MacroRuntime                                                  (L5)
             │  1. Validate params against MacroDefinition schema
             │  2. Check preconditions (file_exists, process_running, …)
             │  3. For each step:
             │       RoutingEngine → select backend by priority       (L3)
             │       Backend.execute(step, resolved_params)           (L2)
             │  4. Check postconditions
             │  5. Collect declared outputs
             │  6. Record telemetry in ExecutionSession
             └─▶  { success, output, error, telemetry }

Layer Mapping

Layer	Name	Implementation
L7	Agent Task Interface	Caller (any AI agent)
L6	Unified CLI Entry	macrocli_cli.py — Click CLI
L5	Macro Execution Runtime	core/runtime.py
L4	Parameterized Macro Model	core/macro_model.py + macro_definitions/*.yaml
L3	Backend Routing Engine	core/routing.py
L2	Execution Backends	backends/ (7 backends)
L1	Target Application	Any GUI-first or closed-source app

Execution Backends

Backend	Priority	Trigger	Use case
native_api	100	backend: native_api	subprocess / shell commands
gui_macro	80	backend: gui_macro	precompiled coordinate replay (pyautogui)
visual_anchor	75	backend: visual_anchor	template-matching click/type (requires [visual])
file_transform	70	backend: file_transform	XML, JSON, text file editing
gui_agent	60	backend: gui_agent	vision-model-driven automation (requires [gui_agent])
semantic_ui	50	backend: semantic_ui	accessibility API + keyboard (xdotool)
recovery	10	backend: recovery	retry + fallback orchestration

The RoutingEngine respects the step's explicit backend: field; if that backend is unavailable it walks down the priority list.

Macro Definition Format

Macros live in cli_anything/macrocli/macro_definitions/ as YAML files:

name: export_png
version: "1.0"
description: Export the active diagram to PNG.

parameters:
  output:
    type: string
    required: true
    example: /tmp/diagram.png

preconditions:
  - process_running: draw.io
  - file_exists: /path/to/input.drawio

steps:
  - id: export
    backend: native_api
    action: run_command
    params:
      command: [draw.io, --export, --output, "${output}", input.drawio]
    timeout_ms: 30000
    on_failure: fail     # or: skip | continue

postconditions:
  - file_exists: ${output}
  - file_size_gt:
      - ${output}
      - 100

outputs:
  - name: exported_file
    path: ${output}

agent_hints:
  danger_level: safe
  side_effects: [creates_file]
  reversible: true

Supported Condition Types

Type	Args	Checks
file_exists	path	os.path.exists(path)
file_size_gt	[path, min_bytes]	os.stat(path).st_size > min_bytes
process_running	name	pgrep -x name or psutil
env_var	name	name in os.environ
always	true/false	constant pass/fail

Package Layout

macrocli/
└── agent-harness/
    ├── setup.py                           entry_point: cli-anything-macrocli
    └── cli_anything/macrocli/
        ├── macrocli_cli.py                Main Click CLI
        ├── macro_definitions/             YAML macro registry
        │   ├── manifest.yaml
        │   └── examples/
        │       ├── export_file.yaml
        │       ├── transform_json.yaml
        │       └── undo_last.yaml
        ├── core/
        │   ├── macro_model.py             MacroDefinition + YAML loader
        │   ├── registry.py               MacroRegistry
        │   ├── routing.py                RoutingEngine
        │   ├── runtime.py                MacroRuntime (full lifecycle)
        │   └── session.py               ExecutionSession + telemetry
        ├── backends/
        │   ├── base.py                   Backend ABC + StepResult
        │   ├── native_api.py             subprocess backend
        │   ├── file_transform.py         XML/JSON/text backend
        │   ├── semantic_ui.py            accessibility backend
        │   ├── visual_anchor.py          template-matching backend
        │   ├── gui_agent.py              vision-model automation backend
        │   ├── gui_macro.py              compiled replay backend
        │   └── recovery.py               retry/fallback backend
        ├── skills/SKILL.md               Agent-readable skill definition
        ├── utils/repl_skin.py            Unified REPL skin (cli-anything standard)
        └── tests/
            ├── test_core.py              Unit tests (49 tests, no external deps)
            └── test_full_e2e.py          E2E + CLI subprocess tests (15 tests)

Installation

cd macrocli/agent-harness
pip install -e .

Runtime dependencies: Python 3.10+, PyYAML, click, prompt-toolkit.

Optional extras:

pip install -e ".[visual]"      # visual_anchor backend (mss, Pillow, numpy, pynput)
pip install -e ".[gui_agent]"   # gui_agent backend     (openai, mss, Pillow)
pip install -e ".[all]"         # everything

gui_agent backend configuration:

The gui_agent backend uses the OpenAI SDK and is compatible with any OpenAI-compatible API. Configure via environment variables:

Variable	Description
MACROCLI_MODEL	Model name (required, e.g. gpt-4o)
MACROCLI_API_KEY	API key for the provider
MACROCLI_BASE_URL	Base URL (only needed for non-OpenAI hosts)

Other optional dependencies:

• xdotool — semantic_ui backend on Linux
• pyautogui — gui_macro backend
• psutil — richer process_running checks

Running Tests

cd macrocli/agent-harness
python3 -m pytest cli_anything/macrocli/tests/ -v -s
# 64 passed

Key Design Decisions

Why YAML macros, not Python? YAML macros are readable by agents without running code, inspectable via macro info, and editable without touching the harness source.

Why 7 backends? Real GUI applications expose many different control surfaces. The routing engine picks the most reliable one available — the agent doesn't need to know which one ran. The visual_anchor backend uses template matching for robust UI element detection, while gui_agent uses vision models for dynamic decision-making when the UI state is unpredictable.

Why preconditions and postconditions? Agents operate in environments where state is uncertain. Failing loudly before execution (preconditions) and verifying after (postconditions) catches problems the agent can act on.

Why on_failure: skip | continue? Some macro steps are best-effort (e.g., confirming a dialog that may or may not appear). Skipping lets the macro continue to the real work.