chore: import upstream snapshot with attribution
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,274 @@
|
||||
# Cross-resource workspace (CLI)
|
||||
|
||||
Drive a multi-mount workspace (`/s3`, `/gdrive`, `/gmail`, `/slack`,
|
||||
`/discord`) end-to-end from the shell using `workspace.yaml`.
|
||||
|
||||
The two CLIs expose the same workspace HTTP API. Each
|
||||
example below shows the **Python** CLI (`mirage`, on `$PATH`) and
|
||||
the **TypeScript** CLI (`./mirage-ts`, a symlink to
|
||||
`typescript/packages/cli/dist/bin/mirage.js` from the repo root).
|
||||
Pick whichever CLI is convenient for the run; the command shapes match.
|
||||
|
||||
## Prereqs
|
||||
|
||||
- `.env.development` at the repo root with `AWS_`\*, `GOOGLE_*`,
|
||||
`SLACK_BOT_TOKEN`, `DISCORD_BOT_TOKEN`.
|
||||
- Python: `mirage` CLI on `$PATH` (e.g. `./python/.venv/bin/mirage`).
|
||||
- TypeScript: `pnpm --filter @struktoai/mirage-cli build` then
|
||||
`ln -sf typescript/packages/cli/dist/bin/mirage.js mirage-ts`
|
||||
at the repo root (already gitignored).
|
||||
|
||||
## 1. Source env and create the workspace
|
||||
|
||||
The YAML's `${...}` placeholders resolve from your shell at create
|
||||
time, so source first.
|
||||
|
||||
```bash
|
||||
set -a && source .env.development && set +a
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage workspace create examples/python/cross/workspace.yaml --id cross
|
||||
./mirage-ts workspace create examples/python/cross/workspace.yaml --id cross
|
||||
```
|
||||
|
||||
## 2. Inspect
|
||||
|
||||
```bash
|
||||
mirage workspace list
|
||||
./mirage-ts workspace list
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage workspace get cross
|
||||
./mirage-ts workspace get cross
|
||||
```
|
||||
|
||||
## 3. Run commands across mounts
|
||||
|
||||
`/gdrive/` is index-first — list it once before reading individual
|
||||
files, otherwise paths resolve to ENOENT.
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross --command "ls /s3/"
|
||||
./mirage-ts execute --workspace_id cross --command "ls /s3/"
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross --command "ls /gdrive/"
|
||||
./mirage-ts execute --workspace_id cross --command "ls /gdrive/"
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross --command "head -n 1 /s3/data/example.jsonl"
|
||||
./mirage-ts execute --workspace_id cross --command "head -n 1 /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross \
|
||||
--command 'cat /s3/data/example.jsonl "/gdrive/AWS CDK.gdoc.json" | wc -l'
|
||||
./mirage-ts execute --workspace_id cross \
|
||||
--command 'cat /s3/data/example.jsonl "/gdrive/AWS CDK.gdoc.json" | wc -l'
|
||||
```
|
||||
|
||||
## 4. Dry-run with `provision`
|
||||
|
||||
```bash
|
||||
mirage provision --workspace_id cross --command "cat /s3/data/example.jsonl | wc -l"
|
||||
./mirage-ts provision --workspace_id cross --command "cat /s3/data/example.jsonl | wc -l"
|
||||
```
|
||||
|
||||
After a real read the same path flips from a network read to a cache
|
||||
hit (`cache_hits=1`):
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross --command "cat /s3/data/example.jsonl > /dev/null"
|
||||
./mirage-ts execute --workspace_id cross --command "cat /s3/data/example.jsonl > /dev/null"
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage provision --workspace_id cross --command "cat /s3/data/example.jsonl"
|
||||
./mirage-ts provision --workspace_id cross --command "cat /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
## 5. Snapshot and restore
|
||||
|
||||
Snapshots redact cloud creds at snapshot time, so loading needs fresh
|
||||
creds via a config file. The same workspace YAML used for create works.
|
||||
|
||||
```bash
|
||||
mirage workspace snapshot cross /tmp/cross.tar
|
||||
./mirage-ts workspace snapshot cross /tmp/cross.tar
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage workspace load /tmp/cross.tar examples/python/cross/workspace.yaml \
|
||||
--id cross_loaded
|
||||
./mirage-ts workspace load /tmp/cross.tar examples/python/cross/workspace.yaml \
|
||||
--id cross_loaded
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage workspace get cross_loaded --verbose
|
||||
./mirage-ts workspace get cross_loaded --verbose
|
||||
```
|
||||
|
||||
## 6. Clean up
|
||||
|
||||
The daemon exits ~30s after the last workspace is deleted.
|
||||
|
||||
```bash
|
||||
mirage workspace delete cross
|
||||
./mirage-ts workspace delete cross
|
||||
```
|
||||
|
||||
```bash
|
||||
mirage workspace delete cross_loaded
|
||||
./mirage-ts workspace delete cross_loaded
|
||||
```
|
||||
|
||||
## 7. Per-mount safeguards (Python CLI)
|
||||
|
||||
A mount can cap what a command streams back with `command_safeguards`,
|
||||
so a runaway `cat`/`grep`/`rg` can't flood the agent or hang forever.
|
||||
Each entry sets `max_lines` / `max_bytes` (output cap) and/or
|
||||
`timeout_seconds` (deadline), with `on_exceed: truncate` (stop, exit 0,
|
||||
add a notice) or `on_exceed: error` (stop, exit 1, add a notice).
|
||||
|
||||
This section is **Python-only**: the TS config schema does not yet carry
|
||||
`command_safeguards`. It runs against its own workspace
|
||||
(`cross_sg`) from [workspace_safeguards.yaml](workspace_safeguards.yaml),
|
||||
so the steps above are untouched. That file guards `/s3` with:
|
||||
`head` → 10 lines / truncate, `grep` → 20 lines / error, `rg` → a 1 ms
|
||||
timeout.
|
||||
|
||||
```bash
|
||||
set -a && source .env.development && set +a
|
||||
mirage workspace create examples/python/cross/workspace_safeguards.yaml --id cross_sg
|
||||
```
|
||||
|
||||
Warm the object once (the first S3 read fetches the whole object and can
|
||||
take a few seconds; later reads are cache hits):
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross_sg --command "head -n 1 /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
`max_lines` + `on_exceed: truncate` — asking for 50 lines yields 10 plus a
|
||||
notice on stderr, exit `0`:
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross_sg --command "head -n 50 /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
`max_lines` + `on_exceed: error` — `grep` matches thousands of lines, trips
|
||||
the 20-line cap, and fails with exit `1`:
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross_sg --command "grep mirage /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
`timeout_seconds` — the 1 ms deadline trips on any real read, exit `124`
|
||||
with `rg: timed out after 0.001s`:
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross_sg --command "rg mirage /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
Commands below their cap are untouched, so the earlier shapes still work
|
||||
unchanged (1 line, no notice):
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross_sg --command "head -n 1 /s3/data/example.jsonl"
|
||||
```
|
||||
|
||||
Clean up:
|
||||
|
||||
```bash
|
||||
mirage workspace delete cross_sg
|
||||
```
|
||||
|
||||
## 8. Versioning (Python CLI)
|
||||
|
||||
The daemon keeps a git-backed history per workspace under
|
||||
`~/.mirage/repos/<id>` (set `MIRAGE_VERSION_ROOT` to relocate). You commit
|
||||
the live state as a version, then `log` / `diff` / `branch` / `checkout`,
|
||||
following git. See the [CLI docs](https://docs.mirage.strukto.ai) for the
|
||||
full reference.
|
||||
|
||||
This section is **Python-only** and runs against its own scratch workspace
|
||||
(`cross_ver`) from
|
||||
[workspace_versioning.yaml](workspace_versioning.yaml), so the steps above
|
||||
are untouched. It uses a writable RAM mount: versioning tracks the workspace
|
||||
tree regardless of backend, and this keeps the demo fast and writes nothing
|
||||
to your real cloud backends.
|
||||
|
||||
> The history outlives the workspace: `workspace delete` does **not** remove
|
||||
> `~/.mirage/repos/<id>`. Re-creating the same id resumes the old history, so
|
||||
> this walkthrough starts from a clean id (or `rm -rf ~/.mirage/repos/cross_ver`
|
||||
> first) to keep output predictable.
|
||||
|
||||
```bash
|
||||
mirage workspace create examples/python/cross/workspace_versioning.yaml --id cross_ver
|
||||
```
|
||||
|
||||
Commit two versions, then `log` (newest first):
|
||||
|
||||
```bash
|
||||
mirage execute --workspace_id cross_ver --command "echo v1 > /notes.txt"
|
||||
mirage workspace commit cross_ver -m "first"
|
||||
|
||||
mirage execute --workspace_id cross_ver --command "echo v2 > /notes.txt"
|
||||
mirage workspace commit cross_ver -m "second"
|
||||
|
||||
mirage workspace log cross_ver
|
||||
```
|
||||
|
||||
`diff` reports changed paths (added / modified / deleted). With no refs it
|
||||
compares the live state to the branch HEAD; with two version ids it compares
|
||||
those versions (substitute the ids `log` printed):
|
||||
|
||||
```bash
|
||||
mirage workspace diff cross_ver # live vs HEAD (clean: all empty)
|
||||
mirage workspace diff cross_ver <v1> <v2> # => modified: ["notes.txt"]
|
||||
```
|
||||
|
||||
`branch` forks at a branch's current version; commit onto it with `-b`. `main`
|
||||
is left untouched:
|
||||
|
||||
```bash
|
||||
mirage workspace branch cross_ver exp
|
||||
mirage execute --workspace_id cross_ver --command "echo on-exp > /notes.txt"
|
||||
mirage workspace commit cross_ver -b exp -m "on exp"
|
||||
mirage workspace log cross_ver -b exp # on exp, second, first
|
||||
mirage workspace log cross_ver # main unchanged: second, first
|
||||
```
|
||||
|
||||
`checkout` restores the live state in place to a version or branch
|
||||
(overwriting uncommitted changes):
|
||||
|
||||
```bash
|
||||
mirage workspace checkout cross_ver <v1>
|
||||
mirage execute --workspace_id cross_ver --command "cat /notes.txt" # => v1
|
||||
```
|
||||
|
||||
`clone --at` builds a new workspace from one of the source's past versions
|
||||
(omit `--at` to clone the live state):
|
||||
|
||||
```bash
|
||||
mirage workspace clone cross_ver --at <v1> --id cross_ver_at
|
||||
mirage execute --workspace_id cross_ver_at --command "cat /notes.txt" # => v1
|
||||
```
|
||||
|
||||
Clean up (and drop the histories so a rerun starts fresh):
|
||||
|
||||
```bash
|
||||
mirage workspace delete cross_ver
|
||||
mirage workspace delete cross_ver_at
|
||||
rm -rf ~/.mirage/repos/cross_ver ~/.mirage/repos/cross_ver_at
|
||||
```
|
||||
|
||||
## SDK alternative
|
||||
|
||||
The same flow driven from Python (with snapshot fingerprinting) lives
|
||||
in [example.py](example.py) + [load_check.py](load_check.py).
|
||||
Reference in New Issue
Block a user