chore: import upstream snapshot with attribution
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: Alibaba OSS
|
||||
icon: /images/aliyun-logo.svg
|
||||
description: Mount an Alibaba Cloud OSS bucket via its S3-compatible API as a virtual filesystem.
|
||||
---
|
||||
The Aliyun resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps an `AliyunConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just derives the right OSS endpoint from
|
||||
`region`. Uses aioboto3 against Alibaba OSS's S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `s3.oss-<region>.aliyuncs.com`
|
||||
(e.g. `s3.oss-cn-hangzhou.aliyuncs.com`). Note the `s3.` prefix: this is the
|
||||
S3-compatible host, distinct from the native `oss-<region>.aliyuncs.com`. Pass
|
||||
`endpoint_url` to override.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.aliyun import AliyunConfig, AliyunResource
|
||||
|
||||
config = AliyunConfig(
|
||||
bucket=os.environ["OSS_BUCKET"],
|
||||
region=os.environ.get("OSS_REGION", "cn-hangzhou"),
|
||||
access_key_id=os.environ["OSS_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["OSS_ACCESS_KEY_SECRET"],
|
||||
# Optional:
|
||||
# endpoint_url="https://s3.oss-cn-hangzhou.aliyuncs.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = AliyunResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.aliyun import AliyunConfig, AliyunResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = AliyunConfig(
|
||||
bucket=os.environ["OSS_BUCKET"],
|
||||
region=os.environ.get("OSS_REGION", "cn-hangzhou"),
|
||||
access_key_id=os.environ["OSS_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["OSS_ACCESS_KEY_SECRET"],
|
||||
)
|
||||
resource = AliyunResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Aliyun reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [Alibaba OSS Setup](/home/setup/aliyun).
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: Backblaze B2
|
||||
icon: /images/backblaze-logo.svg
|
||||
description: Mount a Backblaze B2 bucket via its S3-compatible API as a virtual filesystem.
|
||||
---
|
||||
The Backblaze resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `BackblazeConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just derives the right B2 endpoint from
|
||||
`region`. Uses aioboto3 against Backblaze B2's S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `s3.<region>.backblazeb2.com`. Pass
|
||||
`endpoint_url` to override.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.backblaze import BackblazeConfig, BackblazeResource
|
||||
|
||||
config = BackblazeConfig(
|
||||
bucket=os.environ["B2_BUCKET"],
|
||||
region=os.environ["B2_REGION"], # e.g. us-west-004
|
||||
access_key_id=os.environ["B2_ACCESS_KEY_ID"], # B2 application keyID
|
||||
secret_access_key=os.environ["B2_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://s3.us-west-004.backblazeb2.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = BackblazeResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.backblaze import BackblazeConfig, BackblazeResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = BackblazeConfig(
|
||||
bucket=os.environ["B2_BUCKET"],
|
||||
region=os.environ["B2_REGION"],
|
||||
access_key_id=os.environ["B2_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["B2_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
resource = BackblazeResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /data/ -name '*.csv'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Backblaze reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [Backblaze B2 Setup](/home/setup/backblaze).
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: Ceph (Rados Gateway)
|
||||
icon: /images/ceph-logo.svg
|
||||
description: Mount a self-hosted Ceph Rados Gateway bucket as a virtual filesystem.
|
||||
---
|
||||
The Ceph resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `CephConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just points at your Ceph Rados Gateway
|
||||
`endpoint_url` and uses path-style addressing by default.
|
||||
|
||||
Ceph RGW is self-hosted, so `endpoint_url` is **required** (there is no
|
||||
region-derived host). Uses aioboto3 against the gateway's S3-compatible API.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.ceph import CephConfig, CephResource
|
||||
|
||||
config = CephConfig(
|
||||
bucket=os.environ["CEPH_BUCKET"],
|
||||
endpoint_url=os.environ["CEPH_ENDPOINT_URL"],
|
||||
access_key_id=os.environ["CEPH_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["CEPH_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# region="us-east-1", # default
|
||||
# path_style=True, # default
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = CephResource(config)
|
||||
ws = Workspace({"/ceph": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.ceph import CephConfig, CephResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = CephConfig(
|
||||
bucket=os.environ["CEPH_BUCKET"],
|
||||
endpoint_url=os.environ["CEPH_ENDPOINT_URL"],
|
||||
access_key_id=os.environ["CEPH_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["CEPH_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
resource = CephResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/ceph/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /ceph/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /ceph/ -name '*.json'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Ceph reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [Ceph Setup](/home/setup/ceph).
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
title: Chroma
|
||||
icon: database
|
||||
description: Mount a ChromaDB collection as a read-only virtual filesystem.
|
||||
---
|
||||
|
||||
The Chroma resource exposes an existing ChromaDB collection as text files
|
||||
mounted at a prefix such as `/knowledge/`. It is useful when you already have a
|
||||
chunked knowledge base in Chroma and want agents to use normal filesystem
|
||||
commands like `ls`, `cat`, `grep`, `find`, and `chroma-query`.
|
||||
|
||||
For collection setup, see [Chroma Setup](/python/setup/chroma).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.chroma import ChromaConfig, ChromaResource
|
||||
|
||||
config = ChromaConfig(
|
||||
host=os.environ.get("CHROMA_HOST", "localhost"),
|
||||
port=int(os.environ.get("CHROMA_PORT", "8000")),
|
||||
ssl=os.environ.get("CHROMA_SSL", "false").lower() == "true",
|
||||
collection_name=os.environ["CHROMA_COLLECTION"],
|
||||
slug_field=os.environ.get("CHROMA_SLUG_FIELD", "page_slug"),
|
||||
chunk_index_field=os.environ.get("CHROMA_CHUNK_INDEX_FIELD", "chunk_index"),
|
||||
)
|
||||
resource = ChromaResource(config=config)
|
||||
ws = Workspace({"/knowledge/": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
The resource is read-only and does not support snapshots.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
Chroma paths come from the path tree document stored in the collection with ID
|
||||
`__path_tree__`. The document body must be a JSON object whose keys are virtual
|
||||
file paths below the mount prefix.
|
||||
|
||||
```json
|
||||
{
|
||||
"README.md": {
|
||||
"size": 1024
|
||||
},
|
||||
"guides/quickstart.md": {
|
||||
"size": 4096,
|
||||
"updated_at": "2026-01-03T00:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Mounted at `/knowledge/`, this becomes:
|
||||
|
||||
```text
|
||||
/knowledge/
|
||||
README.md
|
||||
guides/
|
||||
quickstart.md
|
||||
```
|
||||
|
||||
Mirage infers folders from path segments and stores the tree in the index cache.
|
||||
The `size`, `created_at`, and `updated_at` metadata values are used by tree and
|
||||
listing operations when present.
|
||||
|
||||
## Reading Documents
|
||||
|
||||
Each file is assembled from Chroma chunk documents whose metadata slug matches
|
||||
the path. Chunks are sorted by the configured chunk index field and joined with
|
||||
a single newline.
|
||||
|
||||
```bash
|
||||
ls /knowledge/
|
||||
tree /knowledge/
|
||||
find /knowledge/ -type f
|
||||
cat /knowledge/guides/quickstart.md
|
||||
head -n 20 /knowledge/guides/quickstart.md
|
||||
tail -n 20 /knowledge/guides/quickstart.md
|
||||
grep -in "billing" /knowledge/
|
||||
```
|
||||
|
||||
## Exact Search with Grep
|
||||
|
||||
`grep` uses Chroma document filtering as a coarse prefilter when possible, then
|
||||
applies Mirage's grep matching over assembled file text. Scoped paths restrict
|
||||
the candidate files before querying Chroma.
|
||||
|
||||
```bash
|
||||
grep "refund" /knowledge/policies/
|
||||
grep -i "quickstart" /knowledge/
|
||||
grep -n "API key" /knowledge/guides/quickstart.md
|
||||
```
|
||||
|
||||
## Vector Search
|
||||
|
||||
The `chroma-query` command calls Chroma's native vector query API. Use it when
|
||||
semantic similarity matters more than exact text matching.
|
||||
|
||||
```bash
|
||||
chroma-query "how do I get started" /knowledge/
|
||||
chroma-query --top-k 5 "billing policy" /knowledge/policies/
|
||||
chroma-query --top-k 3 "API authentication" /knowledge/guides/quickstart.md
|
||||
```
|
||||
|
||||
Results are emitted one hit per line:
|
||||
|
||||
```text
|
||||
/knowledge/guides/quickstart.md 0.82 Use an API key to authenticate requests.
|
||||
```
|
||||
|
||||
Columns are path, score, and chunk text. The score is derived from Chroma's
|
||||
returned distance as `1 - distance`.
|
||||
|
||||
## Cache
|
||||
|
||||
The resource uses Mirage's index cache for the virtual tree. The first
|
||||
directory listing or path resolution fetches `__path_tree__`; later `ls`,
|
||||
`find`, `tree`, and path resolution reuse the cached tree until the index cache
|
||||
expires or the workspace is recreated.
|
||||
|
||||
File content is still read from Chroma when commands materialize document text.
|
||||
|
||||
## Examples
|
||||
|
||||
Runnable examples live under `examples/python/chroma/`:
|
||||
|
||||
- [`chroma.py`](https://github.com/StruktoAI/mirage/blob/main/examples/python/chroma/chroma.py) — command workflow with `ls`, `tree`, `find`, `cat`, `head`, `tail`, `grep`, and `chroma-query`.
|
||||
- [`chroma_vfs.py`](https://github.com/StruktoAI/mirage/blob/main/examples/python/chroma/chroma_vfs.py) — in-process VFS workflow with `os.listdir()`, `open()`, and `os.path.*`.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ----- |
|
||||
| `ls` | List folders and files from the path tree |
|
||||
| `tree` | Print the mounted path tree |
|
||||
| `find` | Search the virtual tree by name, type, depth, and size |
|
||||
| `cat` | Read full file text assembled from Chroma chunks |
|
||||
| `head` / `tail` | Read the first or last lines/bytes |
|
||||
| `grep` | Exact or regex matching over assembled file text |
|
||||
| `chroma-query` | Vector retrieval through Chroma |
|
||||
@@ -0,0 +1,211 @@
|
||||
---
|
||||
title: Databricks Volume
|
||||
icon: folder-open
|
||||
description: Mount a Databricks Unity Catalog volume as a filesystem.
|
||||
---
|
||||
|
||||
`DatabricksVolumeResource` exposes files from a Unity Catalog volume through
|
||||
Mirage's standard filesystem interface. Agents can list, stat, read, stream,
|
||||
glob, and write files under the configured volume root.
|
||||
|
||||
For auth and environment setup, see [Databricks Volume Setup](/python/setup/databricks).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.databricks_volume import (
|
||||
DatabricksVolumeConfig,
|
||||
DatabricksVolumeResource,
|
||||
)
|
||||
|
||||
resource = DatabricksVolumeResource(DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
root_path="/reports",
|
||||
))
|
||||
ws = Workspace({"/dbx": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
| Field | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `catalog` | required | Unity Catalog catalog name. |
|
||||
| `schema` | required | Unity Catalog schema name. |
|
||||
| `volume` | required | Unity Catalog volume name. |
|
||||
| `root_path` | `/` | Subdirectory inside the volume to expose. |
|
||||
| `host` | `None` | Optional workspace host override. |
|
||||
| `token` | `None` | Optional PAT override. Redacted in snapshots. |
|
||||
| `profile` | `None` | Optional Databricks SDK profile name. |
|
||||
| `timeout` | `30` | Request timeout in seconds. |
|
||||
|
||||
## Mount mode
|
||||
|
||||
`read` or `write`.
|
||||
|
||||
## Filesystem layout
|
||||
|
||||
Mirage maps the configured mount prefix onto the configured volume subtree.
|
||||
|
||||
Given:
|
||||
|
||||
```python
|
||||
DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
root_path="/reports/2026",
|
||||
)
|
||||
```
|
||||
|
||||
and mount prefix `/dbx/`, the volume path:
|
||||
|
||||
```text
|
||||
/Volumes/main/default/agent_files/reports/2026/q1/summary.md
|
||||
```
|
||||
|
||||
appears in Mirage as:
|
||||
|
||||
```text
|
||||
/dbx/q1/summary.md
|
||||
```
|
||||
|
||||
`root_path` is normalized before use, and Mirage rejects any virtual path that
|
||||
would escape above that configured subtree.
|
||||
|
||||
## Supported operations
|
||||
|
||||
Reads: `readdir`, `stat`, `exists`, `read_bytes`, `read_stream`,
|
||||
`range_read`, and glob resolution. Writes: `write`, `create`, `mkdir`,
|
||||
`rmdir`, `unlink`, recursive `rm`, `cp`, and `mv`.
|
||||
|
||||
`mv`/`cp` are non-atomic download + upload — the Files API has no server-side
|
||||
rename.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The Databricks Volume resource supports shell commands that operate on real
|
||||
file content. Reads use the Files API with range requests, so commands like
|
||||
`head -c BYTES` avoid downloading the whole object. The supported set is
|
||||
scoped to commands that work over the volume API (no compression, encoding,
|
||||
or local-only utilities).
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | -------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `diff` | Compare files line by line |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ---------------------------------------------------- |
|
||||
| `cp` | Copy files (non-atomic download + upload) |
|
||||
| `mv` | Move/rename files (non-atomic download + upload) |
|
||||
| `rm` | Remove files (recursive for directories) |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ----------------------- |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
## Snapshot behavior
|
||||
|
||||
`token` is redacted in resource state. Loading a snapshot back requires an
|
||||
override config that provides fresh credentials if the runtime auth chain does
|
||||
not already supply them.
|
||||
|
||||
## Databricks Apps
|
||||
|
||||
For Databricks Apps, prefer SDK-default auth and keep Mirage in-process:
|
||||
|
||||
```python
|
||||
config = DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
)
|
||||
resource = DatabricksVolumeResource(config)
|
||||
ws = Workspace({"/dbx/": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
This does not require FUSE. The agent can access the mounted workspace through
|
||||
Mirage's backend adapters or `Workspace.execute(...)`.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.databricks_volume import (
|
||||
DatabricksVolumeConfig,
|
||||
DatabricksVolumeResource,
|
||||
)
|
||||
|
||||
resource = DatabricksVolumeResource(DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
root_path="/reports",
|
||||
))
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/dbx/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /dbx/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /dbx/ -name '*.md'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute('head -n 20 "/dbx/q1/summary.md"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute('stat "/dbx/q1/summary.md"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See:
|
||||
|
||||
- `examples/python/databricks_volume/databricks_volume.py`
|
||||
- `examples/python/agents/langchain/databricks_volume_deepagent.py`
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Agents in Databricks Apps**: mount a Unity Catalog volume in-process so an
|
||||
agent can read and write governed files without FUSE or hardcoded credentials.
|
||||
- **Reading governed datasets**: expose a reports or dataset subtree through
|
||||
`root_path` and query it with shell commands.
|
||||
- **Sandboxed volume access**: scope an agent to a single volume (and optional
|
||||
`root_path`) so it cannot read or write outside that subtree.
|
||||
- **Writing agent outputs back**: persist generated files to the volume with
|
||||
`write`, `cp`, and `mv`.
|
||||
@@ -0,0 +1,233 @@
|
||||
---
|
||||
title: Dify
|
||||
icon: /images/dify-logo.svg
|
||||
description: Mount a Dify Knowledge dataset as a read-only virtual filesystem.
|
||||
---
|
||||
|
||||
The Dify resource exposes completed Dify Knowledge documents as text files
|
||||
mounted at a prefix such as `/knowledge/`. Each file is assembled from the
|
||||
document's completed, enabled segments.
|
||||
|
||||
For API key setup, see [Dify Setup](/python/setup/dify).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.dify import DifyConfig, DifyResource
|
||||
|
||||
config = DifyConfig(
|
||||
api_key=os.environ["DIFY_API_KEY"],
|
||||
base_url=os.environ.get("DIFY_BASE_URL", "https://api.dify.ai/v1"),
|
||||
dataset_id=os.environ["DIFY_DATASET_ID"],
|
||||
slug_metadata_name=os.environ.get("DIFY_SLUG_METADATA_NAME", "slug"),
|
||||
)
|
||||
resource = DifyResource(config=config)
|
||||
ws = Workspace({"/knowledge/": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
The resource is read-only and does not support snapshots.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
Dify documents are mapped to paths using the configured slug metadata field.
|
||||
The default metadata field name is `slug`. If a document has no configured slug
|
||||
metadata, Mirage falls back to the Dify document name.
|
||||
|
||||
```text
|
||||
/knowledge/
|
||||
README.md
|
||||
guides/
|
||||
quickstart.md
|
||||
api.md
|
||||
policies/
|
||||
support.md
|
||||
```
|
||||
|
||||
Example mapping:
|
||||
|
||||
| Dify document | Metadata | Mirage path |
|
||||
| ------------- | -------- | ----------- |
|
||||
| `Quickstart` | `slug=guides/quickstart.md` | `/knowledge/guides/quickstart.md` |
|
||||
| `README.md` | no slug | `/knowledge/README.md` |
|
||||
|
||||
### Creating Slug Metadata in Dify
|
||||
|
||||
In the Dify Knowledge UI, open a document and add a metadata item whose name
|
||||
matches `slug_metadata_name`. The default name is `slug`.
|
||||
|
||||
```text
|
||||
name: slug
|
||||
value: guides/quickstart.md
|
||||
```
|
||||
|
||||
For a custom Dify metadata name:
|
||||
|
||||
```python
|
||||
config = DifyConfig(
|
||||
api_key=os.environ["DIFY_API_KEY"],
|
||||
base_url=os.environ.get("DIFY_BASE_URL", "https://api.dify.ai/v1"),
|
||||
dataset_id=os.environ["DIFY_DATASET_ID"],
|
||||
slug_metadata_name="path",
|
||||
)
|
||||
```
|
||||
|
||||
Then use:
|
||||
|
||||
```text
|
||||
name: path
|
||||
value: guides/quickstart.md
|
||||
```
|
||||
|
||||
Mirage reads `doc_metadata` from Dify and uses this value as the document's
|
||||
virtual path below the mount prefix.
|
||||
|
||||
Use these rules for slug metadata values:
|
||||
|
||||
- Use `/` to create folders.
|
||||
- Do not use empty segments, `.`, or `..`.
|
||||
- Keep each slug metadata value unique across the dataset.
|
||||
- Do not use a path that is also needed as a folder. For example, `guides` and `guides/quickstart.md` cannot both be document paths.
|
||||
|
||||
Only visible documents are included:
|
||||
|
||||
- `enabled` is `true`
|
||||
- `indexing_status` is `completed`
|
||||
- `archived` is `false`
|
||||
|
||||
## Reading Documents
|
||||
|
||||
`cat`, `head`, `tail`, and `grep` read Dify document segments. Segment content
|
||||
is joined with a single newline between chunks.
|
||||
|
||||
```bash
|
||||
ls /knowledge/
|
||||
find /knowledge/ -type f
|
||||
cat /knowledge/guides/quickstart.md
|
||||
head -n 20 /knowledge/guides/quickstart.md
|
||||
grep -i "billing" /knowledge/guides/quickstart.md
|
||||
wc /knowledge/guides/quickstart.md
|
||||
```
|
||||
|
||||
## Search
|
||||
|
||||
The `search` command calls Dify's dataset retrieval API. Use it when meaning
|
||||
matters more than exact text matching.
|
||||
|
||||
```bash
|
||||
search "how do I reset my password" /knowledge/
|
||||
search --method hybrid --top-k 5 "billing policy" /knowledge/policies/
|
||||
search --method semantic --threshold 0.4 "quickstart" /knowledge/guides/*.md
|
||||
```
|
||||
|
||||
Supported methods:
|
||||
|
||||
| Method | Dify retrieval mode |
|
||||
| ------ | ------------------- |
|
||||
| `semantic` | semantic search |
|
||||
| `fulltext` | full text search |
|
||||
| `hybrid` | hybrid search |
|
||||
| `keyword` | keyword search |
|
||||
|
||||
`top-k` is capped at `100`. `threshold` must be between `0` and `1`.
|
||||
|
||||
Results are emitted one retrieval hit per block:
|
||||
|
||||
```text
|
||||
/knowledge/policies/refunds:0.82
|
||||
Refunds are allowed within 30 days.
|
||||
```
|
||||
|
||||
Mirage uses the configured slug metadata field to derive the path, and falls
|
||||
back to the Dify document name when that metadata is missing. Multiple hits
|
||||
from the same document remain separate records.
|
||||
|
||||
Scoped search uses Dify metadata filtering. Documents with the configured slug
|
||||
metadata field are filtered by that field; name-based documents are filtered by
|
||||
`document_name`, which requires Dify Built-in Fields to be enabled in dataset
|
||||
metadata.
|
||||
|
||||
### Scoped Search Requirements
|
||||
|
||||
Mirage converts a scoped search path into Dify metadata filters:
|
||||
|
||||
| Mirage target | Dify metadata filter |
|
||||
| ------------- | -------------------- |
|
||||
| Document with configured slug metadata | `<slug_metadata_name> in [...]` |
|
||||
| Document without configured slug metadata | `document_name in [...]` |
|
||||
| Folder or glob | `<slug_metadata_name>` / `document_name` filters for all matched files |
|
||||
|
||||
To make scoped search reliable:
|
||||
|
||||
1. Add the configured slug metadata field to every document.
|
||||
1. Enable Dify **Built-in Fields** in the dataset metadata settings.
|
||||
1. Ensure `document_name` is available if you rely on name-based paths.
|
||||
|
||||
If Built-in Fields are disabled, scoped search against documents without the
|
||||
configured slug metadata field may return empty results even though `cat`,
|
||||
`grep`, and `find` can still see the same file.
|
||||
|
||||
## Cache
|
||||
|
||||
The resource uses Mirage's index cache for the virtual tree. Directory listings
|
||||
and path resolution reuse the cached document tree until the index expires or
|
||||
the workspace is recreated. If documents are edited directly in Dify, repeated
|
||||
operations can temporarily see cached paths and metadata.
|
||||
|
||||
Document content is still read from Dify when commands materialize file data.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.dify import DifyConfig, DifyResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = DifyConfig(
|
||||
api_key=os.environ["DIFY_API_KEY"],
|
||||
base_url=os.environ.get("DIFY_BASE_URL", "https://api.dify.ai/v1"),
|
||||
dataset_id=os.environ["DIFY_DATASET_ID"],
|
||||
slug_metadata_name=os.environ.get("DIFY_SLUG_METADATA_NAME", "slug"),
|
||||
)
|
||||
resource = DifyResource(config=config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/knowledge/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /knowledge/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /knowledge/ -type f | head -n 10")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute('search --method hybrid --top-k 5 "getting started" /knowledge/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
A runnable version is available at
|
||||
`examples/python/dify/dify.py`.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ----- |
|
||||
| `ls` | List folders and documents |
|
||||
| `cat` | Read full document text from completed segments |
|
||||
| `head` / `tail` | Read the first or last lines/bytes |
|
||||
| `grep` | Exact or regex matching over streamed document text |
|
||||
| `find` | Search the virtual tree by name, type, depth, and size |
|
||||
| `wc` | Count lines, words, bytes, characters, and max line length |
|
||||
| `search` | Semantic/full-text/hybrid/keyword retrieval through Dify |
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: DigitalOcean Spaces
|
||||
icon: /images/digitalocean-logo.svg
|
||||
description: Mount a DigitalOcean Spaces bucket via its S3-compatible API as a virtual filesystem.
|
||||
---
|
||||
The DigitalOcean resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `DigitalOceanConfig` to an `S3Config` and reuses the exact same
|
||||
backend, commands, and behavior as S3, it just derives the right Spaces
|
||||
endpoint from `region`. Uses aioboto3 against DigitalOcean Spaces'
|
||||
S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `<region>.digitaloceanspaces.com`
|
||||
(e.g. `nyc3.digitaloceanspaces.com`). Pass `endpoint_url` to override.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.digitalocean import DigitalOceanConfig, DigitalOceanResource
|
||||
|
||||
config = DigitalOceanConfig(
|
||||
bucket=os.environ["DO_SPACE"],
|
||||
region=os.environ.get("DO_REGION", "nyc3"),
|
||||
access_key_id=os.environ["DO_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["DO_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://nyc3.digitaloceanspaces.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = DigitalOceanResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.digitalocean import DigitalOceanConfig, DigitalOceanResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = DigitalOceanConfig(
|
||||
bucket=os.environ["DO_SPACE"],
|
||||
region=os.environ.get("DO_REGION", "nyc3"),
|
||||
access_key_id=os.environ["DO_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["DO_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
resource = DigitalOceanResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- DigitalOcean reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [DigitalOcean Setup](/home/setup/digitalocean).
|
||||
@@ -0,0 +1,348 @@
|
||||
---
|
||||
title: Discord
|
||||
description: Mount Discord guilds, channels, messages, attachments, and members as a Mirage virtual filesystem for Python agents.
|
||||
icon: discord
|
||||
---
|
||||
The Discord resource exposes guild, channel, and member data as a virtual
|
||||
filesystem mounted at some prefix such as `/discord/`.
|
||||
|
||||
For token setup, see [Discord Setup](/python/setup/discord).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.discord import DiscordConfig, DiscordResource
|
||||
|
||||
config = DiscordConfig(token=os.environ["DISCORD_BOT_TOKEN"])
|
||||
resource = DiscordResource(config=config)
|
||||
ws = Workspace({"/discord": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/discord/
|
||||
<guild-name>__<guild-id>/
|
||||
channels/
|
||||
<channel-name>__<channel-id>/
|
||||
<yyyy-mm-dd>/
|
||||
chat.jsonl
|
||||
files/
|
||||
<stem>__<attachment-id>.<ext>
|
||||
...
|
||||
...
|
||||
members/
|
||||
<username>__<user-id>.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/discord/
|
||||
My Server__111222333444555666/
|
||||
channels/
|
||||
general__777888999000111222/
|
||||
2026-04-04/
|
||||
chat.jsonl
|
||||
files/
|
||||
screenshot__1488111222333444555.png
|
||||
2026-04-05/
|
||||
chat.jsonl
|
||||
random__777888999000111223/
|
||||
2026-04-11/
|
||||
chat.jsonl
|
||||
members/
|
||||
alice__444555666777888999.json
|
||||
bob__444555666777888900.json
|
||||
```
|
||||
|
||||
Display names keep their original spelling from Discord (spaces,
|
||||
apostrophes, emoji are all preserved). Only `/` is replaced with `∕`
|
||||
(U+2215) so it cannot collide with a directory boundary. The Discord
|
||||
snowflake ID is appended after `__` (double underscore) on guild,
|
||||
channel, member, and attachment names so resource specific commands can
|
||||
extract it without an extra lookup, and so two same named entities never
|
||||
collide. Quote names containing spaces in shell commands. `stat` also
|
||||
exposes the ID in the `extra` dict (see [Finding IDs](#finding-ids)).
|
||||
|
||||
### Guilds
|
||||
|
||||
The root lists one directory per guild the bot has access to.
|
||||
|
||||
### Channels
|
||||
|
||||
`/discord/<guild>/channels/` lists text channels (types 0, 5, 15).
|
||||
Each channel directory contains day-partitioned **directories** for the
|
||||
30 days leading up to the channel's last message. Each day directory
|
||||
holds:
|
||||
|
||||
- `chat.jsonl`, the day's messages (one JSON object per line).
|
||||
- `files/`, attachments posted on that day. Each blob is named
|
||||
`<stem>__<attachment-id>.<ext>`, where the stem keeps the original
|
||||
filename's spelling (only `/` is replaced). The ID suffix keeps the
|
||||
filename collision-free. `cat`'ing a blob downloads it from the
|
||||
Discord CDN.
|
||||
|
||||
The date range is derived from `last_message_id` on the channel object,
|
||||
so inactive channels show dates around their last activity, not today.
|
||||
|
||||
Soft errors (403 missing permissions, 404 unknown channel, 429 rate
|
||||
limit) on a single day are swallowed so listings, `find`, and `grep`
|
||||
keep working across the rest of the tree.
|
||||
|
||||
### Members
|
||||
|
||||
`/discord/<guild>/members/` lists one `.json` file per member.
|
||||
Reading a member file returns the full member payload from the
|
||||
Discord API.
|
||||
|
||||
## Smart Commands
|
||||
|
||||
### grep / rg at different scopes
|
||||
|
||||
When `grep` or `rg` target a channel or guild directory (not a specific file),
|
||||
they use the Discord search API instead of downloading every `.jsonl` file:
|
||||
|
||||
```bash
|
||||
# FILE level - downloads the .jsonl, greps locally
|
||||
grep hello "/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl"
|
||||
|
||||
# CHANNEL level - uses Discord search API (GET /guilds/{id}/messages/search)
|
||||
grep hello "/discord/My Server__111222333444555666/channels/general__777888999000111222/"
|
||||
|
||||
# GUILD level - searches across all channels
|
||||
grep hello "/discord/My Server__111222333444555666/"
|
||||
```
|
||||
|
||||
Scope detection is handled by `mirage/core/discord/scope.py`.
|
||||
|
||||
### head / tail
|
||||
|
||||
`head` and `tail` on file-level paths use the Discord messages API directly
|
||||
(`GET /channels/{id}/messages`) instead of downloading the full day's history.
|
||||
|
||||
## Cache
|
||||
|
||||
The Discord resource uses `IndexCacheStore` (same as RAM/S3/disk/GitHub).
|
||||
Index entries store guild IDs, channel IDs, and `last_message_id` for
|
||||
date range computation. There is no separate content cache - file content
|
||||
caching is handled by the workspace `IOResult` mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.discord import DiscordConfig, DiscordResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = DiscordConfig(token=os.environ["DISCORD_BOT_TOKEN"])
|
||||
resource = DiscordResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/discord": resource}, mode=MountMode.READ)
|
||||
|
||||
# List guilds
|
||||
r = await ws.execute("ls /discord/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
guild = r.stdout_str().strip().split("\n")[0].strip()
|
||||
|
||||
# List channels
|
||||
r = await ws.execute(f'ls "/discord/{guild}/channels/"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
ch = r.stdout_str().strip().splitlines()[0].strip()
|
||||
base = f"/discord/{guild}/channels/{ch}"
|
||||
|
||||
# Read messages from a specific date
|
||||
r = await ws.execute(f'cat "{base}/2026-04-04/chat.jsonl" | head -n 3')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract usernames with jq
|
||||
r = await ws.execute(
|
||||
f'jq -r ".[] | .author.username" "{base}/2026-04-04/chat.jsonl"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Count messages per user
|
||||
r = await ws.execute(
|
||||
f'cat "{base}/2026-04-04/chat.jsonl"'
|
||||
' | jq -r ".[] | .author.username" | sort | uniq -c')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List attachments for a day and download one
|
||||
r = await ws.execute(f'ls "{base}/2026-04-04/files/"')
|
||||
print(await r.stdout_str())
|
||||
r = await ws.execute(
|
||||
f'cat "{base}/2026-04-04/files/screenshot__1488111222333444555.png"')
|
||||
blob = await r.materialize_stdout()
|
||||
print(f"downloaded {len(blob)} bytes")
|
||||
|
||||
# Search across channel (uses Discord search API)
|
||||
r = await ws.execute(f'grep hello "{base}/"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across guild
|
||||
r = await ws.execute(f'grep hello "/discord/{guild}/"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Navigate with cd/pwd
|
||||
await ws.execute(f'cd "{base}"')
|
||||
r = await ws.execute("pwd")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Relative paths after cd
|
||||
r = await ws.execute("ls | tail -n 5")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/python/discord/discord.py` for the full working example.
|
||||
|
||||
## Finding IDs
|
||||
|
||||
Resource-specific commands require Discord snowflake IDs
|
||||
(`channel_id`, `guild_id`, `message_id`). These can be extracted
|
||||
from the filesystem:
|
||||
|
||||
```bash
|
||||
# Guild ID - use stat
|
||||
stat "/discord/My Server__111222333444555666"
|
||||
# → extra={"guild_id": "1256522563555819574"}
|
||||
|
||||
# Channel ID - use stat
|
||||
stat "/discord/My Server__111222333444555666/channels/general__777888999000111222"
|
||||
# → extra={"channel_id": "1256522563555819574"}
|
||||
|
||||
# Message ID - inside JSONL messages
|
||||
jq -r '.[] | "\(.id) [\(.author.username)] \(.content)"' \
|
||||
"/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl"
|
||||
# → 1489887688978075769 [alice] hello world
|
||||
|
||||
# Find a message then reply
|
||||
jq -r '.[] | select(.content | test("hello")) | .id' \
|
||||
"/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl"
|
||||
# → 1489887688978075769
|
||||
discord-send-message --channel_id 1256522563555819574 \
|
||||
--text "Reply" --message_id 1489887688978075769
|
||||
```
|
||||
|
||||
## Working with Large Channels
|
||||
|
||||
Tips for efficient access on busy channels:
|
||||
|
||||
```bash
|
||||
# Check message count per day
|
||||
wc -l "/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl"
|
||||
|
||||
# Read only recent messages
|
||||
tail -n 10 "/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl"
|
||||
|
||||
# Search uses Discord API at channel/guild level (no file download)
|
||||
grep "keyword" "/discord/My Server__111222333444555666/channels/general__777888999000111222/"
|
||||
|
||||
# Extract specific fields
|
||||
jq -r '.[] | "\(.author.username): \(.content)"' \
|
||||
"/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl" | head -n 20
|
||||
|
||||
# Count messages per user
|
||||
cat "/discord/My Server__111222333444555666/channels/general__777888999000111222/2026-04-04/chat.jsonl" \
|
||||
| jq -r '.[] | .author.username' | sort | uniq -c
|
||||
```
|
||||
|
||||
Note: `grep`/`rg` at channel or guild level uses the Discord search
|
||||
API instead of downloading every `.jsonl` file, making it efficient
|
||||
even for large channels.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Discord tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | -------------------------------------------------------------- |
|
||||
| `ls` | List guilds, channels, members, dates, attachments |
|
||||
| `cat` | Read `chat.jsonl`, member `.json`, or download an attachment |
|
||||
| `head` / `tail` | Smart: uses messages API for file scope |
|
||||
| `grep` / `rg` | Smart: uses search API for channel/guild scope (with fallback) |
|
||||
| `jq` | Query JSON; use `.[]` prefix for JSONL files |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, ID via `extra`) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `discord-send-message`
|
||||
|
||||
Post a message to a channel, optionally as a reply.
|
||||
|
||||
```bash
|
||||
discord-send-message --channel_id 1256522563555819574 --text "Hello from MIRAGE"
|
||||
discord-send-message --channel_id 1256522563555819574 --text "Reply" --message_id 1489887688978075769
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ---------------------------- |
|
||||
| `--channel_id` | yes | Discord channel snowflake ID |
|
||||
| `--text` | yes | Message text to send |
|
||||
| `--message_id` | no | Message ID to reply to |
|
||||
|
||||
The channel ID can be found in directory names under
|
||||
`/discord/<guild>/channels/` or via `stat`. Returns the
|
||||
posted message JSON.
|
||||
|
||||
### `discord-add-reaction`
|
||||
|
||||
Add an emoji reaction to a message.
|
||||
|
||||
```bash
|
||||
discord-add-reaction --channel_id 1256522563555819574 --message_id 1489887688978075769 --reaction 👍
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ---------------------------- |
|
||||
| `--channel_id` | yes | Discord channel snowflake ID |
|
||||
| `--message_id` | yes | Message snowflake ID |
|
||||
| `--reaction` | yes | Emoji (unicode or name) |
|
||||
|
||||
### `discord-list-members`
|
||||
|
||||
Search guild members by name.
|
||||
|
||||
```bash
|
||||
discord-list-members --guild_id 1256522563555819574 --query "alice"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------ | -------- | ----------------------- |
|
||||
| `--guild_id` | yes | Discord guild snowflake |
|
||||
| `--query` | yes | Username search query |
|
||||
|
||||
Returns matching members as JSON array.
|
||||
|
||||
### `discord-get-server-info`
|
||||
|
||||
Get full guild metadata from the Discord API.
|
||||
|
||||
```bash
|
||||
discord-get-server-info --guild_id 1256522563555819574
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------ | -------- | ----------------------- |
|
||||
| `--guild_id` | yes | Discord guild snowflake |
|
||||
|
||||
Returns the guild object JSON (name, icon, member count, etc.).
|
||||
@@ -0,0 +1,218 @@
|
||||
---
|
||||
title: Disk
|
||||
description: Mount a local directory as a Mirage resource with read/write shell commands and path traversal protection.
|
||||
icon: hard-drive
|
||||
---
|
||||
The Disk resource mounts a local directory at some prefix such as `/data/`.
|
||||
All operations are backed by real files on disk. Path resolution validates
|
||||
against the root boundary to prevent directory traversal escapes.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.disk import DiskResource
|
||||
|
||||
resource = DiskResource(root="/path/to/dir")
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`DiskResource(root=...)` takes a single `root` path argument pointing to the
|
||||
directory to mount. Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The Disk resource mirrors the structure of the `root` directory. For example,
|
||||
if `root="/srv/files"` contains:
|
||||
|
||||
```text
|
||||
/srv/files/
|
||||
notes.txt
|
||||
config.json
|
||||
reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
```
|
||||
|
||||
Then mounting at `/data/` exposes:
|
||||
|
||||
```text
|
||||
/data/
|
||||
notes.txt
|
||||
config.json
|
||||
reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
```
|
||||
|
||||
Paths like `../../etc/passwd` are rejected - resolution is always confined
|
||||
to the root boundary.
|
||||
|
||||
## Cache
|
||||
|
||||
The Disk resource uses `IndexCacheStore` with `index_ttl = 60` (1 minute).
|
||||
Directory listings are cached for up to 60 seconds before being refreshed
|
||||
from disk.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.disk import DiskResource
|
||||
|
||||
DATA_DIR = Path("/path/to/files")
|
||||
|
||||
tmp = tempfile.mkdtemp()
|
||||
shutil.copytree(DATA_DIR, Path(tmp) / "files", dirs_exist_ok=True)
|
||||
|
||||
resource = DiskResource(root=tmp + "/files")
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /data/example.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /data/ -name '*.json'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("grep example /data/example.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /data/example.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The Disk resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.):
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Local directory access**: Mount local directories for AI agents to read and process
|
||||
- **Sandboxed file access**: Restrict agent file operations to a specific directory tree
|
||||
- **FUSE mounting**: Expose disk files through a virtual FUSE mount for external tools
|
||||
- **Data pipelines**: Process local datasets with shell-like commands
|
||||
- **Development**: Test file operations against real data before deploying to cloud resources
|
||||
@@ -0,0 +1,322 @@
|
||||
---
|
||||
title: Email
|
||||
description: Mount an IMAP mailbox as a Mirage filesystem and send messages through SMTP from Python workspaces.
|
||||
icon: at
|
||||
---
|
||||
The Email resource exposes any IMAP mailbox as a virtual filesystem
|
||||
mounted at some prefix such as `/email/`. Sending is handled via SMTP.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.email import EmailConfig, EmailResource
|
||||
|
||||
config = EmailConfig(
|
||||
imap_host=os.environ["IMAP_HOST"],
|
||||
smtp_host=os.environ["SMTP_HOST"],
|
||||
username=os.environ["EMAIL_USERNAME"],
|
||||
password=os.environ["EMAIL_PASSWORD"],
|
||||
)
|
||||
resource = EmailResource(config=config)
|
||||
ws = Workspace({"/email": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
### Config Reference
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
| ----------- | ------ | ------- | ------------------------ |
|
||||
| `imap_host` | `str` | | IMAP server hostname |
|
||||
| `imap_port` | `int` | `993` | IMAP port |
|
||||
| `smtp_host` | `str` | | SMTP server hostname |
|
||||
| `smtp_port` | `int` | `587` | SMTP port |
|
||||
| `username` | `str` | | Email address / login |
|
||||
| `password` | `str` | | Password or app password |
|
||||
| `use_ssl` | `bool` | `True` | Use SSL for IMAP |
|
||||
|
||||
### Common Resource Settings
|
||||
|
||||
| Resource | IMAP Host | SMTP Host | Notes |
|
||||
| ----------- | ----------------------- | --------------------- | -------------------------- |
|
||||
| Outlook/365 | `outlook.office365.com` | `smtp.office365.com` | App password or OAuth2 |
|
||||
| Yahoo | `imap.mail.yahoo.com` | `smtp.mail.yahoo.com` | App password required |
|
||||
| Fastmail | `imap.fastmail.com` | `smtp.fastmail.com` | App password |
|
||||
| iCloud | `imap.mail.me.com` | `smtp.mail.me.com` | App password |
|
||||
| ProtonMail | `127.0.0.1` (Bridge) | `127.0.0.1` (Bridge) | Requires ProtonMail Bridge |
|
||||
| Self-hosted | Your server hostname | Your server hostname | Whatever you configured |
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/email/
|
||||
<folder>/
|
||||
<yyyy-mm-dd>/
|
||||
<sanitized-subject>__<uid>.email.json
|
||||
<sanitized-subject>__<uid>/ # only if message has attachments
|
||||
<attachment-filename>
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/email/
|
||||
INBOX/
|
||||
2026-04-14/
|
||||
Meeting_Notes__12345.email.json
|
||||
Meeting_Notes__12345/
|
||||
report.pdf
|
||||
screenshot.png
|
||||
Simple_Email__12346.email.json
|
||||
2026-04-13/
|
||||
Hello__12347.email.json
|
||||
Sent/
|
||||
2026-04-14/
|
||||
Reply__12348.email.json
|
||||
Drafts/
|
||||
Archive/
|
||||
```
|
||||
|
||||
Folder directories appear at the root, using IMAP folder names
|
||||
(e.g., `INBOX`, `Sent`, `Drafts`).
|
||||
|
||||
### Date Directories
|
||||
|
||||
Inside each folder, messages are grouped into date subdirectories
|
||||
formatted as `YYYY-MM-DD`. The date is derived from the message's
|
||||
`Date` header.
|
||||
|
||||
### Message Files
|
||||
|
||||
Each message is stored as a `.email.json` file. The filename shape is:
|
||||
|
||||
```text
|
||||
<sanitized-subject>__<uid>.email.json
|
||||
```
|
||||
|
||||
### Attachments
|
||||
|
||||
Messages that have attachments get a companion subdirectory with
|
||||
the same base name (without `.email.json`):
|
||||
|
||||
```text
|
||||
Meeting_Notes__12345.email.json # message JSON
|
||||
Meeting_Notes__12345/ # attachment directory
|
||||
report.pdf
|
||||
screenshot.png
|
||||
```
|
||||
|
||||
## Cache
|
||||
|
||||
The Email resource uses `IndexCacheStore` (same as Gmail, Slack,
|
||||
Discord, and other resources). Index entries store folder names,
|
||||
message UIDs, and message metadata.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.email import EmailConfig, EmailResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = EmailConfig(
|
||||
imap_host=os.environ["IMAP_HOST"],
|
||||
smtp_host=os.environ["SMTP_HOST"],
|
||||
username=os.environ["EMAIL_USERNAME"],
|
||||
password=os.environ["EMAIL_PASSWORD"],
|
||||
)
|
||||
resource = EmailResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/email": resource}, mode=MountMode.READ)
|
||||
|
||||
# List folders
|
||||
r = await ws.execute("ls /email/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List date directories in INBOX
|
||||
r = await ws.execute("ls /email/INBOX/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List messages for a specific date
|
||||
r = await ws.execute("ls /email/INBOX/2026-04-14/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a message
|
||||
r = await ws.execute(
|
||||
"cat /email/INBOX/2026-04-14/Meeting_Notes__12345.email.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract subject with jq
|
||||
r = await ws.execute(
|
||||
'jq ".subject"'
|
||||
" /email/INBOX/2026-04-14/Meeting_Notes__12345.email.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List attachments
|
||||
r = await ws.execute("ls /email/INBOX/2026-04-14/Meeting_Notes__12345/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all messages
|
||||
r = await ws.execute('rg "quarterly" /email/INBOX/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 2 /email/INBOX/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Triage unread messages
|
||||
r = await ws.execute("email-triage --folder INBOX --unseen --max 5")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Send an email
|
||||
r = await ws.execute(
|
||||
'email-send --to "user@example.com"'
|
||||
' --subject "Hello from MIRAGE"'
|
||||
' --body "This email was sent via the MIRAGE email resource."')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Finding UIDs
|
||||
|
||||
Resource-specific commands require message UIDs. These can be
|
||||
extracted from the filesystem:
|
||||
|
||||
```bash
|
||||
# UID is embedded in filename after "__"
|
||||
ls /email/INBOX/2026-04-14/
|
||||
# -> Meeting_Notes__12345.email.json <- uid = 12345
|
||||
|
||||
# Read a message then reply
|
||||
email-read --uid 12345 --folder INBOX
|
||||
email-reply --uid 12345 --folder INBOX --body "Thanks for the notes"
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted email tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List folders, dates, messages, attachments |
|
||||
| `cat` | Read message JSON or attachment content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query message JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` | Extract filename from path |
|
||||
| `dirname` | Extract directory from path |
|
||||
| `realpath` | Resolve path to absolute form |
|
||||
| `nl` | Number lines of output |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `email-send`
|
||||
|
||||
Send a new email.
|
||||
|
||||
```bash
|
||||
email-send --to "user@example.com" --subject "Hello" --body "Hi there"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ----------- | -------- | ----------------------- |
|
||||
| `--to` | yes | Recipient email address |
|
||||
| `--subject` | yes | Email subject line |
|
||||
| `--body` | yes | Email body text |
|
||||
|
||||
Returns the sent message status JSON.
|
||||
|
||||
### `email-reply`
|
||||
|
||||
Reply to a message.
|
||||
|
||||
```bash
|
||||
email-reply --uid 12345 --folder INBOX --body "Thanks for the update"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ---------------------------- |
|
||||
| `--uid` | yes | Message UID |
|
||||
| `--folder` | no | IMAP folder (default: INBOX) |
|
||||
| `--body` | yes | Reply body text |
|
||||
|
||||
Returns the sent reply JSON.
|
||||
|
||||
### `email-reply-all`
|
||||
|
||||
Reply-all to a message.
|
||||
|
||||
```bash
|
||||
email-reply-all --uid 12345 --folder INBOX --body "Acknowledged by the team"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ---------------------------- |
|
||||
| `--uid` | yes | Message UID |
|
||||
| `--folder` | no | IMAP folder (default: INBOX) |
|
||||
| `--body` | yes | Reply body text |
|
||||
|
||||
Returns the sent reply JSON.
|
||||
|
||||
### `email-forward`
|
||||
|
||||
Forward a message to another recipient.
|
||||
|
||||
```bash
|
||||
email-forward --uid 12345 --folder INBOX --to "colleague@example.com"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ---------------------------- |
|
||||
| `--uid` | yes | Message UID |
|
||||
| `--folder` | no | IMAP folder (default: INBOX) |
|
||||
| `--to` | yes | Recipient email address |
|
||||
|
||||
Returns the forwarded message JSON.
|
||||
|
||||
### `email-triage`
|
||||
|
||||
Search and triage emails.
|
||||
|
||||
```bash
|
||||
email-triage --folder INBOX --unseen --max 10
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ---------------------------- |
|
||||
| `--folder` | no | IMAP folder (default: INBOX) |
|
||||
| `--max` | no | Max results (default: 20) |
|
||||
| `--unseen` | no | Only show unread messages |
|
||||
|
||||
Returns matching messages as JSON.
|
||||
|
||||
### `email-read`
|
||||
|
||||
Read a message by its UID.
|
||||
|
||||
```bash
|
||||
email-read --uid 12345 --folder INBOX
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ---------------------------- |
|
||||
| `--uid` | yes | Message UID |
|
||||
| `--folder` | no | IMAP folder (default: INBOX) |
|
||||
|
||||
Returns the full message JSON.
|
||||
@@ -0,0 +1,232 @@
|
||||
---
|
||||
title: GCS
|
||||
description: Mount Google Cloud Storage buckets through Mirage's async S3-compatible resource layer for Python agents.
|
||||
icon: google
|
||||
---
|
||||
The GCS resource mounts a Google Cloud Storage bucket at some prefix
|
||||
such as `/gcs/`. All operations involve network I/O to the remote object
|
||||
store. Uses aioboto3 against GCS's S3-compatible XML API via HMAC keys,
|
||||
inheriting all S3 resource capabilities.
|
||||
|
||||
For credential setup, see [GCS Setup](/home/setup/gcs).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gcs import GCSConfig, GCSResource
|
||||
|
||||
config = GCSConfig(
|
||||
bucket=os.environ["GCS_BUCKET"],
|
||||
access_key_id=os.environ["GCS_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["GCS_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://storage.googleapis.com",
|
||||
# region="auto",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = GCSResource(config)
|
||||
ws = Workspace({"/gcs": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`GCSResource(config)` takes a `GCSConfig` object with the bucket name
|
||||
and HMAC credentials. Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The GCS resource maps object keys to virtual paths under the mount
|
||||
prefix, identical to the S3 resource. GCS "directories" are
|
||||
prefix-based — there are no real directory objects.
|
||||
|
||||
For example, if bucket `mirage-ai` contains:
|
||||
|
||||
```text
|
||||
data/example.json
|
||||
data/example.parquet
|
||||
data/example.jsonl
|
||||
```
|
||||
|
||||
Then mounting at `/gcs/` exposes:
|
||||
|
||||
```text
|
||||
/gcs/
|
||||
data/
|
||||
example.json
|
||||
example.parquet
|
||||
example.jsonl
|
||||
```
|
||||
|
||||
Path mapping: virtual `/gcs/data/example.json` maps to GCS key
|
||||
`data/example.json`.
|
||||
|
||||
## Cache
|
||||
|
||||
The GCS resource uses `IndexCacheStore` with `index_ttl = 600`
|
||||
(10 minutes), same as S3. Directory listings are cached for up to 600
|
||||
seconds before being refreshed from GCS. This reduces API calls for
|
||||
repeated directory traversals.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gcs import GCSConfig, GCSResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = GCSConfig(
|
||||
bucket=os.environ["GCS_BUCKET"],
|
||||
access_key_id=os.environ["GCS_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["GCS_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
|
||||
resource = GCSResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/gcs/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /gcs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /gcs/data/example.json | head -n 10")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /gcs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /gcs/ -name '*.parquet'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /gcs/data/example.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The GCS resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing GCS data**: Mount GCS buckets for agents to read and process datasets
|
||||
- **Data pipelines**: Read and write GCS objects with shell-like commands
|
||||
- **FUSE mounting**: Expose GCS buckets through a virtual FUSE mount for external tools
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
title: Google Docs
|
||||
description: Mount Google Docs as JSON-backed files so Python agents can read document structure and content through Mirage.
|
||||
icon: file-lines
|
||||
---
|
||||
The Google Docs resource exposes Docs documents as a virtual filesystem
|
||||
mounted at some prefix such as `/gdocs/`.
|
||||
|
||||
For Google OAuth setup, see [Google Workspace Setup](/python/setup/google).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gdocs import GDocsConfig, GDocsResource
|
||||
|
||||
config = GDocsConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GDocsResource(config=config)
|
||||
ws = Workspace({"/gdocs": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/gdocs/
|
||||
owned/
|
||||
<YYYY-MM-DD>_<sanitized-title>__<doc-id>.gdoc.json
|
||||
...
|
||||
shared/
|
||||
<YYYY-MM-DD>_<sanitized-title>__<doc-id>.gdoc.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/gdocs/
|
||||
owned/
|
||||
2026-04-04_Project_Plan__1AbCdEf.gdoc.json
|
||||
2026-04-10_Meeting_Notes__2BcDeFg.gdoc.json
|
||||
shared/
|
||||
2026-04-03_Design_Notes__9XyZ.gdoc.json
|
||||
```
|
||||
|
||||
Documents are split into `owned` (documents you created) and `shared`
|
||||
(documents shared with you). The filename shape is:
|
||||
|
||||
```text
|
||||
<YYYY-MM-DD>_<sanitized-title>__<document-id>.gdoc.json
|
||||
```
|
||||
|
||||
If the modified date is unavailable, the date prefix is omitted.
|
||||
Reading a document file returns the full Google Docs API JSON for
|
||||
that document.
|
||||
|
||||
## Cache
|
||||
|
||||
The Google Docs resource uses `IndexCacheStore`. Index entries store
|
||||
document IDs and metadata. There is no separate content cache -- file
|
||||
content caching is handled by the workspace `IOResult` mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gdocs import GDocsConfig, GDocsResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = GDocsConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GDocsResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/gdocs": resource}, mode=MountMode.READ)
|
||||
|
||||
# List structure
|
||||
r = await ws.execute("ls /gdocs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List owned documents
|
||||
r = await ws.execute("ls /gdocs/owned/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a document
|
||||
r = await ws.execute(
|
||||
"cat /gdocs/owned/2026-04-04_Project_Plan__1AbCdEf.gdoc.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract title with jq
|
||||
r = await ws.execute(
|
||||
'jq ".title"'
|
||||
" /gdocs/owned/2026-04-04_Project_Plan__1AbCdEf.gdoc.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all documents
|
||||
r = await ws.execute('rg "quarterly" /gdocs/owned/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 1 /gdocs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Create a new document
|
||||
r = await ws.execute(
|
||||
'gws-docs-documents-create --json \'{"title":"MIRAGE Example Doc"}\'')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Append text to a document
|
||||
r = await ws.execute(
|
||||
'gws-docs-write --document 1AbCdEf --text "Appended via MIRAGE."')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/gdocs/gdocs.py` for the full working example.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Google Docs tree:
|
||||
|
||||
| Command | Notes |
|
||||
| ----------------------------------- | --------------------------- |
|
||||
| `ls` | List owned/shared documents |
|
||||
| `cat` | Read document JSON |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata |
|
||||
| `find` | Recursive search |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` / `dirname` / `realpath` | Path utilities |
|
||||
| `nl` | Number lines |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `gws-docs-documents-create`
|
||||
|
||||
Create a new Google Docs document.
|
||||
|
||||
```bash
|
||||
gws-docs-documents-create --json '{"title":"MIRAGE Example Doc"}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | ---------------------------- |
|
||||
| `--json` | yes | JSON body with `title` field |
|
||||
|
||||
Returns the created document JSON.
|
||||
|
||||
### `gws-docs-documents-batchUpdate`
|
||||
|
||||
Batch update a document.
|
||||
|
||||
```bash
|
||||
gws-docs-documents-batchUpdate --params '{"documentId":"1AbCdEf"}' --json '{"requests":[]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | -------------------------- |
|
||||
| `--params` | yes | JSON with `documentId` |
|
||||
| `--json` | yes | JSON with `requests` array |
|
||||
|
||||
Returns the batch update response JSON.
|
||||
|
||||
### `gws-docs-write`
|
||||
|
||||
Append text to a document.
|
||||
|
||||
```bash
|
||||
gws-docs-write --document 1AbCdEf --text "Appended via MIRAGE."
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------ | -------- | ------------------ |
|
||||
| `--document` | yes | Google document ID |
|
||||
| `--text` | yes | Text to append |
|
||||
|
||||
Returns the update response JSON.
|
||||
@@ -0,0 +1,398 @@
|
||||
---
|
||||
title: Google Drive
|
||||
description: Mount Google Drive folders and files, including Docs, Sheets, and Slides, as a Mirage virtual filesystem.
|
||||
icon: google
|
||||
---
|
||||
The Google Drive resource exposes a Google Drive account as a virtual
|
||||
filesystem mounted at some prefix such as `/gdrive/`.
|
||||
|
||||
For Google OAuth setup, see [Google Workspace Setup](/python/setup/google).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gdrive import GoogleDriveConfig, GoogleDriveResource
|
||||
|
||||
config = GoogleDriveConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GoogleDriveResource(config=config)
|
||||
ws = Workspace({"/gdrive": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/gdrive/
|
||||
<folder>/
|
||||
<file>
|
||||
<subfolder>/
|
||||
...
|
||||
<name>.gdoc.json
|
||||
<name>.gsheet.json
|
||||
<name>.gslide.json
|
||||
<shared-drive>/
|
||||
<file>
|
||||
<folder>/
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/gdrive/
|
||||
Projects/
|
||||
spec.pdf
|
||||
roadmap.gsheet.json
|
||||
Budget/
|
||||
Q1.gsheet.json
|
||||
Q2.gsheet.json
|
||||
Notes/
|
||||
meeting.gdoc.json
|
||||
Presentations/
|
||||
quarterly_review.gslide.json
|
||||
Team Drive/
|
||||
shared-spec.pdf
|
||||
data.csv
|
||||
report.pdf
|
||||
```
|
||||
|
||||
The mount mirrors the actual Google Drive folder hierarchy. The root
|
||||
contains the Drive API `root` folder plus each Shared Drive visible to
|
||||
the user. Shared Drives appear as top-level directories. Duplicate names
|
||||
receive a `[Shared Drive]` suffix and, when needed, a numeric suffix.
|
||||
Subfolders appear as directories, and regular files keep their original names.
|
||||
|
||||
### Synthetic Extensions
|
||||
|
||||
Google Workspace files cannot be downloaded as raw bytes, so they
|
||||
are exposed with synthetic extensions and read via their respective
|
||||
APIs:
|
||||
|
||||
| Extension | Type | Read via |
|
||||
| -------------- | ------------- | ---------- |
|
||||
| `.gdoc.json` | Google Docs | Docs API |
|
||||
| `.gsheet.json` | Google Sheets | Sheets API |
|
||||
| `.gslide.json` | Google Slides | Slides API |
|
||||
|
||||
Regular files (PDFs, images, CSVs, etc.) are downloaded directly
|
||||
from Drive. Large regular files use streaming.
|
||||
|
||||
## Cache
|
||||
|
||||
The Google Drive resource uses `IndexCacheStore` (same as Slack,
|
||||
Gmail, and other resources). Index entries store folder IDs, file
|
||||
IDs, and file metadata. There is no separate content cache -- file
|
||||
content caching is handled by the workspace `IOResult` mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gdrive import GoogleDriveConfig, GoogleDriveResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = GoogleDriveConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GoogleDriveResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/gdrive": resource}, mode=MountMode.READ)
|
||||
|
||||
# List root
|
||||
r = await ws.execute("ls /gdrive/ | head -n 10")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Browse a subfolder
|
||||
r = await ws.execute("ls /gdrive/Projects/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a regular file
|
||||
r = await ws.execute("cat /gdrive/Projects/spec.pdf | head -c 200")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a Google Doc title
|
||||
r = await ws.execute('jq ".title" /gdrive/Notes/meeting.gdoc.json')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a Google Sheet title
|
||||
r = await ws.execute(
|
||||
'jq ".properties.title" /gdrive/Projects/roadmap.gsheet.json')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a Google Slides deck length
|
||||
r = await ws.execute(
|
||||
'jq ".slides | length"'
|
||||
" /gdrive/Presentations/quarterly_review.gslide.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all files
|
||||
r = await ws.execute('rg "quarterly" /gdrive/Projects/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 2 /gdrive/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Create a new Google Doc
|
||||
r = await ws.execute(
|
||||
"gws-docs-documents-create"
|
||||
' --json \'{"title": "New Doc from MIRAGE"}\'')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/google/gdrive.py` for the full working example.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Google Drive tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List folders and files |
|
||||
| `cat` | Read file content (regular or Workspace) |
|
||||
| `head` / `tail` | First/last N lines or bytes |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields on Workspace files |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` | Extract filename from path |
|
||||
| `dirname` | Extract directory from path |
|
||||
| `realpath` | Resolve path to absolute form |
|
||||
| `nl` | Number lines of output |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Deduplicate adjacent lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `awk` | Pattern-directed scanning |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate/delete characters |
|
||||
| `diff` | Compare two files |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Reverse file line order |
|
||||
| `paste` | Merge lines of files |
|
||||
| `join` | Join lines on a common field |
|
||||
| `column` | Columnate output |
|
||||
| `comm` | Compare sorted files line by line |
|
||||
| `fold` | Wrap lines to a given width |
|
||||
| `fmt` | Reformat paragraph text |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `du` | Estimate file space usage |
|
||||
| `shuf` | Randomly permute lines |
|
||||
| `look` | Display lines beginning with a prefix |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `base64` | Base64 encode/decode |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
| `xxd` | Hex dump |
|
||||
| `zcat` | Read compressed files |
|
||||
| `zgrep` | Search compressed files |
|
||||
| `readlink` | Print resolved symbolic links |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `tsort` | Topological sort |
|
||||
| `file` | Detect file type |
|
||||
|
||||
## Data Format Support
|
||||
|
||||
Google Drive may contain data files in binary columnar formats.
|
||||
These are auto-converted to CSV on read. Specialized variants
|
||||
of common commands handle them natively:
|
||||
|
||||
| Format | Extension | Specialized commands |
|
||||
| ------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Parquet | `.parquet` | `cat-parquet`, `head-parquet`, `tail-parquet`, `wc-parquet`, `stat-parquet`, `grep-parquet`, `cut-parquet`, `ls-parquet`, `file-parquet` |
|
||||
| Feather | `.feather` | `cat-feather`, `head-feather`, `tail-feather`, `wc-feather`, `stat-feather`, `grep-feather`, `cut-feather`, `ls-feather`, `file-feather` |
|
||||
| HDF5 | `.hdf5` | `cat-hdf5`, `head-hdf5`, `tail-hdf5`, `wc-hdf5`, `stat-hdf5`, `grep-hdf5`, `cut-hdf5`, `ls-hdf5`, `file-hdf5` |
|
||||
| ORC | `.orc` | `cat-orc`, `head-orc`, `tail-orc`, `wc-orc`, `stat-orc`, `grep-orc`, `cut-orc`, `ls-orc`, `file-orc` |
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
cat-parquet /gdrive/data/sales.parquet
|
||||
head-parquet -n 5 /gdrive/data/sales.parquet
|
||||
grep-parquet "revenue" /gdrive/data/sales.parquet
|
||||
wc-parquet /gdrive/data/sales.parquet
|
||||
```
|
||||
|
||||
## Resource-Specific Commands
|
||||
|
||||
Google Drive registers Google Workspace write commands so that
|
||||
Google-native files can be created and updated from the Drive mount.
|
||||
|
||||
### `gws-docs-documents-create`
|
||||
|
||||
Create a new Google Doc.
|
||||
|
||||
```bash
|
||||
gws-docs-documents-create --json '{"title": "My Doc"}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | ---------------------------- |
|
||||
| `--json` | yes | JSON body with `title` field |
|
||||
|
||||
Returns the created document JSON.
|
||||
|
||||
### `gws-docs-documents-batchUpdate`
|
||||
|
||||
Apply batch updates to a Google Doc.
|
||||
|
||||
```bash
|
||||
gws-docs-documents-batchUpdate \
|
||||
--params '{"documentId": "DOC_ID"}' \
|
||||
--json '{"requests": [...]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ------------------------------- |
|
||||
| `--params` | yes | JSON with `documentId` |
|
||||
| `--json` | yes | JSON body with `requests` array |
|
||||
|
||||
Returns the batch update response JSON.
|
||||
|
||||
### `gws-docs-write`
|
||||
|
||||
Append text to a Google Doc.
|
||||
|
||||
```bash
|
||||
gws-docs-write --document DOC_ID --text "Hello from MIRAGE"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------ | -------- | -------------- |
|
||||
| `--document` | yes | Google Doc ID |
|
||||
| `--text` | yes | Text to append |
|
||||
|
||||
Returns the update response JSON.
|
||||
|
||||
### `gws-sheets-read`
|
||||
|
||||
Read values from a spreadsheet range.
|
||||
|
||||
```bash
|
||||
gws-sheets-read --spreadsheet SHEET_ID --range "Sheet1!A1:C10"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------------- | -------- | ----------------- |
|
||||
| `--spreadsheet` | yes | Spreadsheet ID |
|
||||
| `--range` | yes | A1 notation range |
|
||||
|
||||
Returns the range values.
|
||||
|
||||
### `gws-sheets-write`
|
||||
|
||||
Write values to a spreadsheet range.
|
||||
|
||||
```bash
|
||||
gws-sheets-write \
|
||||
--params '{"spreadsheetId": "SHEET_ID", "range": "Sheet1!A1:C3", "valueInputOption": "USER_ENTERED"}' \
|
||||
--json '{"values": [["a", "b", "c"]]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | --------------------------------------------------------------- |
|
||||
| `--params` | yes | JSON with `spreadsheetId`, `range`, optional `valueInputOption` |
|
||||
| `--json` | yes | JSON body with `values` array |
|
||||
|
||||
Returns the update response JSON.
|
||||
|
||||
### `gws-sheets-append`
|
||||
|
||||
Append rows to a spreadsheet.
|
||||
|
||||
```bash
|
||||
gws-sheets-append --spreadsheet SHEET_ID --range "Sheet1!A1" --values "a,b,c"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------------- | -------- | ---------------------------------------------- |
|
||||
| `--spreadsheet` | yes | Spreadsheet ID |
|
||||
| `--range` | no | A1 notation range (defaults to `A1`) |
|
||||
| `--values` | no | Comma-separated values for a single row |
|
||||
| `--json-values` | no | JSON array of rows (alternative to `--values`) |
|
||||
|
||||
One of `--values` or `--json-values` is required. Returns the
|
||||
append response JSON.
|
||||
|
||||
### `gws-sheets-spreadsheets-create`
|
||||
|
||||
Create a new spreadsheet.
|
||||
|
||||
```bash
|
||||
gws-sheets-spreadsheets-create --json '{"properties": {"title": "My Sheet"}}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | --------------------------------- |
|
||||
| `--json` | yes | JSON body with `properties.title` |
|
||||
|
||||
Returns the created spreadsheet JSON.
|
||||
|
||||
### `gws-sheets-spreadsheets-batchUpdate`
|
||||
|
||||
Apply batch updates to a spreadsheet.
|
||||
|
||||
```bash
|
||||
gws-sheets-spreadsheets-batchUpdate \
|
||||
--params '{"spreadsheetId": "SHEET_ID"}' \
|
||||
--json '{"requests": [...]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ------------------------------- |
|
||||
| `--params` | yes | JSON with `spreadsheetId` |
|
||||
| `--json` | yes | JSON body with `requests` array |
|
||||
|
||||
Returns the batch update response JSON.
|
||||
|
||||
### `gws-slides-presentations-create`
|
||||
|
||||
Create a new Google Slides presentation.
|
||||
|
||||
```bash
|
||||
gws-slides-presentations-create --json '{"title": "My Deck"}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | ---------------------------- |
|
||||
| `--json` | yes | JSON body with `title` field |
|
||||
|
||||
Returns the created presentation JSON.
|
||||
|
||||
### `gws-slides-presentations-batchUpdate`
|
||||
|
||||
Apply batch updates to a presentation.
|
||||
|
||||
```bash
|
||||
gws-slides-presentations-batchUpdate \
|
||||
--params '{"presentationId": "PRES_ID"}' \
|
||||
--json '{"requests": [...]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ------------------------------- |
|
||||
| `--params` | yes | JSON with `presentationId` |
|
||||
| `--json` | yes | JSON body with `requests` array |
|
||||
|
||||
Returns the batch update response JSON.
|
||||
@@ -0,0 +1,171 @@
|
||||
---
|
||||
title: GitHub
|
||||
description: Mount a GitHub repository as a read-only Mirage filesystem for agents to list, read, search, and inspect code.
|
||||
icon: github
|
||||
---
|
||||
The GitHub resource mounts a GitHub repository as a read-only virtual
|
||||
filesystem.
|
||||
|
||||
For token setup, see [GitHub Setup](/python/setup/github).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.github import GitHubConfig, GitHubResource
|
||||
|
||||
config = GitHubConfig(token=os.environ["GITHUB_TOKEN"])
|
||||
resource = GitHubResource(
|
||||
config=config, owner="my-org", repo="my-repo", ref="main")
|
||||
ws = Workspace({"/github": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/github/
|
||||
README.md
|
||||
pyproject.toml
|
||||
src/
|
||||
__init__.py
|
||||
main.py
|
||||
utils.py
|
||||
models/
|
||||
user.py
|
||||
item.py
|
||||
tests/
|
||||
test_main.py
|
||||
```
|
||||
|
||||
The filesystem mirrors the repository tree. No owner/repo/branch in
|
||||
the path - those are specified at mount time.
|
||||
|
||||
## Tree Fetching
|
||||
|
||||
The resource fetches the full recursive tree at init. For repos with
|
||||
|
||||
> 100K entries, it falls back to per-directory fetching.
|
||||
|
||||
## Cache
|
||||
|
||||
The GitHub resource uses `IndexCacheStore` with SHA-based fingerprinting. Content is content-addressed - if the SHA matches, the content is identical.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.github import GitHubConfig, GitHubResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
|
||||
async def main():
|
||||
config = GitHubConfig(token=os.environ["GITHUB_TOKEN"])
|
||||
resource = GitHubResource(
|
||||
config=config, owner="my-org", repo="my-repo", ref="main")
|
||||
ws = Workspace({"/github": resource}, mode=MountMode.READ)
|
||||
|
||||
# List repository root
|
||||
r = await ws.execute("ls /github/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a file
|
||||
r = await ws.execute("cat /github/README.md")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search for a pattern
|
||||
r = await ws.execute('rg "def main" /github/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 2 /github/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# File metadata with SHA
|
||||
r = await ws.execute("stat /github/README.md")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/code/github.py` for the full working example.
|
||||
|
||||
## Finding SHAs
|
||||
|
||||
Git blob SHAs are available via the `stat` command:
|
||||
|
||||
```bash
|
||||
stat /github/README.md
|
||||
# -> extra={"sha": "a1b2c3d4e5f6..."}
|
||||
|
||||
stat /github/src/main.py
|
||||
# -> extra={"sha": "f6e5d4c3b2a1..."}
|
||||
```
|
||||
|
||||
## Working with Large Repos
|
||||
|
||||
Tips for efficient access on large repositories:
|
||||
|
||||
```bash
|
||||
# Find files by name
|
||||
find /github/ -name "*.py"
|
||||
|
||||
# Search with rg (uses GitHub code search API when applicable)
|
||||
rg "TODO" /github/
|
||||
|
||||
# Read only the first lines of a file
|
||||
head -n 20 /github/src/main.py
|
||||
|
||||
# Check file sizes
|
||||
du /github/src/
|
||||
|
||||
# List deeply nested directories
|
||||
tree -L 3 /github/src/
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted GitHub tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List files and directories |
|
||||
| `cat` | Read file contents |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search; rg uses code search API |
|
||||
| `jq` | Query JSON files |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata including git SHA |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `diff` | Compare files |
|
||||
| `du` | Disk usage / file sizes |
|
||||
| `awk` | Text processing |
|
||||
| `sed` | Stream editing |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Deduplicate lines |
|
||||
| `cut` | Extract columns |
|
||||
| `tr` | Translate characters |
|
||||
| `nl` | Number lines |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
| `file` | Detect file type |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
|
||||
## Search Optimization
|
||||
|
||||
`rg` uses the GitHub code search API when the search scope exceeds
|
||||
100 files and the mounted ref is the repository's default branch.
|
||||
This avoids downloading file contents and returns results significantly
|
||||
faster for large repositories.
|
||||
@@ -0,0 +1,194 @@
|
||||
---
|
||||
title: GitHub CI
|
||||
description: Mount GitHub Actions workflows, runs, jobs, logs, and artifacts as a read-only Mirage filesystem.
|
||||
icon: circle-play
|
||||
---
|
||||
The GitHub CI resource mounts GitHub Actions workflows and runs as a
|
||||
read-only virtual filesystem.
|
||||
|
||||
For token setup, see [GitHub CI Setup](/python/setup/github_ci).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.github_ci import GitHubCIConfig, GitHubCIResource
|
||||
|
||||
config = GitHubCIConfig(
|
||||
token=os.environ["GITHUB_TOKEN"],
|
||||
owner="my-org",
|
||||
repo="my-repo",
|
||||
)
|
||||
resource = GitHubCIResource(config=config)
|
||||
ws = Workspace({"/ci": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/ci/
|
||||
workflows/
|
||||
<workflow-name>_<workflow-id>.json
|
||||
runs/
|
||||
<workflow-name>_<run-id>/
|
||||
run.json
|
||||
jobs/
|
||||
<job-name>_<job-id>.json
|
||||
<job-name>_<job-id>.log
|
||||
annotations.jsonl
|
||||
artifacts/
|
||||
<artifact-name>_<artifact-id>.zip
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/ci/
|
||||
workflows/
|
||||
CI_12345678.json
|
||||
Deploy_87654321.json
|
||||
runs/
|
||||
CI_9876543210/
|
||||
run.json
|
||||
jobs/
|
||||
build_11111111.json
|
||||
build_11111111.log
|
||||
test_22222222.json
|
||||
test_22222222.log
|
||||
annotations.jsonl
|
||||
artifacts/
|
||||
coverage-report_55555.zip
|
||||
```
|
||||
|
||||
### Workflows
|
||||
|
||||
`/ci/workflows/` lists all workflows defined in the repository.
|
||||
Each `.json` file contains the workflow metadata (name, path, state).
|
||||
|
||||
### Runs
|
||||
|
||||
`/ci/runs/` lists recent workflow runs within the configured time window
|
||||
(default 30 days). The `created` API parameter filters server-side.
|
||||
|
||||
Each run directory contains:
|
||||
|
||||
- `run.json` - run metadata (status, conclusion, event, branch, actor, timing)
|
||||
- `jobs/` - one `.json` and `.log` pair per job
|
||||
- `annotations.jsonl` - check annotations (warnings, errors) across all jobs
|
||||
- `artifacts/` - downloadable build artifacts
|
||||
|
||||
### Jobs
|
||||
|
||||
Each job has two files:
|
||||
|
||||
- `<job-name>_<job-id>.json` - job metadata with steps and timing
|
||||
- `<job-name>_<job-id>.log` - full job log (plain text)
|
||||
|
||||
### Artifacts
|
||||
|
||||
Artifacts are served as `.zip` files matching the GitHub API download format.
|
||||
|
||||
## Cache
|
||||
|
||||
The GitHub CI resource uses `IndexCacheStore` with `remote_time`-based
|
||||
fingerprinting. Completed runs and jobs are effectively immutable and
|
||||
benefit from long cache TTL.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.github_ci import GitHubCIConfig, GitHubCIResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
|
||||
async def main():
|
||||
config = GitHubCIConfig(
|
||||
token=os.environ["GITHUB_TOKEN"],
|
||||
owner="my-org",
|
||||
repo="my-repo",
|
||||
)
|
||||
resource = GitHubCIResource(config=config)
|
||||
ws = Workspace({"/ci": resource}, mode=MountMode.READ)
|
||||
|
||||
# List top-level
|
||||
r = await ws.execute("ls /ci/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List workflows
|
||||
r = await ws.execute("ls /ci/workflows/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List recent runs
|
||||
r = await ws.execute("ls /ci/runs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read run metadata
|
||||
r = await ws.execute("cat /ci/runs/CI_9876543210/run.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List jobs for a run
|
||||
r = await ws.execute("ls /ci/runs/CI_9876543210/jobs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read job log
|
||||
r = await ws.execute("cat /ci/runs/CI_9876543210/jobs/build_11111111.log")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read annotations
|
||||
r = await ws.execute("cat /ci/runs/CI_9876543210/annotations.jsonl")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 2 /ci/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Find all failed job logs
|
||||
r = await ws.execute("find /ci/runs/ -name '*.log'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted GitHub CI tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------ |
|
||||
| `ls` | List workflows, runs, jobs |
|
||||
| `cat` | Read JSON metadata or job logs |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (type, IDs) |
|
||||
| `find` | Recursive search with `-name` |
|
||||
| `tree` | Directory tree view |
|
||||
|
||||
## Working with CI Data
|
||||
|
||||
```bash
|
||||
# Check run status via jq
|
||||
cat /ci/runs/CI_9876543210/run.json | jq '.conclusion'
|
||||
|
||||
# List all job names
|
||||
ls /ci/runs/CI_9876543210/jobs/ | grep '.json$'
|
||||
|
||||
# Tail a job log for recent output
|
||||
tail -n 50 /ci/runs/CI_9876543210/jobs/build_11111111.log
|
||||
|
||||
# Count annotations
|
||||
wc -l /ci/runs/CI_9876543210/annotations.jsonl
|
||||
|
||||
# Find all .log files
|
||||
find /ci/ -name "*.log"
|
||||
```
|
||||
@@ -0,0 +1,337 @@
|
||||
---
|
||||
title: Gmail
|
||||
description: Mount Gmail mailboxes as a Mirage filesystem so Python agents can browse labels, messages, threads, and attachments.
|
||||
icon: envelope
|
||||
---
|
||||
The Gmail resource exposes a Gmail account as a virtual filesystem
|
||||
mounted at some prefix such as `/gmail/`.
|
||||
|
||||
For Google OAuth setup, see [Google Workspace Setup](/python/setup/google).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gmail import GmailConfig, GmailResource
|
||||
|
||||
config = GmailConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GmailResource(config=config)
|
||||
ws = Workspace({"/gmail": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/gmail/
|
||||
<label>/
|
||||
<yyyy-mm-dd>/
|
||||
<sanitized-subject>__<message-id>.gmail.json
|
||||
<sanitized-subject>__<message-id>/ # only if message has attachments
|
||||
<attachment-filename>
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/gmail/
|
||||
INBOX/
|
||||
2026-04-12/
|
||||
Meeting_Notes__msg123.gmail.json
|
||||
Meeting_Notes__msg123/
|
||||
report.pdf
|
||||
screenshot.png
|
||||
Simple_Email__msg456.gmail.json
|
||||
2026-04-11/
|
||||
Hello__msg789.gmail.json
|
||||
SENT/
|
||||
2026-04-12/
|
||||
Reply__msg012.gmail.json
|
||||
STARRED/
|
||||
My_Custom_Label/
|
||||
```
|
||||
|
||||
Label directories appear at the root. System labels use the Gmail
|
||||
label ID (e.g., `INBOX`, `SENT`, `STARRED`). User-created labels
|
||||
use their label name with spaces replaced by underscores.
|
||||
|
||||
### Date Directories
|
||||
|
||||
Inside each label, messages are grouped into date subdirectories
|
||||
formatted as `YYYY-MM-DD`. The date is derived from the message's
|
||||
`internalDate` (epoch milliseconds) converted to a calendar date.
|
||||
|
||||
### Message Files
|
||||
|
||||
Each message is stored as a `.gmail.json` file. The filename shape is:
|
||||
|
||||
```text
|
||||
<sanitized-subject>__<message-id>.gmail.json
|
||||
```
|
||||
|
||||
Subjects are sanitized for filesystem safety and truncated when
|
||||
necessary. The message ID is embedded after `__` and before
|
||||
`.gmail.json`.
|
||||
|
||||
### Attachments
|
||||
|
||||
Messages that have attachments get a companion subdirectory with
|
||||
the same base name (without `.gmail.json`). Decoded attachment
|
||||
files are placed inside:
|
||||
|
||||
```text
|
||||
Meeting_Notes__msg123.gmail.json # message JSON
|
||||
Meeting_Notes__msg123/ # attachment directory
|
||||
report.pdf
|
||||
screenshot.png
|
||||
```
|
||||
|
||||
## Cache
|
||||
|
||||
The Gmail resource uses `IndexCacheStore` (same as Slack, Discord,
|
||||
and other resources). Index entries store label IDs, message IDs,
|
||||
and message metadata. There is no separate content cache -- file
|
||||
content caching is handled by the workspace `IOResult` mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gmail import GmailConfig, GmailResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = GmailConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GmailResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/gmail": resource}, mode=MountMode.READ)
|
||||
|
||||
# List labels
|
||||
r = await ws.execute("ls /gmail/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List date directories in INBOX
|
||||
r = await ws.execute("ls /gmail/INBOX/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List messages for a specific date
|
||||
r = await ws.execute("ls /gmail/INBOX/2026-04-12/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a message
|
||||
r = await ws.execute(
|
||||
"cat /gmail/INBOX/2026-04-12/Meeting_Notes__msg123.gmail.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract subject with jq
|
||||
r = await ws.execute(
|
||||
'jq ".subject"'
|
||||
" /gmail/INBOX/2026-04-12/Meeting_Notes__msg123.gmail.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List attachments
|
||||
r = await ws.execute("ls /gmail/INBOX/2026-04-12/Meeting_Notes__msg123/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all messages
|
||||
r = await ws.execute('rg "quarterly" /gmail/INBOX/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 2 /gmail/INBOX/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Triage unread messages
|
||||
r = await ws.execute('gws-gmail-triage --query "is:unread" --max 5')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Send an email
|
||||
r = await ws.execute(
|
||||
'gws-gmail-send --to "user@example.com"'
|
||||
' --subject "Hello from MIRAGE"'
|
||||
' --body "This email was sent via the MIRAGE Gmail resource."')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/gmail/gmail.py` for the full working example.
|
||||
|
||||
## Finding IDs
|
||||
|
||||
Resource-specific commands require message IDs. These can be
|
||||
extracted from the filesystem:
|
||||
|
||||
```bash
|
||||
# Message ID -- embedded in filename after "__"
|
||||
ls /gmail/INBOX/2026-04-12/
|
||||
# -> Meeting_Notes__msg123.gmail.json <- message_id = msg123
|
||||
|
||||
# Extract message ID from filename
|
||||
basename /gmail/INBOX/2026-04-12/Meeting_Notes__msg123.gmail.json .gmail.json
|
||||
# -> Meeting_Notes__msg123
|
||||
# The part after "__" is the message ID: msg123
|
||||
|
||||
# Read a message then reply
|
||||
gws-gmail-read --id msg123
|
||||
gws-gmail-reply --message-id msg123 --body "Thanks for the notes"
|
||||
```
|
||||
|
||||
## Working with Large Labels
|
||||
|
||||
Labels with many messages are split into date directories. Tips
|
||||
for efficient access:
|
||||
|
||||
```bash
|
||||
# List available dates
|
||||
ls /gmail/INBOX/
|
||||
|
||||
# Check message count for a specific date
|
||||
ls /gmail/INBOX/2026-04-12/ | wc -l
|
||||
|
||||
# Read only the most recent date
|
||||
ls /gmail/INBOX/ | tail -n 1
|
||||
|
||||
# Search across all dates in a label
|
||||
rg "keyword" /gmail/INBOX/
|
||||
|
||||
# Extract specific fields to reduce output
|
||||
jq -r '.subject' /gmail/INBOX/2026-04-12/*.gmail.json | head -n 20
|
||||
|
||||
# Find messages with attachments
|
||||
find /gmail/INBOX/2026-04-12/ -type d
|
||||
|
||||
# Tree view of a label
|
||||
tree -L 2 /gmail/INBOX/
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Gmail tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List labels, dates, messages, attachments |
|
||||
| `cat` | Read message JSON or attachment content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query message JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` | Extract filename from path |
|
||||
| `dirname` | Extract directory from path |
|
||||
| `realpath` | Resolve path to absolute form |
|
||||
| `nl` | Number lines of output |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `gws-gmail-send`
|
||||
|
||||
Send a new email.
|
||||
|
||||
```bash
|
||||
gws-gmail-send --to "user@example.com" --subject "Hello" --body "Hi there"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ----------- | -------- | ----------------------- |
|
||||
| `--to` | yes | Recipient email address |
|
||||
| `--subject` | yes | Email subject line |
|
||||
| `--body` | yes | Email body text |
|
||||
|
||||
Returns the sent message JSON.
|
||||
|
||||
### `gws-gmail-reply`
|
||||
|
||||
Reply to a message.
|
||||
|
||||
```bash
|
||||
gws-gmail-reply --message-id msg123 --body "Thanks for the update"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ---------------- |
|
||||
| `--message-id` | yes | Gmail message ID |
|
||||
| `--body` | yes | Reply body text |
|
||||
|
||||
Returns the sent reply JSON.
|
||||
|
||||
### `gws-gmail-reply-all`
|
||||
|
||||
Reply-all to a message.
|
||||
|
||||
```bash
|
||||
gws-gmail-reply-all --message-id msg123 --body "Acknowledged by the team"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ---------------- |
|
||||
| `--message-id` | yes | Gmail message ID |
|
||||
| `--body` | yes | Reply body text |
|
||||
|
||||
Returns the sent reply JSON.
|
||||
|
||||
### `gws-gmail-forward`
|
||||
|
||||
Forward a message to another recipient.
|
||||
|
||||
```bash
|
||||
gws-gmail-forward --message-id msg123 --to "colleague@example.com"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ----------------------- |
|
||||
| `--message-id` | yes | Gmail message ID |
|
||||
| `--to` | yes | Recipient email address |
|
||||
|
||||
Returns the forwarded message JSON.
|
||||
|
||||
### `gws-gmail-triage`
|
||||
|
||||
Search and triage emails using Gmail query syntax.
|
||||
|
||||
```bash
|
||||
gws-gmail-triage --query "is:unread" --max 10
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------- | -------- | ----------------------------------- |
|
||||
| `--query` | yes | Gmail search query |
|
||||
| `--max` | no | Maximum number of results to return |
|
||||
|
||||
Returns matching messages as JSON.
|
||||
|
||||
### `gws-gmail-read`
|
||||
|
||||
Read a message by its ID.
|
||||
|
||||
```bash
|
||||
gws-gmail-read --id msg123
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------ | -------- | ---------------- |
|
||||
| `--id` | yes | Gmail message ID |
|
||||
|
||||
Returns the full message JSON.
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
title: Google Sheets
|
||||
description: Mount Google Sheets spreadsheets as JSON-backed files that Python agents can read through Mirage shell commands.
|
||||
icon: table
|
||||
---
|
||||
The Google Sheets resource exposes Sheets spreadsheets as a virtual
|
||||
filesystem mounted at some prefix such as `/gsheets/`.
|
||||
|
||||
For Google OAuth setup, see [Google Workspace Setup](/python/setup/google).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gsheets import GSheetsConfig, GSheetsResource
|
||||
|
||||
config = GSheetsConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GSheetsResource(config=config)
|
||||
ws = Workspace({"/gsheets": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/gsheets/
|
||||
owned/
|
||||
<YYYY-MM-DD>_<sanitized-title>__<spreadsheet-id>.gsheet.json
|
||||
...
|
||||
shared/
|
||||
<YYYY-MM-DD>_<sanitized-title>__<spreadsheet-id>.gsheet.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/gsheets/
|
||||
owned/
|
||||
2026-04-04_Roadmap__1AbCdEf.gsheet.json
|
||||
2026-04-10_Expenses__2BcDeFg.gsheet.json
|
||||
shared/
|
||||
2026-04-03_Budget__9XyZ.gsheet.json
|
||||
```
|
||||
|
||||
Spreadsheets are split into `owned` (spreadsheets you created) and
|
||||
`shared` (spreadsheets shared with you). The filename shape is:
|
||||
|
||||
```text
|
||||
<YYYY-MM-DD>_<sanitized-title>__<spreadsheet-id>.gsheet.json
|
||||
```
|
||||
|
||||
If the modified date is unavailable, the date prefix is omitted.
|
||||
Reading a spreadsheet file returns the full Google Sheets API JSON
|
||||
for that spreadsheet, including sheet metadata.
|
||||
|
||||
## Cache
|
||||
|
||||
The Google Sheets resource uses `IndexCacheStore`. Index entries
|
||||
store spreadsheet IDs and metadata. There is no separate content
|
||||
cache -- file content caching is handled by the workspace `IOResult`
|
||||
mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gsheets import GSheetsConfig, GSheetsResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = GSheetsConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GSheetsResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/gsheets": resource}, mode=MountMode.READ)
|
||||
|
||||
# List structure
|
||||
r = await ws.execute("ls /gsheets/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List owned spreadsheets
|
||||
r = await ws.execute("ls /gsheets/owned/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a spreadsheet
|
||||
r = await ws.execute(
|
||||
"cat /gsheets/owned/2026-04-04_Roadmap__1AbCdEf.gsheet.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract title with jq
|
||||
r = await ws.execute(
|
||||
'jq ".properties.title"'
|
||||
" /gsheets/owned/2026-04-04_Roadmap__1AbCdEf.gsheet.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all spreadsheets
|
||||
r = await ws.execute('rg "revenue" /gsheets/owned/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 1 /gsheets/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read cell values
|
||||
r = await ws.execute(
|
||||
'gws-sheets-read --spreadsheet 1AbCdEf --range "Sheet1!A1:C3"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Append rows
|
||||
r = await ws.execute(
|
||||
"gws-sheets-append --spreadsheet 1AbCdEf --values Alice,30,NYC")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Create a new spreadsheet
|
||||
r = await ws.execute(
|
||||
'gws-sheets-spreadsheets-create'
|
||||
' --json \'{"properties":{"title":"MIRAGE Sheet"}}\'')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/gsheets/gsheets.py` for the full working example.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Google Sheets tree:
|
||||
|
||||
| Command | Notes |
|
||||
| ----------------------------------- | ------------------------------ |
|
||||
| `ls` | List owned/shared spreadsheets |
|
||||
| `cat` | Read spreadsheet JSON |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata |
|
||||
| `find` | Recursive search |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` / `dirname` / `realpath` | Path utilities |
|
||||
| `nl` | Number lines |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `gws-sheets-read`
|
||||
|
||||
Read cell values from a spreadsheet range.
|
||||
|
||||
```bash
|
||||
gws-sheets-read --spreadsheet 1AbCdEf --range "Sheet1!A1:C3"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------------- | -------- | ------------------------- |
|
||||
| `--spreadsheet` | yes | Google spreadsheet ID |
|
||||
| `--range` | yes | A1 notation range to read |
|
||||
|
||||
Returns the cell values as JSON.
|
||||
|
||||
### `gws-sheets-write`
|
||||
|
||||
Write cell values to a spreadsheet range.
|
||||
|
||||
```bash
|
||||
gws-sheets-write --params '{"spreadsheetId":"1AbCdEf","range":"Sheet1!A1"}' --json '{"values":[["x"]]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | --------------------------------------------------------------- |
|
||||
| `--params` | yes | JSON with `spreadsheetId`, `range`, optional `valueInputOption` |
|
||||
| `--json` | yes | JSON with `values` 2D array |
|
||||
|
||||
Returns the update response JSON.
|
||||
|
||||
### `gws-sheets-append`
|
||||
|
||||
Append rows to a spreadsheet.
|
||||
|
||||
```bash
|
||||
gws-sheets-append --spreadsheet 1AbCdEf --values Alice,30,NYC
|
||||
gws-sheets-append --spreadsheet 1AbCdEf --range "Sheet1!A1" --json-values '[["Bob",25,"LA"]]'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------------- | -------- | ------------------------------------ |
|
||||
| `--spreadsheet` | yes | Google spreadsheet ID |
|
||||
| `--range` | no | A1 notation range (defaults to `A1`) |
|
||||
| `--values` | no | Comma-separated values for one row |
|
||||
| `--json-values` | no | JSON 2D array of values |
|
||||
|
||||
Either `--values` or `--json-values` is required. Returns the
|
||||
append response JSON.
|
||||
|
||||
### `gws-sheets-spreadsheets-create`
|
||||
|
||||
Create a new spreadsheet.
|
||||
|
||||
```bash
|
||||
gws-sheets-spreadsheets-create --json '{"properties":{"title":"MIRAGE Sheet"}}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | --------------------------------------- |
|
||||
| `--json` | yes | JSON body with `properties.title` field |
|
||||
|
||||
Returns the created spreadsheet JSON.
|
||||
|
||||
### `gws-sheets-spreadsheets-batchUpdate`
|
||||
|
||||
Batch update a spreadsheet.
|
||||
|
||||
```bash
|
||||
gws-sheets-spreadsheets-batchUpdate --params '{"spreadsheetId":"1AbCdEf"}' --json '{"requests":[]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | -------------------------- |
|
||||
| `--params` | yes | JSON with `spreadsheetId` |
|
||||
| `--json` | yes | JSON with `requests` array |
|
||||
|
||||
Returns the batch update response JSON.
|
||||
@@ -0,0 +1,186 @@
|
||||
---
|
||||
title: Google Slides
|
||||
description: Mount Google Slides presentations as JSON-backed files that Python agents can read through Mirage shell commands.
|
||||
icon: presentation
|
||||
---
|
||||
The Google Slides resource exposes presentations as a virtual
|
||||
filesystem mounted at some prefix such as `/gslides/`.
|
||||
|
||||
For Google OAuth setup, see [Google Workspace Setup](/python/setup/google).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gslides import GSlidesConfig, GSlidesResource
|
||||
|
||||
config = GSlidesConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GSlidesResource(config=config)
|
||||
ws = Workspace({"/gslides": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/gslides/
|
||||
owned/
|
||||
<YYYY-MM-DD>_<sanitized-title>__<presentation-id>.gslide.json
|
||||
...
|
||||
shared/
|
||||
<YYYY-MM-DD>_<sanitized-title>__<presentation-id>.gslide.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/gslides/
|
||||
owned/
|
||||
2026-04-04_QBR_Deck__1AbCdEf.gslide.json
|
||||
2026-04-10_Team_Update__2BcDeFg.gslide.json
|
||||
shared/
|
||||
2026-04-03_Design_Review__9XyZ.gslide.json
|
||||
```
|
||||
|
||||
Presentations are split into `owned` (presentations you created)
|
||||
and `shared` (presentations shared with you). The filename shape is:
|
||||
|
||||
```text
|
||||
<YYYY-MM-DD>_<sanitized-title>__<presentation-id>.gslide.json
|
||||
```
|
||||
|
||||
If the modified date is unavailable, the date prefix is omitted.
|
||||
Reading a presentation file returns the full Google Slides API JSON
|
||||
for that presentation.
|
||||
|
||||
## Cache
|
||||
|
||||
The Google Slides resource uses `IndexCacheStore`. Index entries
|
||||
store presentation IDs and metadata. There is no separate content
|
||||
cache -- file content caching is handled by the workspace `IOResult`
|
||||
mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.gslides import GSlidesConfig, GSlidesResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = GSlidesConfig(
|
||||
client_id=os.environ["GOOGLE_CLIENT_ID"],
|
||||
client_secret=os.environ["GOOGLE_CLIENT_SECRET"],
|
||||
refresh_token=os.environ["GOOGLE_REFRESH_TOKEN"],
|
||||
)
|
||||
resource = GSlidesResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/gslides": resource}, mode=MountMode.READ)
|
||||
|
||||
# List structure
|
||||
r = await ws.execute("ls /gslides/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List owned presentations
|
||||
r = await ws.execute("ls /gslides/owned/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a presentation
|
||||
r = await ws.execute(
|
||||
"cat /gslides/owned/2026-04-04_QBR_Deck__1AbCdEf.gslide.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract title with jq
|
||||
r = await ws.execute(
|
||||
'jq ".title"'
|
||||
" /gslides/owned/2026-04-04_QBR_Deck__1AbCdEf.gslide.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all presentations
|
||||
r = await ws.execute('rg "quarterly" /gslides/owned/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 1 /gslides/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Create a new presentation
|
||||
r = await ws.execute(
|
||||
'gws-slides-presentations-create --json \'{"title":"MIRAGE Deck"}\'')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Batch update a presentation
|
||||
r = await ws.execute(
|
||||
'gws-slides-presentations-batchUpdate'
|
||||
' --params \'{"presentationId":"1AbCdEf"}\''
|
||||
' --json \'{"requests":[]}\'')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/gslides/gslides.py` for the full working example.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Google Slides tree:
|
||||
|
||||
| Command | Notes |
|
||||
| ----------------------------------- | ------------------------------- |
|
||||
| `ls` | List owned/shared presentations |
|
||||
| `cat` | Read presentation JSON |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata |
|
||||
| `find` | Recursive search |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` / `dirname` / `realpath` | Path utilities |
|
||||
| `nl` | Number lines |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `gws-slides-presentations-create`
|
||||
|
||||
Create a new presentation.
|
||||
|
||||
```bash
|
||||
gws-slides-presentations-create --json '{"title":"MIRAGE Deck"}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | ---------------------------- |
|
||||
| `--json` | yes | JSON body with `title` field |
|
||||
|
||||
Returns the created presentation JSON.
|
||||
|
||||
### `gws-slides-presentations-batchUpdate`
|
||||
|
||||
Batch update a presentation.
|
||||
|
||||
```bash
|
||||
gws-slides-presentations-batchUpdate --params '{"presentationId":"1AbCdEf"}' --json '{"requests":[]}'
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | -------------------------- |
|
||||
| `--params` | yes | JSON with `presentationId` |
|
||||
| `--json` | yes | JSON with `requests` array |
|
||||
|
||||
Returns the batch update response JSON.
|
||||
@@ -0,0 +1,252 @@
|
||||
---
|
||||
title: HF Buckets
|
||||
description: Mount Hugging Face Buckets as a Mirage filesystem with async reads and S3-compatible writes for Python agents.
|
||||
icon: /images/huggingface-logo.svg
|
||||
---
|
||||
The HF Buckets resource mounts a [Hugging Face Bucket](https://huggingface.co/docs/buckets)
|
||||
at some prefix such as `/hf/`. Speaks HF's HTTP API natively (async,
|
||||
streaming, no Python SDK dependency).
|
||||
|
||||
For credential setup, see [HF Buckets Setup](/home/setup/hf_buckets).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv add "mirage-ai[hf]"
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_buckets import HfBucketsConfig, HfBucketsResource
|
||||
|
||||
config = HfBucketsConfig(
|
||||
bucket=os.environ["HF_BUCKET_NAME"], # "namespace/bucket-name"
|
||||
token=os.environ["HF_TOKEN"],
|
||||
# Optional:
|
||||
# endpoint="https://huggingface.co",
|
||||
# timeout=30,
|
||||
# key_prefix="data/",
|
||||
)
|
||||
resource = HfBucketsResource(config)
|
||||
ws = Workspace({"/hf": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`HfBucketsResource(config)` takes an `HfBucketsConfig` object with the
|
||||
bucket in `namespace/bucket-name` form plus an optional access token.
|
||||
Both `READ` and `WRITE` modes are supported out of the box.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The HF Buckets resource maps bucket object keys to virtual paths under
|
||||
the mount prefix.
|
||||
|
||||
For example, if bucket `your-user/my-data` contains:
|
||||
|
||||
```text
|
||||
data/file.txt
|
||||
data/config.json
|
||||
reports/q1.csv
|
||||
reports/q2.csv
|
||||
```
|
||||
|
||||
Then mounting at `/hf/` exposes:
|
||||
|
||||
```text
|
||||
/hf/
|
||||
data/
|
||||
file.txt
|
||||
config.json
|
||||
reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
```
|
||||
|
||||
Path mapping: virtual `/hf/data/file.txt` maps to bucket key
|
||||
`data/file.txt`.
|
||||
|
||||
## Cache
|
||||
|
||||
The HF Buckets resource uses `IndexCacheStore` with `index_ttl = 600`
|
||||
(10 minutes). Directory listings are cached and populate file-size/type
|
||||
entries that `stat` reads via a fast path, so a `readdir` followed by
|
||||
per-entry `stat` calls (which is what `ls`, FUSE `getattr`, and most
|
||||
shell commands trigger) costs one HTTP request instead of N.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_buckets import HfBucketsConfig, HfBucketsResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = HfBucketsConfig(
|
||||
bucket=os.environ["HF_BUCKET_NAME"],
|
||||
token=os.environ["HF_TOKEN"],
|
||||
)
|
||||
|
||||
resource = HfBucketsResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/hf/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /hf/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /hf/data/file.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /hf/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /hf/ -name '*.json'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("grep example /hf/data/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /hf/data/file.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The HF Buckets resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `rm` | Remove files |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing HF datasets**: Mount HF Buckets for agents to read and process datasets stored on the Hub
|
||||
- **Data pipelines**: Read and write HF bucket objects with shell-like commands
|
||||
- **Sandboxed bucket access**: Restrict agent operations to a specific bucket and prefix
|
||||
- **FUSE mounting**: Expose HF Buckets through a virtual FUSE mount for external tools
|
||||
|
||||
## Scoping a resource to a key prefix
|
||||
|
||||
Pass `key_prefix: str | None = None` to `HfBucketsConfig` to transparently scope every operation to a subpath of the bucket:
|
||||
|
||||
```python
|
||||
HfBucketsResource(HfBucketsConfig(
|
||||
bucket="your-user/app-data",
|
||||
token=hf_token,
|
||||
key_prefix=f"users/{user_id}/",
|
||||
))
|
||||
```
|
||||
|
||||
When set, every read/write/list/stat operation is transparently scoped to that bucket subpath. Agents see clean paths like `/data/notes.md`; the underlying bucket key is `users/{user_id}/data/notes.md`. Useful for multi-tenant systems.
|
||||
|
||||
**Normalization:** leading slashes are stripped and a trailing slash is added automatically. Both `None` and an empty string are treated as "no prefix."
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: HF Datasets
|
||||
icon: /images/huggingface-logo.svg
|
||||
---
|
||||
The HF Datasets resource mounts a [Hugging Face Dataset](https://huggingface.co/datasets)
|
||||
repo at some prefix such as `/ds/`.
|
||||
All reads are lazy: only the bytes you actually `cat`/`head` get transferred.
|
||||
|
||||
For credential setup, see [HF Datasets Setup](/home/setup/hf_datasets).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv add "mirage-ai[hf]"
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_datasets import HfDatasetsConfig, HfDatasetsResource
|
||||
|
||||
config = HfDatasetsConfig(
|
||||
repo_id=os.environ["HF_DATASET_REPO"], # "namespace/dataset-name"
|
||||
token=os.environ.get("HF_TOKEN"),
|
||||
# Optional:
|
||||
# endpoint="https://huggingface.co",
|
||||
# revision="main",
|
||||
# key_prefix="train/",
|
||||
)
|
||||
resource = HfDatasetsResource(config)
|
||||
ws = Workspace({"/ds": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`HfDatasetsConfig` takes `repo_id` in `namespace/dataset-name` form plus an
|
||||
optional access token. Public datasets need no token.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
Maps dataset repo files to virtual paths under the mount prefix.
|
||||
|
||||
For example, if dataset `AlienKevin/SWE-ZERO-12M-trajectories` contains:
|
||||
|
||||
```text
|
||||
README.md
|
||||
data/train-00000-of-01000.parquet
|
||||
data/train-00001-of-01000.parquet
|
||||
```
|
||||
|
||||
Then mounting at `/ds/` exposes:
|
||||
|
||||
```text
|
||||
/ds/
|
||||
README.md
|
||||
data/
|
||||
train-00000-of-01000.parquet
|
||||
train-00001-of-01000.parquet
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_datasets import HfDatasetsConfig, HfDatasetsResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = HfDatasetsConfig(
|
||||
repo_id=os.environ.get("HF_DATASET_REPO",
|
||||
"AlienKevin/SWE-ZERO-12M-trajectories"),
|
||||
token=os.environ.get("HF_TOKEN"),
|
||||
)
|
||||
resource = HfDatasetsResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/ds": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /ds/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /ds/README.md | head -n 20")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /ds/ -name '*.parquet' | head -n 5")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Same set as [HF Buckets](/python/resource/hf_buckets#shell-commands) — read,
|
||||
text-processing, file ops, path utilities, compression, encoding, and
|
||||
format-specific variants for parquet/feather/orc/hdf5.
|
||||
|
||||
## Cache
|
||||
|
||||
Uses `IndexCacheStore` with `index_ttl = 600` (10 minutes). Directory
|
||||
listings are cached and populate file-size/type entries for `stat`'s fast
|
||||
path, so a `readdir` + per-entry `stat` (which `ls`, FUSE `getattr`, and
|
||||
most shell commands trigger) costs one HTTP request instead of N.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents inspecting datasets**: Mount, browse the README, sample a few
|
||||
rows from parquet shards without downloading the whole dataset
|
||||
- **Dataset triage**: `ls`, `stat`, `find` to see what's in a repo before
|
||||
committing to a full local copy
|
||||
- **Sandboxed access**: Pin a `revision` for reproducibility
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: HF Models
|
||||
icon: /images/huggingface-logo.svg
|
||||
---
|
||||
The HF Models resource mounts a [Hugging Face Model](https://huggingface.co/models)
|
||||
repo at some prefix such as `/m/`.
|
||||
You can inspect configs, tokenizers, and READMEs without downloading the
|
||||
weights, weights stream only when you actually `cat` them.
|
||||
|
||||
For credential setup, see [HF Models Setup](/home/setup/hf_models).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv add "mirage-ai[hf]"
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_models import HfModelsConfig, HfModelsResource
|
||||
|
||||
config = HfModelsConfig(
|
||||
repo_id=os.environ["HF_MODEL_REPO"], # "namespace/model-name"
|
||||
token=os.environ.get("HF_TOKEN"),
|
||||
# Optional:
|
||||
# endpoint="https://huggingface.co",
|
||||
# revision="main",
|
||||
)
|
||||
resource = HfModelsResource(config)
|
||||
ws = Workspace({"/m": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
Maps model repo files (config, tokenizer, weights, etc.) to virtual paths.
|
||||
|
||||
For example, `sapientinc/HRM-Text-1B` mounted at `/m/` exposes:
|
||||
|
||||
```text
|
||||
/m/
|
||||
README.md
|
||||
config.json
|
||||
tokenizer.json
|
||||
tokenizer_config.json
|
||||
model.safetensors ← never downloaded unless you cat it
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_models import HfModelsConfig, HfModelsResource
|
||||
|
||||
config = HfModelsConfig(repo_id="sapientinc/HRM-Text-1B")
|
||||
resource = HfModelsResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/m": resource}, mode=MountMode.READ)
|
||||
|
||||
# ls is cheap: one HTTP list call
|
||||
r = await ws.execute("ls -lh /m/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read the config (small, fast)
|
||||
r = await ws.execute("cat /m/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Stat the weights without downloading them
|
||||
r = await ws.execute("stat /m/model.safetensors")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Same set as [HF Buckets](/python/resource/hf_buckets#shell-commands).
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Model card inspection**: Read configs, tokenizers, READMEs without
|
||||
pulling multi-GB weight files
|
||||
- **Compatibility checks**: `jq` over `config.json` to verify architecture
|
||||
before committing to a download
|
||||
- **Pinned revisions**: Mount a specific commit/tag for reproducibility
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: HF Spaces
|
||||
icon: /images/huggingface-logo.svg
|
||||
---
|
||||
The HF Spaces resource mounts a [Hugging Face Space](https://huggingface.co/spaces)
|
||||
repo (app code, README, config, requirements) at some prefix such as `/s/`.
|
||||
|
||||
For credential setup, see [HF Spaces Setup](/home/setup/hf_spaces).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv add "mirage-ai[hf]"
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_spaces import HfSpacesConfig, HfSpacesResource
|
||||
|
||||
config = HfSpacesConfig(
|
||||
repo_id=os.environ["HF_SPACE_REPO"], # "namespace/space-name"
|
||||
token=os.environ.get("HF_TOKEN"),
|
||||
# Optional:
|
||||
# endpoint="https://huggingface.co",
|
||||
# revision="main",
|
||||
)
|
||||
resource = HfSpacesResource(config)
|
||||
ws = Workspace({"/s": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
For example, `HuggingFaceBio/carbon-demo` mounted at `/s/` might expose:
|
||||
|
||||
```text
|
||||
/s/
|
||||
README.md
|
||||
app.py
|
||||
requirements.txt
|
||||
Dockerfile
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.hf_spaces import HfSpacesConfig, HfSpacesResource
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
resource = HfSpacesResource(
|
||||
HfSpacesConfig(repo_id="HuggingFaceBio/carbon-demo"))
|
||||
ws = Workspace({"/s": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /s/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /s/README.md | head -n 20")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("grep -l import /s/*.py 2>/dev/null")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Same set as [HF Buckets](/python/resource/hf_buckets#shell-commands).
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Space introspection**: Browse an app's structure, requirements, and
|
||||
config before forking/cloning
|
||||
- **App code review**: `grep` / `find` across a Space's Python without
|
||||
cloning the repo
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: Resources
|
||||
icon: grid-2
|
||||
description: Python resources you can mount in a Mirage workspace.
|
||||
---
|
||||
|
||||
Each page describes a Python-supported resource: what config it needs, what the mounted tree looks like, and any resource-specific behavior. Setup docs (credentials, OAuth, etc.) live under [Setup](/home/setup/s3), these pages focus on mounted layout and behavior.
|
||||
|
||||
## Infrastructure
|
||||
|
||||
- [Disk](/python/resource/disk)
|
||||
- [RAM](/python/resource/ram)
|
||||
- [Redis](/python/resource/redis)
|
||||
- [SSH](/python/resource/ssh)
|
||||
|
||||
## Object Storage
|
||||
|
||||
- [S3](/python/resource/s3)
|
||||
- [R2](/python/resource/r2)
|
||||
- [GCS](/python/resource/gcs)
|
||||
- [OCI](/python/resource/oci)
|
||||
- [Supabase](/python/resource/supabase)
|
||||
|
||||
## Cloud Files
|
||||
|
||||
- [Databricks Volume](/python/resource/databricks_volume)
|
||||
|
||||
## Google Workspace
|
||||
|
||||
- [Gmail](/python/resource/gmail)
|
||||
- [Google Drive](/python/resource/gdrive)
|
||||
- [Google Docs](/python/resource/gdocs)
|
||||
- [Google Sheets](/python/resource/gsheets)
|
||||
- [Google Slides](/python/resource/gslides)
|
||||
|
||||
## Code & DevOps
|
||||
|
||||
- [GitHub](/python/resource/github)
|
||||
- [GitHub CI](/python/resource/github_ci)
|
||||
- [Linear](/python/resource/linear)
|
||||
- [Langfuse](/python/resource/langfuse)
|
||||
|
||||
## Messaging
|
||||
|
||||
- [Slack](/python/resource/slack)
|
||||
- [Discord](/python/resource/discord)
|
||||
- [Email](/python/resource/email)
|
||||
|
||||
## Database
|
||||
|
||||
- [MongoDB](/python/resource/mongodb)
|
||||
- [Postgres](/python/resource/postgres)
|
||||
- [LanceDB](/python/resource/lancedb)
|
||||
- [Qdrant](/python/resource/qdrant)
|
||||
|
||||
## Knowledge
|
||||
|
||||
- [Dify](/python/resource/dify)
|
||||
- [Chroma](/python/resource/chroma)
|
||||
|
||||
## Notes
|
||||
|
||||
- [Notion](/python/resource/notion)
|
||||
|
||||
## Others
|
||||
|
||||
- [Trello](/python/resource/trello)
|
||||
- [Adding a New Resource](/python/resource/new)
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
title: LanceDB
|
||||
description: Mount a LanceDB table as a Mirage filesystem, with label folders, multimodal blobs, and a semantic search command for Python agents.
|
||||
icon: database
|
||||
---
|
||||
|
||||
The LanceDB resource exposes a LanceDB table as a virtual filesystem mounted at
|
||||
some prefix such as `/fashion/`. Group-by columns become nested folders, each
|
||||
row becomes a card plus an optional blob file, and semantic search is the
|
||||
`search` command, which returns ranked rows as canonical file paths.
|
||||
|
||||
For connection setup (LanceDB OSS, object storage, Cloud, Enterprise), see
|
||||
[LanceDB Setup](/python/setup/lancedb).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.lancedb import LanceDBConfig, LanceDBResource
|
||||
|
||||
config = LanceDBConfig(
|
||||
uri="/data/fashion.lancedb",
|
||||
table="fashion",
|
||||
group_by=["gender", "articleType", "baseColour"],
|
||||
id_column="id",
|
||||
title_column="productDisplayName",
|
||||
blob_column="image_bytes",
|
||||
blob_ext="jpg",
|
||||
vector_column="vector",
|
||||
search_limit=5,
|
||||
)
|
||||
resource = LanceDBResource(config)
|
||||
ws = Workspace({"/fashion/": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
The mapping is config-driven; nothing about the dataset is hardcoded. Point
|
||||
`group_by` at different columns and the folder tree changes. See the full
|
||||
[config reference](/python/setup/lancedb#config-reference).
|
||||
|
||||
## Filesystem layout
|
||||
|
||||
Every path is translated into a LanceDB query. Descending a folder adds one
|
||||
`WHERE` clause; the leaf level lists rows.
|
||||
|
||||
```text
|
||||
/ # list tables (omitted when `table` is pinned)
|
||||
/<table>/ # distinct group_by[0] values
|
||||
<v1>/ # distinct group_by[1] WHERE group_by[0]=v1
|
||||
.../<vN>/ # all group-by columns bound -> row files
|
||||
<id>.md # rendered card (text)
|
||||
<id>.<ext> # raw blob / image bytes
|
||||
```
|
||||
|
||||
When `table` is set the table level is elided, so the mount root is that table:
|
||||
|
||||
```text
|
||||
/fashion/
|
||||
Men/
|
||||
Shoes/
|
||||
White/
|
||||
3.md
|
||||
3.jpg
|
||||
```
|
||||
|
||||
### Row cards
|
||||
|
||||
A `<id>.md` card renders the row's columns as readable text and points at its
|
||||
blob. The vector and blob columns are omitted from the card body.
|
||||
|
||||
```text
|
||||
# Nike Men White Running Sneakers
|
||||
|
||||
id: 3
|
||||
gender: Men
|
||||
articleType: Shoes
|
||||
baseColour: White
|
||||
productDisplayName: Nike Men White Running Sneakers
|
||||
blob: 3.jpg
|
||||
```
|
||||
|
||||
## Semantic search
|
||||
|
||||
Search is a command, not a path. It returns each ranked row as its **canonical
|
||||
file path** (the same `<id>.md` you would `cat` while browsing) annotated with
|
||||
the vector distance, followed by the card body. Results point back at the real
|
||||
files, so search composes with `cat`, pipes, and `wc`:
|
||||
|
||||
```text
|
||||
$ search "white running sneakers" /fashion
|
||||
/fashion/Men/Shoes/White/3.md:0.2679
|
||||
# Nike Men White Running Sneakers
|
||||
|
||||
id: 3
|
||||
gender: Men
|
||||
articleType: Shoes
|
||||
baseColour: White
|
||||
productDisplayName: Nike Men White Running Sneakers
|
||||
blob: 3.jpg
|
||||
...
|
||||
```
|
||||
|
||||
Flags: `--top-k <n>` (default `search_limit`), `--threshold <max-distance>`,
|
||||
`--method semantic` (the only supported method; `grep`/`rg` stay lexical).
|
||||
|
||||
## Supported commands
|
||||
|
||||
All commands delegate to Mirage's shared implementations.
|
||||
|
||||
| Command | Behaviour on a LanceDB mount |
|
||||
| ------------- | ------------------------------------------------------------- |
|
||||
| `ls` | list tables, label folders, or row files |
|
||||
| `cd` | navigate (each level narrows the filter) |
|
||||
| `tree` | render the label hierarchy |
|
||||
| `cat` | print a row card, or dump raw blob/image bytes |
|
||||
| `stat` | directory vs file, blob size, image mime type |
|
||||
| `find` | walk the tree (e.g. `find /fashion -name '*.md'`) |
|
||||
| `grep` / `rg` | lexical search over the rendered cards |
|
||||
| `search` | semantic (vector) search -> ranked canonical paths + score |
|
||||
| `head` / `tail` | first/last lines of a card |
|
||||
| `wc` | count lines/bytes of a card |
|
||||
|
||||
`grep`/`rg` stay lexical (literal/regex). `search` is the semantic path: it
|
||||
auto-embeds the query via the table's embedding function and returns ranked
|
||||
rows as canonical file paths, which compose with `cat`, `wc`, and pipes.
|
||||
|
||||
## Access pattern
|
||||
|
||||
The mount is **read-only** (`MountMode.READ`); writes are not supported. The two
|
||||
read modes are:
|
||||
|
||||
- **Browse** by label folders: pure metadata `WHERE` filters, no embedding.
|
||||
- **Search** by meaning: `search "<query>" <path>` runs vector search using the
|
||||
table's embedding function and returns canonical row paths.
|
||||
|
||||
Folder listings scan one column with `SELECT DISTINCT` and are capped by
|
||||
`max_rows`, so very large tables should keep `group_by` to low-cardinality
|
||||
columns.
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
title: Langfuse
|
||||
description: Mount Langfuse traces, observations, prompts, datasets, and scores as a Mirage virtual filesystem.
|
||||
icon: chart-line
|
||||
---
|
||||
The Langfuse resource exposes LLM observability data (traces, observations,
|
||||
prompts, datasets, scores) as a virtual filesystem mounted at some prefix
|
||||
such as `/langfuse/`.
|
||||
|
||||
For credential setup, see [Langfuse Setup](/python/setup/langfuse).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.langfuse import LangfuseConfig, LangfuseResource
|
||||
|
||||
config = LangfuseConfig(
|
||||
public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
|
||||
secret_key=os.environ["LANGFUSE_SECRET_KEY"],
|
||||
host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com"),
|
||||
)
|
||||
resource = LangfuseResource(config=config)
|
||||
ws = Workspace({"/langfuse": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
| Config field | Required | Default | Description |
|
||||
| ---------------------- | -------- | ---------------------------- | ------------------------------------------------ |
|
||||
| `public_key` | yes | | Langfuse project public key |
|
||||
| `secret_key` | yes | | Langfuse project secret key |
|
||||
| `host` | no | `https://cloud.langfuse.com` | Langfuse API host URL |
|
||||
| `default_trace_limit` | no | 100 | Default trace/observation cap for listings |
|
||||
| `default_search_limit` | no | 50 | Default result cap for search at directory scope |
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/langfuse/
|
||||
traces/
|
||||
<trace-id>.json
|
||||
...
|
||||
sessions/
|
||||
<session-id>/
|
||||
<trace-id>.json
|
||||
...
|
||||
prompts/
|
||||
<prompt-name>/
|
||||
<version>.json
|
||||
...
|
||||
datasets/
|
||||
<dataset-name>/
|
||||
items.jsonl
|
||||
runs/
|
||||
<run-name>.jsonl
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/langfuse/
|
||||
traces/
|
||||
abc123.json
|
||||
def456.json
|
||||
sessions/
|
||||
chat-session-1/
|
||||
abc123.json
|
||||
ghi789.json
|
||||
prompts/
|
||||
summarize/
|
||||
1.json
|
||||
2.json
|
||||
classify/
|
||||
1.json
|
||||
datasets/
|
||||
eval-v1/
|
||||
items.jsonl
|
||||
runs/
|
||||
run-2026-04-01.jsonl
|
||||
```
|
||||
|
||||
### Traces
|
||||
|
||||
`/langfuse/traces/` lists recent traces (capped at `default_trace_limit`).
|
||||
Each trace is a `.json` file containing the full trace object with nested
|
||||
observations (spans, generations, events).
|
||||
|
||||
### Sessions
|
||||
|
||||
`/langfuse/sessions/` groups traces by session ID. Each session directory
|
||||
contains trace `.json` files belonging to that session.
|
||||
|
||||
### Prompts
|
||||
|
||||
`/langfuse/prompts/` lists prompt names as directories. Each prompt directory
|
||||
contains version files (`1.json`, `2.json`, etc.) with the prompt content
|
||||
and metadata.
|
||||
|
||||
### Datasets
|
||||
|
||||
`/langfuse/datasets/` lists dataset names as directories. Each contains
|
||||
`items.jsonl` (dataset items) and `runs/` with run results.
|
||||
|
||||
## Smart Commands
|
||||
|
||||
### grep at different scopes
|
||||
|
||||
```bash
|
||||
# FILE level - downloads the file, greps locally
|
||||
grep "error" "/langfuse/traces/abc123.json"
|
||||
|
||||
# TRACES level - uses Langfuse trace filter API (name, tags, metadata)
|
||||
grep "error" "/langfuse/traces/"
|
||||
|
||||
# SESSIONS level - uses Langfuse session listing
|
||||
grep "chat" "/langfuse/sessions/"
|
||||
|
||||
# PROMPTS level - filters prompt names
|
||||
grep "classify" "/langfuse/prompts/"
|
||||
|
||||
# ROOT level - searches across all resource types
|
||||
grep "error" "/langfuse/"
|
||||
```
|
||||
|
||||
At directory scope, the resource uses Langfuse's filter API instead of
|
||||
downloading data:
|
||||
|
||||
- **traces/**: filters by trace name, user ID, or tags
|
||||
- **sessions/**: filters by session listing
|
||||
- **prompts/**: filters by prompt name
|
||||
|
||||
### head / tail
|
||||
|
||||
`head` and `tail` on `traces/` use Langfuse API pagination with sort order
|
||||
instead of downloading all traces:
|
||||
|
||||
```bash
|
||||
# Returns 10 most recent traces
|
||||
head -n 10 "/langfuse/traces/"
|
||||
|
||||
# Returns 10 oldest traces
|
||||
tail -n 10 "/langfuse/traces/"
|
||||
```
|
||||
|
||||
## Limits
|
||||
|
||||
| Command | Limit behavior |
|
||||
| ------------------------ | ---------------------------------------------------- |
|
||||
| `ls traces/` | Returns up to `default_trace_limit` traces |
|
||||
| `cat traces/<id>.json` | Full trace (single doc, no limit needed) |
|
||||
| `grep` (directory level) | Server-side filter, capped at `default_search_limit` |
|
||||
| `wc -l traces/` | Not supported (Langfuse has no count endpoint) |
|
||||
|
||||
## Cache
|
||||
|
||||
The Langfuse resource uses `IndexCacheStore` (same as other resources).
|
||||
Index entries store trace IDs, session IDs, and prompt names.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.langfuse import LangfuseConfig, LangfuseResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = LangfuseConfig(
|
||||
public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
|
||||
secret_key=os.environ["LANGFUSE_SECRET_KEY"],
|
||||
)
|
||||
resource = LangfuseResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/langfuse": resource}, mode=MountMode.READ)
|
||||
|
||||
# List top-level resources
|
||||
r = await ws.execute("ls /langfuse/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List recent traces
|
||||
r = await ws.execute("ls /langfuse/traces/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a specific trace
|
||||
traces = r.stdout_str().strip().splitlines()
|
||||
if traces:
|
||||
r = await ws.execute(f'cat "/langfuse/traces/{traces[0]}"')
|
||||
print(r.stdout_str()[:500])
|
||||
|
||||
# List prompts
|
||||
r = await ws.execute("ls /langfuse/prompts/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search traces (uses Langfuse filter API)
|
||||
r = await ws.execute('grep "error" "/langfuse/traces/"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List datasets
|
||||
r = await ws.execute("ls /langfuse/datasets/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------------------ |
|
||||
| `ls` | List traces, sessions, prompts, datasets |
|
||||
| `cat` | Read trace JSON, prompt version, dataset items |
|
||||
| `head` / `tail` | Paginated trace listing with sort |
|
||||
| `grep` / `rg` | Smart: uses Langfuse filter API at directory scope |
|
||||
| `jq` | Query JSON trace/prompt data |
|
||||
| `stat` | Metadata (trace count, prompt versions, dataset items) |
|
||||
| `find` | Search across resources with `-name` |
|
||||
| `tree` | Directory tree view |
|
||||
@@ -0,0 +1,392 @@
|
||||
---
|
||||
title: Linear
|
||||
description: Mount Linear workspaces, teams, projects, issues, and comments as a Mirage filesystem for Python agents.
|
||||
icon: chart-gantt
|
||||
---
|
||||
The Linear resource exposes Linear workspace data as a virtual filesystem
|
||||
mounted at some prefix such as `/linear/`.
|
||||
|
||||
For API key setup, see [Linear Setup](/python/setup/linear).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.linear import LinearConfig, LinearResource
|
||||
|
||||
config = LinearConfig(api_key=os.environ["LINEAR_API_KEY"])
|
||||
resource = LinearResource(config=config)
|
||||
ws = Workspace({"/linear": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/linear/
|
||||
teams/
|
||||
<team-key>__<team-name>__<team-id>/
|
||||
team.json
|
||||
members/
|
||||
<display-name>__<user-id>.json
|
||||
...
|
||||
issues/
|
||||
<issue-key>__<issue-id>/
|
||||
issue.json
|
||||
comments.jsonl
|
||||
...
|
||||
projects/
|
||||
<name>__<project-id>.json
|
||||
...
|
||||
cycles/
|
||||
<name>__<cycle-id>.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/linear/
|
||||
teams/
|
||||
ENG__Engineering__abc123def/
|
||||
team.json
|
||||
members/
|
||||
alice__usr_001.json
|
||||
bob__usr_002.json
|
||||
issues/
|
||||
ENG-123__iss_001/
|
||||
issue.json
|
||||
comments.jsonl
|
||||
ENG-124__iss_002/
|
||||
issue.json
|
||||
comments.jsonl
|
||||
projects/
|
||||
Mirage__proj_001.json
|
||||
cycles/
|
||||
Sprint-12__cyc_001.json
|
||||
```
|
||||
|
||||
Directory and file names embed the Linear ID after `__` so that
|
||||
resource-specific commands can reference the correct resource without
|
||||
extra lookups.
|
||||
|
||||
### Teams
|
||||
|
||||
Each team directory is named:
|
||||
|
||||
```text
|
||||
<team-key>__<team-name>__<team-id>
|
||||
```
|
||||
|
||||
Inside the team directory:
|
||||
|
||||
- `team.json` is the normalized team metadata
|
||||
- `members/` contains one JSON file per member
|
||||
- `issues/` contains one directory per issue
|
||||
- `projects/` contains one JSON file per project, including lightweight
|
||||
related issue references
|
||||
- `cycles/` contains one JSON file per cycle
|
||||
|
||||
### Issues
|
||||
|
||||
Each issue directory is named:
|
||||
|
||||
```text
|
||||
<issue-key>__<issue-id>
|
||||
```
|
||||
|
||||
Each issue directory contains:
|
||||
|
||||
- `issue.json` -- normalized issue metadata with command-aligned fields such as
|
||||
`issue_id`, `issue_key`, `team_id`, `state_id`, and `assignee_id`
|
||||
- `comments.jsonl` -- normalized comment stream ordered by `created_at`
|
||||
|
||||
`comments.jsonl` is a Mirage representation chosen for shell-friendly
|
||||
workflows. It is not a native Linear file format.
|
||||
|
||||
### Members
|
||||
|
||||
Each member file is named:
|
||||
|
||||
```text
|
||||
<display-name>__<user-id>.json
|
||||
```
|
||||
|
||||
Member JSON includes command-aligned identifiers such as `user_id` and `email`
|
||||
so the value can be reused directly in commands like `linear-issue-assign`.
|
||||
|
||||
### Projects and Cycles
|
||||
|
||||
Project and cycle files are named:
|
||||
|
||||
```text
|
||||
<name>__<id>.json
|
||||
```
|
||||
|
||||
Project JSON includes lightweight related issue references. Cycle JSON
|
||||
includes cycle metadata such as start and end dates.
|
||||
|
||||
## Cache
|
||||
|
||||
The Linear resource uses `IndexCacheStore` (same as Discord and other
|
||||
resources). There is no separate content cache -- file content caching
|
||||
is handled by the workspace `IOResult` mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
See `examples/linear/linear.py` for the full working example.
|
||||
|
||||
```bash
|
||||
# List teams
|
||||
ls /linear/teams/
|
||||
|
||||
# Read team metadata
|
||||
cat /linear/teams/ENG__Engineering__abc123/team.json
|
||||
|
||||
# List issues
|
||||
ls /linear/teams/ENG__Engineering__abc123/issues/
|
||||
|
||||
# Read an issue
|
||||
cat /linear/teams/ENG__Engineering__abc123/issues/ENG-123__iss_001/issue.json
|
||||
|
||||
# Read issue comments
|
||||
cat /linear/teams/ENG__Engineering__abc123/issues/ENG-123__iss_001/comments.jsonl
|
||||
|
||||
# Extract issue ID with jq
|
||||
cat /linear/teams/ENG__Engineering__abc123/issues/ENG-123__iss_001/issue.json \
|
||||
| jq '.issue_id'
|
||||
|
||||
# Read last 5 comments
|
||||
tail -n 5 /linear/teams/ENG__Engineering__abc123/issues/ENG-123__iss_001/comments.jsonl
|
||||
|
||||
# List members
|
||||
ls /linear/teams/ENG__Engineering__abc123/members/
|
||||
|
||||
# Read a project with its issues
|
||||
cat /linear/teams/ENG__Engineering__abc123/projects/Mirage__proj_001.json \
|
||||
| jq '.issues'
|
||||
|
||||
# Tree view
|
||||
tree -L 2 /linear/teams/
|
||||
```
|
||||
|
||||
## Finding IDs
|
||||
|
||||
IDs are embedded in directory and file names after `__`:
|
||||
|
||||
```bash
|
||||
# Team ID -- embedded in directory name
|
||||
ls /linear/teams/
|
||||
# -> ENG__Engineering__abc123 <- team_id = abc123
|
||||
|
||||
# Issue ID -- embedded in issue directory name
|
||||
ls /linear/teams/ENG__Engineering__abc123/issues/
|
||||
# -> ENG-123__iss_001 <- issue_id = iss_001
|
||||
|
||||
# Member ID -- embedded in member file name
|
||||
ls /linear/teams/ENG__Engineering__abc123/members/
|
||||
# -> alice__usr_001.json <- user_id = usr_001
|
||||
|
||||
# Use stat for structured metadata
|
||||
stat /linear/teams/ENG__Engineering__abc123/issues/ENG-123__iss_001/issue.json
|
||||
|
||||
# Extract fields with jq
|
||||
cat /linear/teams/ENG__Engineering__abc123/issues/ENG-123__iss_001/issue.json \
|
||||
| jq '{issue_id, issue_key, team_id, state_id, assignee_id}'
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Linear tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List teams, issues, members, projects |
|
||||
| `cat` | Read .json metadata or .jsonl comments |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` | Extract file name from path |
|
||||
| `dirname` | Extract directory from path |
|
||||
| `realpath` | Resolve path to absolute form |
|
||||
|
||||
## Resource-Specific Commands
|
||||
|
||||
### `linear-issue-create`
|
||||
|
||||
Create a new issue. Description can be passed via `--description`, `--description_file`, or stdin.
|
||||
|
||||
```bash
|
||||
linear-issue-create --team_key ENG --title "New bug"
|
||||
linear-issue-create --team_id abc123 --title "New bug" --description "Details here"
|
||||
cat brief.md | linear-issue-create --team_key ENG --title "Agent bug"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------------- | -------- | ----------------------------------- |
|
||||
| `--team_id` | \* | Linear team ID |
|
||||
| `--team_key` | \* | Linear team key (e.g. ENG) |
|
||||
| `--title` | yes | Issue title |
|
||||
| `--description` | no | Inline description text |
|
||||
| `--description_file` | no | Path to file containing description |
|
||||
|
||||
\* One of `--team_id` or `--team_key` is required.
|
||||
|
||||
### `linear-issue-update`
|
||||
|
||||
Update an existing issue. Description can be passed via `--description`, `--description_file`, or stdin.
|
||||
|
||||
```bash
|
||||
linear-issue-update --issue_key ENG-123 --title "Updated title"
|
||||
linear-issue-update --issue_id iss_001 --description "New description"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------------- | -------- | ----------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--title` | no | New title |
|
||||
| `--description` | no | Inline description text |
|
||||
| `--description_file` | no | Path to file containing description |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
|
||||
### `linear-issue-assign`
|
||||
|
||||
Assign an issue to a user.
|
||||
|
||||
```bash
|
||||
linear-issue-assign --issue_key ENG-123 --assignee_email "alice@example.com"
|
||||
linear-issue-assign --issue_id iss_001 --assignee_id usr_001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------------ | -------- | ------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--assignee_id` | \*\* | Linear user ID |
|
||||
| `--assignee_email` | \*\* | User email address |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
\*\* One of `--assignee_id` or `--assignee_email` is required.
|
||||
|
||||
### `linear-issue-transition`
|
||||
|
||||
Transition an issue to a different workflow state.
|
||||
|
||||
```bash
|
||||
linear-issue-transition --issue_key ENG-123 --state_name "In Progress"
|
||||
linear-issue-transition --issue_id iss_001 --state_id state_001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--state_id` | \*\* | Linear workflow state ID |
|
||||
| `--state_name` | \*\* | Workflow state name |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
\*\* One of `--state_id` or `--state_name` is required.
|
||||
|
||||
### `linear-issue-set-priority`
|
||||
|
||||
Set the priority of an issue.
|
||||
|
||||
```bash
|
||||
linear-issue-set-priority --issue_key ENG-123 --priority 1
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ---------------------------------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--priority` | yes | Priority level (0=none, 1=urgent, 2=high, 3=medium, 4=low) |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
|
||||
### `linear-issue-set-project`
|
||||
|
||||
Assign an issue to a project.
|
||||
|
||||
```bash
|
||||
linear-issue-set-project --issue_key ENG-123 --project_id proj_001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--project_id` | yes | Linear project ID |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
|
||||
### `linear-issue-add-label`
|
||||
|
||||
Add a label to an issue.
|
||||
|
||||
```bash
|
||||
linear-issue-add-label --issue_key ENG-123 --label_id lbl_001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--label_id` | yes | Linear label ID |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
|
||||
### `linear-issue-comment-add`
|
||||
|
||||
Add a comment to an issue. Body can be passed via `--body`, `--body_file`, or stdin.
|
||||
|
||||
```bash
|
||||
linear-issue-comment-add --issue_key ENG-123 --body "Looks good"
|
||||
cat note.md | linear-issue-comment-add --issue_key ENG-123
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ------------------------------- |
|
||||
| `--issue_id` | \* | Linear issue ID |
|
||||
| `--issue_key` | \* | Linear issue key (e.g. ENG-123) |
|
||||
| `--body` | \*\* | Inline comment text |
|
||||
| `--body_file` | \*\* | Path to file containing body |
|
||||
|
||||
\* One of `--issue_id` or `--issue_key` is required.
|
||||
\*\* One of `--body`, `--body_file`, or stdin is required.
|
||||
|
||||
### `linear-issue-comment-update`
|
||||
|
||||
Update an existing comment. Body can be passed via `--body`, `--body_file`, or stdin.
|
||||
|
||||
```bash
|
||||
linear-issue-comment-update --comment_id cmt_001 --body "Updated text"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ---------------------------- |
|
||||
| `--comment_id` | yes | Linear comment ID |
|
||||
| `--body` | \*\* | Inline comment text |
|
||||
| `--body_file` | \*\* | Path to file containing body |
|
||||
|
||||
\*\* One of `--body`, `--body_file`, or stdin is required.
|
||||
|
||||
### `linear-search`
|
||||
|
||||
Search issues across the workspace.
|
||||
|
||||
```bash
|
||||
linear-search --query "authentication bug"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------- | -------- | ----------------- |
|
||||
| `--query` | yes | Search query text |
|
||||
|
||||
Returns matching issues as a JSON array.
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: MinIO
|
||||
icon: /images/minio-logo.svg
|
||||
description: Mount a self-hosted MinIO bucket as a virtual filesystem.
|
||||
---
|
||||
The MinIO resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `MinIOConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just points at your MinIO `endpoint_url` and
|
||||
uses path-style addressing by default.
|
||||
|
||||
MinIO is self-hosted, so `endpoint_url` is **required** (there is no
|
||||
region-derived host). Uses aioboto3 against MinIO's S3-compatible API.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.minio import MinIOConfig, MinIOResource
|
||||
|
||||
config = MinIOConfig(
|
||||
bucket=os.environ["MINIO_BUCKET"],
|
||||
endpoint_url=os.environ.get("MINIO_ENDPOINT", "http://localhost:9000"),
|
||||
access_key_id=os.environ["MINIO_ACCESS_KEY"],
|
||||
secret_access_key=os.environ["MINIO_SECRET_KEY"],
|
||||
# Optional:
|
||||
# region="us-east-1", # default
|
||||
# path_style=True, # default; MinIO requires path-style addressing
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = MinIOResource(config)
|
||||
ws = Workspace({"/minio": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.minio import MinIOConfig, MinIOResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = MinIOConfig(
|
||||
bucket=os.environ.get("MINIO_BUCKET", "mirage-demo"),
|
||||
endpoint_url=os.environ.get("MINIO_ENDPOINT", "http://localhost:9000"),
|
||||
access_key_id=os.environ.get("MINIO_ACCESS_KEY", "minioadmin"),
|
||||
secret_access_key=os.environ.get("MINIO_SECRET_KEY", "minioadmin"),
|
||||
)
|
||||
resource = MinIOResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/minio/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /minio/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /minio/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /minio/data/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- MinIO reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies (`ls`, `cat`,
|
||||
`head`, `tail`, `grep`, `rg`, `wc`, `find`, `tree`, `jq`, `stat`, plus
|
||||
parquet/orc/feather table rendering). See the [S3 resource](/python/resource/s3)
|
||||
for the complete command reference, range reads, streaming, and the index
|
||||
cache fast path.
|
||||
- For credential setup, see [MinIO Setup](/home/setup/minio).
|
||||
@@ -0,0 +1,351 @@
|
||||
---
|
||||
title: MongoDB
|
||||
description: Mount MongoDB databases, collections, and documents as a Mirage filesystem with JSON access for Python agents.
|
||||
icon: database
|
||||
---
|
||||
The MongoDB resource exposes MongoDB databases, collections, and documents
|
||||
as a virtual filesystem mounted at some prefix such as `/mongodb/`.
|
||||
|
||||
For connection setup, see [MongoDB Setup](/python/setup/mongodb).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.mongodb import MongoDBConfig, MongoDBResource
|
||||
|
||||
config = MongoDBConfig(
|
||||
uri=os.environ["MONGODB_URI"],
|
||||
default_doc_limit=1000,
|
||||
default_search_limit=100,
|
||||
max_doc_limit=5000,
|
||||
elide_fields={"sample_mflix.movies": ["plot_embedding"]},
|
||||
)
|
||||
resource = MongoDBResource(config=config)
|
||||
ws = Workspace({"/mongodb": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
| Config field | Required | Default | Description |
|
||||
| ---------------------- | -------- | ------- | -------------------------------------------------------------------------- |
|
||||
| `uri` | yes | | MongoDB connection URI |
|
||||
| `databases` | no | | List of database names to mount (omit for all) |
|
||||
| `default_doc_limit` | no | 1000 | Default doc cap for one-shot reads (not used by streaming `cat`) |
|
||||
| `default_search_limit` | no | 100 | Default result cap for collection/db-level `grep` |
|
||||
| `max_doc_limit` | no | 5000 | Hard cap for `head -n K` / `tail -n K` |
|
||||
| `elide_fields` | no | `{}` | `{"<db>.<coll>": ["field", "nested.path"]}`; listed fields are dropped from `documents.jsonl` |
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The mount mirrors MongoDB's `cluster → database → collection / view`
|
||||
model. The database directory always appears in the path, even when
|
||||
`databases` filters to a single entry.
|
||||
|
||||
```text
|
||||
/mongodb/
|
||||
<database>/
|
||||
database.json # collections + views + counts
|
||||
collections/
|
||||
<collection>/
|
||||
schema.json # sampled types + indexes + validator
|
||||
documents.jsonl # streamed BSON Extended JSON
|
||||
views/
|
||||
<view>/
|
||||
schema.json # view-aware (no indexes / validator)
|
||||
documents.jsonl # streamed, same format as collections
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/mongodb/
|
||||
sample_mflix/
|
||||
database.json
|
||||
collections/
|
||||
movies/
|
||||
schema.json
|
||||
documents.jsonl
|
||||
comments/
|
||||
schema.json
|
||||
documents.jsonl
|
||||
views/
|
||||
top_rated_movies/
|
||||
schema.json
|
||||
documents.jsonl
|
||||
```
|
||||
|
||||
### documents.jsonl
|
||||
|
||||
One JSON object per line, encoded with BSON
|
||||
[Relaxed Extended JSON](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/).
|
||||
BSON-specific types round-trip through their canonical `$` wrappers:
|
||||
|
||||
```json
|
||||
{"_id":{"$oid":"573a1390f29313caabcd4135"},"title":"Casablanca","released":{"$date":"1943-01-23T00:00:00Z"},"runtime":102}
|
||||
```
|
||||
|
||||
`cat` / `head` / `tail` / `grep` / `jq` all stream from this file; nothing
|
||||
materializes the full collection in memory.
|
||||
|
||||
Because the file is rendered on demand, `stat` / `ls -l` report no size and
|
||||
`du` returns `0` for it (computing the real size would require rendering the
|
||||
whole collection). Use `wc -c documents.jsonl` for the actual rendered byte
|
||||
count; `stat` still exposes `document_count` for the number of documents.
|
||||
|
||||
### schema.json
|
||||
|
||||
Generated from a 100-document `$sample`. Includes:
|
||||
|
||||
- field path → observed BSON type frequencies (nested paths unioned across the sample)
|
||||
- indexes from `listIndexes` plus access counts from `$indexStats`
|
||||
- the `$jsonSchema` validator if one is registered
|
||||
|
||||
Views skip the index and validator sections.
|
||||
|
||||
### database.json
|
||||
|
||||
Lists every collection and view under the database with their document
|
||||
counts. Useful for `cat /mongodb/<db>/database.json` to get an overview
|
||||
without recursing into each entity.
|
||||
|
||||
### Elided fields
|
||||
|
||||
Fields listed under `elide_fields` are dropped entirely from
|
||||
`documents.jsonl` output. The type stays documented in `schema.json`, so
|
||||
heavy fields (embeddings, large binary, raw text blobs) can be hidden
|
||||
from agent reads without losing the schema signal:
|
||||
|
||||
```python
|
||||
config = MongoDBConfig(
|
||||
uri=os.environ["MONGODB_URI"],
|
||||
elide_fields={
|
||||
"sample_mflix.movies": ["plot_embedding"],
|
||||
"rag.docs": ["metadata.embedding"],
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Nested paths use dot notation. Elision applies to both `cat` (one-shot
|
||||
streaming reads) and `tail -f` (live change-stream follows).
|
||||
|
||||
## Streaming and Limits
|
||||
|
||||
`cat`, `grep`, `head`, and `tail -f` consume documents lazily through a
|
||||
batched PyMongo async cursor (or change stream); the consumer cancels to stop
|
||||
fetching. There is no truncation notice because nothing is forced into
|
||||
memory ahead of the consumer.
|
||||
|
||||
| Command | Behavior |
|
||||
| ---------------------------- | ----------------------------------------------------------------- |
|
||||
| `cat` | Streams the whole collection sorted by `_id`; pipe to `head` to cap |
|
||||
| `head -n K` / `tail -n K` | K is capped at `max_doc_limit`; server-side sort + limit |
|
||||
| `tail -f` | Opens a change stream; yields each new insert as a JSONL line |
|
||||
| `grep` (file level) | Streams from `documents.jsonl`; supports `-m` for short-circuit |
|
||||
| `grep` (collection/db level) | Server-side query, capped at `default_search_limit` |
|
||||
| `jq` | Inherits the streaming `cat` source |
|
||||
| `wc` | Uses `countDocuments()` server-side; zero download |
|
||||
| `stat` | Metadata only (doc count, indexes; views skip the index lookup) |
|
||||
|
||||
## Smart Commands
|
||||
|
||||
### grep at different scopes
|
||||
|
||||
`grep` uses MongoDB's query engine at directory scopes instead of
|
||||
streaming all documents through the regex pipeline:
|
||||
|
||||
```bash
|
||||
# FILE level - streams documents.jsonl, runs the regex locally
|
||||
grep "Godfather" "/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
|
||||
# COLLECTION level - server-side query against the collection
|
||||
grep "Godfather" "/mongodb/sample_mflix/collections/movies/"
|
||||
|
||||
# DATABASE level - searches across every collection in sample_mflix
|
||||
grep "Godfather" "/mongodb/sample_mflix/"
|
||||
|
||||
# ROOT level - fans out across every mounted database
|
||||
grep "Godfather" "/mongodb/"
|
||||
```
|
||||
|
||||
At collection or higher scope the resource picks the best server-side
|
||||
strategy from the indexes available:
|
||||
|
||||
1. **Text index exists** → uses `$text` (ranked by relevance)
|
||||
1. **Atlas Search index exists** → uses `$search` (fuzzy, Lucene-based)
|
||||
1. **Neither** → falls back to `$regex` on sampled string fields
|
||||
|
||||
Scope detection is handled by `mirage/core/mongodb/scope.py`.
|
||||
|
||||
### head / tail / tail -f
|
||||
|
||||
`head` and `tail` use server-side `sort` + `limit`; the requested count
|
||||
is capped at `max_doc_limit`. `tail -f` opens a Mongo change stream
|
||||
filtered to insert events and yields each new document as a JSONL line
|
||||
in the same format as `cat`:
|
||||
|
||||
```bash
|
||||
# First 10 docs (sorted by _id ascending)
|
||||
head -n 10 "/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
|
||||
# Last 10 docs (sorted by _id descending)
|
||||
tail -n 10 "/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
|
||||
# Live-follow new inserts; consumer cancels to stop
|
||||
tail -f "/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
```
|
||||
|
||||
`tail -f` requires the cluster to be a replica set; Atlas already
|
||||
satisfies this. Views fall through to the non-streaming path because
|
||||
change streams aren't defined on views.
|
||||
|
||||
## Cache
|
||||
|
||||
The MongoDB resource uses `IndexCacheStore` (same as RAM/S3/disk/GitHub)
|
||||
for listings: database names, collection names, and document counts.
|
||||
|
||||
Document content is **not** cached. The resource leaves `caches_reads`
|
||||
at its default of `False`, so `cat`, `grep`, `head`, and `tail` always
|
||||
query the live collection
|
||||
instead of serving a stored snapshot. This keeps reads consistent with a
|
||||
mutable database and ensures `tail -f` follows the live change stream
|
||||
rather than replaying cached bytes.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.mongodb import MongoDBConfig, MongoDBResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = MongoDBConfig(uri=os.environ["MONGODB_URI"])
|
||||
resource = MongoDBResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/mongodb": resource}, mode=MountMode.READ)
|
||||
|
||||
# List all databases
|
||||
r = await ws.execute("ls /mongodb/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List the entities under a database (database.json, collections/, views/)
|
||||
r = await ws.execute("ls /mongodb/sample_mflix/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List collections
|
||||
r = await ws.execute("ls /mongodb/sample_mflix/collections/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read first 5 movies
|
||||
r = await ws.execute(
|
||||
'head -n 5 "/mongodb/sample_mflix/collections/movies/documents.jsonl"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read last 5 movies
|
||||
r = await ws.execute(
|
||||
'tail -n 5 "/mongodb/sample_mflix/collections/movies/documents.jsonl"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Inspect the sampled schema and indexes
|
||||
r = await ws.execute(
|
||||
'cat "/mongodb/sample_mflix/collections/movies/schema.json"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract titles with jq
|
||||
r = await ws.execute(
|
||||
'jq -r ".[] | .title" "/mongodb/sample_mflix/collections/movies/documents.jsonl"'
|
||||
)
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across a database (uses MongoDB query engine)
|
||||
r = await ws.execute('grep "Godfather" "/mongodb/sample_mflix/"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Count documents (server-side, no download)
|
||||
r = await ws.execute(
|
||||
'wc -l "/mongodb/sample_mflix/collections/movies/documents.jsonl"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# View the database overview without recursing
|
||||
r = await ws.execute('cat "/mongodb/sample_mflix/database.json"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Runnable examples
|
||||
|
||||
Three working examples live under `examples/python/mongodb/`:
|
||||
|
||||
- [`mongodb.py`](https://github.com/StruktoAI/mirage/blob/main/examples/python/mongodb/mongodb.py) — agent-shell workflow: `ls`, `tree`, `cat`, `head`, `tail`, `wc`, `stat`, `grep`/`rg` at every scope, `jq`, `find`, `cd` + relative paths.
|
||||
- [`mongodb_vfs.py`](https://github.com/StruktoAI/mirage/blob/main/examples/python/mongodb/mongodb_vfs.py) — in-process VFS: `os.listdir` and `open()` walk every readdir level (root, database, `collections/`, `views/`, entity) and read `database.json`, `schema.json`, `documents.jsonl` (collection + view).
|
||||
- [`mongodb_fuse.py`](https://github.com/StruktoAI/mirage/blob/main/examples/python/mongodb/mongodb_fuse.py) — same coverage as the VFS example, but the tree is mounted as a real filesystem so other processes can `cat`/`ls`/`head` the mountpoint directly.
|
||||
|
||||
All three default to the `mirage_test` database seeded by `python/scripts/seed_mongodb_test.py`.
|
||||
|
||||
## Finding IDs
|
||||
|
||||
`_id` is serialized as `{"$oid": "..."}` under Extended JSON:
|
||||
|
||||
```bash
|
||||
# List the first 10 ObjectId values
|
||||
jq -r '.[] | ._id["$oid"]' \
|
||||
"/mongodb/sample_mflix/collections/movies/documents.jsonl" | head -n 10
|
||||
|
||||
# Find a specific document by ID
|
||||
grep "573a1390f29313caabcd42e8" \
|
||||
"/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
|
||||
# Extract a few fields together
|
||||
jq -r '.[] | "\(._id["$oid"]) \(.title) \(.year)"' \
|
||||
"/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
```
|
||||
|
||||
## Working with Large Collections
|
||||
|
||||
Streaming is the default; reach for these patterns when you want to keep
|
||||
the round trip small:
|
||||
|
||||
```bash
|
||||
# Document count (server-side, no download)
|
||||
wc -l "/mongodb/sample_mflix/collections/comments/documents.jsonl"
|
||||
|
||||
# Most recent documents (sorted by _id desc)
|
||||
tail -n 10 "/mongodb/sample_mflix/collections/comments/documents.jsonl"
|
||||
|
||||
# Server-side query at collection scope avoids streaming the whole jsonl
|
||||
grep "love" "/mongodb/sample_mflix/collections/comments/"
|
||||
|
||||
# Drop heavy fields via elide_fields, then take the slice you want
|
||||
head -n 20 "/mongodb/sample_mflix/collections/movies/documents.jsonl"
|
||||
```
|
||||
|
||||
Hide embeddings or large blobs from agent reads with `elide_fields`;
|
||||
`schema.json` still documents the original type so the agent can decide
|
||||
when to ask for the raw bytes through a different path.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted MongoDB tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ----------------------------------------------------------------------- |
|
||||
| `ls` | List databases, collections, views, and per-entity files |
|
||||
| `cat` | Stream `documents.jsonl` (or read `schema.json` / `database.json`) |
|
||||
| `head` / `tail` | Smart: server-side sort + limit (capped at `max_doc_limit`) |
|
||||
| `tail -f` | Live-follow new inserts via Mongo change stream |
|
||||
| `grep` / `rg` | Smart: MongoDB query engine at collection/db scope, regex at file scope |
|
||||
| `jq` | Query JSON; use `.[]` prefix when iterating JSONL files |
|
||||
| `wc` | Smart: uses `countDocuments()` server-side |
|
||||
| `stat` | Metadata (doc count, indexes; views skip the index lookup) |
|
||||
| `find` | List databases/collections/views with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
@@ -0,0 +1,218 @@
|
||||
---
|
||||
title: Adding a New Resource
|
||||
icon: plus
|
||||
description: Step-by-step guide for implementing a new MIRAGE resource.
|
||||
---
|
||||
|
||||
This guide covers what you need to implement when adding a new resource
|
||||
to MIRAGE. Use the Discord or Slack resources as reference.
|
||||
|
||||
## File Structure
|
||||
|
||||
```text
|
||||
mirage/
|
||||
resource/<name>/
|
||||
__init__.py # lazy-loading exports
|
||||
config.py # Pydantic config (credentials)
|
||||
<name>.py # BaseResource subclass
|
||||
accessor/<name>.py # Accessor wrapping config
|
||||
core/<name>/
|
||||
__init__.py
|
||||
_client.py # HTTP client (get/post with rate limiting)
|
||||
readdir.py # directory listing
|
||||
read.py # file reading
|
||||
stat.py # file metadata
|
||||
scope.py # scope detection
|
||||
glob.py # glob pattern resolution
|
||||
... # data fetching modules (history, search, post, etc.)
|
||||
ops/<name>/
|
||||
__init__.py # exports OPS list
|
||||
read.py # @op wrapper
|
||||
readdir.py # @op wrapper
|
||||
stat.py # @op wrapper
|
||||
commands/builtin/<name>/
|
||||
__init__.py # exports COMMANDS list
|
||||
_plan.py # cost estimation helpers
|
||||
cat.py, ls.py, ... # standard commands
|
||||
<name>_send.py, ... # resource-specific commands
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### 1. Config, Accessor, ResourceName
|
||||
|
||||
Create a Pydantic config model to hold credentials and an accessor
|
||||
class that wraps it.
|
||||
|
||||
```python
|
||||
# mirage/resource/<name>/config.py
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class MyConfig(BaseModel):
|
||||
token: str
|
||||
```
|
||||
|
||||
```python
|
||||
# mirage/accessor/<name>.py
|
||||
from mirage.accessor.base import Accessor
|
||||
from mirage.resource.<name>.config import MyConfig
|
||||
|
||||
|
||||
class MyAccessor(Accessor):
|
||||
|
||||
def __init__(self, config: MyConfig) -> None:
|
||||
self.config = config
|
||||
```
|
||||
|
||||
Add `MY_RESOURCE = "<name>"` to the `ResourceName` enum in `mirage/types.py`.
|
||||
|
||||
### 2. HTTP Client
|
||||
|
||||
Wrap the resource's API with rate-limit handling. All resources follow
|
||||
the same pattern: async get/post functions with retry on 429.
|
||||
|
||||
```python
|
||||
# mirage/core/<name>/_client.py
|
||||
async def my_get(config, endpoint, params=None) -> dict: ...
|
||||
async def my_post(config, endpoint, body=None) -> dict: ...
|
||||
```
|
||||
|
||||
### 3. Core VFS
|
||||
|
||||
Implement the three VFS operations that map API data to a filesystem:
|
||||
|
||||
- **`readdir.py`** -- Returns `list[str]` of child paths for a directory.
|
||||
- **`read.py`** -- Returns `bytes` content of a file.
|
||||
- **`stat.py`** -- Returns `FileStat` with name, type, and extras (e.g., IDs).
|
||||
|
||||
All three accept `(accessor, path, index, prefix)` and use `IndexCacheStore`
|
||||
to cache name-to-ID mappings.
|
||||
|
||||
### 4. Scope Detection and GlobScope Optimization
|
||||
|
||||
`GlobScope` carries the raw path and pattern **before** expansion. This
|
||||
lets commands decide how to resolve paths efficiently -- skipping
|
||||
expensive glob expansion when the resource has a native API for the
|
||||
operation.
|
||||
|
||||
**`scope.py`** parses the unexpanded path to determine the level:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class MyScope:
|
||||
level: str # "root", "category", "item", "file"
|
||||
item_id: str | None = None
|
||||
```
|
||||
|
||||
**How commands use scope for optimization:**
|
||||
|
||||
```python
|
||||
# Example: grep at different scopes
|
||||
|
||||
@command("grep", resource="my_resource", spec=SPECS["grep"])
|
||||
async def grep(accessor, paths, *texts, **_extra):
|
||||
pattern = texts[0]
|
||||
scope = detect_scope(paths[0], index)
|
||||
|
||||
if scope.level in ("category", "item"):
|
||||
# CHEAP: use native search API (1 API call)
|
||||
results = await search_api(accessor.config, scope.item_id, pattern)
|
||||
return format_results(results), IOResult()
|
||||
|
||||
# EXPENSIVE fallback: expand glob, download each file, grep locally
|
||||
paths = await resolve_glob(accessor, paths, index=index)
|
||||
for p in paths:
|
||||
data = await read(accessor, p.original, index, prefix=p.prefix)
|
||||
# ... grep the bytes ...
|
||||
```
|
||||
|
||||
**When to use this pattern:**
|
||||
|
||||
| Scenario | Approach |
|
||||
| ------------------------------------------ | --------------------------------------------------------- |
|
||||
| Resource has a search API (Discord, Slack) | Use scope to route to native API at directory level |
|
||||
| Resource has no search API | Always fall through to file-level reads; scope is a noop |
|
||||
| `head`/`tail` with direct message fetch | Use scope to detect file level, fetch N messages directly |
|
||||
|
||||
**When the resource has no search API**, keep `scope.py` as a noop for
|
||||
structural consistency. The file still parses path parts but does not
|
||||
trigger any API calls:
|
||||
|
||||
```python
|
||||
# Noop scope -- no search API, no resource-relative paths
|
||||
def detect_scope(path: str | GlobScope) -> MyScope:
|
||||
key = path.strip("/")
|
||||
parts = key.split("/")
|
||||
# Just parse path structure, no index lookups
|
||||
...
|
||||
```
|
||||
|
||||
### 5. Glob Resolution
|
||||
|
||||
```python
|
||||
# mirage/core/<name>/glob.py
|
||||
async def resolve_glob(accessor, paths, index=None) -> list[GlobScope]:
|
||||
# Expand patterns via readdir + fnmatch
|
||||
```
|
||||
|
||||
### 6. Ops Layer
|
||||
|
||||
Thin wrappers that bridge core functions to the command framework:
|
||||
|
||||
```python
|
||||
@op("read", resource="<name>")
|
||||
async def read(accessor, scope, **kwargs) -> bytes:
|
||||
return await core_read(accessor, scope.original,
|
||||
kwargs.get("index"), prefix=scope.prefix)
|
||||
```
|
||||
|
||||
### 7. Commands
|
||||
|
||||
Copy from an existing resource (Discord/Slack), then:
|
||||
|
||||
1. Replace accessor and core imports.
|
||||
1. Set `resource="<name>"` in decorators.
|
||||
1. Add or remove scope-based optimizations depending on API capabilities.
|
||||
1. Add resource-specific commands (send message, etc.).
|
||||
|
||||
**Provision (dry-run estimates).** Factory-built commands
|
||||
(`make_generic_commands`) get family-default estimators for free, so
|
||||
`ws.execute(cmd, provision=True)` works without extra wiring. For
|
||||
bespoke commands, pass `provision=` to the decorator, reusing the
|
||||
shared helpers from `mirage.commands.builtin.generic_bind`:
|
||||
|
||||
```python
|
||||
from mirage.commands.builtin.generic_bind.provision import (
|
||||
make_file_read_provision, metadata_provision)
|
||||
from mirage.core.<name>.stat import stat as my_stat
|
||||
|
||||
@command("cat", resource="<name>", spec=SPECS["cat"],
|
||||
provision=make_file_read_provision(my_stat))
|
||||
async def cat(...): ...
|
||||
```
|
||||
|
||||
Omit `provision=` and the planner honestly reports
|
||||
`precision=unknown`. If your resource renders virtual files (a
|
||||
`chat.jsonl` built from messages), have `stat` report `size=None`
|
||||
rather than 0 so estimates degrade to floors instead of lying.
|
||||
|
||||
### 8. Resource Class
|
||||
|
||||
```python
|
||||
class MyResource(BaseResource):
|
||||
name: str = ResourceName.MY_RESOURCE
|
||||
caches_reads: bool = True
|
||||
|
||||
def __init__(self, config: MyConfig) -> None:
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.accessor = MyAccessor(self.config)
|
||||
from mirage.commands.builtin.<name> import COMMANDS
|
||||
from mirage.ops.<name> import OPS
|
||||
|
||||
for fn in COMMANDS:
|
||||
self.register(fn)
|
||||
for fn in OPS:
|
||||
self.register_op(fn)
|
||||
```
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
title: Nextcloud
|
||||
icon: cloud
|
||||
---
|
||||
The Nextcloud resource mounts a [WebDAV](http://www.webdav.org/specs/rfc4918.html)
|
||||
server (Nextcloud, ownCloud, Hetzner Storage Share, or any generic WebDAV
|
||||
endpoint) at some prefix such as `/nc/`. Reads are async and streaming, with
|
||||
range requests for partial reads.
|
||||
|
||||
For credential setup, see [Nextcloud Setup](/home/setup/nextcloud).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv add "mirage-ai[nextcloud]"
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.nextcloud import NextcloudConfig, NextcloudResource
|
||||
|
||||
config = NextcloudConfig(
|
||||
url=os.environ["NEXTCLOUD_URL"],
|
||||
username=os.environ["NEXTCLOUD_USERNAME"],
|
||||
password=os.environ["NEXTCLOUD_PASSWORD"],
|
||||
# Optional:
|
||||
# verify_ssl=True,
|
||||
# timeout=30,
|
||||
)
|
||||
resource = NextcloudResource(config)
|
||||
ws = Workspace({"/nc/": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
`NextcloudResource(config)` takes a `NextcloudConfig` object with the
|
||||
WebDAV URL plus optional Basic Auth credentials. Both `READ` and `WRITE`
|
||||
modes are supported. Use an **app password** (Settings → Security in
|
||||
Nextcloud) rather than your account password.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
WebDAV resources map directly to virtual paths under the mount prefix.
|
||||
|
||||
For example, if your Nextcloud root contains:
|
||||
|
||||
```text
|
||||
Documents/notes.md
|
||||
Documents/contract.pdf
|
||||
Photos/2024/cat.jpg
|
||||
```
|
||||
|
||||
Then mounting at `/nc/` exposes:
|
||||
|
||||
```text
|
||||
/nc/
|
||||
Documents/
|
||||
notes.md
|
||||
contract.pdf
|
||||
Photos/
|
||||
2024/
|
||||
cat.jpg
|
||||
```
|
||||
|
||||
Path mapping: virtual `/nc/Documents/notes.md` issues HTTP requests
|
||||
against `<NEXTCLOUD_URL>/Documents/notes.md`.
|
||||
|
||||
## Cache
|
||||
|
||||
The Nextcloud resource uses `IndexCacheStore`. Directory listings come
|
||||
from a single `PROPFIND Depth: 1` and populate file size, type, and
|
||||
ETag entries that `stat` reads via a fast path, so a `readdir` followed
|
||||
by per-entry `stat` calls (which is what `ls`, FUSE `getattr`, and most
|
||||
shell commands trigger) costs one HTTP request instead of N.
|
||||
|
||||
## Fingerprinting and Snapshots
|
||||
|
||||
`SUPPORTS_SNAPSHOT = True`. Per-file fingerprints come from the WebDAV
|
||||
`getetag` property, so snapshot drift detection works automatically:
|
||||
when a remote file changes, its ETag changes, and Mirage notices on the
|
||||
next access.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.nextcloud import NextcloudConfig, NextcloudResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = NextcloudConfig(
|
||||
url=os.environ["NEXTCLOUD_URL"],
|
||||
username=os.environ["NEXTCLOUD_USERNAME"],
|
||||
password=os.environ["NEXTCLOUD_PASSWORD"],
|
||||
)
|
||||
resource = NextcloudResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/nc/": resource}, mode=MountMode.WRITE)
|
||||
|
||||
r = await ws.execute("ls /nc/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /nc/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /nc/ -name '*.md'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /nc/Documents/notes.md")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("grep TODO /nc/Documents/notes.md")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /nc/Documents/notes.md")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The Nextcloud resource supports the full set of shell commands since it
|
||||
operates on real file content (text, binary, JSON, CSV, etc.). Reads
|
||||
benefit from HTTP `Range` requests so commands like `head -c BYTES`
|
||||
don't pull the whole file.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------------------------------- |
|
||||
| `cp` | Copy files (WebDAV `COPY` method) |
|
||||
| `mv` | Move/rename files (WebDAV `MOVE` method) |
|
||||
| `rm` | Remove files (WebDAV `DELETE`; recursive for directories) |
|
||||
| `mkdir` | Create directories (WebDAV `MKCOL`) |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Streaming
|
||||
|
||||
- **Reads**: streamed in chunks. Downstream commands like `head -n 1`
|
||||
early-cancel, so you only pay for the first chunk over the wire.
|
||||
- **Range reads**: `head -c BYTES` and other partial reads issue HTTP
|
||||
`Range` requests that Nextcloud and most WebDAV servers honor.
|
||||
- **Writes**: buffer the payload before upload. Streaming uploads
|
||||
(Nextcloud's `uploads/` resumable protocol) are a possible future
|
||||
follow-up.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing personal cloud storage**: Mount Nextcloud so
|
||||
agents can read documents, notes, and structured data on a self-hosted
|
||||
cloud.
|
||||
- **Self-hosted alternative to Dropbox/Box**: Same shell-command surface,
|
||||
with full read+write support.
|
||||
- **Multi-protocol WebDAV**: Works against any RFC 4918-compliant server,
|
||||
not just Nextcloud.
|
||||
- **FUSE mounting**: Expose Nextcloud through a virtual FUSE mount for
|
||||
external tools.
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
title: Notion
|
||||
icon: book
|
||||
description: Mount Notion pages as a virtual filesystem.
|
||||
---
|
||||
The Notion resource exposes a Notion workspace as a virtual filesystem
|
||||
mounted at a prefix such as `/notion/`.
|
||||
|
||||
For API key setup, see [Notion Setup](/python/setup/notion).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.notion import NotionConfig, NotionResource
|
||||
|
||||
config = NotionConfig(api_key=os.environ["NOTION_API_KEY"])
|
||||
resource = NotionResource(config=config)
|
||||
ws = Workspace({"/notion": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
| Field | Required | Default | Description |
|
||||
| ---------- | -------- | ------------------------------ | ----------------------- |
|
||||
| `api_key` | yes | | Notion integration token |
|
||||
| `base_url` | no | `https://api.notion.com/v1` | API base URL |
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/notion/
|
||||
pages/
|
||||
<page-title>__<page-id>/
|
||||
page.json
|
||||
<child-page-title>__<child-id>/
|
||||
page.json
|
||||
...
|
||||
databases/
|
||||
<database-title>__<database-id>/
|
||||
database.json
|
||||
<row-page-title>__<page-id>/
|
||||
page.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/notion/
|
||||
pages/
|
||||
Project-Roadmap__a1b2c3d4/
|
||||
page.json
|
||||
Q1-Goals__e5f6g7h8/
|
||||
page.json
|
||||
Q2-Goals__i9j0k1l2/
|
||||
page.json
|
||||
Meeting-Notes__m3n4o5p6/
|
||||
page.json
|
||||
databases/
|
||||
Tasks__4a3b21915e77/
|
||||
database.json
|
||||
Write-proposal__62212c5affe6/
|
||||
page.json
|
||||
Build-dashboards__f988c5a145ef/
|
||||
page.json
|
||||
Roadmap__2c4d6a7f5bf3/
|
||||
database.json
|
||||
```
|
||||
|
||||
The `pages/` hierarchy mirrors Notion's standalone page tree. The
|
||||
`databases/` hierarchy lists databases shared with the integration and
|
||||
then the row pages returned by querying each database. Each database
|
||||
directory contains `database.json` with the database metadata and its
|
||||
typed property schema (the columns), but not the rows themselves; the
|
||||
rows are the row-page directories alongside it, so `ls` the database
|
||||
directory to enumerate them. Each page directory contains a `page.json`
|
||||
file with the page metadata and content. Child pages appear as nested
|
||||
directories.
|
||||
|
||||
`page.json` carries the page metadata (`page_id`, `title`, `url`,
|
||||
timestamps, `parent_type`/`parent_id`, `created_by`/`last_edited_by`),
|
||||
a `markdown` field with the page body rendered as Markdown, and the raw
|
||||
`blocks`. Blocks with children embed them recursively under a
|
||||
`children` key, and the Markdown renders nested blocks with
|
||||
indentation.
|
||||
|
||||
`database.json` carries the database metadata (`database_id`, `title`,
|
||||
`url`, timestamps, `parent`, `archived`, `is_inline`) and `properties`,
|
||||
the database's typed column schema. It does not embed the rows; read a
|
||||
row's `page.json` for its content, since a database row is itself a page
|
||||
with `parent_type` of `database_id`.
|
||||
|
||||
### database.json
|
||||
|
||||
The database's identity plus its typed column schema (Notion's own
|
||||
property objects), with no rows inline. `ls` the database directory to
|
||||
enumerate row pages:
|
||||
|
||||
```json
|
||||
{
|
||||
"database_id": "2c4d6a7f-5bf3-8036-bed2-d1c95826f76b",
|
||||
"title": "Item",
|
||||
"url": "https://www.notion.so/2c4d6a7f5bf38036bed2d1c95826f76b",
|
||||
"created_time": "2025-12-09T23:36:00.000Z",
|
||||
"last_edited_time": "2026-04-15T12:30:00.000Z",
|
||||
"parent": { "type": "workspace", "workspace": true },
|
||||
"archived": false,
|
||||
"is_inline": false,
|
||||
"properties": {
|
||||
"Name": { "id": "title", "type": "title", "title": {} },
|
||||
"Number": { "id": "e%5BNQ", "type": "number", "number": { "format": "number" } },
|
||||
"Date": { "id": "%3BgQq", "type": "date", "date": {} }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
To create a row, `notion-page-create` with a `database_id` parent and a
|
||||
`properties` object whose keys match this schema.
|
||||
|
||||
### page.json
|
||||
|
||||
A page (including a database row). Notion blocks render to `markdown`
|
||||
and stay available raw under `blocks`:
|
||||
|
||||
```json
|
||||
{
|
||||
"page_id": "2c4d6a7f-5bf3-8022-826b-ed4700254ed2",
|
||||
"title": "item_0",
|
||||
"url": "https://www.notion.so/item_0-2c4d6a7f5bf38022826bed4700254ed2",
|
||||
"created_time": "2025-12-09T23:36:00.000Z",
|
||||
"last_edited_time": "2025-12-09T23:44:00.000Z",
|
||||
"parent_type": "database_id",
|
||||
"parent_id": "2c4d6a7f-5bf3-8036-bed2-d1c95826f76b",
|
||||
"archived": false,
|
||||
"created_by": "faa6ebe0-0686-466d-96b3-2480ddda715b",
|
||||
"last_edited_by": "faa6ebe0-0686-466d-96b3-2480ddda715b",
|
||||
"markdown": "",
|
||||
"blocks": []
|
||||
}
|
||||
```
|
||||
|
||||
## Cache
|
||||
|
||||
Uses `IndexCacheStore` for page metadata. No separate content
|
||||
cache - file content caching is handled by the workspace `IOResult`
|
||||
mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.notion import NotionConfig, NotionResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = NotionConfig(api_key=os.environ["NOTION_API_KEY"])
|
||||
resource = NotionResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/notion": resource}, mode=MountMode.WRITE)
|
||||
|
||||
# List top-level pages
|
||||
r = await ws.execute("ls /notion/pages/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List shared databases and rows in a database
|
||||
r = await ws.execute("ls /notion/databases/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read a page
|
||||
r = await ws.execute(
|
||||
'cat "/notion/pages/Project-Roadmap__a1b2c3d4/page.json"'
|
||||
)
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across all pages
|
||||
r = await ws.execute('grep "deadline" /notion/pages/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 2 /notion/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search pages with the Notion search API
|
||||
r = await ws.execute('notion-search --query "Roadmap" --limit 5')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Create a new page
|
||||
r = await ws.execute(
|
||||
'notion-page-create --json \'{"parent":{"page_id":"a1b2c3d4"},'
|
||||
'"properties":{"title":[{"text":{"content":"New Page"}}]}}\''
|
||||
)
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Append content to an existing page
|
||||
r = await ws.execute(
|
||||
'notion-block-append --params \'{"block_id":"a1b2c3d4"}\''
|
||||
' --json \'{"children":[{"type":"paragraph","paragraph":'
|
||||
'{"rich_text":[{"text":{"content":"Appended paragraph"}}]}}]}\''
|
||||
)
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Notion tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List pages and child pages |
|
||||
| `cat` | Read page.json content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Search across pages |
|
||||
| `jq` | Query page JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `notion-page-create`
|
||||
|
||||
Create a new page under a parent page. Prints the new page normalized as JSON.
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | ---------------------------------------------------------------------- |
|
||||
| `--json` | yes | Page body with `parent` (required) and `properties`, per the Notion API |
|
||||
|
||||
In the TypeScript SDK this command takes `--parent <parent-path>` and
|
||||
`--title "title"` instead of a raw `--json` body.
|
||||
|
||||
### `notion-block-append`
|
||||
|
||||
Append content blocks to an existing page. Prints the refreshed page as JSON.
|
||||
|
||||
| Option | Required | Description |
|
||||
| ---------- | -------- | ------------------------------------------------- |
|
||||
| `--params` | yes | JSON object with the target `block_id` (page ID) |
|
||||
| `--json` | yes | JSON object with the `children` blocks to append |
|
||||
|
||||
### `notion-comment-add`
|
||||
|
||||
Add a comment to a page. Prints the created comment as JSON.
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------- | -------- | ---------------------------------------------------- |
|
||||
| `--json` | yes | Comment body with `parent` (required) and `rich_text` |
|
||||
|
||||
### `notion-search`
|
||||
|
||||
Search pages with the Notion search API.
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------- | -------- | ---------------------------- |
|
||||
| `--query` | yes | Search query |
|
||||
| `--limit` | no | Max results (default 20) |
|
||||
@@ -0,0 +1,238 @@
|
||||
---
|
||||
title: OCI Object Storage
|
||||
icon: server
|
||||
description: Mount Oracle Cloud Infrastructure Object Storage as a virtual filesystem.
|
||||
---
|
||||
The OCI resource mounts an Oracle Cloud Infrastructure Object Storage
|
||||
bucket at some prefix such as `/oci/`. All operations involve network
|
||||
I/O to the remote object store. Uses aioboto3 against OCI's
|
||||
S3-compatible API.
|
||||
|
||||
For credential setup, see [OCI Setup](/home/setup/oci).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.oci import OCIConfig, OCIResource
|
||||
|
||||
config = OCIConfig(
|
||||
bucket=os.environ["OCI_BUCKET"],
|
||||
namespace=os.environ["OCI_NAMESPACE"],
|
||||
region=os.environ["OCI_REGION"],
|
||||
access_key_id=os.environ["OCI_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["OCI_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://{namespace}.compat.objectstorage.{region}.oraclecloud.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = OCIResource(config=config)
|
||||
ws = Workspace({"/oci": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`OCIResource(config)` takes an `OCIConfig` object with the bucket name,
|
||||
OCI tenancy `namespace`, `region`, and Customer Secret Key credentials.
|
||||
The S3-compatible endpoint is auto-built from `namespace` + `region`
|
||||
unless `endpoint_url` is supplied. Both `READ` and `WRITE` modes are
|
||||
supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The OCI resource maps object keys to virtual paths under the mount
|
||||
prefix. OCI "directories" are prefix-based — there are no real
|
||||
directory objects.
|
||||
|
||||
For example, if bucket `my-bucket` contains:
|
||||
|
||||
```text
|
||||
data/users.csv
|
||||
data/logs/2026-04-01.jsonl
|
||||
config.json
|
||||
```
|
||||
|
||||
Then mounting at `/oci/` exposes:
|
||||
|
||||
```text
|
||||
/oci/
|
||||
data/
|
||||
users.csv
|
||||
logs/
|
||||
2026-04-01.jsonl
|
||||
config.json
|
||||
```
|
||||
|
||||
Path mapping: virtual `/oci/data/users.csv` maps to OCI key
|
||||
`data/users.csv`.
|
||||
|
||||
## Cache
|
||||
|
||||
The OCI resource uses `IndexCacheStore` with `index_ttl = 600`
|
||||
(10 minutes). Directory listings are cached for up to 600 seconds before
|
||||
being refreshed from OCI.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.oci import OCIConfig, OCIResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = OCIConfig(
|
||||
bucket=os.environ["OCI_BUCKET"],
|
||||
namespace=os.environ["OCI_NAMESPACE"],
|
||||
region=os.environ["OCI_REGION"],
|
||||
access_key_id=os.environ["OCI_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["OCI_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
|
||||
resource = OCIResource(config=config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/oci/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /oci/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /oci/data/users.csv | head -n 5")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /oci/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /oci/ -name '*.jsonl'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /oci/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The OCI resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing OCI data**: Mount OCI buckets for agents to read and process datasets
|
||||
- **Data pipelines**: Read and write OCI objects with shell-like commands
|
||||
- **FUSE mounting**: Expose OCI buckets through a virtual FUSE mount for external tools
|
||||
@@ -0,0 +1,216 @@
|
||||
---
|
||||
title: OneDrive
|
||||
description: Mount Microsoft OneDrive or SharePoint document libraries as a Mirage filesystem with async access, versioning, and shell commands.
|
||||
icon: microsoft
|
||||
---
|
||||
The OneDrive resource mounts a Microsoft OneDrive (or SharePoint document
|
||||
library) at some prefix such as `/onedrive/`. All operations involve network
|
||||
I/O to Microsoft Graph. Files are served as raw bytes (no Office filetype
|
||||
conversion). Uses aiohttp for async Graph access.
|
||||
|
||||
Compatible with: personal OneDrive, OneDrive for Business, and SharePoint
|
||||
document libraries (driveItems on Microsoft Graph v1.0).
|
||||
|
||||
For credential setup (tokens, drive ids, app vs delegated auth), see the
|
||||
[OneDrive Setup](/home/setup/onedrive) guide.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.onedrive import OneDriveResource, OneDriveConfig
|
||||
|
||||
config = OneDriveConfig(
|
||||
access_token=os.environ["MS_GRAPH_DRIVE_TOKEN"],
|
||||
# Optional:
|
||||
# drive_id="b!...", # target a specific drive (required for app-only tokens)
|
||||
# site_id="contoso.sharepoint.com,...", # use a SharePoint site's default drive
|
||||
# key_prefix="Documents/", # mount a sub-folder as the root
|
||||
# timeout=30,
|
||||
)
|
||||
|
||||
resource = OneDriveResource(config)
|
||||
ws = Workspace({"/onedrive": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
`OneDriveResource(config)` takes an `OneDriveConfig` object with a Microsoft
|
||||
Graph bearer token and optional drive/site targeting. Both `READ` and `WRITE`
|
||||
modes are supported.
|
||||
|
||||
### Config Reference
|
||||
|
||||
| Field | Required | Description |
|
||||
| -------------- | -------- | --------------------------------------------------------------------- |
|
||||
| `access_token` | Yes | Microsoft Graph OAuth2 bearer token |
|
||||
| `drive_id` | No | Target a specific drive. Required for app-only tokens (no `/me/drive`) |
|
||||
| `site_id` | No | Resolve a SharePoint site's default drive instead of `/me/drive` |
|
||||
| `key_prefix` | No | Mount a sub-folder of the drive as the root |
|
||||
| `timeout` | No | Request timeout in seconds (default `30`) |
|
||||
|
||||
Drive resolution order: `drive_id`, then the `site_id` site's default drive,
|
||||
then `/me/drive`.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The OneDrive resource maps Graph driveItems (path-addressed) to virtual paths
|
||||
under the mount prefix. Folders are real driveItems, so the tree matches what
|
||||
you see in OneDrive.
|
||||
|
||||
For example, if the drive contains:
|
||||
|
||||
```text
|
||||
Documents/file.txt
|
||||
Documents/config.json
|
||||
Reports/q1.csv
|
||||
Reports/q2.csv
|
||||
```
|
||||
|
||||
Then mounting at `/onedrive/` exposes:
|
||||
|
||||
```text
|
||||
/onedrive/
|
||||
Documents/
|
||||
file.txt
|
||||
config.json
|
||||
Reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
```
|
||||
|
||||
Path mapping: virtual `/onedrive/Documents/file.txt` maps to the driveItem at
|
||||
`/root:/Documents/file.txt`.
|
||||
|
||||
## Versioning and Snapshots
|
||||
|
||||
OneDrive keeps per-file version history. The resource exposes it the same way
|
||||
the S3 backend does:
|
||||
|
||||
- **Fingerprint** is the driveItem `cTag`, so normal reads and `stat` reflect
|
||||
the current content without extra Graph calls.
|
||||
- **Snapshots** pin each path to a Graph driveItem version id. Replaying a
|
||||
snapshot reads `/versions/{id}/content`, giving time-travel to the exact
|
||||
bytes captured at snapshot time.
|
||||
- **Restore** writes a previous version back as the current one.
|
||||
|
||||
This makes the OneDrive resource `SUPPORTS_SNAPSHOT = True`.
|
||||
|
||||
## Cache
|
||||
|
||||
The OneDrive resource caches directory listings via `IndexCacheStore` to reduce
|
||||
repeated Graph calls during traversal.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.onedrive import OneDriveResource, OneDriveConfig
|
||||
|
||||
config = OneDriveConfig(
|
||||
access_token=os.environ["MS_GRAPH_DRIVE_TOKEN"],
|
||||
drive_id=os.environ.get("MS_GRAPH_DRIVE_ID") or None,
|
||||
)
|
||||
|
||||
resource = OneDriveResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/onedrive/": resource}, mode=MountMode.WRITE)
|
||||
|
||||
r = await ws.execute("ls /onedrive/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
await ws.execute("echo 'hello from mirage' > /onedrive/note.txt")
|
||||
|
||||
r = await ws.execute("cat /onedrive/note.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /onedrive/note.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /onedrive/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
A runnable version lives at `examples/python/onedrive/onedrive.py`.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The OneDrive resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit from
|
||||
range reads to avoid downloading entire items.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing org documents**: Mount OneDrive/SharePoint for agents to read and process files
|
||||
- **Versioned document workflows**: Snapshot and replay exact file states over time
|
||||
- **Sandboxed access**: Restrict agent operations to a specific drive and sub-folder via `key_prefix`
|
||||
- **FUSE mounting**: Expose a OneDrive through a virtual FUSE mount for external tools
|
||||
|
||||
## Scoping a resource to a folder
|
||||
|
||||
Pass `key_prefix: str | None = None` to `OneDriveConfig` to transparently scope
|
||||
every operation to a sub-folder of the drive:
|
||||
|
||||
```python
|
||||
OneDriveResource(OneDriveConfig(
|
||||
access_token=token,
|
||||
drive_id=drive_id,
|
||||
key_prefix=f"users/{user_id}/",
|
||||
))
|
||||
```
|
||||
|
||||
When set, every read/write/list/stat/copy/rename/delete operation is scoped to
|
||||
that sub-folder. Agents see clean paths like `/data/notes.md`; the underlying
|
||||
driveItem is `users/{user_id}/data/notes.md`. Useful for multi-tenant systems.
|
||||
|
||||
**Normalization:** leading slashes are stripped and a trailing slash is added
|
||||
automatically. Both `None` and an empty string are treated as "no prefix."
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: Postgres
|
||||
icon: database
|
||||
description: Mount a Postgres database as a read-only filesystem of schemas, tables, and rows.
|
||||
---
|
||||
|
||||
`PostgresResource` connects to a Postgres database via a DSN and exposes its schemas and tables as a tree the agent can list, read, and grep.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage.resource.postgres import PostgresConfig, PostgresResource
|
||||
|
||||
resource = PostgresResource(PostgresConfig(
|
||||
dsn="postgres://user:pass@host:5432/db?sslmode=require",
|
||||
schemas=["public", "analytics"], # optional allowlist
|
||||
default_row_limit=1000,
|
||||
max_read_rows=10_000,
|
||||
max_read_bytes=10 * 1024 * 1024,
|
||||
default_search_limit=100,
|
||||
))
|
||||
```
|
||||
|
||||
| Field | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `dsn` | required | Postgres connection string. Redacted in snapshots. |
|
||||
| `schemas` | `None` | Optional list to limit which schemas appear in the tree. |
|
||||
| `default_row_limit` | `1000` | Default LIMIT applied to ad-hoc reads. |
|
||||
| `max_read_rows` | `10_000` | Hard ceiling per read. |
|
||||
| `max_read_bytes` | `10 MiB` | Hard ceiling per read. |
|
||||
| `default_search_limit` | `100` | Default LIMIT for search-style queries. |
|
||||
|
||||
For DSN format and permissions, see [Postgres Setup](/home/setup/postgres).
|
||||
|
||||
## Mount mode
|
||||
|
||||
`read` only. Postgres mounts are read-only at the resource level.
|
||||
|
||||
## Snapshot behavior
|
||||
|
||||
The DSN is redacted on snapshot. Loading a snapshot back requires a config file that supplies a fresh DSN.
|
||||
|
||||
## File size and `du`
|
||||
|
||||
`rows.jsonl` is rendered on demand, so `stat` / `ls -l` / `du` report the table's **physical storage size** (`pg_total_relation_size`: heap + indexes + TOAST + page padding), not the size of the rendered JSONL. That number can differ a lot from the content and shifts with Postgres version and vacuum state. For the actual rendered byte count, use `wc -c rows.jsonl` (which streams and counts the bytes).
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
title: Qdrant
|
||||
description: Mount a Qdrant collection as a Mirage filesystem, with payload folders, multimodal blobs, and a semantic search command for Python agents.
|
||||
icon: database
|
||||
---
|
||||
|
||||
The Qdrant resource exposes a [Qdrant](https://qdrant.tech/) collection as a virtual filesystem mounted
|
||||
at some prefix such as `/q/`. Group-by payload fields become nested folders,
|
||||
each point becomes a `.json` payload file (plus a `.txt` text file and an optional blob), and semantic search is the
|
||||
`search` command, which returns ranked points as canonical file paths.
|
||||
|
||||
For connection setup (self-hosted, Qdrant Cloud, search), see
|
||||
[Qdrant Setup](/python/setup/qdrant).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.qdrant import QdrantConfig, QdrantResource
|
||||
|
||||
config = QdrantConfig(
|
||||
url="https://xyz.cloud.qdrant.io",
|
||||
api_key="...",
|
||||
collection="fashion",
|
||||
group_by=["gender", "articleType", "baseColour"],
|
||||
id_field="id",
|
||||
text_field="productDisplayName",
|
||||
blob_field="image_b64",
|
||||
blob_ext="jpg",
|
||||
search_limit=5,
|
||||
)
|
||||
resource = QdrantResource(config)
|
||||
ws = Workspace({"/fashion/": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
The mapping is config-driven; nothing about the dataset is hardcoded. Point
|
||||
`group_by` at different payload fields and the folder tree changes.
|
||||
|
||||
## Filesystem layout
|
||||
|
||||
Every path is translated into a Qdrant query. Descending a folder adds one
|
||||
payload filter; the leaf level lists points.
|
||||
|
||||
```text
|
||||
/ # list collections (omitted when `collection` is pinned)
|
||||
/<collection>/ # distinct group_by[0] values
|
||||
<v1>/ # distinct group_by[1] where group_by[0]=v1
|
||||
.../<vN>/ # all group-by fields bound -> point files
|
||||
<id>.json # full payload (the metadata)
|
||||
<id>.txt # embedded source text (when text_field set)
|
||||
<id>.<ext> # raw blob / image bytes (when blob_field set)
|
||||
```
|
||||
|
||||
`<id>` is the Qdrant point id. When `collection` is set the collection level is
|
||||
elided, so the mount root is that collection:
|
||||
|
||||
```text
|
||||
/fashion/
|
||||
Men/
|
||||
Shoes/
|
||||
White/
|
||||
3.json
|
||||
3.txt
|
||||
3.jpg
|
||||
```
|
||||
|
||||
### Point files
|
||||
|
||||
A point is shown as its underlying data in its original format, never as the
|
||||
embedding vector:
|
||||
|
||||
- `<id>.txt` is the embedded source text (the `text_field` value), exactly what
|
||||
the vector was built from.
|
||||
- `<id>.json` is the full payload as compact JSON (the metadata), with the vector
|
||||
and the raw blob omitted.
|
||||
- `<id>.<ext>` is the raw blob bytes when `blob_field` is configured.
|
||||
|
||||
```text
|
||||
$ cat /fashion/Men/Shoes/White/3.txt
|
||||
Nike Men White Running Sneakers
|
||||
|
||||
$ cat /fashion/Men/Shoes/White/3.json
|
||||
{"gender":"Men","articleType":"Shoes","baseColour":"White","productDisplayName":"Nike Men White Running Sneakers","id":3}
|
||||
```
|
||||
|
||||
## Semantic search
|
||||
|
||||
Search is a command, not a path. It returns each ranked point as its **canonical
|
||||
content path** (the `<id>.txt`, or `<id>.json` when no `text_field` is set)
|
||||
annotated with the similarity score, followed by the content:
|
||||
|
||||
```text
|
||||
$ search "white running sneakers" /fashion
|
||||
/fashion/Men/Shoes/White/3.txt:0.7421
|
||||
Nike Men White Running Sneakers
|
||||
```
|
||||
|
||||
Flags: `--top-k <n>` (default `search_limit`), `--threshold <min-score>`,
|
||||
`--method semantic` (the only supported method; `grep`/`rg` stay lexical).
|
||||
|
||||
## Supported commands
|
||||
|
||||
All commands delegate to Mirage's shared implementations.
|
||||
|
||||
| Command | Behaviour on a Qdrant mount |
|
||||
| ------------- | ------------------------------------------------------------- |
|
||||
| `ls` | list collections, payload folders, or point files |
|
||||
| `cd` | navigate (each level narrows the filter) |
|
||||
| `tree` | render the folder hierarchy |
|
||||
| `cat` | print a point's text/JSON, or dump raw blob/image bytes |
|
||||
| `stat` | directory vs file, blob size, image mime type |
|
||||
| `find` | walk the tree (e.g. `find /fashion -name '*.txt'`) |
|
||||
| `grep` / `rg` | lexical search over the text/JSON files |
|
||||
| `search` | semantic (vector) search -> ranked content paths + score |
|
||||
| `head` / `tail` | first/last lines of a file |
|
||||
| `wc` | count lines/bytes of a file |
|
||||
|
||||
## Access pattern
|
||||
|
||||
The mount is **read-only** (`MountMode.READ`); writes are not supported. The two
|
||||
read modes are:
|
||||
|
||||
- **Browse** by payload folders: scroll filters on `group_by` fields, no embedding.
|
||||
- **Search** by meaning: `search "<query>" <path>` runs vector search and returns
|
||||
canonical point paths.
|
||||
|
||||
Folder listings are capped by `max_rows`. A filtered listing scrolls first and
|
||||
only creates keyword payload indexes for the `group_by` fields if Qdrant reports
|
||||
one is required, so already-indexed collections work under read-only keys. Keep
|
||||
`group_by` to low-cardinality fields for large collections.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: QingStor
|
||||
icon: cloud
|
||||
description: Mount a QingStor (QingCloud) Object Storage bucket via its S3-compatible API.
|
||||
---
|
||||
The QingStor resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `QingStorConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just derives the right endpoint from `region`
|
||||
(the QingStor zone). Uses aioboto3 against QingStor's S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `s3.<region>.qingstor.com`
|
||||
(e.g. `s3.pek3a.qingstor.com`). This is QingStor's S3-compatible host, distinct
|
||||
from the native `<zone>.qingstor.com` API. Pass `endpoint_url` to override.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.qingstor import QingStorConfig, QingStorResource
|
||||
|
||||
config = QingStorConfig(
|
||||
bucket=os.environ["QINGSTOR_BUCKET"],
|
||||
region=os.environ.get("QINGSTOR_ZONE", "pek3a"),
|
||||
access_key_id=os.environ["QINGSTOR_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["QINGSTOR_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://s3.pek3a.qingstor.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = QingStorResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.qingstor import QingStorConfig, QingStorResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = QingStorConfig(
|
||||
bucket=os.environ["QINGSTOR_BUCKET"],
|
||||
region=os.environ.get("QINGSTOR_ZONE", "pek3a"),
|
||||
endpoint_url=os.environ.get("QINGSTOR_ENDPOINT_URL"),
|
||||
access_key_id=os.environ["QINGSTOR_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["QINGSTOR_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
resource = QingStorResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /data/ -name '*.json' | head -n 5")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- QingStor reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [QingStor Setup](/home/setup/qingstor).
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
title: Cloudflare R2
|
||||
icon: cloud
|
||||
description: Mount Cloudflare R2 storage as a virtual filesystem.
|
||||
---
|
||||
The R2 resource mounts a Cloudflare R2 bucket at some prefix such as
|
||||
`/r2/`. All operations involve network I/O to the remote object store.
|
||||
Uses aioboto3 against R2's S3-compatible API.
|
||||
|
||||
For credential setup, see [R2 Setup](/home/setup/r2).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.r2 import R2Config, R2Resource
|
||||
|
||||
config = R2Config(
|
||||
bucket=os.environ["R2_BUCKET"],
|
||||
account_id=os.environ["R2_ACCOUNT_ID"],
|
||||
access_key_id=os.environ["R2_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["R2_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://...r2.cloudflarestorage.com",
|
||||
# aws_profile="my-profile",
|
||||
# region="auto",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = R2Resource(config=config)
|
||||
ws = Workspace({"/r2": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`R2Resource(config)` takes an `R2Config` object with the bucket name and
|
||||
either `account_id` (endpoint is auto-built as
|
||||
`https://{account_id}.r2.cloudflarestorage.com`) or an explicit
|
||||
`endpoint_url`. Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The R2 resource maps object keys to virtual paths under the mount
|
||||
prefix. R2 "directories" are prefix-based — there are no real directory
|
||||
objects.
|
||||
|
||||
For example, if bucket `my-bucket` contains:
|
||||
|
||||
```text
|
||||
assets/logo.png
|
||||
assets/style.css
|
||||
data/export.parquet
|
||||
data/metrics.csv
|
||||
```
|
||||
|
||||
Then mounting at `/r2/` exposes:
|
||||
|
||||
```text
|
||||
/r2/
|
||||
assets/
|
||||
logo.png
|
||||
style.css
|
||||
data/
|
||||
export.parquet
|
||||
metrics.csv
|
||||
```
|
||||
|
||||
Path mapping: virtual `/r2/assets/logo.png` maps to R2 key
|
||||
`assets/logo.png`.
|
||||
|
||||
## Cache
|
||||
|
||||
The R2 resource uses `IndexCacheStore` with `index_ttl = 600`
|
||||
(10 minutes). Directory listings are cached for up to 600 seconds before
|
||||
being refreshed from R2. This reduces API calls for repeated directory
|
||||
traversals.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.r2 import R2Config, R2Resource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = R2Config(
|
||||
bucket=os.environ["R2_BUCKET"],
|
||||
account_id=os.environ["R2_ACCOUNT_ID"],
|
||||
access_key_id=os.environ["R2_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["R2_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
|
||||
resource = R2Resource(config=config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/r2/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /r2/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /r2/data/metrics.csv | head -n 10")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /r2/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /r2/ -name '*.csv'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /r2/assets/logo.png")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The R2 resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing R2 data**: Mount R2 buckets for agents to read and process datasets
|
||||
- **Data pipelines**: Read and write R2 objects with shell-like commands
|
||||
- **FUSE mounting**: Expose R2 buckets through a virtual FUSE mount for external tools
|
||||
@@ -0,0 +1,218 @@
|
||||
---
|
||||
title: RAM
|
||||
description: Use an in-memory Mirage filesystem for temporary files, tests, examples, and writable agent scratch space.
|
||||
icon: memory
|
||||
---
|
||||
The RAM resource exposes an in-memory filesystem mounted at some prefix
|
||||
such as `/data/`. All data lives in process memory and is lost on exit.
|
||||
|
||||
No setup or credentials required.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.ram import RAMResource
|
||||
|
||||
resource = RAMResource()
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
No configuration object needed - `RAMResource()` takes no arguments.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The RAM filesystem starts empty. Files and directories are created
|
||||
on demand via commands like `touch`, `mkdir`, `tee`, `cp`, etc.
|
||||
|
||||
```text
|
||||
/data/
|
||||
notes.txt
|
||||
config.json
|
||||
reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
uploads/
|
||||
image.png
|
||||
```
|
||||
|
||||
Paths are normalized with a leading `/`. The root directory `/` always
|
||||
exists.
|
||||
|
||||
## Cache
|
||||
|
||||
The RAM resource uses `IndexCacheStore` with `index_ttl = 0` (no
|
||||
expiry). Since all data is in-memory, the index is always fresh,
|
||||
no network calls or stale cache concerns.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.ram import RAMResource
|
||||
|
||||
resource = RAMResource()
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.WRITE)
|
||||
|
||||
# Create files
|
||||
await ws.execute('echo "hello world" | tee /data/hello.txt')
|
||||
await ws.execute('echo \'{"name": "alice", "age": 30}\' | tee /data/user.json')
|
||||
await ws.execute("mkdir /data/reports")
|
||||
await ws.execute('echo "revenue,100" | tee /data/reports/q1.csv')
|
||||
|
||||
# Read
|
||||
r = await ws.execute("cat /data/hello.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Query JSON
|
||||
r = await ws.execute('jq ".name" /data/user.json')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search
|
||||
r = await ws.execute("grep hello /data/hello.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree
|
||||
r = await ws.execute("tree /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# File info
|
||||
r = await ws.execute("stat /data/hello.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Copy, move, remove
|
||||
await ws.execute("cp /data/hello.txt /data/hello_copy.txt")
|
||||
await ws.execute("mv /data/hello_copy.txt /data/renamed.txt")
|
||||
await ws.execute("rm /data/renamed.txt")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The RAM resource supports the full set of shell commands since it
|
||||
handles real file content (text, binary, JSON, CSV, etc.):
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Testing**: Fast filesystem for unit tests without I/O
|
||||
- **Staging**: Temporary data area during pipelines
|
||||
- **Prototyping**: Develop against a filesystem before switching to S3/disk
|
||||
- **Caching**: In-process data exchange between commands
|
||||
- **Ephemeral workspaces**: Create, process, discard - no cleanup needed
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
title: Redis
|
||||
icon: bolt
|
||||
description: Mount a Redis-backed virtual filesystem.
|
||||
---
|
||||
The Redis resource creates a persistent virtual filesystem backed by
|
||||
Redis. It supports full read and write operations.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.redis import RedisResource
|
||||
|
||||
resource = RedisResource(
|
||||
url="redis://localhost:6379/0",
|
||||
key_prefix="mirage:fs:",
|
||||
)
|
||||
ws = Workspace({"/redis": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
| Parameter | Required | Default | Description |
|
||||
| ------------ | -------- | ------------------------ | ----------------------- |
|
||||
| `url` | no | `redis://localhost:6379/0` | Redis connection URL |
|
||||
| `key_prefix` | no | `mirage:fs:` | Key namespace prefix |
|
||||
|
||||
Keys are stored as:
|
||||
- `{prefix}file:{path}` - file content (raw bytes)
|
||||
- `{prefix}dir` - Redis set of directory paths
|
||||
- `{prefix}modified:{path}` - modification timestamps
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/redis/
|
||||
<file>
|
||||
<directory>/
|
||||
<file>
|
||||
<subdirectory>/
|
||||
<file>
|
||||
```
|
||||
|
||||
The filesystem is fully user-defined - create any directory structure
|
||||
with `mkdir`, `touch`, `tee`, etc.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/redis/
|
||||
config.json
|
||||
logs/
|
||||
app.log
|
||||
error.log
|
||||
cache/
|
||||
session-abc.json
|
||||
```
|
||||
|
||||
## Cache
|
||||
|
||||
Index TTL is `0` - always reads fresh state from Redis.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.redis import RedisResource
|
||||
|
||||
resource = RedisResource(url="redis://localhost:6379/0")
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/redis": resource}, mode=MountMode.WRITE)
|
||||
|
||||
# Create directories and files
|
||||
await ws.execute("mkdir -p /redis/data/")
|
||||
await ws.execute('echo "hello world" | tee /redis/data/hello.txt')
|
||||
|
||||
# Read back
|
||||
r = await ws.execute("cat /redis/data/hello.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List files
|
||||
r = await ws.execute("ls /redis/data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Copy and move
|
||||
await ws.execute("cp /redis/data/hello.txt /redis/data/copy.txt")
|
||||
await ws.execute("mv /redis/data/copy.txt /redis/data/renamed.txt")
|
||||
|
||||
# Search
|
||||
r = await ws.execute('grep "hello" /redis/data/')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Clean up
|
||||
await ws.execute("rm -r /redis/data/")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
| Command | Notes |
|
||||
| ---------------------- | ------------------------------------------ |
|
||||
| `ls` | List files and directories |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty files |
|
||||
| `cp` / `mv` / `rm` | Copy, move, delete |
|
||||
| `tee` | Write stdin to file |
|
||||
| `diff` / `cmp` | Compare files |
|
||||
| `sort` / `cut` / `tr` | Text processing |
|
||||
| `tar` / `zip` / `gzip` | Compression |
|
||||
| `base64` / `md5` / `sha256sum` | Encoding and checksums |
|
||||
@@ -0,0 +1,248 @@
|
||||
---
|
||||
title: S3
|
||||
description: Mount AWS S3 or S3-compatible buckets as a Mirage filesystem with async access, caching, and shell commands.
|
||||
icon: aws
|
||||
---
|
||||
The S3 resource mounts an Amazon S3 bucket (or any S3-compatible service) at
|
||||
some prefix such as `/s3/`. All operations involve network I/O to the remote
|
||||
object store. Uses aioboto3 for async S3 access.
|
||||
|
||||
Compatible with: AWS S3, MinIO, Cloudflare R2, Supabase Storage,
|
||||
DigitalOcean Spaces, and any S3-compatible service.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.s3 import S3Resource, S3Config
|
||||
|
||||
config = S3Config(
|
||||
bucket="my-bucket",
|
||||
region="us-east-1",
|
||||
# Optional:
|
||||
# aws_access_key_id="...",
|
||||
# aws_secret_access_key="...",
|
||||
# aws_profile="my-profile",
|
||||
# endpoint_url="http://localhost:9000", # MinIO
|
||||
# path_style=True, # For MinIO/R2
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
|
||||
resource = S3Resource(config)
|
||||
ws = Workspace({"/s3": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`S3Resource(config)` takes an `S3Config` object with the bucket name and
|
||||
optional credentials, endpoint, and connection settings. Both `READ` and
|
||||
`WRITE` modes are supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The S3 resource maps S3 object keys to virtual paths under the mount prefix.
|
||||
S3 "directories" are prefix-based - there are no real directory objects.
|
||||
|
||||
For example, if bucket `my-bucket` contains:
|
||||
|
||||
```text
|
||||
data/file.txt
|
||||
data/config.json
|
||||
reports/q1.csv
|
||||
reports/q2.csv
|
||||
```
|
||||
|
||||
Then mounting at `/s3/` exposes:
|
||||
|
||||
```text
|
||||
/s3/
|
||||
data/
|
||||
file.txt
|
||||
config.json
|
||||
reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
```
|
||||
|
||||
Path mapping: virtual `/s3/data/file.txt` maps to S3 key `data/file.txt`.
|
||||
|
||||
## Cache
|
||||
|
||||
The S3 resource uses `IndexCacheStore` with `index_ttl = 600` (10 minutes).
|
||||
Directory listings are cached for up to 600 seconds before being refreshed
|
||||
from S3. This reduces API calls for repeated directory traversals.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.s3 import S3Resource, S3Config
|
||||
|
||||
config = S3Config(
|
||||
bucket="my-bucket",
|
||||
region="us-east-1",
|
||||
)
|
||||
|
||||
resource = S3Resource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/s3/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /s3/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /s3/data/file.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /s3/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /s3/ -name '*.json'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("grep example /s3/data/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /s3/data/file.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The S3 resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing cloud data**: Mount S3 buckets for agents to read and process remote datasets
|
||||
- **Data pipelines**: Read and write S3 objects with shell-like commands
|
||||
- **Sandboxed cloud storage access**: Restrict agent operations to a specific bucket and prefix
|
||||
- **FUSE mounting**: Expose S3 buckets through a virtual FUSE mount for external tools
|
||||
|
||||
## Scoping a resource to a key prefix
|
||||
|
||||
Pass `key_prefix: str | None = None` to `S3Config` to transparently scope every operation to a subpath of the bucket:
|
||||
|
||||
```python
|
||||
S3Resource(S3Config(
|
||||
bucket="app-data",
|
||||
region="eu-west-1",
|
||||
key_prefix=f"users/{user_id}/",
|
||||
))
|
||||
```
|
||||
|
||||
When set, every read/write/list/stat/copy/rename/delete operation is transparently scoped to that bucket subpath. Agents see clean paths like `/data/notes.md`; the underlying bucket key is `users/{user_id}/data/notes.md`. Useful for multi-tenant systems. Pairs naturally with STS AssumeRole session policies for AWS-side enforcement. Unset behavior is unchanged.
|
||||
|
||||
**Normalization:** leading slashes are stripped and a trailing slash is added automatically. Both `None` and an empty string are treated as "no prefix."
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: Scaleway
|
||||
icon: /images/scaleway-logo.svg
|
||||
description: Mount a Scaleway Object Storage bucket via its S3-compatible API.
|
||||
---
|
||||
The Scaleway resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `ScalewayConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just derives the right endpoint from `region`.
|
||||
Uses aioboto3 against Scaleway Object Storage's S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `s3.<region>.scw.cloud`
|
||||
(e.g. `s3.fr-par.scw.cloud`). Pass `endpoint_url` to override.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.scaleway import ScalewayConfig, ScalewayResource
|
||||
|
||||
config = ScalewayConfig(
|
||||
bucket=os.environ["SCW_BUCKET"],
|
||||
region=os.environ.get("SCW_REGION", "fr-par"),
|
||||
access_key_id=os.environ["SCW_ACCESS_KEY"],
|
||||
secret_access_key=os.environ["SCW_SECRET_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://s3.fr-par.scw.cloud",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = ScalewayResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.scaleway import ScalewayConfig, ScalewayResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = ScalewayConfig(
|
||||
bucket=os.environ["SCW_BUCKET"],
|
||||
region=os.environ.get("SCW_REGION", "fr-par"),
|
||||
access_key_id=os.environ["SCW_ACCESS_KEY"],
|
||||
secret_access_key=os.environ["SCW_SECRET_KEY"],
|
||||
)
|
||||
resource = ScalewayResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Scaleway reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [Scaleway Setup](/home/setup/scaleway).
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: SeaweedFS
|
||||
icon: /images/seaweedfs-logo.svg
|
||||
description: Mount a SeaweedFS bucket as a virtual filesystem.
|
||||
---
|
||||
The SeaweedFS resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `SeaweedFSConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just points at your SeaweedFS S3 gateway
|
||||
`endpoint_url` and uses path-style addressing by default.
|
||||
|
||||
SeaweedFS exposes an S3-compatible gateway (default port `8333`), so
|
||||
`endpoint_url` is **required** (there is no region-derived host). Uses aioboto3
|
||||
against the SeaweedFS S3 API.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.seaweedfs import SeaweedFSConfig, SeaweedFSResource
|
||||
|
||||
config = SeaweedFSConfig(
|
||||
bucket=os.environ["SEAWEEDFS_BUCKET"],
|
||||
endpoint_url=os.environ.get("SEAWEEDFS_ENDPOINT", "http://localhost:8333"),
|
||||
access_key_id=os.environ["SEAWEEDFS_ACCESS_KEY"],
|
||||
secret_access_key=os.environ["SEAWEEDFS_SECRET_KEY"],
|
||||
# Optional:
|
||||
# region="us-east-1", # default
|
||||
# path_style=True, # default; SeaweedFS uses path-style addressing
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = SeaweedFSResource(config)
|
||||
ws = Workspace({"/seaweedfs": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.seaweedfs import SeaweedFSConfig, SeaweedFSResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = SeaweedFSConfig(
|
||||
bucket=os.environ.get("SEAWEEDFS_BUCKET", "mirage-demo"),
|
||||
endpoint_url=os.environ.get("SEAWEEDFS_ENDPOINT", "http://localhost:8333"),
|
||||
access_key_id=os.environ.get("SEAWEEDFS_ACCESS_KEY", "any"),
|
||||
secret_access_key=os.environ.get("SEAWEEDFS_SECRET_KEY", "any"),
|
||||
)
|
||||
resource = SeaweedFSResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/seaweedfs/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /seaweedfs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /seaweedfs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /seaweedfs/data/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- SeaweedFS reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies (`ls`, `cat`,
|
||||
`head`, `tail`, `grep`, `rg`, `wc`, `find`, `tree`, `jq`, `stat`, plus
|
||||
parquet/orc/feather table rendering). See the [S3 resource](/python/resource/s3)
|
||||
for the complete command reference, range reads, streaming, and the index
|
||||
cache fast path.
|
||||
- Blaxel Agent Drive is backed by SeaweedFS. To mount a drive's S3 gateway,
|
||||
point `endpoint_url` at the drive's `s3Url` and supply credentials for it.
|
||||
- For credential setup, see [SeaweedFS Setup](/home/setup/seaweedfs).
|
||||
@@ -0,0 +1,207 @@
|
||||
---
|
||||
title: SharePoint
|
||||
description: Mount Microsoft SharePoint Online sites and document libraries as a Mirage filesystem with multi-site discovery, versioning, and shell commands.
|
||||
icon: microsoft
|
||||
---
|
||||
|
||||
The SharePoint resource mounts Microsoft SharePoint Online at some prefix such
|
||||
as `/sharepoint/`. Unlike the single-drive OneDrive resource, it exposes the
|
||||
full enterprise topology: many sites, each with many document libraries. Site
|
||||
and drive names are resolved to Microsoft Graph ids automatically, so you mount
|
||||
once and browse everything you have access to. All operations involve network
|
||||
I/O to Microsoft Graph. Files are served as raw bytes (no Office filetype
|
||||
conversion). Uses aiohttp for async Graph access.
|
||||
|
||||
For credential setup (tokens, app vs delegated auth, permissions), see the
|
||||
[SharePoint Setup](/home/setup/sharepoint) guide.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.sharepoint import SharePointResource, SharePointConfig
|
||||
|
||||
config = SharePointConfig(
|
||||
access_token=os.environ["MS_GRAPH_DRIVE_TOKEN"],
|
||||
# Optional:
|
||||
# tenant_host="contoso.sharepoint.com", # restrict to one tenant host
|
||||
# site_filter="Engineering", # narrow the site search
|
||||
# timeout=30,
|
||||
# max_retries=5,
|
||||
)
|
||||
|
||||
resource = SharePointResource(config)
|
||||
ws = Workspace({"/sharepoint": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
`SharePointResource(config)` takes a `SharePointConfig` object with a Microsoft
|
||||
Graph bearer token. No drive id is required: the resource discovers sites and
|
||||
drives for you. Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
### Config Reference
|
||||
|
||||
| Field | Required | Description |
|
||||
| -------------- | -------- | ----------------------------------------------------------------- |
|
||||
| `access_token` | Yes | Microsoft Graph OAuth2 bearer token (string or callable provider) |
|
||||
| `tenant_host` | No | Restrict resolution to a single SharePoint host |
|
||||
| `site_filter` | No | Search term passed to `/sites?search=` to narrow discovery |
|
||||
| `timeout` | No | Request timeout in seconds (default `30`) |
|
||||
| `max_retries` | No | Max retries on `429`/`5xx` with backoff (default `5`) |
|
||||
|
||||
`access_token` accepts either a static string or a `Callable[[], str]` provider.
|
||||
With a callable, the resource refreshes the token on `401`, so long-running
|
||||
mounts survive token expiry.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
SharePoint paths follow the structure `/{site}/{library}/{path}`. The first two
|
||||
levels are virtual (sites and drives), and everything below is a real Graph
|
||||
driveItem.
|
||||
|
||||
- **Level 0** (`/sharepoint/`): lists all accessible sites
|
||||
- **Level 1** (`/sharepoint/{site}/`): lists document libraries (drives) in that site
|
||||
- **Level 2+** (`/sharepoint/{site}/{library}/...`): lists files and folders
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
/sharepoint/
|
||||
Engineering/
|
||||
Documents/
|
||||
spec.md
|
||||
deck.pptx
|
||||
Reports/
|
||||
q1.csv
|
||||
Marketing/
|
||||
Shared Documents/
|
||||
brand.pdf
|
||||
```
|
||||
|
||||
Path mapping: virtual `/sharepoint/Engineering/Documents/spec.md` resolves the
|
||||
site `Engineering` to a site id, the library `Documents` to a drive id, then
|
||||
reads the driveItem at `/root:/spec.md`.
|
||||
|
||||
## Versioning and Snapshots
|
||||
|
||||
SharePoint keeps per-file version history. The resource exposes it the same way
|
||||
the OneDrive and S3 backends do:
|
||||
|
||||
- **Fingerprint** is the driveItem `cTag`, so normal reads and `stat` reflect
|
||||
the current content without extra Graph calls.
|
||||
- **Snapshots** pin each path to a Graph driveItem version id. Replaying a
|
||||
snapshot reads `/versions/{id}/content`, giving time-travel to the exact bytes
|
||||
captured at snapshot time.
|
||||
- **Restore** writes a previous version back as the current one.
|
||||
|
||||
This makes the SharePoint resource `SUPPORTS_SNAPSHOT = True`.
|
||||
|
||||
## Cache
|
||||
|
||||
The SharePoint resource caches site, drive, and directory listings via
|
||||
`IndexCacheStore` to reduce repeated Graph calls during traversal.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.sharepoint import SharePointResource, SharePointConfig
|
||||
|
||||
config = SharePointConfig(access_token=os.environ["MS_GRAPH_DRIVE_TOKEN"])
|
||||
resource = SharePointResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/sharepoint/": resource}, mode=MountMode.WRITE)
|
||||
|
||||
# list sites, then drives in the first site
|
||||
r = await ws.execute("ls /sharepoint/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("ls /sharepoint/Engineering/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
await ws.execute(
|
||||
"echo 'hello from mirage' > /sharepoint/Engineering/Documents/note.txt")
|
||||
|
||||
r = await ws.execute("cat /sharepoint/Engineering/Documents/note.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /sharepoint/Engineering/Documents/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
Runnable versions live at `examples/python/sharepoint/sharepoint.py` (command
|
||||
matrix) and `examples/python/sharepoint/sharepoint_vfs.py` (FUSE mount).
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The SharePoint resource supports the full set of shell commands since it
|
||||
operates on real file content (text, binary, JSON, CSV, etc.). Large files
|
||||
benefit from range reads to avoid downloading entire items.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing org documents**: Mount enterprise SharePoint for agents to read and process files across many sites
|
||||
- **Multi-site discovery**: Browse every accessible site and library from a single mount, no drive ids to manage
|
||||
- **Versioned document workflows**: Snapshot and replay exact file states over time
|
||||
- **FUSE mounting**: Expose SharePoint through a virtual FUSE mount for external tools
|
||||
|
||||
## SharePoint vs OneDrive
|
||||
|
||||
Both backends talk to Microsoft Graph driveItems and behave identically below
|
||||
the drive level. Choose by topology:
|
||||
|
||||
- Use **[OneDrive](/python/resource/onedrive)** for a single drive (personal
|
||||
OneDrive, OneDrive for Business, or one targeted SharePoint library via
|
||||
`drive_id` / `site_id`).
|
||||
- Use **SharePoint** for enterprise scenarios with many sites and libraries that
|
||||
you want to discover and traverse without knowing drive ids up front.
|
||||
@@ -0,0 +1,331 @@
|
||||
---
|
||||
title: Slack
|
||||
description: Mount Slack channels, DMs, messages, users, and shared files as a Mirage virtual filesystem for Python agents.
|
||||
icon: slack
|
||||
---
|
||||
The Slack resource exposes a Slack workspace as a virtual filesystem
|
||||
mounted at some prefix such as `/slack/`.
|
||||
|
||||
For token setup, see [Slack Setup](/python/setup/slack).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.slack import SlackConfig, SlackResource
|
||||
|
||||
config = SlackConfig(token=os.environ["SLACK_BOT_TOKEN"])
|
||||
resource = SlackResource(config=config)
|
||||
ws = Workspace({"/slack": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/slack/
|
||||
channels/
|
||||
<channel-name>__<channel-id>/
|
||||
<yyyy-mm-dd>/
|
||||
chat.jsonl
|
||||
files/
|
||||
<name>__<F-id>.<ext>
|
||||
dms/
|
||||
<user-name>__<dm-id>/
|
||||
<yyyy-mm-dd>/
|
||||
chat.jsonl
|
||||
files/
|
||||
<name>__<F-id>.<ext>
|
||||
users/
|
||||
<username>__<user-id>.json
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/slack/
|
||||
channels/
|
||||
general__C04KEPWF6V7/
|
||||
2026-04-04/
|
||||
chat.jsonl
|
||||
files/
|
||||
2026-04-05/
|
||||
chat.jsonl
|
||||
files/
|
||||
random__C04JVGZM7UN/
|
||||
2026-04-11/
|
||||
chat.jsonl
|
||||
files/
|
||||
dms/
|
||||
alice__D0AQDJ5FP9V/
|
||||
2026-04-04/
|
||||
chat.jsonl
|
||||
files/
|
||||
slackbot__D0ARPA1QSPJ/
|
||||
2026-04-04/
|
||||
chat.jsonl
|
||||
files/
|
||||
users/
|
||||
alice__U12345678.json
|
||||
bob__U87654321.json
|
||||
```
|
||||
|
||||
Directory names embed the Slack ID so that write commands
|
||||
(`slack-post-message --channel_id`, etc.) can reference the correct
|
||||
resource without extra lookups.
|
||||
|
||||
### Channels
|
||||
|
||||
`/slack/channels/` lists public and private channels the bot has
|
||||
access to. The channel ID is appended after `__`. Each channel
|
||||
directory contains day-partitioned directories for the last 90 days
|
||||
(or since channel creation, whichever is shorter). Each date directory
|
||||
contains `chat.jsonl` plus a `files/` directory for attachments shared
|
||||
that day.
|
||||
|
||||
The date range is derived from the channel's `created` timestamp.
|
||||
|
||||
### DMs
|
||||
|
||||
`/slack/dms/` lists direct message conversations. The DM ID is
|
||||
appended after `__`. Like channels, DM directories contain daily
|
||||
directories with `chat.jsonl` and `files/`.
|
||||
|
||||
### Users
|
||||
|
||||
`/slack/users/` lists one `.json` file per non-deleted, non-bot user.
|
||||
Reading a user file calls `get_user_profile()` and returns the full
|
||||
profile JSON from the Slack API.
|
||||
|
||||
## Cache
|
||||
|
||||
The Slack resource uses `IndexCacheStore` (same as Discord and other
|
||||
resources). Index entries store channel IDs, DM IDs, user IDs, and
|
||||
channel creation timestamps for date range computation. There is no
|
||||
separate content cache - file content caching is handled by the
|
||||
workspace `IOResult` mechanism.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.slack import SlackConfig, SlackResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = SlackConfig(token=os.environ["SLACK_BOT_TOKEN"])
|
||||
resource = SlackResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/slack": resource}, mode=MountMode.READ)
|
||||
|
||||
# List structure
|
||||
r = await ws.execute("ls /slack/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# List channels
|
||||
r = await ws.execute("ls /slack/channels/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
ch = r.stdout_str().strip().splitlines()[0].strip()
|
||||
base = f"/slack/channels/{ch}"
|
||||
|
||||
# Read messages from a specific date
|
||||
r = await ws.execute(f'cat "{base}/2026-04-04/chat.jsonl" | head -n 3')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read user profile
|
||||
r = await ws.execute("cat /slack/users/alice__U12345678.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Extract message text with jq
|
||||
r = await ws.execute(
|
||||
f'cat "{base}/2026-04-04/chat.jsonl"'
|
||||
' | jq -r ".[] | .text" | head -n 5')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across channel (scans all date files)
|
||||
r = await ws.execute(f'rg message "{base}/"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Tree view
|
||||
r = await ws.execute("tree -L 1 /slack/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Navigate with cd/pwd
|
||||
await ws.execute(f'cd "{base}"')
|
||||
r = await ws.execute("pwd")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See `examples/python/slack/slack.py` for the full working example.
|
||||
|
||||
## Finding IDs
|
||||
|
||||
Resource-specific commands require Slack IDs (`channel_id`, `user_id`,
|
||||
`ts`). These can be extracted from the filesystem:
|
||||
|
||||
```bash
|
||||
# Channel ID - embedded in directory name after "__"
|
||||
ls /slack/channels/
|
||||
# → general__C04KEPWF6V7 ← channel_id = C04KEPWF6V7
|
||||
|
||||
# User ID - embedded in filename after "__"
|
||||
ls /slack/users/
|
||||
# → alice__U04K21SEVR9.json ← user_id = U04K21SEVR9
|
||||
|
||||
# Message timestamp (ts) - inside JSONL messages
|
||||
jq -r '.[] | "\(.ts) [\(.user)] \(.text)"' \
|
||||
"/slack/channels/general__C04KEPWF6V7/2026-04-04/chat.jsonl"
|
||||
# → 1712345678.123456 [U04K21SEVR9] hello world
|
||||
|
||||
# Find a specific message then reply
|
||||
jq -r '.[] | select(.text | test("hello")) | .ts' \
|
||||
"/slack/channels/general__C04KEPWF6V7/2026-04-04/chat.jsonl"
|
||||
# → 1712345678.123456
|
||||
slack-reply-to-thread --channel_id C04KEPWF6V7 \
|
||||
--ts 1712345678.123456 --text "Reply to hello"
|
||||
```
|
||||
|
||||
## Working with Large Channels
|
||||
|
||||
Channels with many messages produce large `chat.jsonl` files per day.
|
||||
Tips for efficient access:
|
||||
|
||||
```bash
|
||||
# Check message count per day
|
||||
wc -l "/slack/channels/general__C04KEPWF6V7/2026-04-04/chat.jsonl"
|
||||
|
||||
# Read only recent messages
|
||||
tail -n 10 "/slack/channels/general__C04KEPWF6V7/2026-04-04/chat.jsonl"
|
||||
|
||||
# Search across all dates (scans each file)
|
||||
rg "keyword" "/slack/channels/general__C04KEPWF6V7/"
|
||||
|
||||
# Extract specific fields to reduce output
|
||||
jq -r '.[] | "\(.ts) [\(.user)] \(.text)"' \
|
||||
"/slack/channels/general__C04KEPWF6V7/2026-04-04/chat.jsonl" | head -n 20
|
||||
|
||||
# Count messages per user
|
||||
cat "/slack/channels/general__C04KEPWF6V7/2026-04-04/chat.jsonl" \
|
||||
| jq -r '.[] | .user' | sort | uniq -c
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Slack tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List channels, DMs, users, dates |
|
||||
| `cat` | Read `chat.jsonl`, user `.json`, or files |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON; use `.[]` prefix for JSONL |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
|
||||
Resource-specific commands:
|
||||
|
||||
### `slack-post-message`
|
||||
|
||||
Post a message to a channel.
|
||||
|
||||
```bash
|
||||
slack-post-message --channel_id C04KEPWF6V7 --text "Hello from MIRAGE"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | -------------------- |
|
||||
| `--channel_id` | yes | Slack channel ID |
|
||||
| `--text` | yes | Message text to send |
|
||||
|
||||
Returns the posted message JSON.
|
||||
|
||||
### `slack-reply-to-thread`
|
||||
|
||||
Reply to a specific thread.
|
||||
|
||||
```bash
|
||||
slack-reply-to-thread --channel_id C04KEPWF6V7 --ts 1712345678.123456 --text "Thread reply"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ----------------------- |
|
||||
| `--channel_id` | yes | Slack channel ID |
|
||||
| `--ts` | yes | Thread parent timestamp |
|
||||
| `--text` | yes | Reply text |
|
||||
|
||||
The `--ts` value is the message timestamp (e.g., from a `.jsonl`
|
||||
entry's `ts` field). Returns the posted reply JSON.
|
||||
|
||||
### `slack-add-reaction`
|
||||
|
||||
Add an emoji reaction to a message.
|
||||
|
||||
```bash
|
||||
slack-add-reaction --channel_id C04KEPWF6V7 --ts 1712345678.123456 --reaction thumbsup
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ----------------------------- |
|
||||
| `--channel_id` | yes | Slack channel ID |
|
||||
| `--ts` | yes | Message timestamp to react to |
|
||||
| `--reaction` | yes | Emoji name (without colons) |
|
||||
|
||||
### `slack-get-users`
|
||||
|
||||
Search for users by name, real name, or email.
|
||||
|
||||
```bash
|
||||
slack-get-users --query "alice"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------- | -------- | ------------ |
|
||||
| `--query` | yes | Search query |
|
||||
|
||||
Returns matching users as JSON array.
|
||||
|
||||
### `slack-get-user-profile`
|
||||
|
||||
Get a single user's full profile.
|
||||
|
||||
```bash
|
||||
slack-get-user-profile --user_id U04K21SEVR9
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ----------- | -------- | ------------- |
|
||||
| `--user_id` | yes | Slack user ID |
|
||||
|
||||
Returns the user profile JSON. The user ID can be found in
|
||||
filenames under `/slack/users/` (e.g., `alice__U04K21SEVR9.json`).
|
||||
|
||||
### `slack-search`
|
||||
|
||||
Search messages across the workspace.
|
||||
|
||||
```bash
|
||||
slack-search --query "incident report"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| --------- | -------- | ------------ |
|
||||
| `--query` | yes | Search query |
|
||||
|
||||
Returns search results as JSON.
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
title: SSH
|
||||
icon: terminal
|
||||
description: Mount a remote filesystem over SSH/SFTP.
|
||||
---
|
||||
The SSH resource mounts a remote server's filesystem over SFTP.
|
||||
It supports full read and write operations.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.ssh import SSHConfig, SSHResource
|
||||
|
||||
config = SSHConfig(
|
||||
host="myserver",
|
||||
username="deploy",
|
||||
identity_file="~/.ssh/id_ed25519",
|
||||
root="/var/data",
|
||||
)
|
||||
resource = SSHResource(config=config)
|
||||
ws = Workspace({"/remote": resource}, mode=MountMode.WRITE)
|
||||
```
|
||||
|
||||
| Field | Required | Default | Description |
|
||||
| --------------- | -------- | ------- | ------------------------------------ |
|
||||
| `host` | yes | | SSH host (name or IP) |
|
||||
| `hostname` | no | | Override resolved hostname |
|
||||
| `port` | no | `22` | SSH port |
|
||||
| `username` | no | | SSH username |
|
||||
| `identity_file` | no | | Path to private key |
|
||||
| `root` | no | `/` | Remote directory to mount |
|
||||
| `timeout` | no | `30` | Connection timeout in seconds |
|
||||
| `known_hosts` | no | | Path to known_hosts file |
|
||||
|
||||
The `host` field matches entries in `~/.ssh/config`, so existing SSH
|
||||
configurations are automatically picked up.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/remote/
|
||||
<remote-directory-tree>
|
||||
```
|
||||
|
||||
The mounted tree mirrors the remote filesystem starting at `root`.
|
||||
|
||||
Example with `root="/var/data"`:
|
||||
|
||||
```text
|
||||
/remote/
|
||||
logs/
|
||||
app.log
|
||||
nginx/
|
||||
access.log
|
||||
error.log
|
||||
config/
|
||||
app.yaml
|
||||
uploads/
|
||||
image.png
|
||||
```
|
||||
|
||||
## Cache
|
||||
|
||||
Uses `IndexCacheStore` for directory listings. Freshness is checked
|
||||
via `{mtime}:{size}` fingerprints - files are re-fetched only when
|
||||
the remote has changed.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.ssh import SSHConfig, SSHResource
|
||||
|
||||
config = SSHConfig(
|
||||
host="myserver",
|
||||
username="deploy",
|
||||
identity_file="~/.ssh/id_ed25519",
|
||||
root="/var/log",
|
||||
)
|
||||
resource = SSHResource(config=config)
|
||||
|
||||
|
||||
async def main():
|
||||
ws = Workspace({"/logs": resource}, mode=MountMode.READ)
|
||||
|
||||
# List remote directory
|
||||
r = await ws.execute("ls /logs/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Read last 20 lines of a log
|
||||
r = await ws.execute("tail -n 20 /logs/nginx/access.log")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Search across log files
|
||||
r = await ws.execute('grep "ERROR" /logs/app.log')
|
||||
print(await r.stdout_str())
|
||||
|
||||
# Find large files
|
||||
r = await ws.execute("find /logs/ -name '*.log'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
# File metadata
|
||||
r = await ws.execute("stat /logs/app.log")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
| Command | Notes |
|
||||
| ---------------------- | ------------------------------------------ |
|
||||
| `ls` | List remote files and directories |
|
||||
| `cat` | Read remote file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (use targeted paths) |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (size, mtime) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `mkdir` | Create remote directories |
|
||||
| `touch` | Create empty remote files |
|
||||
| `cp` / `mv` / `rm` | Copy, move, delete on remote |
|
||||
| `tee` | Write stdin to remote file |
|
||||
| `diff` / `cmp` | Compare remote files |
|
||||
| `sort` / `cut` / `tr` | Text processing |
|
||||
| `tar` / `zip` / `gzip` | Compression |
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
title: Supabase Storage
|
||||
icon: bolt
|
||||
description: Mount Supabase Storage as a virtual filesystem.
|
||||
---
|
||||
The Supabase resource mounts a Supabase Storage bucket at some prefix
|
||||
such as `/supabase/`. All operations involve network I/O to the remote
|
||||
object store. Uses aioboto3 against Supabase's S3-compatible API.
|
||||
|
||||
For credential setup, see [Supabase Setup](/home/setup/supabase).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.supabase import SupabaseConfig, SupabaseResource
|
||||
|
||||
config = SupabaseConfig(
|
||||
bucket=os.environ["SUPABASE_BUCKET"],
|
||||
region=os.environ["SUPABASE_REGION"],
|
||||
project_ref=os.environ["SUPABASE_PROJECT_REF"],
|
||||
access_key_id=os.environ["SUPABASE_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["SUPABASE_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://{project_ref}.storage.supabase.co/storage/v1/s3",
|
||||
# session_token="...",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = SupabaseResource(config=config)
|
||||
ws = Workspace({"/supabase": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`SupabaseResource(config)` takes a `SupabaseConfig` object with the
|
||||
bucket name, `region`, and either `project_ref` (endpoint is auto-built
|
||||
as `https://{project_ref}.storage.supabase.co/storage/v1/s3`) or an
|
||||
explicit `endpoint_url`. Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The Supabase resource maps object keys to virtual paths under the mount
|
||||
prefix. Supabase "directories" are prefix-based — there are no real
|
||||
directory objects.
|
||||
|
||||
For example, if bucket `my-bucket` contains:
|
||||
|
||||
```text
|
||||
avatars/user-001.png
|
||||
avatars/user-002.png
|
||||
documents/report.pdf
|
||||
documents/data.csv
|
||||
```
|
||||
|
||||
Then mounting at `/supabase/` exposes:
|
||||
|
||||
```text
|
||||
/supabase/
|
||||
avatars/
|
||||
user-001.png
|
||||
user-002.png
|
||||
documents/
|
||||
report.pdf
|
||||
data.csv
|
||||
```
|
||||
|
||||
Path mapping: virtual `/supabase/avatars/user-001.png` maps to Supabase
|
||||
key `avatars/user-001.png`.
|
||||
|
||||
## Cache
|
||||
|
||||
The Supabase resource uses `IndexCacheStore` with `index_ttl = 600`
|
||||
(10 minutes). Directory listings are cached for up to 600 seconds before
|
||||
being refreshed from Supabase.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.supabase import SupabaseConfig, SupabaseResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = SupabaseConfig(
|
||||
bucket=os.environ["SUPABASE_BUCKET"],
|
||||
region=os.environ["SUPABASE_REGION"],
|
||||
project_ref=os.environ["SUPABASE_PROJECT_REF"],
|
||||
access_key_id=os.environ["SUPABASE_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["SUPABASE_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
|
||||
resource = SupabaseResource(config=config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/supabase/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /supabase/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /supabase/documents/data.csv | head -n 10")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree -L 2 /supabase/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /supabase/ -name '*.pdf'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /supabase/avatars/user-001.png")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The Supabase resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing Supabase data**: Mount Supabase buckets for agents to read and process datasets
|
||||
- **Data pipelines**: Read and write Supabase objects with shell-like commands
|
||||
- **FUSE mounting**: Expose Supabase buckets through a virtual FUSE mount for external tools
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
title: Tencent COS
|
||||
icon: /images/tencent-logo.svg
|
||||
description: Mount a Tencent Cloud Object Storage bucket via its S3-compatible API.
|
||||
---
|
||||
The Tencent resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `TencentConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just derives the right COS endpoint from
|
||||
`region`. Uses aioboto3 against Tencent COS's S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `cos.<region>.myqcloud.com`
|
||||
(e.g. `cos.ap-guangzhou.myqcloud.com`). Pass `endpoint_url` to override.
|
||||
|
||||
<Note>
|
||||
Tencent COS bucket names include the APPID suffix, e.g.
|
||||
`my-bucket-1250000000`. Use the full name (with the suffix) for `bucket`.
|
||||
</Note>
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.tencent import TencentConfig, TencentResource
|
||||
|
||||
config = TencentConfig(
|
||||
bucket=os.environ["COS_BUCKET"], # e.g. my-bucket-1250000000
|
||||
region=os.environ.get("COS_REGION", "ap-guangzhou"),
|
||||
access_key_id=os.environ["COS_SECRET_ID"],
|
||||
secret_access_key=os.environ["COS_SECRET_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://cos.ap-guangzhou.myqcloud.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = TencentResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.tencent import TencentConfig, TencentResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = TencentConfig(
|
||||
bucket=os.environ["COS_BUCKET"],
|
||||
region=os.environ.get("COS_REGION", "ap-guangzhou"),
|
||||
access_key_id=os.environ["COS_SECRET_ID"],
|
||||
secret_access_key=os.environ["COS_SECRET_KEY"],
|
||||
)
|
||||
resource = TencentResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /data/ -name '*.json' | head -n 5")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Tencent COS reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [Tencent COS Setup](/home/setup/tencent).
|
||||
@@ -0,0 +1,396 @@
|
||||
---
|
||||
title: Trello
|
||||
description: Mount Trello workspaces, boards, lists, cards, members, and labels as a Mirage filesystem for Python agents.
|
||||
icon: table-columns
|
||||
---
|
||||
The Trello resource exposes workspaces, boards, lists, cards, members, and
|
||||
labels as a virtual filesystem mounted at some prefix such as `/trello/`.
|
||||
|
||||
For API key setup, see [Trello Setup](/python/setup/trello).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.trello import TrelloConfig, TrelloResource
|
||||
|
||||
config = TrelloConfig(
|
||||
api_key=os.environ["TRELLO_API_KEY"],
|
||||
api_token=os.environ["TRELLO_API_TOKEN"],
|
||||
)
|
||||
resource = TrelloResource(config=config)
|
||||
ws = Workspace({"/trello": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```text
|
||||
/trello/
|
||||
workspaces/
|
||||
<workspace-name>__<workspace-id>/
|
||||
workspace.json
|
||||
boards/
|
||||
<board-name>__<board-id>/
|
||||
board.json
|
||||
members/
|
||||
<full-name>__<member-id>.json
|
||||
...
|
||||
labels/
|
||||
<label-name>__<label-id>.json
|
||||
...
|
||||
lists/
|
||||
<list-name>__<list-id>/
|
||||
list.json
|
||||
cards/
|
||||
<card-name>__<card-id>/
|
||||
card.json
|
||||
comments.jsonl
|
||||
...
|
||||
...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
/trello/
|
||||
workspaces/
|
||||
engineering__abc123def/
|
||||
workspace.json
|
||||
boards/
|
||||
product-roadmap__brd_001/
|
||||
board.json
|
||||
members/
|
||||
alice__mem_001.json
|
||||
bob__mem_002.json
|
||||
labels/
|
||||
bug__lbl_001.json
|
||||
feature__lbl_002.json
|
||||
lists/
|
||||
backlog__lst_001/
|
||||
list.json
|
||||
cards/
|
||||
fix-login__crd_001/
|
||||
card.json
|
||||
comments.jsonl
|
||||
add-search__crd_002/
|
||||
card.json
|
||||
comments.jsonl
|
||||
in-progress__lst_002/
|
||||
list.json
|
||||
cards/
|
||||
...
|
||||
```
|
||||
|
||||
Directory and file names embed the Trello ID after `__` so that
|
||||
resource-specific commands can reference the correct resource without
|
||||
extra lookups.
|
||||
|
||||
### Workspaces
|
||||
|
||||
Each workspace directory is named:
|
||||
|
||||
```text
|
||||
<workspace-name>__<workspace-id>
|
||||
```
|
||||
|
||||
Inside the workspace directory:
|
||||
|
||||
- `workspace.json` is the normalized workspace metadata
|
||||
- `boards/` contains one directory per board
|
||||
|
||||
### Boards
|
||||
|
||||
Each board directory is named:
|
||||
|
||||
```text
|
||||
<board-name>__<board-id>
|
||||
```
|
||||
|
||||
Inside the board directory:
|
||||
|
||||
- `board.json` is the normalized board metadata
|
||||
- `members/` contains one JSON file per board member
|
||||
- `labels/` contains one JSON file per board label
|
||||
- `lists/` contains one directory per list
|
||||
|
||||
### Lists
|
||||
|
||||
Each list directory is named:
|
||||
|
||||
```text
|
||||
<list-name>__<list-id>
|
||||
```
|
||||
|
||||
Each list directory contains:
|
||||
|
||||
- `list.json` -- normalized list metadata with fields such as `list_id`,
|
||||
`list_name`, `board_id`, `closed`, `pos`
|
||||
- `cards/` contains one directory per card
|
||||
|
||||
### Cards
|
||||
|
||||
Each card directory is named:
|
||||
|
||||
```text
|
||||
<card-name>__<card-id>
|
||||
```
|
||||
|
||||
Each card directory contains:
|
||||
|
||||
- `card.json` -- normalized card metadata with command-aligned fields such as
|
||||
`card_id`, `board_id`, `list_id`, `member_ids`, `label_ids`, `due`, `url`
|
||||
- `comments.jsonl` -- normalized comment stream ordered by `created_at`
|
||||
|
||||
`comments.jsonl` is a Mirage representation chosen for shell-friendly
|
||||
workflows. It is not a native Trello file format.
|
||||
|
||||
### Members
|
||||
|
||||
Each member file is named:
|
||||
|
||||
```text
|
||||
<full-name>__<member-id>.json
|
||||
```
|
||||
|
||||
Member JSON includes command-aligned identifiers such as `member_id` and
|
||||
`username` so the value can be reused directly in commands like
|
||||
`trello-card-assign`.
|
||||
|
||||
### Labels
|
||||
|
||||
Each label file is named:
|
||||
|
||||
```text
|
||||
<label-name>__<label-id>.json
|
||||
```
|
||||
|
||||
Label JSON includes `label_id`, `label_name`, `color`, and `board_id`.
|
||||
|
||||
## Cache
|
||||
|
||||
Uses `IndexCacheStore` (same as other resources).
|
||||
|
||||
## Example
|
||||
|
||||
See `examples/trello/trello.py` for the full working example.
|
||||
|
||||
```bash
|
||||
# List workspaces
|
||||
ls /trello/workspaces/
|
||||
|
||||
# Read workspace metadata
|
||||
cat /trello/workspaces/engineering__abc123/workspace.json
|
||||
|
||||
# List boards
|
||||
ls /trello/workspaces/engineering__abc123/boards/
|
||||
|
||||
# Read board metadata
|
||||
cat /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/board.json
|
||||
|
||||
# List lists
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/
|
||||
|
||||
# Read a card
|
||||
cat /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/fix-login__crd_001/card.json
|
||||
|
||||
# Read card comments
|
||||
cat /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/fix-login__crd_001/comments.jsonl
|
||||
|
||||
# Extract card ID with jq
|
||||
cat /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/fix-login__crd_001/card.json \
|
||||
| jq '.card_id'
|
||||
|
||||
# Read last 5 comments
|
||||
tail -n 5 /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/fix-login__crd_001/comments.jsonl
|
||||
|
||||
# List members
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/members/
|
||||
|
||||
# List labels
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/labels/
|
||||
|
||||
# Tree view
|
||||
tree -L 2 /trello/workspaces/
|
||||
```
|
||||
|
||||
## Finding IDs
|
||||
|
||||
IDs are embedded in directory and file names after `__`:
|
||||
|
||||
```bash
|
||||
# Workspace ID -- embedded in directory name
|
||||
ls /trello/workspaces/
|
||||
# -> engineering__abc123 <- workspace_id = abc123
|
||||
|
||||
# Board ID -- embedded in board directory name
|
||||
ls /trello/workspaces/engineering__abc123/boards/
|
||||
# -> roadmap__brd_001 <- board_id = brd_001
|
||||
|
||||
# List ID -- embedded in list directory name
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/
|
||||
# -> backlog__lst_001 <- list_id = lst_001
|
||||
|
||||
# Card ID -- embedded in card directory name
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/
|
||||
# -> fix-login__crd_001 <- card_id = crd_001
|
||||
|
||||
# Member ID -- embedded in member file name
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/members/
|
||||
# -> alice__mem_001.json <- member_id = mem_001
|
||||
|
||||
# Label ID -- embedded in label file name
|
||||
ls /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/labels/
|
||||
# -> bug__lbl_001.json <- label_id = lbl_001
|
||||
|
||||
# Use stat for structured metadata
|
||||
stat /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/fix-login__crd_001/card.json
|
||||
|
||||
# Extract fields with jq
|
||||
cat /trello/workspaces/engineering__abc123/boards/roadmap__brd_001/lists/backlog__lst_001/cards/fix-login__crd_001/card.json \
|
||||
| jq '{card_id, board_id, list_id, member_ids, label_ids}'
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
Standard commands available on the mounted Trello tree:
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `ls` | List workspaces, boards, lists, cards |
|
||||
| `cat` | Read .json metadata or .jsonl comments |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `basename` | Extract file name from path |
|
||||
| `dirname` | Extract directory from path |
|
||||
| `realpath` | Resolve path to absolute form |
|
||||
|
||||
## Resource-Specific Commands
|
||||
|
||||
### `trello-card-create`
|
||||
|
||||
Create a new card. Description can be passed via `--desc`, `--desc_file`, or stdin.
|
||||
|
||||
```bash
|
||||
trello-card-create --list_id LST123 --name "Fix login"
|
||||
trello-card-create --list_id LST123 --name "Fix login" --desc "Details here"
|
||||
cat brief.md | trello-card-create --list_id LST123 --name "Agent bug"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ----------------------------------- |
|
||||
| `--list_id` | yes | Trello list ID |
|
||||
| `--name` | yes | Card name |
|
||||
| `--desc` | no | Inline description text |
|
||||
| `--desc_file` | no | Path to file containing description |
|
||||
|
||||
### `trello-card-update`
|
||||
|
||||
Update an existing card. Description can be passed via `--desc`, `--desc_file`, or stdin.
|
||||
|
||||
```bash
|
||||
trello-card-update --card_id CRD123 --name "Updated title"
|
||||
trello-card-update --card_id CRD123 --desc "New description"
|
||||
trello-card-update --card_id CRD123 --closed true
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ----------------------------------- |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--name` | no | New card name |
|
||||
| `--desc` | no | Inline description text |
|
||||
| `--desc_file` | no | Path to file containing description |
|
||||
| `--due` | no | Due date |
|
||||
| `--closed` | no | Archive card (true/false) |
|
||||
|
||||
### `trello-card-move`
|
||||
|
||||
Move a card to a different list.
|
||||
|
||||
```bash
|
||||
trello-card-move --card_id CRD123 --list_id LST999
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ----------- | -------- | -------------- |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--list_id` | yes | Trello list ID |
|
||||
|
||||
### `trello-card-assign`
|
||||
|
||||
Assign a member to a card.
|
||||
|
||||
```bash
|
||||
trello-card-assign --card_id CRD123 --member_id MEM001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ---------------- |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--member_id` | yes | Trello member ID |
|
||||
|
||||
### `trello-card-comment-add`
|
||||
|
||||
Add a comment to a card. Text can be passed via `--text`, `--text_file`, or stdin.
|
||||
|
||||
```bash
|
||||
trello-card-comment-add --card_id CRD123 --text "Looks good"
|
||||
cat note.md | trello-card-comment-add --card_id CRD123
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------- | -------- | ---------------------------- |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--text` | \* | Inline comment text |
|
||||
| `--text_file` | \* | Path to file containing text |
|
||||
|
||||
\* One of `--text`, `--text_file`, or stdin is required.
|
||||
|
||||
### `trello-card-comment-update`
|
||||
|
||||
Update an existing comment. Text can be passed via `--text`, `--text_file`, or stdin.
|
||||
|
||||
```bash
|
||||
trello-card-comment-update --comment_id CMT001 --card_id CRD123 --text "Updated text"
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| -------------- | -------- | ---------------------------- |
|
||||
| `--comment_id` | yes | Trello comment ID |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--text` | \* | Inline comment text |
|
||||
| `--text_file` | \* | Path to file containing text |
|
||||
|
||||
\* One of `--text`, `--text_file`, or stdin is required.
|
||||
|
||||
### `trello-card-label-add`
|
||||
|
||||
Add a label to a card.
|
||||
|
||||
```bash
|
||||
trello-card-label-add --card_id CRD123 --label_id LBL001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------ | -------- | --------------- |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--label_id` | yes | Trello label ID |
|
||||
|
||||
### `trello-card-label-remove`
|
||||
|
||||
Remove a label from a card.
|
||||
|
||||
```bash
|
||||
trello-card-label-remove --card_id CRD123 --label_id LBL001
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
| ------------ | -------- | --------------- |
|
||||
| `--card_id` | yes | Trello card ID |
|
||||
| `--label_id` | yes | Trello label ID |
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: Wasabi
|
||||
icon: /images/wasabi-logo.svg
|
||||
description: Mount a Wasabi cloud storage bucket as a virtual filesystem.
|
||||
---
|
||||
The Wasabi resource is a thin wrapper over the [S3 resource](/python/resource/s3).
|
||||
It maps a `WasabiConfig` to an `S3Config` and reuses the exact same backend,
|
||||
commands, and behavior as S3, it just derives the right Wasabi endpoint from
|
||||
`region`. Uses aioboto3 against Wasabi's S3-compatible API.
|
||||
|
||||
The endpoint is computed from `region` as `s3.<region>.wasabisys.com`
|
||||
(`s3.wasabisys.com` for `us-east-1`). Pass `endpoint_url` to override.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.wasabi import WasabiConfig, WasabiResource
|
||||
|
||||
config = WasabiConfig(
|
||||
bucket=os.environ["WASABI_BUCKET"],
|
||||
region=os.environ.get("WASABI_REGION", "us-east-2"),
|
||||
access_key_id=os.environ["WASABI_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["WASABI_SECRET_ACCESS_KEY"],
|
||||
# Optional:
|
||||
# endpoint_url="https://s3.us-east-2.wasabisys.com",
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
resource = WasabiResource(config)
|
||||
ws = Workspace({"/data": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`region` defaults to `us-east-1`. Both `READ` and `WRITE` modes are supported.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.wasabi import WasabiConfig, WasabiResource
|
||||
|
||||
load_dotenv(".env.development")
|
||||
|
||||
config = WasabiConfig(
|
||||
bucket=os.environ["WASABI_BUCKET"],
|
||||
region=os.environ.get("WASABI_REGION", "us-east-2"),
|
||||
access_key_id=os.environ["WASABI_ACCESS_KEY_ID"],
|
||||
secret_access_key=os.environ["WASABI_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
resource = WasabiResource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/data/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /data/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Wasabi reports `ResourceName.S3` and routes through the same `core/s3`
|
||||
implementation, so the full S3 shell-command set applies. See the
|
||||
[S3 resource](/python/resource/s3) for the complete command reference,
|
||||
range reads, streaming, and the index cache fast path.
|
||||
- For credential setup, see [Wasabi Setup](/home/setup/wasabi).
|
||||
Reference in New Issue
Block a user