chore: import upstream snapshot with attribution
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,248 @@
|
||||
---
|
||||
title: S3
|
||||
description: Mount AWS S3 or S3-compatible buckets as a Mirage filesystem with async access, caching, and shell commands.
|
||||
icon: aws
|
||||
---
|
||||
The S3 resource mounts an Amazon S3 bucket (or any S3-compatible service) at
|
||||
some prefix such as `/s3/`. All operations involve network I/O to the remote
|
||||
object store. Uses aioboto3 for async S3 access.
|
||||
|
||||
Compatible with: AWS S3, MinIO, Cloudflare R2, Supabase Storage,
|
||||
DigitalOcean Spaces, and any S3-compatible service.
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.s3 import S3Resource, S3Config
|
||||
|
||||
config = S3Config(
|
||||
bucket="my-bucket",
|
||||
region="us-east-1",
|
||||
# Optional:
|
||||
# aws_access_key_id="...",
|
||||
# aws_secret_access_key="...",
|
||||
# aws_profile="my-profile",
|
||||
# endpoint_url="http://localhost:9000", # MinIO
|
||||
# path_style=True, # For MinIO/R2
|
||||
# timeout=30,
|
||||
# proxy="http://proxy:8080",
|
||||
)
|
||||
|
||||
resource = S3Resource(config)
|
||||
ws = Workspace({"/s3": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
`S3Resource(config)` takes an `S3Config` object with the bucket name and
|
||||
optional credentials, endpoint, and connection settings. Both `READ` and
|
||||
`WRITE` modes are supported.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
The S3 resource maps S3 object keys to virtual paths under the mount prefix.
|
||||
S3 "directories" are prefix-based - there are no real directory objects.
|
||||
|
||||
For example, if bucket `my-bucket` contains:
|
||||
|
||||
```text
|
||||
data/file.txt
|
||||
data/config.json
|
||||
reports/q1.csv
|
||||
reports/q2.csv
|
||||
```
|
||||
|
||||
Then mounting at `/s3/` exposes:
|
||||
|
||||
```text
|
||||
/s3/
|
||||
data/
|
||||
file.txt
|
||||
config.json
|
||||
reports/
|
||||
q1.csv
|
||||
q2.csv
|
||||
```
|
||||
|
||||
Path mapping: virtual `/s3/data/file.txt` maps to S3 key `data/file.txt`.
|
||||
|
||||
## Cache
|
||||
|
||||
The S3 resource uses `IndexCacheStore` with `index_ttl = 600` (10 minutes).
|
||||
Directory listings are cached for up to 600 seconds before being refreshed
|
||||
from S3. This reduces API calls for repeated directory traversals.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.s3 import S3Resource, S3Config
|
||||
|
||||
config = S3Config(
|
||||
bucket="my-bucket",
|
||||
region="us-east-1",
|
||||
)
|
||||
|
||||
resource = S3Resource(config)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/s3/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /s3/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("cat /s3/data/file.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("tree /s3/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /s3/ -name '*.json'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("grep example /s3/data/config.json")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("stat /s3/data/file.txt")
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The S3 resource supports the full set of shell commands since it operates
|
||||
on real file content (text, binary, JSON, CSV, etc.). Large files benefit
|
||||
from range reads to avoid downloading entire objects.
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
| `du` | Disk usage summary |
|
||||
| `file` | Detect file type |
|
||||
| `strings` | Extract printable strings from binary |
|
||||
| `xxd` | Hex dump |
|
||||
| `md5` | MD5 checksum |
|
||||
| `sha256sum` | SHA-256 checksum |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | ------------------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `join` | Join lines on a common field |
|
||||
| `paste` | Merge lines side by side |
|
||||
| `column` | Columnate output |
|
||||
| `fold` | Wrap lines to a specified width |
|
||||
| `expand` | Convert tabs to spaces |
|
||||
| `unexpand` | Convert spaces to tabs |
|
||||
| `fmt` | Simple text formatter |
|
||||
| `rev` | Reverse lines |
|
||||
| `tac` | Concatenate and print in reverse |
|
||||
| `look` | Display lines beginning with a given string |
|
||||
| `shuf` | Shuffle lines |
|
||||
| `tsort` | Topological sort |
|
||||
| `comm` | Compare two sorted files |
|
||||
| `cmp` | Compare two files byte by byte |
|
||||
| `diff` | Compare files line by line |
|
||||
| `patch` | Apply a diff patch |
|
||||
| `iconv` | Character encoding conversion |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | ------------------------------------- |
|
||||
| `cp` | Copy files |
|
||||
| `mv` | Move/rename files |
|
||||
| `rm` | Remove files |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
| `ln` | Create symbolic links |
|
||||
| `tee` | Write stdin to file and stdout |
|
||||
| `mktemp` | Create temporary file |
|
||||
| `split` | Split file into pieces |
|
||||
| `csplit` | Split file by context |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ---------- | -------------------------- |
|
||||
| `basename` | Strip directory from path |
|
||||
| `dirname` | Strip filename from path |
|
||||
| `realpath` | Resolve path |
|
||||
| `readlink` | Print symbolic link target |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
### Compression
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | --------------------- |
|
||||
| `gzip` | Compress files |
|
||||
| `gunzip` | Decompress gzip files |
|
||||
| `zip` | Create zip archives |
|
||||
| `unzip` | Extract zip archives |
|
||||
| `tar` | Archive files |
|
||||
| `zcat` | Cat compressed files |
|
||||
| `zgrep` | Grep compressed files |
|
||||
|
||||
### Encoding
|
||||
|
||||
| Command | Notes |
|
||||
| -------- | -------------------- |
|
||||
| `base64` | Base64 encode/decode |
|
||||
|
||||
### Data Format Support
|
||||
|
||||
Commands with format-specific variants for structured data files:
|
||||
|
||||
| Format | Extension | Variants |
|
||||
| ------- | ---------- | ---------------------------------------------- |
|
||||
| Parquet | `.parquet` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| Feather | `.feather` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| ORC | `.orc` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
| HDF5 | `.hdf5` | cat, head, tail, wc, stat, cut, grep, ls, file |
|
||||
|
||||
These variants auto-detect the format by extension and convert to
|
||||
tabular text (CSV) for processing.
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **AI agents accessing cloud data**: Mount S3 buckets for agents to read and process remote datasets
|
||||
- **Data pipelines**: Read and write S3 objects with shell-like commands
|
||||
- **Sandboxed cloud storage access**: Restrict agent operations to a specific bucket and prefix
|
||||
- **FUSE mounting**: Expose S3 buckets through a virtual FUSE mount for external tools
|
||||
|
||||
## Scoping a resource to a key prefix
|
||||
|
||||
Pass `key_prefix: str | None = None` to `S3Config` to transparently scope every operation to a subpath of the bucket:
|
||||
|
||||
```python
|
||||
S3Resource(S3Config(
|
||||
bucket="app-data",
|
||||
region="eu-west-1",
|
||||
key_prefix=f"users/{user_id}/",
|
||||
))
|
||||
```
|
||||
|
||||
When set, every read/write/list/stat/copy/rename/delete operation is transparently scoped to that bucket subpath. Agents see clean paths like `/data/notes.md`; the underlying bucket key is `users/{user_id}/data/notes.md`. Useful for multi-tenant systems. Pairs naturally with STS AssumeRole session policies for AWS-side enforcement. Unset behavior is unchanged.
|
||||
|
||||
**Normalization:** leading slashes are stripped and a trailing slash is added automatically. Both `None` and an empty string are treated as "no prefix."
|
||||
Reference in New Issue
Block a user