Files
strukto-ai--mirage/docs/home/cache.mdx
T
wehub-resource-sync bcbd1bdb22
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
chore: import upstream snapshot with attribution
2026-07-13 12:30:44 +08:00

107 lines
3.8 KiB
Plaintext

---
title: Cache
description: The two-layer workspace cache, index and file, with RAM and Redis stores.
icon: database
---
## What It Does
Every `Workspace` ships with a **two-layer cache** so repeated work against remote backends (S3, GDrive, Slack, ...) hits local state instead of the network:
- **Index cache.** Listings and metadata. The first directory walk hits the API; subsequent ones serve from the index until the TTL expires.
- **File cache.** Object bytes. The first read streams from origin; later pipelines read from cache.
## Stores
Each layer is a pluggable store with two built-ins:
- **RAM** (default): in-process, zero setup, 512 MB file cache and 10-minute index TTL. Best for single-process apps and notebooks.
- **Redis**: shared across workers, processes, and machines. Best for serverless, multi-replica services, or for cache state that survives restarts.
<CodeGroup>
```python Python
from mirage import Workspace
from mirage.cache.file.config import RedisCacheConfig
from mirage.cache.index.config import RedisIndexConfig
from mirage.resource.s3 import S3Config, S3Resource
ws = Workspace(
{"/s3": S3Resource(S3Config(bucket="my-bucket"))},
cache=RedisCacheConfig(url="redis://localhost:6379/0", limit="8GB"),
index=RedisIndexConfig(url="redis://localhost:6379/0", ttl=600),
)
```
```typescript TypeScript
import { RedisFileCacheStore, S3Resource, Workspace } from '@struktoai/mirage-node'
const ws = new Workspace(
{ '/s3': new S3Resource({ bucket: 'my-bucket' }) },
{
cache: new RedisFileCacheStore({ url: 'redis://localhost:6379/0', cacheLimit: '8GB' }),
index: { type: 'redis', url: 'redis://localhost:6379/0', ttl: 600 },
},
)
```
</CodeGroup>
## Eviction & Limits
The two layers are bounded differently:
| Layer | Holds | Default | Bound | Eviction |
| --- | --- | --- | --- | --- |
| **File cache** | object bytes per virtual path | RAM, 512 MB | `cache_limit` (Py) / `cacheLimit` (TS) | LRU: least-recently-used bytes drop once the total exceeds the limit |
| **Index cache** | directory listings + `FileStat` metadata | RAM, 10-min TTL | `ttl` (seconds) | time-based: entries expire after the TTL, then re-fetch on next access |
Raising the file limit keeps more bytes warm at the cost of memory; lengthening the index TTL serves listings longer between API walks at the cost of staleness.
## Miss/Hit Lifecycle
<CodeGroup>
```python Python
from mirage import Workspace
from mirage.resource.s3 import S3Config, S3Resource
ws = Workspace({"/s3": S3Resource(S3Config(bucket="my-bucket"))})
# 1. Index miss → S3 LIST. Listing stored in index cache.
await ws.execute("ls /s3/data/")
# 2. Index hit → 0 network calls.
await ws.execute('find /s3/data/ -name "*.jsonl"')
# 3. File miss → S3 GET. Bytes stored in file cache.
await ws.execute("cat /s3/data/log.jsonl | wc -l")
# 4. File hit → 0 network calls.
await ws.execute("grep alert /s3/data/log.jsonl")
```
```typescript TypeScript
import { S3Resource, Workspace } from '@struktoai/mirage-node'
const ws = new Workspace({ '/s3': new S3Resource({ bucket: 'my-bucket' }) })
// 1. Index miss → S3 LIST. Listing stored in index cache.
await ws.execute('ls /s3/data/')
// 2. Index hit → 0 network calls.
await ws.execute('find /s3/data/ -name "*.jsonl"')
// 3. File miss → S3 GET. Bytes stored in file cache.
await ws.execute('cat /s3/data/log.jsonl | wc -l')
// 4. File hit → 0 network calls.
await ws.execute('grep alert /s3/data/log.jsonl')
```
</CodeGroup>
## Relationship To Snapshots
The file cache is exactly what a [snapshot](/home/snapshot) serializes: `ws.snapshot()` writes the cached bytes for every touched path into the tar, and `Workspace.load()` restores them into the file cache so a replayed run reads from local state. The index cache is not snapshotted; it rebuilds lazily after load.