bcbd1bdb22
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
186 lines
12 KiB
Plaintext
186 lines
12 KiB
Plaintext
---
|
|
title: Limitations
|
|
icon: triangle-exclamation
|
|
description: Runtime-specific constraints in the TypeScript SDK (Node and Browser) and how to work around them.
|
|
---
|
|
|
|
Most of Mirage's behavior is identical across Python and TypeScript. The two TS runtimes (Node and Browser) each impose a handful of constraints that don't exist in Python. All are documented here so you can plan around them before they surprise you in production.
|
|
|
|
<Note>
|
|
`python3` has its own set of WASM-runtime-level divergences from Python Mirage's subprocess model. See [Python](/typescript/python#limitations) for the full list.
|
|
</Note>
|
|
|
|
## Node
|
|
|
|
### 1. `fs-monkey` only patches CJS `require('fs')`, not ESM `node:fs`
|
|
|
|
**The problem.** `patchNodeFs()` routes `fs` calls through the workspace VFS so that third-party libraries "just work" against mounted paths. It works by replacing `require.cache`'s `fs` entry (a CJS-only mechanism). ESM is fundamentally different:
|
|
|
|
- `import { readFile } from 'node:fs/promises'` resolves at parse time to V8's internal binding.
|
|
- There is no public hook to replace that binding after the fact.
|
|
- Loader hooks (`--loader=…`) could intercept the resolution, but they're a build-time decision, not a runtime monkey-patch.
|
|
|
|
**What this means in practice.**
|
|
|
|
```ts
|
|
// ✅ Works, this is CJS and goes through require('fs')
|
|
const { readFileSync } = require('fs')
|
|
patchNodeFs(ws)
|
|
readFileSync('/data/x.txt') // routes through RAM resource
|
|
|
|
// ❌ Does not work, this is ESM and bypasses the patch
|
|
import { readFileSync } from 'node:fs'
|
|
patchNodeFs(ws)
|
|
readFileSync('/data/x.txt') // hits the real filesystem, throws ENOENT
|
|
```
|
|
|
|
**Why Python doesn't hit this.** Python has no equivalent concept of "ESM vs CJS". `with Workspace() as ws:` swaps `builtins.open` and `sys.modules["os"]`: one set of mutable globals, one patch point, works for every caller.
|
|
|
|
**Workarounds.**
|
|
|
|
<Tabs>
|
|
<Tab title="Use FUSE instead">
|
|
If you want ESM-imported `node:fs` to see your mounted data, expose the workspace as a real filesystem. Mount FUSE, then every `fs` call, ESM or CJS, goes through the kernel.
|
|
|
|
```ts
|
|
import { readFile } from 'node:fs/promises'
|
|
const ws = new Workspace({ '/data': new Mount(new RAMResource(), { fuse: true }) })
|
|
await ws.fuseReady()
|
|
// The `/data` subtree is exposed at the mountpoint root, so paths are relative to it.
|
|
const bytes = await readFile(`${ws.fuseMountpoint}/x.txt`)
|
|
```
|
|
</Tab>
|
|
<Tab title="Use Mirage's VFS API directly">
|
|
The most reliable approach: call `ws.execute()` or the resource methods. No magic, no monkey-patching, works from ESM or CJS.
|
|
|
|
```ts
|
|
const res = await ws.execute('cat /data/x.txt')
|
|
const bytes = res.stdout
|
|
```
|
|
</Tab>
|
|
<Tab title="Restrict your code to CJS">
|
|
If you really need `fs` patching to work, keep the consuming code in CJS. Note that most modern Node projects default to ESM, so this is a narrow escape hatch.
|
|
|
|
```js
|
|
// package.json: no "type": "module"
|
|
const { readFileSync } = require('fs')
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
**Possible future fix.** A published Node loader hook that resolves `node:fs` through the workspace would remove the ESM limitation, at the cost of forcing consumers to opt into the loader (`node --loader @struktoai/mirage-node/loader main.mjs`). Not planned currently.
|
|
|
|
---
|
|
|
|
### 2. FUSE files from API-backed resources cap at 100 MiB
|
|
|
|
**The problem.** API-backed resources (Trello, Linear, Slack, MongoDB, etc.) return `stat.size = null` because the byte size isn't known until the API has been called. Python passes `direct_io=True` to libfuse at mount time so the kernel ignores reported size and issues `read()` until it returns 0 (Slack's daily history can be tens of MB and it just works).
|
|
|
|
`@zkochan/fuse-native` doesn't expose `direct_io`. There's no per-file flag (the `open` C bridge can only return `fh`, not modify `fuse_file_info.direct_io`), and the `-o direct_io` mount option is rejected by macFUSE/libosxfuse and crashes the channel. So when Mirage's FUSE layer hits a `size=null` file, it has to report **some** non-zero size to make the kernel issue reads, otherwise `cat board.json` would print empty.
|
|
|
|
We report a 100 MiB sentinel. The read handler returns 0 past the actual data length, so `cat`, `wc -c`, and friends correctly see EOF for files smaller than the sentinel. **Files larger than 100 MiB get truncated.**
|
|
|
|
**What this means in practice.**
|
|
|
|
```ts
|
|
const ws = new Workspace({ '/slack': new Mount(new SlackResource({ apiKey }), { fuse: true }) })
|
|
await ws.fuseReady()
|
|
// The `/slack` subtree is exposed at the mountpoint root, so paths are relative to it.
|
|
const slackMp = ws.fuseMountpoint
|
|
// ✅ Works, most channels have <100 MiB of daily history
|
|
await readFile(`${slackMp}/channels/general__C123/2026-04-12/chat.jsonl`)
|
|
|
|
// ⚠️ Truncated to 100 MiB if the actual day exceeds it
|
|
await readFile(`${slackMp}/channels/super-busy-channel__C456/2026-04-12/chat.jsonl`)
|
|
```
|
|
|
|
`ls -l` shows `100M` for any unfetched API-backed file. Once a file has been opened, the real size is cached and subsequent `ls -l` shows the actual byte count.
|
|
|
|
The cap exists because Node's `fs/promises.readFile` allocates a Buffer of the reported size and decodes it as utf-8 at the end; V8's string length limit is ~512 MiB, so a larger sentinel throws `RangeError: Invalid string length` (which is what you'd hit before any real read).
|
|
|
|
**Knock-on effect for JSON files.** Because the kernel zero-pads the unread tail of the 100 MiB buffer, `JSON.parse(readFile(...))` on a FUSE-mounted JSON file fails at the first ` ` byte right after the real payload with `SyntaxError: Unexpected non-whitespace character after JSON`. Streaming tools (`cat`, `grep`, `head`, `wc -c`, `jq`) stop at EOF and are unaffected; only consumers that decode the full buffer as UTF-8 and then parse it choke on the padding. MongoDB's FUSE-exposed `schema.json`, `database.json`, and `documents.jsonl` are the most visible cases, see [TS MongoDB → FUSE caveat](/typescript/setup/mongodb#fuse-caveat-padded-reads-of-schemajson-documentsjsonl) for workarounds.
|
|
|
|
**Why Python doesn't hit this.** Python's `mfusepy` accepts `direct_io=True` and passes it to libfuse2 as a mount option. macFUSE supports it through that path even though it rejects the same option from `@zkochan/fuse-native`'s option string. With direct_io enabled, the kernel doesn't care what size getattr reports.
|
|
|
|
**Workarounds.**
|
|
|
|
<Tabs>
|
|
<Tab title="Read via the VFS API">
|
|
The Mirage VFS doesn't go through FUSE at all (no sentinel, no truncation). Use this for any file that might exceed 100 MiB.
|
|
|
|
```ts
|
|
const bytes = await ws.fs.readFile('/slack/channels/super-busy-channel__C456/2026-04-12/chat.jsonl')
|
|
// Or via shell:
|
|
const res = await ws.execute('cat /slack/channels/super-busy-channel__C456/2026-04-12/chat.jsonl')
|
|
```
|
|
</Tab>
|
|
<Tab title="Process incrementally">
|
|
For Slack history specifically, narrow the date or use a more targeted command (`grep`, `jq`, line-count stats) so the result fits well under the cap.
|
|
|
|
```ts
|
|
await ws.execute('jq -r ".[] | .text" /slack/channels/general__C123/2026-04-12/chat.jsonl')
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
**Possible future fix.** Patch `@zkochan/fuse-native`'s C bridge to set `info->direct_io = 1` in the `open` callback (a one-line change applied via `pnpm patch`). That achieves Python parity and removes the 100 MiB cap entirely. Tracked but not yet implemented.
|
|
|
|
---
|
|
|
|
## Browser
|
|
|
|
The browser SDK runs entirely in-page: no kernel, no subprocesses, no Node `fs`. That removes the Node sections above (neither FUSE nor `fs-monkey` apply) but introduces its own constraints.
|
|
|
|
### 1. No FUSE
|
|
|
|
Browsers can't mount filesystems. A workspace with a fuse `Mount` throws on construction. Use `ws.execute(...)` (virtual executor) and `ws.fs.readFile/writeFile` instead. Every builtin (`cat`, `grep`, `jq`, `awk`, `python3`, etc.) is reimplemented in-process, so most agent code paths work unchanged from Node.
|
|
|
|
### 2. OPFS quotas and persistence
|
|
|
|
`OPFSResource` writes through the [Origin Private File System](https://web.dev/articles/origin-private-file-system). Two things to know:
|
|
|
|
- **Storage quota.** The browser sets per-origin quotas (typically a fraction of free disk, single-digit GB on most setups). Hitting it raises `QuotaExceededError`. Call `navigator.storage.estimate()` to inspect.
|
|
- **Eviction.** Origins that aren't [persisted](https://web.dev/articles/persistent-storage) can be cleared by the browser under storage pressure. For long-lived workspaces, request `navigator.storage.persist()` early.
|
|
|
|
### 3. CORS for HTTP-backed resources
|
|
|
|
Mounts that hit third-party APIs (S3, GitHub, Linear, etc.) make `fetch` calls from the page. Anything not configured to allow your origin via CORS will fail with the usual browser error. Workarounds:
|
|
|
|
- **Browser-native auth flows.** Resources like Box, Dropbox, GDrive, GDocs ship PKCE OAuth examples that work entirely in-browser.
|
|
- **Pre-signed URLs.** For S3/R2/GCS, generate pre-signed URLs server-side and pass them in. The Mirage browser examples include a Vite dev-server `presigner` plugin as reference.
|
|
- **Same-origin proxy.** Stand up a tiny proxy on your own domain that forwards to the upstream API with the right auth headers.
|
|
|
|
### 4. Python writes need JSPI
|
|
|
|
The Python FS shim (`open()` against mounted paths) flushes writes back through an async bridge. That requires [JSPI](https://github.com/WebAssembly/js-promise-integration). Reads of preloaded files still work without it, but `close()` on a write throws `RuntimeError: Cannot stack switch`.
|
|
|
|
- **Chrome / Edge 137+** (May 2025): works out of the box.
|
|
- **Firefox**: behind `javascript.options.wasm_js_promise_integration`.
|
|
- **Safari**: not yet shipped.
|
|
|
|
See [Python FS shim](/typescript/python-fs#runtime-requirements) for the full matrix.
|
|
|
|
### 5. No SSH, Postgres, MongoDB, LanceDB, Email, FUSE peers
|
|
|
|
These resources are only exposed by `@struktoai/mirage-node` because their drivers are Node-only. Importing them from `@struktoai/mirage-browser` is a build error. For browser apps, route those reads through your backend (or use the HTTP-driver variants where they exist, e.g. MongoDB via the bundled mongo-proxy in the examples).
|
|
|
|
---
|
|
|
|
## Quick reference
|
|
|
|
| Scenario | Runtime | Status |
|
|
|---|---|---|
|
|
| `patchNodeFs()` with `require('fs')` (CJS) | Node | ✅ works |
|
|
| `patchNodeFs()` with `import from 'node:fs'` (ESM) | Node | ❌ silently bypassed; use FUSE or the VFS API |
|
|
| FUSE `cat` of a Trello/Linear/Slack file ≤ 100 MiB | Node | ✅ works |
|
|
| FUSE `cat` of an API-backed file > 100 MiB | Node | ❌ truncated; use `ws.fs.readFile` or `ws.execute('cat …')` |
|
|
| `ws.execute(...)` virtual builtins | Browser | ✅ works |
|
|
| `Workspace` with a fuse `Mount` | Browser | ❌ throws; not supported |
|
|
| OPFS reads/writes within quota | Browser | ✅ works |
|
|
| OPFS over quota | Browser | ❌ `QuotaExceededError`; check `navigator.storage.estimate()` |
|
|
| HTTP-backed resources without CORS allow-list | Browser | ❌ blocked; use PKCE, presigned URLs, or a same-origin proxy |
|
|
| Python `open()` writes back through mount | Browser | ✅ with JSPI (Chrome 137+); ❌ otherwise |
|
|
| `@struktoai/mirage-node`-only resources (SSH, Postgres, MongoDB, LanceDB, Email, FUSE) | Browser | ❌ Node-only drivers |
|
|
|
|
The Node limitations are runtime-level, not Mirage design decisions. Python's equivalent behavior is strictly better in each case, so if a workflow absolutely requires ESM-level `fs` patching or large API-backed FUSE reads, consider whether the Python SDK fits better for that specific piece. The Browser limitations are by design (no kernel, no subprocess), so the workarounds there are about choosing the right runtime for the task.
|