Ok now let's refactor the mirage/backends

let's group similar backend together to same subfolder

for example s3 and r2 are object storage (maybe a better name) etc.

and loader etc we should put to another folder etc

let's brainstorm and do the refactoring and renaming and new subfolders


The user wants to refactor the mirage/backends directory by grouping similar backends into subfolders. They want to brainstorm first. Let me invoke the brainstorming skill as required.


Using the brainstorming skill to explore the refactoring approach before implementation.


Launching skill: superpowers:brainstorming


Base directory for this skill: /Users/agent/.claude/plugins/cache/superpowers-marketplace/superpowers/4.0.3/skills/brainstorming

# Brainstorming Ideas Into Designs

## Overview

Help turn ideas into fully formed designs and specs through natural collaborative dialogue.

Start by understanding the current project context, then ask questions one at a time to refine the idea. Once you understand what you're building, present the design in small sections (200-300 words), checking after each section whether it looks right so far.

## The Process

**Understanding the idea:**
- Check out the current project state first (files, docs, recent commits)
- Ask questions one at a time to refine the idea
- Prefer multiple choice questions when possible, but open-ended is fine too
- Only one question per message - if a topic needs more exploration, break it into multiple questions
- Focus on understanding: purpose, constraints, success criteria

**Exploring approaches:**
- Propose 2-3 different approaches with trade-offs
- Present options conversationally with your recommendation and reasoning
- Lead with your recommended option and explain why

**Presenting the design:**
- Once you believe you understand what you're building, present the design
- Break it into sections of 200-300 words
- Ask after each section whether it looks right so far
- Cover: architecture, components, data flow, error handling, testing
- Be ready to go back and clarify if something doesn't make sense

## After the Design

**Documentation:**
- Write the validated design to `docs/plans/YYYY-MM-DD-<topic>-design.md`
- Use elements-of-style:writing-clearly-and-concisely skill if available
- Commit the design document to git

**Implementation (if continuing):**
- Ask: "Ready to set up for implementation?"
- Use superpowers:using-git-worktrees to create isolated workspace
- Use superpowers:writing-plans to create detailed implementation plan

## Key Principles

- **One question at a time** - Don't overwhelm with multiple questions
- **Multiple choice preferred** - Easier to answer than open-ended when possible
- **YAGNI ruthlessly** - Remove unnecessary features from all designs
- **Explore alternatives** - Always propose 2-3 approaches before settling
- **Incremental validation** - Present design in sections, validate each
- **Be flexible** - Go back and clarify when something doesn't make sense


Let me start by understanding the current state of the backends directory.


Bash

List all files in backends directory


/Users/agent/strukto/mirage/mirage/backends/base.py
/Users/agent/strukto/mirage/mirage/backends/loader.py
/Users/agent/strukto/mirage/mirage/backends/r2.py
/Users/agent/strukto/mirage/mirage/backends/__init__.py
/Users/agent/strukto/mirage/mirage/backends/proxy.py
/Users/agent/strukto/mirage/mirage/backends/http.py
/Users/agent/strukto/mirage/mirage/backends/local.py
/Users/agent/strukto/mirage/mirage/backends/memory.py
/Users/agent/strukto/mirage/mirage/backends/s3.py


total 120
drwxr-xr-x@ 12 agent  staff    384 Mar 17 06:34 .
drwxr-xr-x@ 13 agent  staff    416 Mar 17 06:34 ..
-rw-r--r--@  1 agent  staff    336 Mar 16 16:30 __init__.py
drwxr-xr-x@ 14 agent  staff    448 Mar 17 04:17 __pycache__
-rw-r--r--@  1 agent  staff   4143 Mar 14 09:39 base.py
-rw-r--r--@  1 agent  staff   4381 Mar 17 03:58 http.py
-rw-r--r--@  1 agent  staff   1357 Mar 15 02:22 loader.py
-rw-r--r--@  1 agent  staff   3918 Mar 17 04:02 local.py
-rw-r--r--@  1 agent  staff   7034 Mar 17 04:02 memory.py
-rw-r--r--@  1 agent  staff   2689 Mar 16 23:31 proxy.py
-rw-r--r--@  1 agent  staff   1244 Mar 16 01:23 r2.py
-rw-r--r--@  1 agent  staff  13310 Mar 17 04:17 s3.py


Agent

Explore all backend files

Prompt: Read all the backend files in mirage/backends/ to understand what each one does and what classes/interfaces they expose. Read: base.py, loader.py, r2.py, s3.py, proxy.py, http.py, local.py, memory.py, __init__.py. For each file, tell me: the main class name, what it inherits from, and a brief description of what it does. Also check what imports reference these backends from other parts of the codebase (grep for "from mirage.backends" and "import mirage.backends").


Perfect! Now I have all the information needed. Let me compile a comprehensive report:

## Backend Files Summary

Here's a comprehensive breakdown of each backend file in `/Users/agent/strukto/mirage/mirage/backends/`:

### 1. **base.py** - BaseBackend
- **Main Class:** `BaseBackend`
- **Inherits from:** None (base class)
- **Description:** Abstract base class that defines the interface for all backends. Contains required operations (read, readdir, stat, open, release, write, create, unlink, rmdir, mkdir, rename, truncate) that subclasses must implement. Also provides helper methods built from those primitives (cat, ls, ls_l, grep, head, tail, presign, put).

### 2. **memory.py** - MemoryBackend
- **Main Class:** `MemoryBackend`
- **Inherits from:** `BaseBackend`
- **Description:** In-memory backend using plain Python dicts. No external dependencies. Stores file content in `_files` dict and tracks directories in `_dirs` set. Useful for tests, ephemeral scratch space, and caching. Also contains `_guess_type()` helper function to infer file types from extensions.

### 3. **local.py** - LocalBackend
- **Main Class:** `LocalBackend`
- **Inherits from:** `BaseBackend`
- **Description:** Filesystem backend using pathlib/os. Maps virtual paths to local filesystem paths relative to a root directory. No external dependencies. Prevents path traversal outside root directory.

### 4. **s3.py** - S3Backend
- **Main Class:** `S3Backend`
- **Inherits from:** `BaseBackend`
- **Description:** S3-compatible backend using boto3. Requires optional dependency. Implements optimized operations like S3 Select for grep/head on large files, batch ls_l using list_objects_v2, and presigned URLs. Key design: uses Range headers for partial reads, full PutObject for writes (S3 has no partial write).

### 5. **r2.py** - R2Backend
- **Main Class:** `R2Backend`
- **Inherits from:** `S3Backend`
- **Description:** Cloudflare R2 backend that extends S3Backend. Auto-computes the endpoint URL from account_id. Disables S3 Select (R2 doesn't support SelectObjectContent). Requires optional dependency.

### 6. **http.py** - HTTPBackend
- **Main Class:** `HTTPBackend`
- **Inherits from:** `BaseBackend`
- **Description:** Read-only HTTP backend using httpx. Maps virtual paths to HTTP URLs. Requires optional dependency. Requires user-supplied `readdir_fn` callback since HTTP has no standard directory listing protocol. All write operations raise NotImplementedError.

### 7. **proxy.py** - ProxyBackend
- **Main Class:** `ProxyBackend`
- **Inherits from:** `BaseBackend`
- **Description:** Remote proxy backend that delegates operations to a remote mirage server via HTTP. Uses base64-encoded data in JSON payloads for read/write. Does not support open/release operations.

### 8. **loader.py** - Utility Module (not a backend)
- **Main Function:** `load_backend_class(spec: str)`
- **Description:** Dynamically loads backend classes from either file paths (e.g., "./my_backend.py:MyClass") or module dotpaths (e.g., "mypackage.backends:MyClass"). Used for custom backend plugins.

### 9. **__init__.py** - Package Initialization
- **Exports:** BaseBackend, LocalBackend, MemoryBackend, ProxyBackend (eagerly imported)
- **Lazy Imports:** S3Backend, R2Backend, HTTPBackend are mentioned as lazily imported to avoid hard dependencies

---

## Import Usage Throughout Codebase

The backends are imported in these key locations:

### Core Framework Files:
- **mirage/workspace/mount.py** - Imports `BaseBackend`
- **mirage/workspace/__init__.py** - Imports `BaseBackend` and `_guess_type` helper
- **mirage/cli/config.py** - Imports all backends conditionally (lazy imports for S3, R2, HTTP)
- **mirage/cli/commands/workspace.py** - Imports `load_backend_class` and `ProxyBackend`
- **mirage/cli/commands/command.py** - Imports `load_backend_class`
- **mirage/runtime/local.py** - Imports `LocalBackend`
- **mirage/runtime/remote.py** - Imports `ProxyBackend`
- **mirage/workspace/cache/policy.py** - Imports `BaseBackend`
- **mirage/workspace/sync.py** - Imports `BaseBackend`
- **mirage/server/service/local.py** - Imports `load_backend_class`
- **mirage/__init__.py** - Exports `LocalBackend` and `MemoryBackend` at package level

### Test Files:
- Extensive use in tests for MemoryBackend (test fixtures), LocalBackend (filesystem tests), and ProxyBackend (remote tests)
- Tests include: test_local.py, test_proxy.py, test_loader.py, test_configs.py

### Examples:
- examples/r2.py - Uses R2Backend
- examples/claude-memory.py, examples/claude-sdk.py - Use MemoryBackend

agentId: ab90e4dc934b387be (for resuming to continue this agent's work if needed)
<usage>total_tokens: 31005
tool_uses: 14
duration_ms: 31609</usage>

Prompt: Read all the backend files in mirage/backends/ to understand what each one does and what classes/interfaces they expose. Read: base.py, loader.py, r2.py, s3.py, proxy.py, http.py, local.py, memory.py, __init__.py. For each file, tell me: the main class name, what it inherits from, and a brief description of what it does. Also check what imports reference these backends from other parts of the codebase (grep for "from mirage.backends" and "import mirage.backends").


Now I have a good understanding of all the backends. Let me think about the grouping:

1. **base.py** - Base class, should stay at the top level
2. **memory.py** - In-memory storage
3. **local.py** - Local filesystem storage
4. **s3.py** - S3 object storage
5. **r2.py** - R2 object storage (extends S3)
6. **http.py** - Read-only HTTP backend
7. **proxy.py** - Remote proxy backend
8. **loader.py** - Utility for loading backends

Possible groupings:

- **Object storage**: s3.py, r2.py (cloud object storage)
- **Local storage**: local.py, memory.py (local/in-process storage)
- **Network/Remote**: http.py, proxy.py (network-based backends)
- **Core**: base.py, loader.py (foundational stuff)

Or I could organize it with base.py and loader.py at the top level, then create subdirectories for object_storage (s3.py, r2.py), local (local.py, memory.py), and network (http.py, proxy.py).

I should ask the user which structure they prefer before moving forward.


Good, I've got a clear picture of all 8 backends. Here's what we're working with:

| File | Class | Description |
|------|-------|-------------|
| `base.py` | `BaseBackend` | Abstract base class |
| `memory.py` | `MemoryBackend` | In-memory dict storage |
| `local.py` | `LocalBackend` | Local filesystem |
| `s3.py` | `S3Backend` | AWS S3 via boto3 |
| `r2.py` | `R2Backend` (extends S3) | Cloudflare R2 |
| `http.py` | `HTTPBackend` | Read-only HTTP |
| `proxy.py` | `ProxyBackend` | Remote mirage server proxy |
| `loader.py` | `load_backend_class()` | Dynamic class loader utility |

I see three natural groupings:

1. **Object storage** — `s3.py`, `r2.py` (cloud object stores, R2 even inherits from S3)
2. **Local** — `local.py`, `memory.py` (in-process, no network)
3. **Network** — `http.py`, `proxy.py` (remote access over HTTP)

Then `base.py` and `loader.py` stay at the top level as foundational pieces.

Does this grouping feel right, or would you organize them differently? For instance, should `memory.py` be in its own category since it's more of a "virtual" backend, or is grouping it with `local.py` fine?


Read all the backend files in mirage/backends/ to understand what each one does and what classes/interfaces they expose. Read: base.py, loader.py, r2.py, s3.py, proxy.py, http.py, local.py, memory.py, __init__.py. For each file, tell me: the main class name, what it inherits from, and a brief description of what it does. Also check what imports reference these backends from other parts of the codebase (grep for "from mirage.backends" and "import mirage.backends").

Prompt: Read all the backend files in mirage/backends/ to understand what each one does and what classes/interfaces they expose. Read: base.py, loader.py, r2.py, s3.py, proxy.py, http.py, local.py, memory.py, __init__.py. For each file, tell me: the main class name, what it inherits from, and a brief description of what it does. Also check what imports reference these backends from other parts of the codebase (grep for "from mirage.backends" and "import mirage.backends").


Error: EISDIR: illegal operation on a directory, read '/Users/agent/strukto/mirage/mirage/backends'


<tool_use_error>Sibling tool call errored</tool_use_error>


Bash

find /Users/agent/strukto/mirage -type d -name backends

Find the backends directory


/Users/agent/strukto/mirage/tests/backends
/Users/agent/strukto/mirage/mirage/backends


List files in backends directory


     1→from mirage.backends.base import BaseBackend
     2→from mirage.backends.local import LocalBackend
     3→from mirage.backends.memory import MemoryBackend
     4→from mirage.backends.proxy import ProxyBackend
     5→
     6→__all__ = ["BaseBackend", "LocalBackend", "MemoryBackend", "ProxyBackend"]
     7→
     8→# S3Backend, R2Backend, HTTPBackend imported lazily to avoid hard deps
     9→


     1→import re
     2→
     3→from mirage.types import FileStat
     4→
     5→
     6→class BaseBackend:
     7→    """
     8→    Default command-level methods assembled from BackendOps primitives.
     9→    Backends override these for native efficiency (e.g. S3 uses list_objects_v2
    10→    for ls_l instead of per-file stat calls).
    11→    """
    12→
    13→    # ── ops that subclasses MUST implement ─────────────────────────────────
    14→
    15→    def read(self, path: str, offset: int, size: int) -> bytes:
    16→        raise NotImplementedError
    17→
    18→    def readdir(self,
    19→                path: str,
    20→                cursor: str | None = None) -> tuple[list[str], str | None]:
    21→        raise NotImplementedError
    22→
    23→    def stat(self, path: str) -> FileStat:
    24→        raise NotImplementedError
    25→
    26→    def open(self, path: str, flags: int) -> int:
    27→        raise NotImplementedError
    28→
    29→    def release(self, fh: int) -> None:
    30→        raise NotImplementedError
    31→
    32→    def write(self, path: str, offset: int, data: bytes) -> int:
    33→        raise NotImplementedError
    34→
    35→    def create(self, path: str) -> None:
    36→        raise NotImplementedError
    37→
    38→    def unlink(self, path: str) -> None:
    39→        raise NotImplementedError
    40→
    41→    def rmdir(self, path: str) -> None:
    42→        raise NotImplementedError
    43→
    44→    def mkdir(self, path: str) -> None:
    45→        raise NotImplementedError
    46→
    47→    def rename(self, src: str, dst: str) -> None:
    48→        raise NotImplementedError
    49→
    50→    def truncate(self, path: str, size: int) -> None:
    51→        raise NotImplementedError
    52→
    53→    # ── command-level helpers built from ops ────────────────────────────────
    54→
    55→    def exists(self, path: str) -> bool:
    56→        try:
    57→            self.stat(path)
    58→            return True
    59→        except (FileNotFoundError, KeyError):
    60→            return False
    61→
    62→    def cat(self, path: str) -> bytes:
    63→        s = self.stat(path)
    64→        size = s.size or 0
    65→        if size == 0:
    66→            return b""
    67→        return self.read(path, 0, size)
    68→
    69→    def ls(self, path: str) -> list[str]:
    70→        names: list[str] = []
    71→        cursor: str | None = None
    72→        while True:
    73→            batch, cursor = self.readdir(path, cursor)
    74→            names.extend(batch)
    75→            if cursor is None:
    76→                break
    77→        return names
    78→
    79→    def ls_l(self, path: str) -> list[FileStat]:
    80→        """Stat each entry. Backends override for batch efficiency."""
    81→        return [self.stat(name) for name in self.ls(path)]
    82→
    83→    def grep(self, pattern: str, path: str, flags: int = 0) -> list[str]:
    84→        """Full fetch + Python regex filter.
    85→
    86→        Backends override for native search."""
    87→        data = self.cat(path)
    88→        compiled = re.compile(pattern, flags)
    89→        return [
    90→            line for line in data.decode(errors="replace").splitlines()
    91→            if compiled.search(line)
    92→        ]
    93→
    94→    def head(self, path: str, lines: int = 10) -> bytes:
    95→        """Read first N lines via partial read (64KB chunk)."""
    96→        chunk_size = 64 * 1024
    97→        s = self.stat(path)
    98→        read_size = min(chunk_size, s.size or 0)
    99→        if read_size == 0:
   100→            return b""
   101→        data = self.read(path, 0, read_size)
   102→        parts = data.split(b"\n", lines)
   103→        return b"\n".join(parts[:lines])
   104→
   105→    def tail(self, path: str, lines: int = 10) -> bytes:
   106→        """Read last N lines via partial read (64KB chunk)."""
   107→        chunk_size = 64 * 1024
   108→        s = self.stat(path)
   109→        total = s.size or 0
   110→        if total == 0:
   111→            return b""
   112→        offset = max(0, total - chunk_size)
   113→        data = self.read(path, offset, total - offset)
   114→        parts = data.split(b"\n")
   115→        return b"\n".join(parts[-lines:])
   116→
   117→    def presign(self, path: str, expires: int = 3600) -> str:
   118→        raise NotImplementedError(
   119→            f"{type(self).__name__} does not support presigned URLs")
   120→
   121→    def put(self, path: str, data: bytes) -> None:
   122→        """Write full content to path (create if absent, replace if exists)."""
   123→        if not self.exists(path):
   124→            self.create(path)
   125→        self.truncate(path, 0)
   126→        if data:
   127→            self.write(path, 0, data)
   128→


     1→import importlib
     2→import importlib.util
     3→from typing import Any
     4→
     5→
     6→def load_backend_class(spec: str) -> type[Any]:
     7→    """Load a backend class from a spec string.
     8→
     9→    Supports two formats (auto-detected):
    10→    - Script file: ``"./my_backend.py:MyClass"``
    11→    - Module dotpath: ``"mypackage.backends:MyClass"``
    12→
    13→    Args:
    14→        spec (str): ``"source:class_name"`` where source is a file path
    15→            or module dotpath.
    16→
    17→    Returns:
    18→        type: The loaded class.
    19→    """
    20→    if ":" not in spec:
    21→        raise ValueError(
    22→            f"invalid backend spec {spec!r}, expected 'source:ClassName'")
    23→
    24→    source, class_name = spec.rsplit(":", 1)
    25→
    26→    if "/" in source or source.endswith(".py"):
    27→        module_spec = importlib.util.spec_from_file_location(
    28→            "_mirage_user_backend", source)
    29→        if module_spec is None or module_spec.loader is None:
    30→            raise ValueError(f"cannot load script {source!r}")
    31→        module = importlib.util.module_from_spec(module_spec)
    32→        try:
    33→            module_spec.loader.exec_module(module)
    34→        except FileNotFoundError:
    35→            raise ValueError(f"cannot load script {source!r}")
    36→    else:
    37→        module = importlib.import_module(source)
    38→
    39→    if not hasattr(module, class_name):
    40→        raise ValueError(f"{class_name!r} not found in {source!r}")
    41→
    42→    return getattr(module, class_name)
    43→


     1→from pydantic import BaseModel, ConfigDict
     2→
     3→from mirage.backends.s3 import S3Backend, S3Config
     4→
     5→
     6→class R2Config(BaseModel):
     7→    model_config = ConfigDict(frozen=True)
     8→
     9→    account_id: str
    10→    bucket: str
    11→    access_key_id: str
    12→    secret_access_key: str
    13→    region: str = "auto"
    14→    timeout: int = 30
    15→
    16→
    17→class R2Backend(S3Backend):
    18→    """
    19→    Cloudflare R2 backend — extends S3Backend with auto-computed endpoint.
    20→
    21→    Differences from S3Backend:
    22→    - endpoint_url auto-computed from account_id
    23→    - S3 Select DISABLED (R2 does not support SelectObjectContent)
    24→    - region: R2 location hint (wnam, enam, weur, eeur, apac, oc, auto)
    25→
    26→    Requires: pip install mirage[r2]
    27→    """
    28→
    29→    def __init__(self, config: R2Config) -> None:
    30→        endpoint = f"https://{config.account_id}.r2.cloudflarestorage.com"
    31→        s3_config = S3Config(
    32→            bucket=config.bucket,
    33→            region=config.region,
    34→            endpoint_url=endpoint,
    35→            aws_access_key_id=config.access_key_id,
    36→            aws_secret_access_key=config.secret_access_key,
    37→            timeout=config.timeout,
    38→            select_threshold_bytes=None,  # R2 does not support S3 Select
    39→        )
    40→        super().__init__(s3_config)
    41→        self._endpoint = endpoint
    42→


     1→import re
     2→from datetime import timezone
     3→
     4→from pydantic import BaseModel, ConfigDict
     5→
     6→from mirage.backends.base import BaseBackend
     7→from mirage.backends.memory import _guess_type
     8→from mirage.types import FileStat, FileType
     9→
    10→
    11→class S3Config(BaseModel):
    12→    model_config = ConfigDict(frozen=True)
    13→
    14→    bucket: str
    15→    region: str | None = None
    16→    endpoint_url: str | None = None
    17→    aws_access_key_id: str | None = None
    18→    aws_secret_access_key: str | None = None
    19→    aws_profile: str | None = None
    20→    timeout: int = 30
    21→    proxy: str | None = None
    22→    select_threshold_bytes: int | None = 1 * 1024 * 1024  # 1 MB
    23→
    24→
    25→class S3Backend(BaseBackend):
    26→    """
    27→    S3-compatible backend using boto3.
    28→
    29→    Requires: pip install mirage[s3]
    30→
    31→    Key design decisions:
    32→    - read()  → GetObject with Range header (partial reads)
    33→    - readdir() → list_objects_v2 with paginator (handles >1000 keys)
    34→    - write()  → full PutObject (S3 has no partial write)
    35→    - grep()   → S3 Select for large files, full fetch for small files
    36→    - ls_l()   → list_objects_v2 already returns size+mtime, skip per-file stat
    37→    """
    38→
    39→    def __init__(self, config: S3Config) -> None:
    40→        self.config = config
    41→        self._client = self._make_client()
    42→
    43→    def _make_client(self):
    44→        try:
    45→            import boto3
    46→            from botocore.config import Config
    47→        except ImportError:
    48→            raise ImportError(
    49→                "boto3 is required for S3Backend: pip install 'mirage[s3]'")
    50→
    51→        kwargs: dict = {"service_name": "s3"}
    52→        c = self.config
    53→        if c.region:
    54→            kwargs["region_name"] = c.region
    55→        if c.endpoint_url:
    56→            kwargs["endpoint_url"] = c.endpoint_url
    57→        if c.aws_access_key_id and c.aws_secret_access_key:
    58→            kwargs["aws_access_key_id"] = c.aws_access_key_id
    59→            kwargs["aws_secret_access_key"] = c.aws_secret_access_key
    60→
    61→        cfg_kwargs: dict = {
    62→            "connect_timeout": c.timeout,
    63→            "read_timeout": c.timeout
    64→        }
    65→        if c.proxy:
    66→            cfg_kwargs["proxies"] = {"https": c.proxy, "http": c.proxy}
    67→
    68→        kwargs["config"] = Config(**cfg_kwargs)
    69→
    70→        if c.aws_profile:
    71→            session = boto3.Session(profile_name=c.aws_profile)
    72→            return session.client(**kwargs)
    73→        return boto3.client(**kwargs)
    74→
    75→    # ── key helpers ─────────────────────────────────────────────────────────
    76→
    77→    def _key(self, path: str) -> str:
    78→        """Convert virtual path to S3 key (strip leading slash)."""
    79→        return path.lstrip("/")
    80→
    81→    def _prefix(self, path: str) -> str:
    82→        """Directory prefix — always ends with '/' unless root."""
    83→        key = self._key(path)
    84→        if key and not key.endswith("/"):
    85→            key += "/"
    86→        return key
    87→
    88→    # ── ops ─────────────────────────────────────────────────────────────────
    89→
    90→    def read(self, path: str, offset: int, size: int) -> bytes:
    91→        end = offset + size - 1
    92→        resp = self._client.get_object(
    93→            Bucket=self.config.bucket,
    94→            Key=self._key(path),
    95→            Range=f"bytes={offset}-{end}",
    96→        )
    97→        return resp["Body"].read()
    98→
    99→    def readdir(self,
   100→                path: str,
   101→                cursor: str | None = None) -> tuple[list[str], str | None]:
   102→        prefix = self._prefix(path)
   103→        kwargs: dict = {
   104→            "Bucket": self.config.bucket,
   105→            "Prefix": prefix,
   106→            "Delimiter": "/",
   107→        }
   108→        if cursor:
   109→            kwargs["ContinuationToken"] = cursor
   110→
   111→        resp = self._client.list_objects_v2(**kwargs)
   112→
   113→        names: list[str] = []
   114→        # common prefixes = subdirectories
   115→        for cp in resp.get("CommonPrefixes") or []:
   116→            p = cp["Prefix"]
   117→            child = p.rstrip("/")
   118→            if child:
   119→                names.append(child)
   120→        # object keys = files
   121→        for obj in resp.get("Contents") or []:
   122→            key = obj["Key"]
   123→            relative = key[len(prefix):]
   124→            if relative and "/" not in relative:  # direct children only
   125→                names.append(key)
   126→
   127→        next_cursor = resp.get("NextContinuationToken")
   128→        return sorted(names), next_cursor
   129→
   130→    def stat(self, path: str) -> FileStat:
   131→        try:
   132→            resp = self._client.head_object(Bucket=self.config.bucket,
   133→                                            Key=self._key(path))
   134→            modified = resp["LastModified"].astimezone(
   135→                timezone.utc).isoformat()
   136→            return FileStat(
   137→                name=path.rstrip("/").rsplit("/", 1)[-1],
   138→                size=resp["ContentLength"],
   139→                modified=modified,
   140→                type=_guess_type(path),
   141→                extra={"etag": resp.get("ETag", "").strip('"')},
   142→            )
   143→        except self._client.exceptions.NoSuchKey:
   144→            pass
   145→        except Exception as exc:
   146→            if not (hasattr(exc, "response")
   147→                    and exc.response.get("Error", {}).get("Code") == "404"):
   148→                raise
   149→        # key not found — check if it's a directory prefix
   150→        prefix = self._prefix(path)
   151→        resp = self._client.list_objects_v2(Bucket=self.config.bucket,
   152→                                            Prefix=prefix,
   153→                                            MaxKeys=1)
   154→        if resp.get("Contents") or resp.get("CommonPrefixes"):
   155→            return FileStat(
   156→                name=path.rstrip("/").rsplit("/", 1)[-1] or "/",
   157→                size=None,
   158→                type=FileType.DIRECTORY,
   159→            )
   160→        raise FileNotFoundError(path)
   161→
   162→    def open(self, path: str, flags: int) -> int:
   163→        raise NotImplementedError("open/release are FUSE-only")
   164→
   165→    def release(self, fh: int) -> None:
   166→        raise NotImplementedError("open/release are FUSE-only")
   167→
   168→    def write(self, path: str, offset: int, data: bytes) -> int:
   169→        # S3 has no partial write — we must do full replace
   170→        if offset != 0:
   171→            # fetch existing, splice, re-upload
   172→            existing = self.cat(path)
   173→            data = existing[:offset] + data + existing[offset + len(data):]
   174→        self._client.put_object(Bucket=self.config.bucket,
   175→                                Key=self._key(path),
   176→                                Body=data)
   177→        return len(data)
   178→
   179→    def create(self, path: str) -> None:
   180→        self._client.put_object(Bucket=self.config.bucket,
   181→                                Key=self._key(path),
   182→                                Body=b"")
   183→
   184→    def unlink(self, path: str) -> None:
   185→        self._client.delete_object(Bucket=self.config.bucket,
   186→                                   Key=self._key(path))
   187→
   188→    def rmdir(self, path: str) -> None:
   189→        prefix = self._prefix(path)
   190→        paginator = self._client.get_paginator("list_objects_v2")
   191→        for page in paginator.paginate(Bucket=self.config.bucket,
   192→                                       Prefix=prefix):
   193→            keys = [{"Key": obj["Key"]} for obj in page.get("Contents") or []]
   194→            if keys:
   195→                self._client.delete_objects(Bucket=self.config.bucket,
   196→                                            Delete={"Objects": keys})
   197→
   198→    def mkdir(self, path: str) -> None:
   199→        # S3 has no real directories — key prefix convention
   200→        prefix = self._prefix(path)
   201→        if prefix:
   202→            self._client.put_object(Bucket=self.config.bucket,
   203→                                    Key=prefix,
   204→                                    Body=b"")
   205→
   206→    def rename(self, src: str, dst: str) -> None:
   207→        self._client.copy_object(
   208→            Bucket=self.config.bucket,
   209→            CopySource={
   210→                "Bucket": self.config.bucket,
   211→                "Key": self._key(src)
   212→            },
   213→            Key=self._key(dst),
   214→        )
   215→        self._client.delete_object(Bucket=self.config.bucket,
   216→                                   Key=self._key(src))
   217→
   218→    def truncate(self, path: str, size: int) -> None:
   219→        data = self.cat(path)
   220→        if size < len(data):
   221→            data = data[:size]
   222→        else:
   223→            data = data + b"\x00" * (size - len(data))
   224→        self._client.put_object(Bucket=self.config.bucket,
   225→                                Key=self._key(path),
   226→                                Body=data)
   227→
   228→    # ── native overrides ────────────────────────────────────────────────────
   229→
   230→    def ls_l(self, path: str) -> list[FileStat]:
   231→        """list_objects_v2 returns size+mtime — skip per-file stat."""
   232→        prefix = self._prefix(path)
   233→        stats: list[FileStat] = []
   234→        paginator = self._client.get_paginator("list_objects_v2")
   235→        for page in paginator.paginate(Bucket=self.config.bucket,
   236→                                       Prefix=prefix,
   237→                                       Delimiter="/"):
   238→            for cp in page.get("CommonPrefixes") or []:
   239→                name = cp["Prefix"][len(prefix):].rstrip("/")
   240→                stats.append(
   241→                    FileStat(name=name, size=None, type=FileType.DIRECTORY))
   242→            for obj in page.get("Contents") or []:
   243→                key = obj["Key"]
   244→                name = key[len(prefix):]
   245→                if name and "/" not in name:
   246→                    modified = obj["LastModified"].astimezone(
   247→                        timezone.utc).isoformat()
   248→                    stats.append(
   249→                        FileStat(name=name,
   250→                                 size=obj["Size"],
   251→                                 modified=modified,
   252→                                 type=_guess_type(name)))
   253→        return stats
   254→
   255→    def head(self, path: str, lines: int = 10) -> bytes:
   256→        threshold = self.config.select_threshold_bytes
   257→        if threshold is not None:
   258→            try:
   259→                s = self.stat(path)
   260→                if s.size and s.size > threshold:
   261→                    return self._head_select(path, lines)
   262→            except FileNotFoundError:
   263→                pass
   264→        return super().head(path, lines)
   265→
   266→    def _head_select(self, path: str, lines: int) -> bytes:
   267→        sql = f"SELECT * FROM S3Object LIMIT {int(lines)}"
   268→        try:
   269→            resp = self._client.select_object_content(
   270→                Bucket=self.config.bucket,
   271→                Key=self._key(path),
   272→                ExpressionType="SQL",
   273→                Expression=sql,
   274→                InputSerialization={
   275→                    "CSV": {
   276→                        "FileHeaderInfo": "NONE"
   277→                    },
   278→                    "CompressionType": "NONE"
   279→                },
   280→                OutputSerialization={"CSV": {}},
   281→            )
   282→            parts: list[str] = []
   283→            for event in resp["Payload"]:
   284→                if "Records" in event:
   285→                    parts.extend(
   286→                        event["Records"]["Payload"].decode().splitlines())
   287→            return "\n".join(parts[:lines]).encode()
   288→        except Exception:
   289→            return super().head(path, lines)
   290→
   291→    def presign(self, path: str, expires: int = 3600) -> str:
   292→        return self._client.generate_presigned_url(
   293→            "get_object",
   294→            Params={
   295→                "Bucket": self.config.bucket,
   296→                "Key": self._key(path)
   297→            },
   298→            ExpiresIn=expires,
   299→        )
   300→
   301→    def grep(self, pattern: str, path: str, flags: int = 0) -> list[str]:
   302→        """S3 Select for large files; full fetch for small files."""
   303→        threshold = self.config.select_threshold_bytes
   304→        if threshold is not None:
   305→            try:
   306→                s = self.stat(path)
   307→                if s.size and s.size > threshold:
   308→                    return self._grep_select(pattern, path)
   309→            except FileNotFoundError:
   310→                pass
   311→        return super().grep(pattern, path, flags)
   312→
   313→    def _grep_select(self, pattern: str, path: str) -> list[str]:
   314→        """S3 Select LIKE filter — regex unsupported, re-filtered in Python."""
   315→        escaped = (pattern.replace("'",
   316→                                   "''").replace("%",
   317→                                                 "\\%").replace("_", "\\_"))
   318→        sql = f"SELECT * FROM S3Object WHERE _1 LIKE '%{escaped}%'"
   319→        try:
   320→            resp = self._client.select_object_content(
   321→                Bucket=self.config.bucket,
   322→                Key=self._key(path),
   323→                ExpressionType="SQL",
   324→                Expression=sql,
   325→                InputSerialization={
   326→                    "CSV": {
   327→                        "FileHeaderInfo": "NONE"
   328→                    },
   329→                    "CompressionType": "NONE"
   330→                },
   331→                OutputSerialization={"CSV": {}},
   332→            )
   333→            lines: list[str] = []
   334→            for event in resp["Payload"]:
   335→                if "Records" in event:
   336→                    lines.extend(
   337→                        event["Records"]["Payload"].decode().splitlines())
   338→            compiled = re.compile(pattern)
   339→            return [line for line in lines if compiled.search(line)]
   340→        except Exception:
   341→            return super().grep(pattern, path)
   342→


     1→import base64
     2→
     3→import httpx
     4→
     5→from mirage.backends.base import BaseBackend
     6→from mirage.types import FileStat
     7→
     8→
     9→class ProxyBackend(BaseBackend):
    10→
    11→    def __init__(self,
    12→                 base_url: str,
    13→                 ws_id: str,
    14→                 timeout: int = 30,
    15→                 http_client: httpx.Client | None = None) -> None:
    16→        self._base_url = base_url.rstrip("/")
    17→        self._ws_id = ws_id
    18→        self._http = http_client or httpx.Client(timeout=timeout)
    19→
    20→    def _post(self, op: str, payload: dict) -> dict:
    21→        url = f"{self._base_url}/workspaces/{self._ws_id}/io/{op}"
    22→        resp = self._http.post(url, json=payload)
    23→        if resp.status_code == 404:
    24→            detail = resp.json().get("detail", "")
    25→            if "workspace" in detail:
    26→                raise KeyError(detail)
    27→            raise FileNotFoundError(payload.get("path", ""))
    28→        resp.raise_for_status()
    29→        return resp.json()
    30→
    31→    def read(self, path: str, offset: int, size: int) -> bytes:
    32→        r = self._post("read", {"path": path, "offset": offset, "size": size})
    33→        return base64.b64decode(r["data"])
    34→
    35→    def readdir(self,
    36→                path: str,
    37→                cursor: str | None = None) -> tuple[list[str], str | None]:
    38→        r = self._post("readdir", {"path": path, "cursor": cursor})
    39→        return r["entries"], r.get("cursor")
    40→
    41→    def stat(self, path: str) -> FileStat:
    42→        r = self._post("stat", {"path": path})
    43→        return FileStat(**r)
    44→
    45→    def open(self, path: str, flags: int) -> int:
    46→        raise NotImplementedError("open not supported on ProxyBackend")
    47→
    48→    def release(self, fh: int) -> None:
    49→        raise NotImplementedError("release not supported on ProxyBackend")
    50→
    51→    def write(self, path: str, offset: int, data: bytes) -> int:
    52→        encoded = base64.b64encode(data).decode()
    53→        r = self._post("write", {
    54→            "path": path,
    55→            "offset": offset,
    56→            "data": encoded
    57→        })
    58→        return r["written"]
    59→
    60→    def create(self, path: str) -> None:
    61→        self._post("create", {"path": path})
    62→
    63→    def unlink(self, path: str) -> None:
    64→        self._post("unlink", {"path": path})
    65→
    66→    def rmdir(self, path: str) -> None:
    67→        self._post("rmdir", {"path": path})
    68→
    69→    def mkdir(self, path: str) -> None:
    70→        self._post("mkdir", {"path": path})
    71→
    72→    def rename(self, src: str, dst: str) -> None:
    73→        self._post("rename", {"src": src, "dst": dst})
    74→
    75→    def truncate(self, path: str, size: int) -> None:
    76→        self._post("truncate", {"path": path, "size": size})
    77→
    78→    def fuse_cat(self, path: str) -> bytes:
    79→        r = self._post("fuse_cat", {"path": path})
    80→        return base64.b64decode(r["data"])
    81→


     1→from pydantic import BaseModel, ConfigDict, Field
     2→
     3→from mirage.backends.base import BaseBackend
     4→from mirage.backends.memory import _guess_type
     5→from mirage.types import FileStat
     6→
     7→
     8→class HTTPConfig(BaseModel):
     9→    model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)
    10→
    11→    base_url: str
    12→    headers: dict[str, str] = Field(default_factory=dict)
    13→    timeout: int = 30
    14→    readdir_fn: object = None
    15→
    16→
    17→class HTTPBackend(BaseBackend):
    18→    """
    19→    Read-only HTTP backend using httpx.
    20→
    21→    Maps virtual paths to URLs: base_url + path
    22→    e.g. base_url="https://data.example.com", path="/q1.json"
    23→         → GET https://data.example.com/q1.json
    24→
    25→    Requires: pip install mirage[http]
    26→
    27→    Write ops (write, create, unlink, etc.) raise NotImplementedError —
    28→    HTTP is read-only by design. Override to support writable HTTP APIs.
    29→
    30→    readdir() requires a user-supplied `readdir_fn` since HTTP has no
    31→    standard directory listing protocol.
    32→    """
    33→
    34→    def __init__(self, config: HTTPConfig) -> None:
    35→        self.config = config
    36→        self._client = self._make_client()
    37→
    38→    def _make_client(self):
    39→        try:
    40→            import httpx
    41→        except ImportError:
    42→            raise ImportError(
    43→                "httpx is required for HTTPBackend: pip install 'mirage[http]'"
    44→            )
    45→        return httpx.Client(
    46→            base_url=self.config.base_url,
    47→            headers=self.config.headers,
    48→            timeout=self.config.timeout,
    49→            follow_redirects=True,
    50→        )
    51→
    52→    def _url(self, path: str) -> str:
    53→        return self.config.base_url.rstrip("/") + "/" + path.lstrip("/")
    54→
    55→    # ── ops ─────────────────────────────────────────────────────────────────
    56→
    57→    def read(self, path: str, offset: int, size: int) -> bytes:
    58→        headers = {"Range": f"bytes={offset}-{offset + size - 1}"}
    59→        resp = self._client.get(self._url(path), headers=headers)
    60→        if resp.status_code == 416:
    61→            return b""
    62→        resp.raise_for_status()
    63→        return resp.content
    64→
    65→    def readdir(self,
    66→                path: str,
    67→                cursor: str | None = None) -> tuple[list[str], str | None]:
    68→        if self.config.readdir_fn is None:
    69→            raise NotImplementedError(
    70→                "HTTPBackend.readdir() requires a readdir_fn in HTTPConfig. "
    71→                "Provide a callable(path) -> list[str] for your API.")
    72→        names = self.config.readdir_fn(path)
    73→        return names, None
    74→
    75→    def stat(self, path: str) -> FileStat:
    76→        resp = self._client.head(self._url(path))
    77→        if resp.status_code == 404:
    78→            raise FileNotFoundError(path)
    79→        resp.raise_for_status()
    80→        size = int(resp.headers.get("content-length", 0)) or None
    81→        modified = resp.headers.get("last-modified")
    82→        etag = resp.headers.get("etag", "").strip('"') or None
    83→        extra = {"etag": etag} if etag else {}
    84→        return FileStat(
    85→            name=path.rstrip("/").rsplit("/", 1)[-1],
    86→            size=size,
    87→            modified=modified,
    88→            type=_guess_type(path),
    89→            extra=extra,
    90→        )
    91→
    92→    def open(self, path: str, flags: int) -> int:
    93→        raise NotImplementedError("open/release are FUSE-only")
    94→
    95→    def release(self, fh: int) -> None:
    96→        raise NotImplementedError("open/release are FUSE-only")
    97→
    98→    # ── write ops — not supported ────────────────────────────────────────────
    99→
   100→    def write(self, path: str, offset: int, data: bytes) -> int:
   101→        raise NotImplementedError("HTTPBackend is read-only")
   102→
   103→    def create(self, path: str) -> None:
   104→        raise NotImplementedError("HTTPBackend is read-only")
   105→
   106→    def unlink(self, path: str) -> None:
   107→        raise NotImplementedError("HTTPBackend is read-only")
   108→
   109→    def rmdir(self, path: str) -> None:
   110→        raise NotImplementedError("HTTPBackend is read-only")
   111→
   112→    def mkdir(self, path: str) -> None:
   113→        raise NotImplementedError("HTTPBackend is read-only")
   114→
   115→    def rename(self, src: str, dst: str) -> None:
   116→        raise NotImplementedError("HTTPBackend is read-only")
   117→
   118→    def truncate(self, path: str, size: int) -> None:
   119→        raise NotImplementedError("HTTPBackend is read-only")
   120→


     1→import os
     2→from datetime import datetime, timezone
     3→from pathlib import Path
     4→
     5→from mirage.backends.base import BaseBackend
     6→from mirage.backends.memory import _guess_type
     7→from mirage.types import FileStat, FileType
     8→
     9→
    10→class LocalBackend(BaseBackend):
    11→    """
    12→    Backend backed by the local filesystem via pathlib/os.
    13→    No external dependencies.
    14→
    15→    All virtual paths are resolved relative to `root`.
    16→    Example: root="/data", path="/reports/q1.csv" → /data/reports/q1.csv
    17→    """
    18→
    19→    def __init__(self, root: str) -> None:
    20→        self.root = Path(root).resolve()
    21→
    22→    # ── helpers ─────────────────────────────────────────────────────────────
    23→
    24→    def _resolve(self, path: str) -> Path:
    25→        # strip leading slash so joinpath works correctly
    26→        relative = path.lstrip("/")
    27→        resolved = (self.root / relative).resolve()
    28→        # prevent path traversal outside root
    29→        resolved.relative_to(self.root)
    30→        return resolved
    31→
    32→    def _to_stat(self, p: Path) -> FileStat:
    33→        st = p.stat()
    34→        modified = datetime.fromtimestamp(st.st_mtime,
    35→                                          tz=timezone.utc).isoformat()
    36→        if p.is_dir():
    37→            return FileStat(name=p.name,
    38→                            size=None,
    39→                            modified=modified,
    40→                            type=FileType.DIRECTORY)
    41→        return FileStat(name=p.name,
    42→                        size=st.st_size,
    43→                        modified=modified,
    44→                        type=_guess_type(p.name))
    45→
    46→    # ── ops ─────────────────────────────────────────────────────────────────
    47→
    48→    def read(self, path: str, offset: int, size: int) -> bytes:
    49→        p = self._resolve(path)
    50→        with p.open("rb") as f:
    51→            f.seek(offset)
    52→            return f.read(size)
    53→
    54→    def readdir(self,
    55→                path: str,
    56→                cursor: str | None = None) -> tuple[list[str], str | None]:
    57→        p = self._resolve(path)
    58→        if not p.is_dir():
    59→            raise NotADirectoryError(str(p))
    60→        base = "/" + path.strip("/")
    61→        entries = sorted(base.rstrip("/") + "/" + e.name for e in p.iterdir())
    62→        # cursor is an offset index for pagination
    63→        start = int(cursor) if cursor else 0
    64→        page = entries[start:]
    65→        return page, None
    66→
    67→    def stat(self, path: str) -> FileStat:
    68→        p = self._resolve(path)
    69→        if not p.exists():
    70→            raise FileNotFoundError(str(p))
    71→        return self._to_stat(p)
    72→
    73→    def open(self, path: str, flags: int) -> int:
    74→        raise NotImplementedError("open/release are FUSE-only")
    75→
    76→    def release(self, fh: int) -> None:
    77→        raise NotImplementedError("open/release are FUSE-only")
    78→
    79→    def write(self, path: str, offset: int, data: bytes) -> int:
    80→        p = self._resolve(path)
    81→        with p.open("r+b") as f:
    82→            f.seek(offset)
    83→            f.write(data)
    84→        return len(data)
    85→
    86→    def create(self, path: str) -> None:
    87→        p = self._resolve(path)
    88→        p.parent.mkdir(parents=True, exist_ok=True)
    89→        p.touch(exist_ok=True)
    90→
    91→    def unlink(self, path: str) -> None:
    92→        p = self._resolve(path)
    93→        p.unlink()
    94→
    95→    def rmdir(self, path: str) -> None:
    96→        p = self._resolve(path)
    97→        p.rmdir()  # raises OSError if non-empty — matches POSIX semantics
    98→
    99→    def mkdir(self, path: str) -> None:
   100→        p = self._resolve(path)
   101→        p.mkdir(parents=True, exist_ok=True)
   102→
   103→    def rename(self, src: str, dst: str) -> None:
   104→        s = self._resolve(src)
   105→        d = self._resolve(dst)
   106→        s.rename(d)
   107→
   108→    def truncate(self, path: str, size: int) -> None:
   109→        p = self._resolve(path)
   110→        os.truncate(p, size)
   111→


     1→import hashlib
     2→from datetime import datetime, timezone
     3→
     4→from mirage.backends.base import BaseBackend
     5→from mirage.types import FileStat, FileType
     6→
     7→
     8→class MemoryBackend(BaseBackend):
     9→    """
    10→    In-memory backend backed by a plain dict. No external dependencies.
    11→    Useful for tests, ephemeral scratch space, and as a cache target.
    12→
    13→    Internal layout:
    14→        _files: dict[str, bytes]  — path → content
    15→        _dirs:  set[str]          — explicit directory paths
    16→    """
    17→
    18→    def __init__(self) -> None:
    19→        self._files: dict[str, bytes] = {}
    20→        self._dirs: set[str] = {"/"}
    21→        self._modified: dict[str, str] = {}  # path → ISO timestamp
    22→
    23→    # ── helpers ─────────────────────────────────────────────────────────────
    24→
    25→    def _norm(self, path: str) -> str:
    26→        """Normalise to absolute path without trailing slash."""
    27→        p = "/" + path.strip("/")
    28→        return p
    29→
    30→    def _now(self) -> str:
    31→        return datetime.now(timezone.utc).isoformat()
    32→
    33→    def _parent(self, path: str) -> str:
    34→        parts = path.rsplit("/", 1)
    35→        return parts[0] or "/"
    36→
    37→    # ── ops ─────────────────────────────────────────────────────────────────
    38→
    39→    def read(self, path: str, offset: int, size: int) -> bytes:
    40→        p = self._norm(path)
    41→        if p not in self._files:
    42→            raise FileNotFoundError(p)
    43→        return self._files[p][offset:offset + size]
    44→
    45→    def readdir(self,
    46→                path: str,
    47→                cursor: str | None = None) -> tuple[list[str], str | None]:
    48→        p = self._norm(path)
    49→        if p not in self._dirs:
    50→            raise FileNotFoundError(p)
    51→        prefix = p.rstrip("/") + "/"
    52→        # direct children only
    53→        seen: set[str] = set()
    54→        for key in list(self._files) + list(self._dirs):
    55→            if key == p:
    56→                continue
    57→            if key.startswith(prefix):
    58→                remainder = key[len(prefix):]
    59→                child = remainder.split("/")[0]
    60→                if child:
    61→                    seen.add(prefix + child)
    62→        names = sorted(seen)
    63→        # simple offset-based pagination via cursor
    64→        start = int(cursor) if cursor else 0
    65→        page = names[start:]
    66→        return page, None  # no pagination needed for in-memory
    67→
    68→    def stat(self, path: str) -> FileStat:
    69→        p = self._norm(path)
    70→        if p in self._dirs:
    71→            return FileStat(name=p.rsplit("/", 1)[-1] or "/",
    72→                            size=None,
    73→                            modified=self._modified.get(p),
    74→                            type=FileType.DIRECTORY)
    75→        if p in self._files:
    76→            data = self._files[p]
    77→            return FileStat(
    78→                name=p.rsplit("/", 1)[-1],
    79→                size=len(data),
    80→                modified=self._modified.get(p),
    81→                type=_guess_type(p),
    82→            )
    83→        raise FileNotFoundError(p)
    84→
    85→    def open(self, path: str, flags: int) -> int:
    86→        raise NotImplementedError("open/release are FUSE-only")
    87→
    88→    def release(self, fh: int) -> None:
    89→        raise NotImplementedError("open/release are FUSE-only")
    90→
    91→    def write(self, path: str, offset: int, data: bytes) -> int:
    92→        p = self._norm(path)
    93→        if p not in self._files:
    94→            raise FileNotFoundError(p)
    95→        existing = self._files[p]
    96→        # pad if offset beyond current size
    97→        if offset > len(existing):
    98→            existing = existing + b"\x00" * (offset - len(existing))
    99→        self._files[p] = existing[:offset] + data + existing[offset +
   100→                                                             len(data):]
   101→        self._modified[p] = self._now()
   102→        return len(data)
   103→
   104→    def create(self, path: str) -> None:
   105→        p = self._norm(path)
   106→        parent = self._parent(p)
   107→        if parent != "/" and parent not in self._dirs:
   108→            raise FileNotFoundError(
   109→                f"parent directory does not exist: {parent}")
   110→        self._files[p] = b""
   111→        self._modified[p] = self._now()
   112→
   113→    def unlink(self, path: str) -> None:
   114→        p = self._norm(path)
   115→        if p not in self._files:
   116→            raise FileNotFoundError(p)
   117→        del self._files[p]
   118→        self._modified.pop(p, None)
   119→
   120→    def rmdir(self, path: str) -> None:
   121→        p = self._norm(path)
   122→        if p not in self._dirs:
   123→            raise FileNotFoundError(p)
   124→        prefix = p.rstrip("/") + "/"
   125→        children = [
   126→            k for k in list(self._files) + list(self._dirs)
   127→            if k.startswith(prefix) and k != p
   128→        ]
   129→        if children:
   130→            raise OSError(f"directory not empty: {p}")
   131→        self._dirs.discard(p)
   132→
   133→    def mkdir(self, path: str) -> None:
   134→        p = self._norm(path)
   135→        parent = self._parent(p)
   136→        if parent != "/" and parent not in self._dirs:
   137→            raise FileNotFoundError(
   138→                f"parent directory does not exist: {parent}")
   139→        self._dirs.add(p)
   140→        self._modified[p] = self._now()
   141→
   142→    def rename(self, src: str, dst: str) -> None:
   143→        s, d = self._norm(src), self._norm(dst)
   144→        if s in self._files:
   145→            self._files[d] = self._files.pop(s)
   146→            self._modified[d] = self._modified.pop(s, self._now())
   147→        elif s in self._dirs:
   148→            self._dirs.discard(s)
   149→            self._dirs.add(d)
   150→            self._modified[d] = self._modified.pop(s, self._now())
   151→            # move all children
   152→            prefix = s.rstrip("/") + "/"
   153→            for key in list(self._files):
   154→                if key.startswith(prefix):
   155→                    new_key = d.rstrip("/") + "/" + key[len(prefix):]
   156→                    self._files[new_key] = self._files.pop(key)
   157→        else:
   158→            raise FileNotFoundError(s)
   159→
   160→    def truncate(self, path: str, size: int) -> None:
   161→        p = self._norm(path)
   162→        if p not in self._files:
   163→            raise FileNotFoundError(p)
   164→        data = self._files[p]
   165→        if size < len(data):
   166→            self._files[p] = data[:size]
   167→        else:
   168→            self._files[p] = data + b"\x00" * (size - len(data))
   169→        self._modified[p] = self._now()
   170→
   171→    # ── convenience ─────────────────────────────────────────────────────────
   172→
   173→    def checksum(self, path: str) -> str:
   174→        p = self._norm(path)
   175→        return hashlib.md5(self._files[p]).hexdigest()
   176→
   177→
   178→def _guess_type(path: str) -> str | None:
   179→    ext = path.rsplit(".", 1)[-1].lower() if "." in path else ""
   180→    return {
   181→        "json": "json",
   182→        "jsonl": "json",
   183→        "csv": "csv",
   184→        "tsv": "csv",
   185→        "txt": "text",
   186→        "md": "text",
   187→        "py": "text",
   188→        "js": "text",
   189→        "ts": "text",
   190→        "yaml": "text",
   191→        "yml": "text",
   192→        "toml": "text",
   193→    }.get(ext)
   194→


examples/r2.py:15:from mirage.backends.r2 import R2Backend, R2Config
examples/claude-memory.py:21:from mirage.backends.memory import MemoryBackend
tests/commands/test_init.py:1:from mirage.backends.memory import MemoryBackend
examples/claude-sdk.py:24:from mirage.backends.memory import MemoryBackend
tests/commands/test_read.py:5:from mirage.backends.local import LocalBackend
tests/fuse_ops/test_remote.py:3:from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_default.py:1:from mirage.backends.memory import MemoryBackend
mirage/cli/commands/workspace.py:10:from mirage.backends.loader import load_backend_class
mirage/cli/commands/workspace.py:207:    from mirage.backends.proxy import ProxyBackend
tests/cli/test_config.py:3:from mirage.backends.local import LocalBackend
tests/cli/test_config.py:4:from mirage.backends.memory import MemoryBackend
mirage/cli/commands/command.py:42:    from mirage.backends.loader import load_backend_class
mirage/runtime/local.py:1:from mirage.backends.local import LocalBackend
tests/server/test_arg_resolver.py:1:from mirage.backends.memory import MemoryBackend
mirage/runtime/remote.py:5:from mirage.backends.proxy import ProxyBackend
tests/fuse_ops/test_format_ops.py:6:from mirage.backends.memory import MemoryBackend
mirage/workspace/mount.py:1:from mirage.backends.base import BaseBackend
mirage/cli/config.py:5:from mirage.backends.base import BaseBackend
mirage/cli/config.py:6:from mirage.backends.local import LocalBackend
mirage/cli/config.py:7:from mirage.backends.memory import MemoryBackend
mirage/cli/config.py:17:        from mirage.backends.s3 import S3Backend, S3Config
mirage/cli/config.py:22:        from mirage.backends.r2 import R2Backend, R2Config
mirage/cli/config.py:33:        from mirage.backends.http import HTTPBackend, HTTPConfig
tests/observe/test_inspector.py:3:from mirage.backends.memory import MemoryBackend
mirage/workspace/__init__.py:5:from mirage.backends.base import BaseBackend
mirage/workspace/__init__.py:6:from mirage.backends.memory import _guess_type
tests/fuse_ops/test_registry.py:63:    from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_registry.py:76:    from mirage.backends.local import LocalBackend
tests/fuse_ops/test_registry.py:88:    from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_registry.py:100:    from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_registry.py:111:    from mirage.backends.memory import MemoryBackend
mirage/workspace/cache/policy.py:3:from mirage.backends.base import BaseBackend
mirage/workspace/sync.py:4:from mirage.backends.base import BaseBackend
mirage/server/service/local.py:11:from mirage.backends.loader import load_backend_class
mirage/observe/inspector.py:6:from mirage.backends.memory import MemoryBackend
mirage/backends/r2.py:3:from mirage.backends.s3 import S3Backend, S3Config
mirage/backends/s3.py:6:from mirage.backends.base import BaseBackend
mirage/backends/s3.py:7:from mirage.backends.memory import _guess_type
tests/integration/test_remote_transfer.py:3:from mirage.backends.memory import MemoryBackend
mirage/backends/http.py:3:from mirage.backends.base import BaseBackend
mirage/backends/http.py:4:from mirage.backends.memory import _guess_type
tests/conftest.py:3:from mirage.backends.memory import MemoryBackend
mirage/backends/__init__.py:1:from mirage.backends.base import BaseBackend
mirage/backends/__init__.py:2:from mirage.backends.local import LocalBackend
mirage/backends/__init__.py:3:from mirage.backends.memory import MemoryBackend
mirage/backends/__init__.py:4:from mirage.backends.proxy import ProxyBackend
mirage/backends/local.py:5:from mirage.backends.base import BaseBackend
mirage/backends/local.py:6:from mirage.backends.memory import _guess_type
mirage/backends/proxy.py:5:from mirage.backends.base import BaseBackend
tests/runtime/test_memory.py:3:from mirage.backends.memory import MemoryBackend
mirage/backends/memory.py:4:from mirage.backends.base import BaseBackend
tests/runtime/test_accessors.py:3:from mirage.backends.memory import MemoryBackend
tests/server/test_registry.py:6:from mirage.backends.memory import MemoryBackend
tests/runtime/test_runtime.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_init.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_lock.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_mount.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_sync.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_write_ops.py:3:from mirage.backends.memory import MemoryBackend
tests/backends/test_local.py:3:from mirage.backends.local import LocalBackend
tests/backends/test_proxy.py:4:from mirage.backends.proxy import ProxyBackend
tests/backends/test_loader.py:3:from mirage.backends.loader import load_backend_class
tests/backends/test_loader.py:8:    from mirage.backends.memory import MemoryBackend
tests/backends/test_loader.py:14:    script.write_text("from mirage.backends.memory import MemoryBackend\n"
tests/backends/test_configs.py:4:from mirage.backends.r2 import R2Config
tests/backends/test_configs.py:5:from mirage.backends.s3 import S3Config
mirage/__init__.py:1:from mirage.backends.local import LocalBackend
mirage/__init__.py:2:from mirage.backends.memory import MemoryBackend
fuse/tests/test_fs.py:12:from mirage.backends.memory import MemoryBackend
fuse/README.md:60:from mirage.backends.memory import MemoryBackend


do


The user wants me to proceed with the implementation. Let me use the writing-plans skill to create a detailed plan, then execute it.

Actually, the user said "do" - they want me to just go ahead and do the refactoring with the groupings I proposed. Let me plan and execute this.

Let me think about what needs to happen:

1. Create subdirectories: `object_storage/`, `local/`, `network/`
2. Move files into subdirectories
3. Add `__init__.py` to each subdirectory
4. Update all imports across the codebase
5. Update the main `backends/__init__.py`

Now I'm confirming the exact groupings: object_storage gets s3.py and r2.py, local gets local.py and memory.py, network gets http.py and proxy.py, while base.py and loader.py stay at the top level. Since the user wants me to just execute, I'll start the refactoring directly.

I need to track down all the import statements that'll break during this reorganization—BaseBackend is imported in mount.py and __init__.py, the CLI config pulls in all backends, and several runtime modules reference specific backend classes, so I'll need to update those paths systematically. I need to check what import patterns are actually being used across the codebase before I start moving files around, since that'll determine which imports need updating and which can stay the same through re-exports.


TodoWrite

Grep all import patterns for backends across codebase

Create subdirectories: object_storage/, local/, network/

Move files into subdirectories and add __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable

Grep all import patterns for backends across codebase

Create subdirectories: object_storage/, local/, network/

Move files into subdirectories and add __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


examples/r2.py:15:from mirage.backends.r2 import R2Backend, R2Config
examples/claude-memory.py:21:from mirage.backends.memory import MemoryBackend
examples/claude-sdk.py:24:from mirage.backends.memory import MemoryBackend
mirage/workspace/mount.py:1:from mirage.backends.base import BaseBackend
mirage/workspace/__init__.py:5:from mirage.backends.base import BaseBackend
mirage/workspace/__init__.py:6:from mirage.backends.memory import _guess_type
tests/fuse_ops/test_remote.py:3:from mirage.backends.memory import MemoryBackend
tests/commands/test_init.py:1:from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_default.py:1:from mirage.backends.memory import MemoryBackend
tests/commands/test_read.py:5:from mirage.backends.local import LocalBackend
mirage/runtime/local.py:1:from mirage.backends.local import LocalBackend
tests/observe/test_inspector.py:3:from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_format_ops.py:6:from mirage.backends.memory import MemoryBackend
mirage/backends/memory.py:4:from mirage.backends.base import BaseBackend
mirage/runtime/remote.py:5:from mirage.backends.proxy import ProxyBackend
mirage/backends/__init__.py:1:from mirage.backends.base import BaseBackend
mirage/backends/__init__.py:2:from mirage.backends.local import LocalBackend
mirage/backends/__init__.py:3:from mirage.backends.memory import MemoryBackend
mirage/backends/__init__.py:4:from mirage.backends.proxy import ProxyBackend
mirage/backends/http.py:3:from mirage.backends.base import BaseBackend
mirage/backends/http.py:4:from mirage.backends.memory import _guess_type
mirage/workspace/cache/policy.py:3:from mirage.backends.base import BaseBackend
mirage/backends/s3.py:6:from mirage.backends.base import BaseBackend
mirage/backends/s3.py:7:from mirage.backends.memory import _guess_type
mirage/workspace/sync.py:4:from mirage.backends.base import BaseBackend
mirage/__init__.py:1:from mirage.backends.local import LocalBackend
mirage/__init__.py:2:from mirage.backends.memory import MemoryBackend
mirage/backends/r2.py:3:from mirage.backends.s3 import S3Backend, S3Config
mirage/backends/proxy.py:5:from mirage.backends.base import BaseBackend
tests/fuse_ops/test_registry.py:63:    from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_registry.py:76:    from mirage.backends.local import LocalBackend
tests/fuse_ops/test_registry.py:88:    from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_registry.py:100:    from mirage.backends.memory import MemoryBackend
tests/fuse_ops/test_registry.py:111:    from mirage.backends.memory import MemoryBackend
mirage/backends/local.py:5:from mirage.backends.base import BaseBackend
mirage/backends/local.py:6:from mirage.backends.memory import _guess_type
mirage/observe/inspector.py:6:from mirage.backends.memory import MemoryBackend
mirage/server/service/local.py:11:from mirage.backends.loader import load_backend_class
tests/integration/test_remote_transfer.py:3:from mirage.backends.memory import MemoryBackend
tests/cli/test_config.py:3:from mirage.backends.local import LocalBackend
tests/cli/test_config.py:4:from mirage.backends.memory import MemoryBackend
mirage/cli/config.py:5:from mirage.backends.base import BaseBackend
mirage/cli/config.py:6:from mirage.backends.local import LocalBackend
mirage/cli/config.py:7:from mirage.backends.memory import MemoryBackend
mirage/cli/config.py:17:        from mirage.backends.s3 import S3Backend, S3Config
mirage/cli/config.py:22:        from mirage.backends.r2 import R2Backend, R2Config
mirage/cli/config.py:33:        from mirage.backends.http import HTTPBackend, HTTPConfig
mirage/cli/commands/command.py:42:    from mirage.backends.loader import load_backend_class
mirage/cli/commands/workspace.py:10:from mirage.backends.loader import load_backend_class
mirage/cli/commands/workspace.py:207:    from mirage.backends.proxy import ProxyBackend
fuse/README.md:60:from mirage.backends.memory import MemoryBackend
fuse/tests/test_fs.py:12:from mirage.backends.memory import MemoryBackend
tests/server/test_registry.py:6:from mirage.backends.memory import MemoryBackend
tests/conftest.py:3:from mirage.backends.memory import MemoryBackend
tests/runtime/test_memory.py:3:from mirage.backends.memory import MemoryBackend
tests/runtime/test_accessors.py:3:from mirage.backends.memory import MemoryBackend
tests/runtime/test_runtime.py:3:from mirage.backends.memory import MemoryBackend
tests/server/test_arg_resolver.py:1:from mirage.backends.memory import MemoryBackend
tests/workspace/test_mount.py:3:from mirage.backends.memory import MemoryBackend
tests/backends/test_loader.py:3:from mirage.backends.loader import load_backend_class
tests/backends/test_loader.py:8:    from mirage.backends.memory import MemoryBackend
tests/backends/test_loader.py:14:    script.write_text("from mirage.backends.memory import MemoryBackend\n"
tests/backends/test_local.py:3:from mirage.backends.local import LocalBackend
tests/backends/test_proxy.py:4:from mirage.backends.proxy import ProxyBackend
tests/backends/test_configs.py:4:from mirage.backends.r2 import R2Config
tests/backends/test_configs.py:5:from mirage.backends.s3 import S3Config
tests/workspace/test_write_ops.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_sync.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_init.py:3:from mirage.backends.memory import MemoryBackend
tests/workspace/test_lock.py:3:from mirage.backends.memory import MemoryBackend


No matches found


from mirage.backends.base import BaseBackend
from mirage.backends.local import LocalBackend
from mirage.backends.memory import MemoryBackend
from mirage.backends.proxy import ProxyBackend

__all__ = ["BaseBackend", "LocalBackend", "MemoryBackend", "ProxyBackend"]

# S3Backend, R2Backend, HTTPBackend imported lazily to avoid hard deps


Now I have a clear picture. Let me plan the moves:

**New structure:**
```
mirage/backends/
├── __init__.py          (re-exports everything)
├── base.py              (stays)
├── loader.py            (stays)
├── object_storage/
│   ├── __init__.py
│   ├── s3.py
│   └── r2.py
├── local/
│   ├── __init__.py
│   ├── local.py         (or filesystem.py?)
│   └── memory.py
└── network/
    ├── __init__.py
    ├── http.py
    └── proxy.py
```

I'm second-guessing the naming though—`local/local.py` feels redundant. I could go with `filesystem.py` instead, or maybe just use `storage/` as the parent folder. The user wants me to move forward with this, so I'll stick with `local/` and `filesystem.py` for clarity.

Actually, wait—the user specifically said not to rename files unless asked. So I'll keep the original filenames and just organize them into subfolders. That means `local/local.py` stays as-is, which is fine since Python uses this pattern elsewhere. The structure will have `object_storage/` for S3 and R2, and `local/` for the local backends.

Now I need to handle the reorganization: creating the directories, moving files with git mv, updating all the internal imports within those files, setting up the `__init__.py` files for each subpackage, and then updating the main backends `__init__.py` to reflect the new structure. I'm mapping out which imports need to change—like `local.py` importing from `memory.py` will need to become `from mirage.backends.local.memory import _guess_type` since they're moving into the same subfolder. to update memory import to `from mirage.backends.local.memory import _guess_type`
- `s3.py`: `from mirage.backends.base import BaseBackend` and `from mirage.backends.memory import _guess_type` → `from mirage.backends.local.memory import _guess_type`
- `http.py`: `from mirage.backends.base import BaseBackend` and `from mirage.backends.memory import _guess_type` → `from mirage.backends.local.memory import _guess_type`
- `r2.py`: `from mirage.backends.s3 import S3Backend, S3Config` → `from mirage.backends.object_storage.s3 import S3Backend, S3Config`
- `proxy.py`: `from mirage.backends.base import BaseBackend` (stays same)

Now I'm going through all the external imports systematically. The base and loader modules stay in place, but anything importing from memory needs to point to the local subfolder, and LocalBackend itself needs to come from the local.local module.

Actually, I realize I could solve the awkward `local.local` import by having the `local/__init__.py` re-export LocalBackend, so callers can just do `from mirage.backends.local import LocalBackend`. That would be cleaner and the user did ask for refactoring, so renaming is fair game. local` is now a package instead of a module. But that's actually fine - it would work the same way! The `local/` directory replaces `local.py`, and `local/__init__.py` can export `LocalBackend`.

So let me structure it so the sub-package __init__.py files re-export the key classes. That way external imports like `from mirage.backends.local import LocalBackend` still work!

But wait - if I have `local/__init__.py` that exports `LocalBackend`, then `from mirage.backends.local import LocalBackend` works. And `from mirage.backends.memory import MemoryBackend` would need to become... hmm, memory.py is inside the `local/` folder.

Actually, the cleanest approach: make each subfolder's `__init__.py` re-export the key symbols so that `from mirage.backends.object_storage import S3Backend` etc. works. Then I need to decide whether to maintain backward compatibility or just update all the import paths throughout the codebase. Since the user didn't require backward compat, I'll go with updating imports to be explicit about the new structure - something like `from mirage.backends.local.memory import MemoryBackend` or `from mirage.backends.object_storage.s3 import S3Backend`. Though I could also make the sub-package `__init__.py` files cleaner by re-exporting the main classes there.

Actually, re-exporting in the sub-package `__init__.py` is the way to go - it keeps imports clean without needing backward compat. I'll skip renaming files since the user said not to change filenames unless asked. ... needs to import from there directly. So my final structure organizes the backends into three subpackages—object storage, local, and network—each with their own `__init__.py` to cleanly export the public API, while keeping the implementation files organized by type.

Now I'm planning the import migration strategy: external code will reference the new subpackage paths, internal files need their cross-references updated, and the top-level `__init__.py` will re-export everything from the subpackages. I'll use git to move the files, write out all the `__init__.py` files to handle the exports, then systematically update the imports across the codebase.

The key decision is settling on the canonical import paths—I want external users to be able to import from `mirage.backends.local` rather than the deeper `mirage.backends.local.memory`, so I'll make sure the subpackage `__init__.py` files expose the main classes at that level.

For private helpers like `_guess_type`, I'll keep those as direct imports from their source modules since they're internal implementation details. Now I'm mapping out all the old import statements to their new equivalents so I can systematically update the codebase. I'm also updating the proxy import to follow the same pattern, moving it under the network subpackage. For the re-exports, I'm planning to have each subpackage's `__init__.py` expose its backends for convenience, while keeping the top-level `backends/__init__.py` relatively unchanged. I'm leaning toward using full explicit paths in external imports rather than the subpackage shortcuts—it's clearer and more maintainable.

Actually, I'm reconsidering. Using the subpackage `__init__` approach for re-exports is cleaner overall, so external code can just import directly from `mirage.backends.local`, `mirage.backends.object_storage`, and `mirage.backends.network` without worrying about the internal file structure.

For internal cross-package references like the `_guess_type` helper that's shared across multiple backends, I'll use full module paths to keep things explicit. Let me start executing this refactor systematically—first moving the files with git, then creating the `__init__` files for each subpackage, and finally updating all the imports throughout the codebase. Now I'm going through the remaining import updates across the network modules and workspace, updating all references to `MemoryBackend` and `_guess_type` to point to the new `local.memory` module, and fixing the `LocalBackend` import to use the full path through `local.local`. I'm realizing the local backend imports are already clean since the package-level `__init__.py` files expose them properly, so those don't need changes. But I need to reconsider whether the current structure actually uses `local` as a file or a package directory. I'm settling on the canonical import paths for each backend module, making sure sub-package `__init__.py` files re-export the main classes so users can import from the package level rather than diving into individual modules. For internal helpers like `_guess_type`, I'll use the full module path to keep things explicit. I'm working through the import path updates for the reorganized backend structure, mapping each backend to its new nested location—memory and local utilities under a local subpackage, proxy under network, and S3/R2 under object_storage.

Now I'm deciding whether to use direct module imports or re-exports through the subpackage `__init__` files. Since S3 and R2 have optional dependencies that need lazy loading, I'll keep the imports pointing directly to the specific modules rather than trying to re-export them at the subpackage level. Updating the import paths throughout the codebase to reflect the new structure—local subpackage imports stay the same, while object_storage and network backends need their paths adjusted to point to their new locations within those subpackages. I'm realizing this refactoring is getting complicated with all the import path changes across multiple files. Let me take a systematic approach—I'll start with the file structure changes using git operations, then handle all the import updates together, making sure to catch the hardcoded import strings in test files and documentation as well. I'm going to be pragmatic here—for mass find-and-replace across 30+ files, bash sed is the right tool despite the preference for Edit. I'll create the directories and move files with bash, write the new __init__.py files, then use sed to update all the imports in bulk, and finally verify with grep that the old imports are gone before running tests.


Grep all import patterns for backends across codebase

Create subdirectories and move files with git mv

Creating subdirectories and moving files

Write sub-package __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests

Create subdirectories and move files with git mv

Creating subdirectories and moving files

Write sub-package __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


Bash

mkdir -p mirage/backends/object_storage mirage/backends/local mirage/backends/network && git mv mirage/backends/s3.py mirage/backends/object_storage/s3.py && git mv mirage/backends/r2.py mirage/backends/object_storage/r2.py && git mv mirage/backends/local.py mirage/backends/local/local.py && git mv mirage/backends/memory.py mirage/backends/local/memory.py && git mv mirage/backends/http.py mirage/backends/network/http.py && git mv mirage/backends/proxy.py mirage/backends/network/proxy.py

Create subdirectories and git mv files into them


TodoWrite

Grep all import patterns for backends across codebase

Create subdirectories and move files with git mv

Creating subdirectories and moving files

Write sub-package __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


Write sub-package __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests

Write sub-package __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


Write

from mirage.backends.local.local import LocalBackend
from mirage.backends.local.memory import MemoryBackend, _guess_type

__all__ = ["LocalBackend", "MemoryBackend", "_guess_type"]


__all__: list[str] = []


File created successfully at: /Users/agent/strukto/mirage/mirage/backends/local/__init__.py

from mirage.backends.local.local import LocalBackend
from mirage.backends.local.memory import MemoryBackend, _guess_type

__all__ = ["LocalBackend", "MemoryBackend", "_guess_type"]


from mirage.backends.network.proxy import ProxyBackend

__all__ = ["ProxyBackend"]


File created successfully at: /Users/agent/strukto/mirage/mirage/backends/object_storage/__init__.py

__all__: list[str] = []


File created successfully at: /Users/agent/strukto/mirage/mirage/backends/network/__init__.py

from mirage.backends.network.proxy import ProxyBackend

__all__ = ["ProxyBackend"]


TodoWrite

Grep all import patterns for backends across codebase

Create subdirectories and move files with git mv

Creating subdirectories and moving files

Write sub-package __init__.py files

Update all imports across the codebase

Update top-level backends/__init__.py

Run pre-commit and tests


Now I'll update all imports. The key replacements are:

- `mirage.backends.memory` → `mirage.backends.local.memory` (for direct module imports)
- `mirage.backends.local import LocalBackend` → `mirage.backends.local.local import LocalBackend` (internal only; external can still use `mirage.backends.local`)
- `mirage.backends.s3` → `mirage.backends.object_storage.s3`
- `mirage.backends.r2` → `mirage.backends.object_storage.r2`
- `mirage.backends.http` → `mirage.backends.network.http`
- `mirage.backends.proxy` → `mirage.backends.network.proxy`

Let me start with the internal backend files, then external files.


     1→import os
     2→from datetime import datetime, timezone
     3→from pathlib import Path
     4→
     5→from mirage.backends.base import BaseBackend
     6→from mirage.backends.memory import _guess_type
     7→from mirage.types import FileStat, FileType
     8→
     9→
    10→class LocalBackend(BaseBackend):

import os
from datetime import datetime, timezone
from pathlib import Path

from mirage.backends.base import BaseBackend
from mirage.backends.memory import _guess_type
from mirage.types import FileStat, FileType

class LocalBackend(BaseBackend):


     1→import hashlib
     2→from datetime import datetime, timezone
     3→
     4→from mirage.backends.base import BaseBackend
     5→from mirage.types import FileStat, FileType
     6→
     7→
     8→class MemoryBackend(BaseBackend):
     9→    """
    10→    In-memory backend backed by a plain dict. No external dependencies.

import hashlib
from datetime import datetime, timezone

from mirage.backends.base import BaseBackend
from mirage.types import FileStat, FileType

class MemoryBackend(BaseBackend):
    """
    In-memory backend backed by a plain dict. No external dependencies.


     1→import re
     2→from datetime import timezone
     3→
     4→from pydantic import BaseModel, ConfigDict
     5→
     6→from mirage.backends.base import BaseBackend
     7→from mirage.backends.memory import _guess_type
     8→from mirage.types import FileStat, FileType
     9→
    10→

import re
from datetime import timezone

from pydantic import BaseModel, ConfigDict

from mirage.backends.base import BaseBackend
from mirage.backends.memory import _guess_type
from mirage.types import FileStat, FileType


     1→from pydantic import BaseModel, ConfigDict
     2→
     3→from mirage.backends.s3 import S3Backend, S3Config
     4→
     5→
     6→class R2Config(BaseModel):
     7→    model_config = ConfigDict(frozen=True)
     8→
     9→    account_id: str
    10→    bucket: str

from pydantic import BaseModel, ConfigDict

from mirage.backends.s3 import S3Backend, S3Config

class R2Config(BaseModel):
    model_config = ConfigDict(frozen=True)

    account_id: str
    bucket: str


     1→from pydantic import BaseModel, ConfigDict, Field
     2→
     3→from mirage.backends.base import BaseBackend
     4→from mirage.backends.memory import _guess_type
     5→from mirage.types import FileStat
     6→
     7→
     8→class HTTPConfig(BaseModel):
     9→    model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)
    10→

from pydantic import BaseModel, ConfigDict, Field

from mirage.backends.base import BaseBackend
from mirage.backends.memory import _guess_type
from mirage.types import FileStat

class HTTPConfig(BaseModel):
    model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)


     1→import base64
     2→
     3→import httpx
     4→
     5→from mirage.backends.base import BaseBackend
     6→from mirage.types import FileStat
     7→
     8→
     9→class ProxyBackend(BaseBackend):
    10→

import base64

import httpx

from mirage.backends.base import BaseBackend
from mirage.types import FileStat

class ProxyBackend(BaseBackend):


Edit

from mirage.backends.memory import _guess_type

from mirage.backends.local.memory import _guess_type


from mirage.backends.memory import _guess_type

from mirage.backends.local.memory import _guess_type


The file /Users/agent/strukto/mirage/mirage/backends/local/local.py has been updated successfully.

import os
from datetime import datetime, timezone
from pathlib import Path

from mirage.backends.base import BaseBackend
from mirage.backends.memory import _guess_type
from mirage.types import FileStat, FileType

class LocalBackend(BaseBackend):
    """
    Backend backed by the local filesystem via pathlib/os.
    No external dependencies.

    All virtual paths are resolved relative to `root`.
    Example: root="/data", path="/reports/q1.csv" → /data/reports/q1.csv
    """

    def __init__(self, root: str) -> None:
        self.root = Path(root).resolve()

    # ── helpers ─────────────────────────────────────────────────────────────

    def _resolve(self, path: str) -> Path:
        # strip leading slash so joinpath works correctly
        relative = path.lstrip("/")
        resolved = (self.root / relative).resolve()
        # prevent path traversal outside root
        resolved.relative_to(self.root)
        return resolved

    def _to_stat(self, p: Path) -> FileStat:
        st = p.stat()
        modified = datetime.fromtimestamp(st.st_mtime,
                                          tz=timezone.utc).isoformat()
        if p.is_dir():
            return FileStat(name=p.name,
                            size=None,
                            modified=modified,
                            type=FileType.DIRECTORY)
        return FileStat(name=p.name,
                        size=st.st_size,
                        modified=modified,
                        type=_guess_type(p.name))

    # ── ops ─────────────────────────────────────────────────────────────────

    def read(self, path: str, offset: int, size: int) -> bytes:
        p = self._resolve(path)
        with p.open("rb") as f:
            f.seek(offset)
            return f.read(size)

    def readdir(self,
                path: str,
                cursor: str | None = None) -> tuple[list[str], str | None]:
        p = self._resolve(path)
        if not p.is_dir():
            raise NotADirectoryError(str(p))
        base = "/" + path.strip("/")
        entries = sorted(base.rstrip("/") + "/" + e.name for e in p.iterdir())
        # cursor is an offset index for pagination
        start = int(cursor) if cursor else 0
        page = entries[start:]
        return page, None

    def stat(self, path: str) -> FileStat:
        p = self._resolve(path)
        if not p.exists():
            raise FileNotFoundError(str(p))
        return self._to_stat(p)

    def open(self, path: str, flags: int) -> int:
        raise NotImplementedError("open/release are FUSE-only")

    def release(self, fh: int) -> None:
        raise NotImplementedError("open/release are FUSE-only")

    def write(self, path: str, offset: int, data: bytes) -> int:
        p = self._resolve(path)
        with p.open("r+b") as f:
            f.seek(offset)
            f.write(data)
        return len(data)

    def create(self, path: str) -> None:
        p = self._resolve(path)
        p.parent.mkdir(parents=True, exist_ok=True)
        p.touch(exist_ok=True)

    def unlink(self, path: str) -> None:
        p = self._resolve(path)
        p.unlink()

    def rmdir(self, path: str) -> None:
        p = self._resolve(path)
        p.rmdir()  # raises OSError if non-empty — matches POSIX semantics

    def mkdir(self, path: str) -> None:
        p = self._resolve(path)
        p.mkdir(parents=True, exist_ok=True)

    def rename(self, src: str, dst: str) -> None:
        s = self._resolve(src)
        d = self._resolve(dst)
        s.rename(d)

    def truncate(self, path: str, size: int) -> None:
        p = self._resolve(path)
        os.truncate(p, size)

-from mirage.backends.memory import _guess_type

+from mirage.backends.local.memory import _guess_type