chore: import upstream snapshot with attribution
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,211 @@
|
||||
---
|
||||
title: Databricks Volume
|
||||
icon: folder-open
|
||||
description: Mount a Databricks Unity Catalog volume as a filesystem.
|
||||
---
|
||||
|
||||
`DatabricksVolumeResource` exposes files from a Unity Catalog volume through
|
||||
Mirage's standard filesystem interface. Agents can list, stat, read, stream,
|
||||
glob, and write files under the configured volume root.
|
||||
|
||||
For auth and environment setup, see [Databricks Volume Setup](/python/setup/databricks).
|
||||
|
||||
## Config
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.databricks_volume import (
|
||||
DatabricksVolumeConfig,
|
||||
DatabricksVolumeResource,
|
||||
)
|
||||
|
||||
resource = DatabricksVolumeResource(DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
root_path="/reports",
|
||||
))
|
||||
ws = Workspace({"/dbx": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
| Field | Default | Notes |
|
||||
| --- | --- | --- |
|
||||
| `catalog` | required | Unity Catalog catalog name. |
|
||||
| `schema` | required | Unity Catalog schema name. |
|
||||
| `volume` | required | Unity Catalog volume name. |
|
||||
| `root_path` | `/` | Subdirectory inside the volume to expose. |
|
||||
| `host` | `None` | Optional workspace host override. |
|
||||
| `token` | `None` | Optional PAT override. Redacted in snapshots. |
|
||||
| `profile` | `None` | Optional Databricks SDK profile name. |
|
||||
| `timeout` | `30` | Request timeout in seconds. |
|
||||
|
||||
## Mount mode
|
||||
|
||||
`read` or `write`.
|
||||
|
||||
## Filesystem layout
|
||||
|
||||
Mirage maps the configured mount prefix onto the configured volume subtree.
|
||||
|
||||
Given:
|
||||
|
||||
```python
|
||||
DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
root_path="/reports/2026",
|
||||
)
|
||||
```
|
||||
|
||||
and mount prefix `/dbx/`, the volume path:
|
||||
|
||||
```text
|
||||
/Volumes/main/default/agent_files/reports/2026/q1/summary.md
|
||||
```
|
||||
|
||||
appears in Mirage as:
|
||||
|
||||
```text
|
||||
/dbx/q1/summary.md
|
||||
```
|
||||
|
||||
`root_path` is normalized before use, and Mirage rejects any virtual path that
|
||||
would escape above that configured subtree.
|
||||
|
||||
## Supported operations
|
||||
|
||||
Reads: `readdir`, `stat`, `exists`, `read_bytes`, `read_stream`,
|
||||
`range_read`, and glob resolution. Writes: `write`, `create`, `mkdir`,
|
||||
`rmdir`, `unlink`, recursive `rm`, `cp`, and `mv`.
|
||||
|
||||
`mv`/`cp` are non-atomic download + upload — the Files API has no server-side
|
||||
rename.
|
||||
|
||||
## Shell Commands
|
||||
|
||||
The Databricks Volume resource supports shell commands that operate on real
|
||||
file content. Reads use the Files API with range requests, so commands like
|
||||
`head -c BYTES` avoid downloading the whole object. The supported set is
|
||||
scoped to commands that work over the volume API (no compression, encoding,
|
||||
or local-only utilities).
|
||||
|
||||
### Read Commands
|
||||
|
||||
| Command | Notes |
|
||||
| --------------- | ------------------------------------------ |
|
||||
| `cat` | Read file content |
|
||||
| `head` / `tail` | First/last N lines |
|
||||
| `grep` / `rg` | Pattern search (file or directory level) |
|
||||
| `jq` | Query JSON fields |
|
||||
| `wc` | Line/word/byte counts |
|
||||
| `stat` | File metadata (name, size, type, modified) |
|
||||
| `find` | Recursive search with `-name`, `-maxdepth` |
|
||||
| `tree` | Directory tree view |
|
||||
| `nl` | Number lines |
|
||||
|
||||
### Text Processing
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | -------------------------------- |
|
||||
| `awk` | Pattern scanning and processing |
|
||||
| `sed` | Stream editor |
|
||||
| `tr` | Translate or delete characters |
|
||||
| `sort` | Sort lines |
|
||||
| `uniq` | Remove duplicate lines |
|
||||
| `cut` | Extract fields/columns |
|
||||
| `diff` | Compare files line by line |
|
||||
|
||||
### File Operations
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ---------------------------------------------------- |
|
||||
| `cp` | Copy files (non-atomic download + upload) |
|
||||
| `mv` | Move/rename files (non-atomic download + upload) |
|
||||
| `rm` | Remove files (recursive for directories) |
|
||||
| `mkdir` | Create directories |
|
||||
| `touch` | Create empty file or update timestamp |
|
||||
|
||||
### Path Utilities
|
||||
|
||||
| Command | Notes |
|
||||
| ------- | ----------------------- |
|
||||
| `ls` | List directory contents |
|
||||
|
||||
## Snapshot behavior
|
||||
|
||||
`token` is redacted in resource state. Loading a snapshot back requires an
|
||||
override config that provides fresh credentials if the runtime auth chain does
|
||||
not already supply them.
|
||||
|
||||
## Databricks Apps
|
||||
|
||||
For Databricks Apps, prefer SDK-default auth and keep Mirage in-process:
|
||||
|
||||
```python
|
||||
config = DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
)
|
||||
resource = DatabricksVolumeResource(config)
|
||||
ws = Workspace({"/dbx/": resource}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
This does not require FUSE. The agent can access the mounted workspace through
|
||||
Mirage's backend adapters or `Workspace.execute(...)`.
|
||||
|
||||
## Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.databricks_volume import (
|
||||
DatabricksVolumeConfig,
|
||||
DatabricksVolumeResource,
|
||||
)
|
||||
|
||||
resource = DatabricksVolumeResource(DatabricksVolumeConfig(
|
||||
catalog="main",
|
||||
schema="default",
|
||||
volume="agent_files",
|
||||
root_path="/reports",
|
||||
))
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
ws = Workspace({"/dbx/": resource}, mode=MountMode.READ)
|
||||
|
||||
r = await ws.execute("ls /dbx/")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute("find /dbx/ -name '*.md'")
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute('head -n 20 "/dbx/q1/summary.md"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
r = await ws.execute('stat "/dbx/q1/summary.md"')
|
||||
print(await r.stdout_str())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
See:
|
||||
|
||||
- `examples/python/databricks_volume/databricks_volume.py`
|
||||
- `examples/python/agents/langchain/databricks_volume_deepagent.py`
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Agents in Databricks Apps**: mount a Unity Catalog volume in-process so an
|
||||
agent can read and write governed files without FUSE or hardcoded credentials.
|
||||
- **Reading governed datasets**: expose a reports or dataset subtree through
|
||||
`root_path` and query it with shell commands.
|
||||
- **Sandboxed volume access**: scope an agent to a single volume (and optional
|
||||
`root_path`) so it cannot read or write outside that subtree.
|
||||
- **Writing agent outputs back**: persist generated files to the volume with
|
||||
`write`, `cp`, and `mv`.
|
||||
Reference in New Issue
Block a user