bcbd1bdb22
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
138 lines
4.8 KiB
Plaintext
138 lines
4.8 KiB
Plaintext
---
|
|
title: LanceDB
|
|
description: Mount a LanceDB table as a Mirage filesystem, with label folders, multimodal blobs, and a semantic search command for Python agents.
|
|
icon: database
|
|
---
|
|
|
|
The LanceDB resource exposes a LanceDB table as a virtual filesystem mounted at
|
|
some prefix such as `/fashion/`. Group-by columns become nested folders, each
|
|
row becomes a card plus an optional blob file, and semantic search is the
|
|
`search` command, which returns ranked rows as canonical file paths.
|
|
|
|
For connection setup (LanceDB OSS, object storage, Cloud, Enterprise), see
|
|
[LanceDB Setup](/python/setup/lancedb).
|
|
|
|
## Config
|
|
|
|
```python
|
|
from mirage import MountMode, Workspace
|
|
from mirage.resource.lancedb import LanceDBConfig, LanceDBResource
|
|
|
|
config = LanceDBConfig(
|
|
uri="/data/fashion.lancedb",
|
|
table="fashion",
|
|
group_by=["gender", "articleType", "baseColour"],
|
|
id_column="id",
|
|
title_column="productDisplayName",
|
|
blob_column="image_bytes",
|
|
blob_ext="jpg",
|
|
vector_column="vector",
|
|
search_limit=5,
|
|
)
|
|
resource = LanceDBResource(config)
|
|
ws = Workspace({"/fashion/": resource}, mode=MountMode.READ)
|
|
```
|
|
|
|
The mapping is config-driven; nothing about the dataset is hardcoded. Point
|
|
`group_by` at different columns and the folder tree changes. See the full
|
|
[config reference](/python/setup/lancedb#config-reference).
|
|
|
|
## Filesystem layout
|
|
|
|
Every path is translated into a LanceDB query. Descending a folder adds one
|
|
`WHERE` clause; the leaf level lists rows.
|
|
|
|
```text
|
|
/ # list tables (omitted when `table` is pinned)
|
|
/<table>/ # distinct group_by[0] values
|
|
<v1>/ # distinct group_by[1] WHERE group_by[0]=v1
|
|
.../<vN>/ # all group-by columns bound -> row files
|
|
<id>.md # rendered card (text)
|
|
<id>.<ext> # raw blob / image bytes
|
|
```
|
|
|
|
When `table` is set the table level is elided, so the mount root is that table:
|
|
|
|
```text
|
|
/fashion/
|
|
Men/
|
|
Shoes/
|
|
White/
|
|
3.md
|
|
3.jpg
|
|
```
|
|
|
|
### Row cards
|
|
|
|
A `<id>.md` card renders the row's columns as readable text and points at its
|
|
blob. The vector and blob columns are omitted from the card body.
|
|
|
|
```text
|
|
# Nike Men White Running Sneakers
|
|
|
|
id: 3
|
|
gender: Men
|
|
articleType: Shoes
|
|
baseColour: White
|
|
productDisplayName: Nike Men White Running Sneakers
|
|
blob: 3.jpg
|
|
```
|
|
|
|
## Semantic search
|
|
|
|
Search is a command, not a path. It returns each ranked row as its **canonical
|
|
file path** (the same `<id>.md` you would `cat` while browsing) annotated with
|
|
the vector distance, followed by the card body. Results point back at the real
|
|
files, so search composes with `cat`, pipes, and `wc`:
|
|
|
|
```text
|
|
$ search "white running sneakers" /fashion
|
|
/fashion/Men/Shoes/White/3.md:0.2679
|
|
# Nike Men White Running Sneakers
|
|
|
|
id: 3
|
|
gender: Men
|
|
articleType: Shoes
|
|
baseColour: White
|
|
productDisplayName: Nike Men White Running Sneakers
|
|
blob: 3.jpg
|
|
...
|
|
```
|
|
|
|
Flags: `--top-k <n>` (default `search_limit`), `--threshold <max-distance>`,
|
|
`--method semantic` (the only supported method; `grep`/`rg` stay lexical).
|
|
|
|
## Supported commands
|
|
|
|
All commands delegate to Mirage's shared implementations.
|
|
|
|
| Command | Behaviour on a LanceDB mount |
|
|
| ------------- | ------------------------------------------------------------- |
|
|
| `ls` | list tables, label folders, or row files |
|
|
| `cd` | navigate (each level narrows the filter) |
|
|
| `tree` | render the label hierarchy |
|
|
| `cat` | print a row card, or dump raw blob/image bytes |
|
|
| `stat` | directory vs file, blob size, image mime type |
|
|
| `find` | walk the tree (e.g. `find /fashion -name '*.md'`) |
|
|
| `grep` / `rg` | lexical search over the rendered cards |
|
|
| `search` | semantic (vector) search -> ranked canonical paths + score |
|
|
| `head` / `tail` | first/last lines of a card |
|
|
| `wc` | count lines/bytes of a card |
|
|
|
|
`grep`/`rg` stay lexical (literal/regex). `search` is the semantic path: it
|
|
auto-embeds the query via the table's embedding function and returns ranked
|
|
rows as canonical file paths, which compose with `cat`, `wc`, and pipes.
|
|
|
|
## Access pattern
|
|
|
|
The mount is **read-only** (`MountMode.READ`); writes are not supported. The two
|
|
read modes are:
|
|
|
|
- **Browse** by label folders: pure metadata `WHERE` filters, no embedding.
|
|
- **Search** by meaning: `search "<query>" <path>` runs vector search using the
|
|
table's embedding function and returns canonical row paths.
|
|
|
|
Folder listings scan one column with `SELECT DISTINCT` and are capped by
|
|
`max_rows`, so very large tables should keep `group_by` to low-cardinality
|
|
columns.
|