chore: import upstream snapshot with attribution
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
Integ / changes (push) Has been skipped
Pre-commit / pre-commit (push) Failing after 1s
CLI exit codes / changes (push) Has been skipped
Test (Install) / changes (push) Has been skipped
Test (Python) / changes (push) Has been skipped
Test (TypeScript) / changes (push) Has been skipped
CLI exit codes / cli-gate (push) Has been cancelled
Test (Install) / test-install-gate (push) Has been cancelled
Integ / integ-gate (push) Has been cancelled
Test (Python) / test-python-gate (push) Has been cancelled
Test (TypeScript) / test-typescript-gate (push) Has been cancelled
Test (Install) / python-minimal (3.12) (push) Has been cancelled
Test (Install) / python-minimal (3.11) (push) Has been cancelled
Test (Install) / python-extra (agno, mirage.agents.agno) (push) Has been cancelled
Test (Install) / python-extra (chroma, mirage.resource.chroma) (push) Has been cancelled
Test (Install) / python-extra (pdf, mirage.core.filetype.pdf) (push) Has been cancelled
Integ / integ (push) Has been cancelled
Integ / integ-database (push) Has been cancelled
Integ / integ-database-ts (push) Has been cancelled
Integ / integ-data (push) Has been cancelled
Integ / integ-ssh (push) Has been cancelled
Integ / integ-ssh-ts (push) Has been cancelled
Test (Python) / audit (push) Has been cancelled
Test (TypeScript) / test (push) Has been cancelled
Test (TypeScript) / python-fs-shim (push) Has been cancelled
CLI exit codes / Python CLI (push) Has been cancelled
CLI exit codes / TypeScript CLI (push) Has been cancelled
CLI exit codes / Cross-language snapshot interop (push) Has been cancelled
Test (Python) / test (push) Has been cancelled
Test (Python) / import-isolation (deepagents, openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Python) / import-isolation (deepagents, pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Integ / integ-ts (push) Has been cancelled
Integ / integ-fuse (push) Has been cancelled
Test (Install) / python-extra (databricks, mirage.resource.databricks_volume) (push) Has been cancelled
Test (Install) / python-extra (deepagents, mirage.agents.langchain) (push) Has been cancelled
Test (Install) / python-extra (email, mirage.resource.email) (push) Has been cancelled
Test (Install) / python-extra (fuse, mirage.fuse.mount) (push) Has been cancelled
Test (Install) / python-extra (hdf5, mirage.core.filetype.hdf5) (push) Has been cancelled
Test (Install) / python-extra (hf, mirage.resource.hf_buckets) (push) Has been cancelled
Test (Install) / python-extra (lancedb, mirage.resource.lancedb) (push) Has been cancelled
Test (Install) / python-extra (langfuse, mirage.resource.langfuse) (push) Has been cancelled
Test (Install) / python-extra (mongodb, mirage.resource.mongodb) (push) Has been cancelled
Test (Install) / python-extra (nextcloud, mirage.resource.nextcloud) (push) Has been cancelled
Test (Install) / python-extra (openai, mirage.agents.openai_agents) (push) Has been cancelled
Test (Install) / python-extra (openhands, mirage.agents.openhands, 3.12) (push) Has been cancelled
Test (Install) / python-extra (parquet, mirage.core.filetype.parquet) (push) Has been cancelled
Test (Install) / python-extra (postgres, mirage.resource.postgres) (push) Has been cancelled
Test (Install) / python-extra (pydantic-ai, mirage.agents.pydantic_ai) (push) Has been cancelled
Test (Install) / python-extra (qdrant, mirage.resource.qdrant) (push) Has been cancelled
Test (Install) / python-extra (redis, mirage.resource.redis) (push) Has been cancelled
Test (Install) / python-extra (s3, mirage.resource.s3) (push) Has been cancelled
Test (Install) / python-extra (ssh, mirage.resource.ssh) (push) Has been cancelled
Test (Install) / ts-minimal (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: LanceDB
|
||||
icon: database
|
||||
description: Set up the LanceDB resource in Python, including LanceDB OSS, object storage, LanceDB Cloud, and LanceDB Enterprise, with semantic search.
|
||||
---
|
||||
|
||||
The LanceDB resource mounts a LanceDB table as a filesystem: group-by columns
|
||||
become folders, rows become files, and semantic search is the `search` command.
|
||||
See [LanceDB Resource](/python/resource/lancedb) for the full layout and command
|
||||
list.
|
||||
|
||||
## Dependencies
|
||||
|
||||
```bash
|
||||
uv add lancedb
|
||||
```
|
||||
|
||||
`lancedb` ships the embedded engine and the async client Mirage uses. It pulls
|
||||
in `pyarrow`; no separate server is required.
|
||||
|
||||
Semantic search needs an embedding function inside the table. For real
|
||||
multimodal (CLIP) embeddings, add the model deps to your **builder**
|
||||
environment only (Mirage core never imports them):
|
||||
|
||||
```bash
|
||||
uv add open-clip-torch torch
|
||||
```
|
||||
|
||||
## Where the data lives
|
||||
|
||||
A LanceDB database is a directory of Lance files. The `uri` decides where it is
|
||||
stored, and the same `LanceDBConfig` works for every tier.
|
||||
|
||||
### LanceDB OSS (local disk)
|
||||
|
||||
```python
|
||||
from mirage import MountMode, Workspace
|
||||
from mirage.resource.lancedb import LanceDBConfig, LanceDBResource
|
||||
|
||||
config = LanceDBConfig(
|
||||
uri="/data/fashion.lancedb",
|
||||
table="fashion",
|
||||
group_by=["gender", "articleType", "baseColour"],
|
||||
id_column="id",
|
||||
title_column="productDisplayName",
|
||||
blob_column="image_bytes",
|
||||
blob_ext="jpg",
|
||||
vector_column="vector",
|
||||
)
|
||||
ws = Workspace({"/fashion/": LanceDBResource(config)}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
### Object storage (S3 / GCS / Azure)
|
||||
|
||||
Point `uri` at a bucket. Credentials come from the environment by default, or
|
||||
pass them through `storage_options`.
|
||||
|
||||
```python
|
||||
config = LanceDBConfig(
|
||||
uri="s3://my-bucket/fashion.lancedb",
|
||||
table="fashion",
|
||||
group_by=["gender", "articleType", "baseColour"],
|
||||
id_column="id",
|
||||
vector_column="vector",
|
||||
storage_options={"region": "us-east-1"},
|
||||
)
|
||||
```
|
||||
|
||||
### LanceDB Cloud
|
||||
|
||||
Use a `db://` URI plus an API key and region. The API key can also come from the
|
||||
`LANCEDB_API_KEY` environment variable.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
config = LanceDBConfig(
|
||||
uri="db://my-database",
|
||||
api_key=os.environ["LANCEDB_API_KEY"],
|
||||
region="us-east-1",
|
||||
table="fashion",
|
||||
group_by=["gender", "articleType", "baseColour"],
|
||||
id_column="id",
|
||||
vector_column="vector",
|
||||
)
|
||||
ws = Workspace({"/fashion/": LanceDBResource(config)}, mode=MountMode.READ)
|
||||
```
|
||||
|
||||
### LanceDB Enterprise
|
||||
|
||||
Enterprise is the same as Cloud plus a custom endpoint via `host_override`.
|
||||
|
||||
```python
|
||||
config = LanceDBConfig(
|
||||
uri="db://my-database",
|
||||
api_key=os.environ["LANCEDB_API_KEY"],
|
||||
host_override="https://my-database.us-east-1.api.lancedb.com",
|
||||
region="us-east-1",
|
||||
table="fashion",
|
||||
group_by=["gender", "articleType", "baseColour"],
|
||||
id_column="id",
|
||||
vector_column="vector",
|
||||
)
|
||||
```
|
||||
|
||||
`region` and `host_override` are only applied for `db://` URIs; they are ignored
|
||||
for local and object-storage mounts.
|
||||
|
||||
## Search setup
|
||||
|
||||
Search is powered by the table's own embedding function, not by Mirage. The
|
||||
`search` command is available when `vector_column` is set; the table must have
|
||||
been created with an embedding function registered on a source field.
|
||||
|
||||
A minimal CLIP-backed table (run once in your builder environment):
|
||||
|
||||
```python
|
||||
import lancedb
|
||||
from lancedb.embeddings import get_registry
|
||||
from lancedb.pydantic import LanceModel, Vector
|
||||
|
||||
func = get_registry().get("open-clip").create()
|
||||
|
||||
class Product(LanceModel):
|
||||
id: int
|
||||
gender: str
|
||||
articleType: str
|
||||
baseColour: str
|
||||
productDisplayName: str = func.SourceField() # text the model embeds
|
||||
image_bytes: bytes
|
||||
vector: Vector(func.ndims()) = func.VectorField()
|
||||
|
||||
db = lancedb.connect("/data/fashion.lancedb")
|
||||
table = db.create_table("fashion", schema=Product)
|
||||
table.add(rows) # rows include image_bytes
|
||||
```
|
||||
|
||||
Once mounted, querying is the `search` command. LanceDB embeds the query text
|
||||
with the same model and runs vector search, returning ranked rows as canonical
|
||||
file paths with a score, then their cards:
|
||||
|
||||
```bash
|
||||
search "red running shoes" /fashion # ranked <path>.md:score + card body
|
||||
cat /fashion/Men/Shoes/White/3.md # follow a result to the real file
|
||||
```
|
||||
|
||||
A runnable, dependency-free version (a lightweight keyword embedding instead of
|
||||
CLIP) lives in `examples/python/lancedb/`.
|
||||
|
||||
## Config reference
|
||||
|
||||
| Field | Required | Default | Description |
|
||||
| ---------------- | -------- | ------------- | ------------------------------------------------------------------ |
|
||||
| `uri` | Yes | | Local path, `s3://`/`gs://`/`az://`/`hf://`, or `db://` (Cloud) |
|
||||
| `api_key` | No | | LanceDB Cloud/Enterprise API key (or `LANCEDB_API_KEY`) |
|
||||
| `region` | No | `us-east-1` | Cloud region (`db://` only) |
|
||||
| `host_override` | No | | Enterprise endpoint URL (`db://` only) |
|
||||
| `storage_options`| No | | Object-storage options/credentials |
|
||||
| `table` | No | | Pin one table; the mount root becomes that table |
|
||||
| `group_by` | No | `[]` | Columns that become nested folder levels |
|
||||
| `id_column` | No | `id` | Column used to name row files |
|
||||
| `title_column` | No | | Column used as the card heading |
|
||||
| `blob_column` | No | | Column served as the raw blob/image file |
|
||||
| `blob_ext` | No | `bin` | Extension for the blob file (`jpg`, `png`, ...) |
|
||||
| `vector_column` | No | | Vector column; presence enables the `search` command |
|
||||
| `search_limit` | No | `10` | Default top-k returned by `search` |
|
||||
| `max_rows` | No | `1000` | Cap on rows scanned per folder listing |
|
||||
|
||||
The mount is read-only. See [LanceDB Resource](/python/resource/lancedb) for the
|
||||
filesystem layout and supported commands.
|
||||
Reference in New Issue
Block a user