wehub-resource-sync
227 lines
8.6 KiB
Markdown
227 lines
8.6 KiB
Markdown
# MV2 File Format Specification
|
|
|
|
Version 2.1
|
|
|
|
## Overview
|
|
|
|
MV2 is a single-file format for AI memory storage. Everything lives in one file: header, write-ahead log, data segments, search indices, and metadata. No sidecar files.
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ .mv2 FILE │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Header │ 4 KB │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Embedded WAL │ 1-64 MB (capacity-dependent) │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Data Segments │ Variable │
|
|
│ - Frame payloads │
|
|
│ - Compressed content │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Lex Index Segment │ Tantivy index (optional) │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Vec Index Segment │ HNSW vectors (optional) │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ Time Index Segment │ Chronological ordering │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ TOC (Footer) │ Segment catalog + checksums │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Header (4096 bytes)
|
|
|
|
The header occupies the first 4 KB of the file.
|
|
|
|
| Offset | Size | Field | Description |
|
|
|--------|------|-------|-------------|
|
|
| 0 | 4 | `magic` | `MV2\0` (0x4D 0x56 0x32 0x00) |
|
|
| 4 | 2 | `version` | Format version (little-endian) |
|
|
| 6 | 1 | `spec_major` | Spec major version (2) |
|
|
| 7 | 1 | `spec_minor` | Spec minor version (1) |
|
|
| 8 | 8 | `footer_offset` | Byte offset to TOC |
|
|
| 16 | 8 | `wal_offset` | Byte offset to WAL (always 4096) |
|
|
| 24 | 8 | `wal_size` | WAL region size in bytes |
|
|
| 32 | 8 | `wal_checkpoint_pos` | Last checkpointed sequence |
|
|
| 40 | 8 | `wal_sequence` | Current WAL sequence number |
|
|
| 48 | 32 | `toc_checksum` | SHA-256 of TOC segment |
|
|
| 80 | 4016 | reserved | Zero-filled, reserved for future use |
|
|
|
|
All multi-byte integers are little-endian.
|
|
|
|
## Write-Ahead Log (WAL)
|
|
|
|
The embedded WAL provides crash recovery. It starts at byte 4096 and has a capacity determined by the file's target size:
|
|
|
|
| File Capacity | WAL Size |
|
|
|---------------|----------|
|
|
| < 100 MB | 1 MB |
|
|
| < 1 GB | 4 MB |
|
|
| < 10 GB | 16 MB |
|
|
| >= 10 GB | 64 MB |
|
|
|
|
### WAL Entry Format
|
|
|
|
```
|
|
┌──────────────────────────────────────┐
|
|
│ sequence │ 8 bytes (u64 LE) │
|
|
│ entry_type │ 1 byte │
|
|
│ payload_len │ 4 bytes (u32 LE) │
|
|
│ payload │ variable │
|
|
│ checksum │ 4 bytes (CRC32) │
|
|
└──────────────────────────────────────┘
|
|
```
|
|
|
|
Entry types:
|
|
- `0x01` - Frame append
|
|
- `0x02` - Frame update
|
|
- `0x03` - Frame delete (tombstone)
|
|
- `0x04` - Index update
|
|
|
|
### Checkpoint Behavior
|
|
|
|
- Checkpoint triggers at 75% WAL occupancy or every 1,000 transactions
|
|
- Checkpoint flushes WAL entries to data segments
|
|
- `seal()` forces immediate checkpoint
|
|
- Recovery replays entries with `sequence > wal_checkpoint_pos`
|
|
|
|
## Frame Structure
|
|
|
|
Each frame represents a single piece of content.
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `frame_id` | u64 | Unique identifier (monotonic) |
|
|
| `uri` | String | Hierarchical path (`mv2://path/to/doc`) |
|
|
| `title` | String? | Optional display title |
|
|
| `created_at` | u64 | Unix timestamp (seconds) |
|
|
| `encoding` | u8 | Content encoding (see below) |
|
|
| `payload` | bytes | Compressed content |
|
|
| `payload_checksum` | [u8; 32] | SHA-256 of uncompressed payload |
|
|
| `tags` | Map<String, String> | User-defined key-value pairs |
|
|
| `status` | u8 | 0=active, 1=tombstoned |
|
|
|
|
### Encoding Types
|
|
|
|
| Value | Name | Description |
|
|
|-------|------|-------------|
|
|
| 0 | Raw | Uncompressed bytes |
|
|
| 1 | Zstd | Zstandard compression |
|
|
| 2 | Lz4 | LZ4 compression |
|
|
|
|
## Data Segments
|
|
|
|
Frames are grouped into segments for efficient storage and retrieval.
|
|
|
|
### Segment Header
|
|
|
|
```
|
|
┌──────────────────────────────────────┐
|
|
│ magic │ 4 bytes │
|
|
│ version │ 2 bytes │
|
|
│ segment_type │ 1 byte │
|
|
│ frame_count │ 4 bytes │
|
|
│ compressed │ 1 byte (bool) │
|
|
│ checksum │ 32 bytes │
|
|
└──────────────────────────────────────┘
|
|
```
|
|
|
|
Segment types:
|
|
- `0x01` - Data segment (frames)
|
|
- `0x02` - Lex index segment
|
|
- `0x03` - Vec index segment
|
|
- `0x04` - Time index segment
|
|
|
|
## Time Index
|
|
|
|
The time index enables chronological queries and time-travel.
|
|
|
|
### Time Index Entry
|
|
|
|
| Field | Size | Description |
|
|
|-------|------|-------------|
|
|
| `frame_id` | 8 | Frame identifier |
|
|
| `timestamp` | 8 | Unix timestamp |
|
|
| `offset` | 8 | Byte offset in data segment |
|
|
|
|
Magic: `MVTI` (0x4D 0x56 0x54 0x49)
|
|
|
|
## Lex Index (Full-Text Search)
|
|
|
|
When the `lex` feature is enabled, the file contains a Tantivy index segment.
|
|
|
|
Indexed fields:
|
|
- `body` - Full text content
|
|
- `title` - Document title
|
|
- `uri` - Document URI
|
|
- `tags` - Flattened tag values
|
|
|
|
Supports:
|
|
- BM25 ranking
|
|
- Phrase queries
|
|
- Boolean operators
|
|
- Date range filters
|
|
|
|
## Vec Index (Vector Search)
|
|
|
|
When the `vec` feature is enabled, the file contains an HNSW index segment.
|
|
|
|
| Parameter | Value |
|
|
|-----------|-------|
|
|
| Dimensions | 384 (BGE-small) |
|
|
| Distance | Cosine similarity |
|
|
| M | 16 |
|
|
| ef_construction | 200 |
|
|
|
|
## Table of Contents (TOC)
|
|
|
|
The TOC is the final segment, pointed to by `footer_offset` in the header.
|
|
|
|
```
|
|
┌──────────────────────────────────────┐
|
|
│ magic │ "MVTC" │
|
|
│ version │ 2 bytes │
|
|
│ segment_count │ 4 bytes │
|
|
│ segments[] │ SegmentDescriptor[] │
|
|
│ manifests │ IndexManifests │
|
|
│ checksum │ 32 bytes │
|
|
└──────────────────────────────────────┘
|
|
```
|
|
|
|
### Segment Descriptor
|
|
|
|
| Field | Size | Description |
|
|
|-------|------|-------------|
|
|
| `segment_type` | 1 | Type identifier |
|
|
| `offset` | 8 | Byte offset in file |
|
|
| `length` | 8 | Segment size in bytes |
|
|
| `checksum` | 32 | SHA-256 of segment |
|
|
|
|
## URI Scheme
|
|
|
|
All content is addressable via `mv2://` URIs:
|
|
|
|
```
|
|
mv2://[track/][path/]name
|
|
```
|
|
|
|
Examples:
|
|
- `mv2://meetings/2024-01-15`
|
|
- `mv2://docs/api/reference.md`
|
|
- `mv2://media/photo.png`
|
|
|
|
## Invariants
|
|
|
|
1. **Single-file guarantee**: No `.wal`, `.shm`, `.lock`, or other sidecar files
|
|
2. **Append-only frames**: Existing frames are never modified in place
|
|
3. **Determinism**: Same API calls produce identical bytes
|
|
4. **Crash safety**: WAL ensures durability across unexpected termination
|
|
5. **Self-describing**: TOC contains all metadata needed to parse the file
|
|
|
|
## Version History
|
|
|
|
| Version | Changes |
|
|
|---------|---------|
|
|
| 2.1 | Current version. Embedded WAL, temporal track support |
|
|
| 2.0 | Single-file format, removed external indices |
|
|
| 1.x | Legacy format (deprecated) |
|