wehub-resource-sync
182 lines
5.0 KiB
Markdown
182 lines
5.0 KiB
Markdown
# LMCache Rust Raw Block I/O
|
|
|
|
This crate provides the low-level raw device I/O layer for LMCache via Rust +
|
|
PyO3. It is used by both:
|
|
|
|
- the legacy non-MP `RustRawBlockBackend`
|
|
- the MP `raw_block` L2 adapter (`RawBlockL2Adapter`) via `RawBlockCore`
|
|
|
|
The Rust crate intentionally stays narrow: it owns the raw device handle and
|
|
exposes blocking `pwrite_from_buffer` / `pread_into` primitives. Slotting,
|
|
checkpointing, recovery, and MP task orchestration all live in Python.
|
|
|
|
## I/O Engines
|
|
|
|
`RawBlockDevice` accepts `io_engine`:
|
|
|
|
- `posix` (default): synchronous Linux `pread` / `pwrite`.
|
|
- `io_uring`: direct Rust io_uring syscall path using the existing worker,
|
|
batch, and `wait_iouring` machinery.
|
|
|
|
`use_iouring=True` remains accepted for backward compatibility. If `io_engine`
|
|
is explicitly set, it wins over the legacy flag.
|
|
|
|
## io_uring_cmd (NVMe Passthrough)
|
|
|
|
When `io_engine="io_uring"`, you can optionally enable `use_uring_cmd=True` to
|
|
use NVMe passthrough via the io_uring command interface for direct device access.
|
|
|
|
**io_uring_cmd notes:**
|
|
|
|
- Requires NVMe character device node (`/dev/ngXnY`) instead of the block device
|
|
node (`/dev/nvmeXnY`) for direct NVMe passthrough command.
|
|
- Requires `io_engine="io_uring"` to be set.
|
|
- Supports `max_data_transfer_size` parameter to split large transfers into
|
|
smaller chunks that fit within device limits.
|
|
- When `use_uring_cmd=True`, `use_odirect` is ignored for NVMe namespace
|
|
character devices.
|
|
|
|
## MP Mode Integration
|
|
|
|
In MP mode, the stack looks like this:
|
|
|
|
```text
|
|
StoreController / PrefetchController
|
|
|
|
|
v
|
|
RawBlockL2Adapter
|
|
|
|
|
v
|
|
RawBlockCore
|
|
|
|
|
v
|
|
lmcache_rust_raw_block_io
|
|
|
|
|
v
|
|
raw device / file
|
|
```
|
|
|
|
This split lets LMCache reuse the same on-device metadata and recovery model in
|
|
both non-MP and MP mode without duplicating the raw-block implementation.
|
|
|
|
## Zero-Copy Data Path
|
|
|
|
```text
|
|
LMCache LocalCPUBackend (aligned pinned CPU tensor)
|
|
|
|
|
| Python buffer / memoryview (no payload memcpy)
|
|
v
|
|
RustRawBlockBackend (PyO3 boundary)
|
|
|
|
|
| direct pointer path when O_DIRECT constraints are met
|
|
| fallback: bounce only for unaligned tail/block
|
|
v
|
|
RawBlockDevice::pwrite_from_buffer / pread_into
|
|
|
|
|
v
|
|
Block device or file
|
|
```
|
|
|
|
## How To Compare Performance
|
|
|
|
To compare `local_disk` vs `rust_raw_block` on a real NVMe device:
|
|
- Run `local_disk` on an ext4 mount of the device.
|
|
- Unmount it.
|
|
- Run `rust_raw_block` directly on the raw block device.
|
|
|
|
Use the benchmark commands in:
|
|
- `benchmarks/storage_backend_io/README.md`
|
|
|
|
No fixed numbers are included here because results are host/device/workload dependent.
|
|
|
|
## Limitations
|
|
|
|
- Linux only (`pread` / `pwrite`, O_DIRECT semantics).
|
|
- O_DIRECT requires aligned offset, size, and user buffer address.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
cd rust/raw_block
|
|
pip install maturin
|
|
maturin develop --release
|
|
```
|
|
|
|
## Minimal Usage
|
|
|
|
```python
|
|
from lmcache_rust_raw_block_io import RawBlockDevice
|
|
|
|
dev = RawBlockDevice("/dev/nvme0n1", True, use_odirect=True, alignment=4096)
|
|
dev.pwrite_from_buffer(offset=0, data=b"hello", total_len=4096)
|
|
|
|
buf = bytearray(4096)
|
|
dev.pread_into(offset=0, out=buf, payload_len=5, total_len=4096)
|
|
```
|
|
|
|
io_uring:
|
|
|
|
```python
|
|
dev = RawBlockDevice(
|
|
"/dev/nvme0n1",
|
|
True,
|
|
use_odirect=True,
|
|
alignment=4096,
|
|
io_engine="io_uring",
|
|
iouring_queue_depth=256,
|
|
)
|
|
```
|
|
|
|
io_uring with io_uring_cmd (NVMe passthrough):
|
|
|
|
```python
|
|
dev = RawBlockDevice(
|
|
"/dev/ng0n1", # Note: NVMe character device node
|
|
True,
|
|
use_odirect=False,
|
|
alignment=4096,
|
|
io_engine="io_uring",
|
|
use_uring_cmd=True,
|
|
iouring_queue_depth=256,
|
|
max_data_transfer_size=131072, # Optional: split large transfers
|
|
)
|
|
```
|
|
|
|
## MP Adapter Example
|
|
|
|
To use the MP adapter from `lmcache server`, pass a `raw_block` L2 adapter
|
|
config:
|
|
|
|
```bash
|
|
lmcache server \
|
|
--l1-size-gb 10 \
|
|
--eviction-policy LRU \
|
|
--l1-align-bytes 4096 \
|
|
--l2-adapter '{
|
|
"type": "raw_block",
|
|
"device_path": "/dev/nvme0n1",
|
|
"slot_bytes": 1048576,
|
|
"block_align": 4096,
|
|
"header_bytes": 4096,
|
|
"meta_total_bytes": 268435456,
|
|
"use_odirect": true,
|
|
"io_engine": "io_uring",
|
|
"num_store_workers": 2,
|
|
"num_lookup_workers": 1,
|
|
"num_load_workers": 4
|
|
}'
|
|
```
|
|
|
|
Notes:
|
|
|
|
- `device_path` should point to an unmounted raw block device or a dedicated
|
|
file used only by LMCache.
|
|
- For `use_uring_cmd=true`, `device_path` must use the NVMe character
|
|
device node (e.g., `/dev/ng0n1`) instead of the block device node.
|
|
- With `use_odirect=true`, LMCache MP L1 alignment must be at least
|
|
`block_align`.
|
|
- Restart recovery uses the metadata checkpoint region on the same device.
|
|
- Raw-block slot reclamation is driven by the shared/global L2 eviction
|
|
controller or explicit `delete()` calls.
|
|
- `raw_block` remains the adapter type for all supported engines.
|