chore: import upstream snapshot with attribution

python/sglang/srt/mem_cache/storage/nixl/README.md

@@ -0,0 +1,614 @@
+# NIXL Integration for HiCache
+
+This directory contains the **NIXL (NVIDIA Inference Xfer Library)** integration for **HiCache**, enabling high-performance storage across multiple backends.
+
+NIXL provides a unified API for accessing various storage plugins, including but not limited to:
+
+- POSIX for file based operations, including AIO / io_uring / POSIX AIO.
+- **Deepseek's 3FS APIs** for high-throughput file operations
+- **GPU Direct Storage (GDS)** for direct data movement between storage and GPU memory, bypassing CPU memory copies
+- **Amazon S3-compatible object storage** for key-value access patterns
+
+NIXL also supports additional backends such as **AZURE_BLOB**, **GUSLI**, and **UCX**. Additional backend integrations are planned for future releases.
+
+## NIXL Resources
+
+- **Project Repository**: [NIXL on GitHub](https://github.com/ai-dynamo/nixl)
+- **Documentation**: [NIXL Documentation](https://github.com/ai-dynamo/nixl/tree/main/docs)
+
+## Overview
+
+The NIXL integration consists of these main files:
+
+- **`hicache_nixl.py`** - Main HiCache storage connector using NIXL
+- **`nixl_utils.py`** - Utility classes for backend selection, registration, and file management
+- **`nixl_cleaner.py`** - Background FILE-backend disk cleaner
+
+At runtime, HiCache uses NIXL as a transfer layer between host memory and either:
+
+- **FILE-backed storage plugins** such as 3FS / POSIX / GDS / GDS_MT
+- **OBJ-backed storage plugins** such as S3-compatible object stores
+
+The connector supports both the legacy tensor-oriented API (`get` / `set`) and the newer page-oriented API (`batch_get_v1` / `batch_set_v1`) used by modern HiCache backends.
+
+## Components
+
+### HiCacheNixl
+The main storage connector that provides:
+- Single and batch tensor set/get operations
+- Automatic backend selection (3FS > POSIX > GDS_MT > GDS > OBJ)
+- High-performance file-based (or) object based storage access using NIXL
+- Automatic zero-copy enablement when HiCache host memory layout is `page_first` or `page_first_direct`
+- MLA-aware storage naming and backend-local MLA backup skipping on non-zero TP ranks
+- Runtime diagnostics for mem-pool type, MLA mode, TP rank, and backup-skip state
+
+### NixlUtils
+Consolidated utility classes:
+- **NixlBackendSelection** - Handles backend selection and creation
+- **NixlBackendConfig** - Handles backend configuration
+- **NixlFileManager** - Handles file system operations
+
+### NixlRegistry (`nixl_registry.py`)
+Owns the `(agent, mem_type, file_manager)` triple and exposes `host(...)` and `storage(...)` context managers that register on entry, yield the NIXL `xfer_descs`, and deregister + close fds on exit. Internally composes two single-resource primitives (`_open_files` and `_registered`) so leak-freeness is verifiable per primitive.
+
+The current implementation performs per-transfer registration for file / object targets and explicitly closes FILE descriptors after registration / transfer setup to avoid descriptor leaks.
+
+### L3 Cleaner (`nixl_cleaner.py`)
+For FILE-backed plugins, TP rank 0 starts a best-effort background cleaner that scans the bucketed storage directories and deletes the oldest logical cache-key groups when disk usage exceeds the configured high watermark. Deleted files are handled by the cache layer as ordinary storage misses and can be recomputed.
+
+Set the top-level `l3_cleaner_enabled` config key to `false` when an external cleaner is responsible for L3 cache eviction.
+
+## Using NIXL as the HiCache Storage Backend
+
+### 1. How Backend Plugin Selection Works
+
+The NIXL backend can support **multiple storage plugins** (e.g., POSIX, GDS, GDS_MT, 3FS, object store, etc).
+
+* Each plugin has its own configuration section in the TOML file.
+* The connector accepts configuration in two forms:
+
+  * a **fully qualified** form such as `{"plugin": {"posix": {...}, "gds": {...}}}`
+  * a **flat** form such as `{"use_uring": "true"}`, which applies to the selected plugin
+* A plugin is considered **usable** if:
+
+  * Its required library is available on the system (POSIX, GDS, GDS_MT are natively supported by NIXL).
+  * Its configuration is valid.
+  * It is marked as `active = true` in the configuration file (if applicable).
+* Some plugins (e.g., 3FS, GDS) require additional system libraries or hardware support.
+* If the config explicitly enables multiple plugins, the connector chooses the **first active plugin** in the config.
+* If no plugin is explicitly selected in config, the connector falls back to the environment variable `SGLANG_HICACHE_NIXL_BACKEND_PLUGIN`, and finally to `auto`.
+* In `auto` mode, NIXL selects the backend based on **internal priority and availability**.
+
+If a plugin is configured but its dependencies are missing, it will be skipped.
+
+
+### 2. Setting the Storage Directory (Optional)
+
+For POSIX / GDS / GDS_MT file-based backends, the default storage location is `/tmp/hicache_storage`. However, you can customize where cached data is stored:
+
+```bash
+# When specifying multiple storage directories. SGLang routes each cache object to one
+# directory with a stable hash.
+export SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR=/path/to/storage/dir1,/path/to/storage/dir2,/path/to/storage/dir3
+```
+
+These directories are used only for **FILE-backed** plugins. **OBJ-backed** plugins use object keys instead of local files.
+
+### 3. How to Provide Configuration for Backends
+
+There are three ways to specify configurations for the backends: default config, file based config, and command-line (JSON string based) config.
+
+#### 1. Using Default Configuration
+
+To enable HiCache with the NIXL backend, start the SGLang server with:
+
+```bash
+python3 -m sglang.launch_server \
+  --model-path <model> \
+  --host <ip> \
+  --port <port> \
+  --page-size 64 \
+  --enable-hierarchical-cache \
+  --hicache-ratio 2 \
+  --hicache-size 64 \
+  --hicache-write-policy write_through \
+  --hicache-storage-backend nixl
+```
+
+By default, NIXL will use its internal backend selection logic to choose an available storage plugin (and use default configs for the selected storage plugin).
+
+For object storage backends, make sure the bucket is configured either in `--hicache-storage-backend-extra-config` or via:
+
+```bash
+export AWS_DEFAULT_BUCKET=<bucket-name>
+```
+
+
+#### 2. Using a Configuration File (Recommended)
+
+For non-trivial setups with complex configurations, it is recommended to use a **TOML configuration file** to define which backend plugin to use and its configurations, via `--hicache-storage-backend-extra-config`:
+
+Below is an example command (note: detailed configs are defined in the config file):
+
+```bash
+python3 -m sglang.launch_server \
+  --model-path <model> \
+  --host <ip> \
+  --port <port> \
+  --page-size 64 \
+  --enable-hierarchical-cache \
+  --hicache-ratio 2 \
+  --hicache-size 64 \
+  --hicache-write-policy write_through \
+  --hicache-storage-backend nixl \
+  --hicache-storage-backend-extra-config "@config.nixl.toml"
+```
+
+> **Important**
+>
+> * The `@` prefix tells SGLang to load the configuration from a file.
+> * The file can be in **TOML format** (other formats, JSON / YAML, are also supported).
+> * This is the preferred way to configure NIXL storage backends.
+
+The structure of the config file is described in further details in [Configuration File Spec](#Configuration-File-Specification).
+
+
+#### 3. Using Command-line JSON String
+
+For debugging or quick testing, you may pass a **JSON-style string** directly via `--hicache-storage-backend-extra-config`.
+
+This requires explicitly specifying the plugin type via an environment variable, and this method can be applicable to **only a few** plugins (e.g., POSIX, GDS, GDS_MT)
+
+The below example shows how to use command-line string to use the POSIX plugin where URING is enabled for async POSIX storage, with O_DIRECT enabled (the default).
+
+```bash
+export SGLANG_HICACHE_NIXL_BACKEND_PLUGIN=POSIX
+
+python3 -m sglang.launch_server \
+  --model-path <model> \
+  --host <ip> \
+  --port <port> \
+  --page-size 64 \
+  --enable-hierarchical-cache \
+  --hicache-ratio 2 \
+  --hicache-size 64 \
+  --hicache-write-policy write_through \
+  --hicache-storage-backend nixl \
+  --hicache-storage-backend-extra-config '{"use_uring": "true"}'
+```
+
+To disable O_DIRECT (e.g. for debugging or unsupported filesystems), set the top-level `use_direct_io` key:
+
+```bash
+export SGLANG_HICACHE_NIXL_BACKEND_PLUGIN=POSIX
+
+python3 -m sglang.launch_server \
+  ... \
+  --hicache-storage-backend-extra-config '{"use_direct_io": false, "use_uring": "true"}'
+```
+
+⚠️ **Note**:
+This method is convenient for testing / experimenting. For production or multi-plugin setups, it is always recommended to use the config file based approach.
+
+Also note that the flat inline config form is interpreted as plugin-specific parameters for the selected plugin.
+
+
+## Running Unit Tests
+
+### Prerequisites
+- NIXL library installed and available (latest main required for supporting object query)
+- PyTorch installed
+- Python 3.8+
+
+### Unit tests from current directory
+From the current directory run:
+
+#### Run all NIXL tests:
+```bash
+PYTHONPATH=. python -m pytest test_hicache_nixl_storage.py -o asyncio_mode=strict
+```
+
+#### Run with verbose output:
+```bash
+PYTHONPATH=. python -m pytest test_hicache_nixl_storage.py -v -o asyncio_mode=strict
+```
+
+Note: The `-v` flag provides more detailed output, showing each test case name and its result.
+
+#### Run a specific test:
+```bash
+PYTHONPATH=. python -m pytest test_hicache_nixl_storage.py -v -k test_single_set_get -o asyncio_mode=strict
+```
+
+Note: The `-o asyncio_mode=strict` flag is added to suppress warnings about asyncio configuration. This is not required for test functionality but provides cleaner output.
+
+## Test Coverage
+
+Tests for this integration, a test suite can be found at `test_hicache_nixl_storage.py` which covers:
+
+### HiCache Integration Tests
+- Single tensor set/get operations
+- Batch tensor set/get operations
+- Mixed single and batch operations
+- Data integrity for various tensor types
+
+### File Management Tests
+- Basic file operations
+- NIXL tuple creation
+- Error handling in file operations
+
+### Registration and MLA / Query Tests
+- Tensor registration with memory type detection
+- File registration using file paths
+- MLA backup-skip behavior for `batch_set_v1`
+- Zero-copy `batch_exists()` accounting for MLA and MHA
+
+## Expected Output
+
+When tests run successfully, you should see:
+- NIXL agent initialization messages
+- Backend selection messages (e.g., "Backend POSIX was instantiated")
+- Test results with "ok" for passed tests
+- Summary showing "Ran X tests in Y seconds" and "OK"
+
+## Troubleshooting
+
+### Import Errors
+If you encounter `ModuleNotFoundError`, ensure:
+- You're running from the correct directory
+- `PYTHONPATH` is set correctly
+- NIXL library is properly installed
+
+### NIXL Errors
+If NIXL operations fail:
+- Check that NIXL is properly installed
+- Verify that required plugins are available
+- Ensure file permissions are correct for test directories
+- For OBJ plugins, verify `bucket` or `AWS_DEFAULT_BUCKET` is set
+- Check the NIXL diagnostic log emitted when the mem pool is registered; it includes:
+  - `mem_pool_device_type`
+  - `is_mla_model`
+  - `tp_rank`
+  - `backup_skip`
+
+### MLA Write Behavior
+For MLA models, the NIXL backend now mirrors HF3FS's backend-local protection:
+- TP rank 0 performs the actual storage write
+- non-zero TP ranks skip backup writes locally in `batch_set` / `batch_set_v1`
+- MLA storage names omit TP rank so all ranks refer to the same logical storage object or file
+
+## File Structure
+
+```text
+python/sglang/srt/mem_cache/storage/nixl/
+├── hicache_nixl.py              # Main HiCache storage connector
+├── nixl_cleaner.py              # Background FILE-backend disk cleaner
+├── nixl_utils.py                # NIXL utility classes
+├── test_hicache_nixl_storage.py # Unit tests
+├── nixl.config.toml.sample      # Example configuration
+└── README.md                    # This file
+```
+
+## Dependencies
+
+- **NIXL**: NVIDIA Inference Xfer Library (version 0.4 or later)
+  - Required plugins: POSIX (minimum), 3FS/GDS (optional for better performance)
+  - See [NIXL Installation Guide](https://github.com/ai-dynamo/nixl/blob/main/README.md#installation)
+- **PyTorch**: For tensor operations (version 1.8 or later)
+- **Python 3.8+**: For type hints and modern features
+
+## Supported Features
+
+### Memory Types
+- **Tensor side**: multi-dimensional tensors of all numeric types (int32, int64, float32, float64) are supported.
+  - Tensors can be on CPU or GPU (as long as a GPU capable backend such as GDS_MT is available).
+  - Currently each tensor is mapped to a file or key, but it can be extended to support multiple keys per file or key.
+  - The page-oriented `*_v1` path also supports zero-copy transfers using `(address, size)` metadata from the host memory pool.
+
+- **Storage side**: file and object are supported through their relevant backends (e.g., 3FS or OBJ).
+
+### HiCache / NIXL Data Model
+
+- **FILE backends** use local file paths under `SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR`. When multiple comma-separated directories are configured, each logical cache key is routed to one base directory with a stable hash and stored as `base_dir/<bucket>/<key>`.
+- **OBJ backends** use object keys directly
+- **MHA naming** includes TP rank and TP size, so each rank stores its own KV data
+- **MLA naming** omits TP rank, so all ranks refer to one shared logical KV object / file
+- In zero-copy mode:
+  - **MHA** expands each logical page into `_k` and `_v` entries
+  - **MLA** expands each logical page into a single `_k` entry because MLA stores one interleaved KV representation
+- The L3 cleaner groups physical files by the logical base key after removing TP-rank and zero-copy `_k` / `_v` suffixes. This keeps MHA, MLA, and DSA file cleanup aligned with the names emitted by `HiCacheNixl`.
+
+### Zero-Copy Behavior
+
+- Zero-copy is enabled automatically when the HiCache host layout is `page_first` or `page_first_direct`
+- The connector uses `mem_pool_host.get_page_buffer_meta(...)` to obtain `(address, size)` metadata
+- `batch_exists()` uses the same logical key expansion rules as `batch_get_v1()` / `batch_set_v1()`
+- Non-zero MLA TP ranks skip `batch_set` / `batch_set_v1()` locally as a backend-side fallback guard
+
+### Backend Priority
+
+The NIXL backend selection follows this priority order:
+1. **3FS** - Highest performance (if available)
+    - Best for high-throughput file operations using Deepseek 3FS APIs
+2. **POSIX** - Standard file I/O (fallback)
+    - Universal compatibility
+    - Good for development and testing - Leverages both libaio/liburing
+3. **GDS_MT** - Multi-threaded GDS (if available)
+    - Optimized for concurrent operations
+    - Supports GPU Direct storage with multiple light weight threads
+4. **GDS** - GPU Direct Storage (if available)
+    - Direct GPU-storage data path
+    - Best for filesystems benefiting from batch operations and smaller IOs.
+5. **OBJ** - Amazon S3 based Object Storage
+    - Key-value based storage
+The system automatically selects the best available backend, with POSIX as the default fallback.
+
+
+
+## Configuration File Specification
+
+This section defines the structure, supported sections, configuration keys, data types, defaults, and semantics for the NIXL HiCache backend configuration file (`config.nixl.toml`).
+
+The configuration file is written in **TOML** and consists of multiple **plugin-specific sections** under the `plugin.*` namespace. Each section configures one storage backend plugin. Only one plugin should be enabled via setting `active = true` in the corresponding plugin-specific section.
+
+An example of the configuration is provided in [`nixl.config.toml.sample`](./nixl.config.toml.sample).
+
+### 1. General Structure
+
+```toml
+[plugin.<backend_name>]
+<key> = <value>
+```
+
+* `<backend_name>` identifies the storage backend plugin.
+* Each plugin is configured independently.
+* Plugins are selected at runtime based on:
+
+  * Availability of required libraries/hardware
+  * Plugin configuration validity
+  * Internal backend priority rules
+* Unless otherwise stated, all configuration keys are **optional** and have sensible defaults.
+
+For object storage, `bucket` may also be omitted from the config if `AWS_DEFAULT_BUCKET` is already defined in the environment.
+
+### 1a. Top-Level Configuration Keys
+
+The following keys are placed at the **top level** of the config file (not inside any `[plugin.*]` section) and apply globally to the NIXL backend:
+
+| Key              | Type    | Default  | Description |
+| ---------------- | ------- | -------- | ----------- |
+| `use_direct_io`  | boolean | `true`   | Open cache files with `O_DIRECT` to bypass the OS page cache. Reduces memory pressure and improves NVMe throughput. Falls back to buffered I/O with a warning if `O_DIRECT` is unavailable on the current OS. Can also be overridden via the `SGLANG_HICACHE_NIXL_USE_DIRECT_IO` environment variable. |
+| `l3_cleaner_enabled` | boolean | `true` | Enable the built-in background cleaner for FILE-backed L3 storage. Set to `false` when using an external cleaner. |
+| `l3_cleaner_high_watermark` | float | `80.0` | Start cleanup when the built-in cleaner is enabled and the filesystem containing a configured storage directory reaches this disk-usage percentage. |
+| `l3_cleaner_low_watermark` | float | `70.0` | Stop cleanup after hot filesystems drop below this disk-usage percentage. Must be lower than `l3_cleaner_high_watermark`. |
+
+**Page-alignment and `O_DIRECT`**
+
+When `use_direct_io = true` with any file-based backend (POSIX, GDS, GDS_MT, 3FS), the kernel requires every I/O buffer pointer to be OS-page-aligned (4 KiB). SGLang handles this automatically:
+
+* **Zero-copy mode** (`page_first` / `page_first_direct` layout): the host memory pool is always mmap-backed and therefore page-aligned. If the per-page stride is also a multiple of 4 KiB, zero-copy transfers are used as-is.
+* **Copy mode** (all other layouts, or if stride alignment cannot be satisfied): SGLang pre-allocates page-aligned bounce buffers via `mmap` and falls back to copy mode, logging a warning. No user action is required -- this is fully automatic.
+
+To disable `O_DIRECT` (e.g. for debugging or when the filesystem does not support it):
+
+```toml
+use_direct_io = false
+
+[plugin.posix]
+use_uring = "true"
+active = true
+```
+
+or via environment variable: `SGLANG_HICACHE_NIXL_USE_DIRECT_IO=0`.
+
+To tune FILE-backend cleanup watermarks:
+
+```toml
+l3_cleaner_enabled = true
+l3_cleaner_high_watermark = 85.0
+l3_cleaner_low_watermark = 75.0
+
+[plugin.posix]
+use_uring = "true"
+active = true
+```
+
+To use an external cleaner instead of the built-in cleaner:
+
+```toml
+l3_cleaner_enabled = false
+
+[plugin.posix]
+use_uring = "true"
+active = true
+```
+
+
+### 2. POSIX File System Backend (`plugin.posix`)
+
+#### Section
+
+```toml
+[plugin.posix]
+```
+
+#### Description
+
+Configures the POSIX file-system-based backend.
+This backend supports multiple asynchronous I/O mechanisms and automatically selects the most performant option supported by the system.
+
+**Backend priority (highest to lowest):**
+
+1. Linux AIO
+2. `io_uring`
+3. POSIX AIO
+
+
+#### Configuration Keys
+
+| Key             | Type    | Default   | Description                                                                                              |
+| --------------- | ------- | --------- | -------------------------------------------------------------------------------------------------------- |
+| `use_uring`     | string  | `"false"` | Enables Linux `io_uring` for asynchronous I/O when set to `"true"`. Recommended on modern Linux kernels. |
+| `use_posix_aio` | string  | `"false"` | Enables POSIX AIO as an alternative async I/O mechanism.                                                 |
+| `use_aio`       | string  | `"false"` | Enables generic Linux AIO.                                                                               |
+| `active`        | boolean |    N/A    | Controls whether this plugin is eligible for backend selection.                                          |
+
+**Notes**
+
+* Boolean-like options use **string values** (`"true"` / `"false"`) for compatibility.
+* **Only one backend** (i.e., only one of `use_uring`, `use_aio`, `use_posix_aio`) should be included in the config.
+
+
+### 3. NVIDIA GPUDirect Storage Backend (`plugin.gds`)
+
+#### Section
+
+```toml
+[plugin.gds]
+```
+
+#### Description
+
+Configures NVIDIA GPUDirect Storage (GDS) backend.
+This backend enables direct data transfers between storage and GPU memory.
+
+**Requirements**
+
+* NVIDIA GPU with GDS support
+* Compatible NVIDIA driver and CUDA runtime
+* Supported filesystem
+
+
+#### Configuration Keys
+
+| Key                | Type    | Default            | Description                                            |
+| ------------------ | ------- | ------------------ | ------------------------------------------------------ |
+| `batch_pool_size`  | integer | `128`              | Number of I/O requests maintained in the request pool. |
+| `batch_limit`      | integer | `128`              | Maximum number of requests issued in a single batch.   |
+| `max_request_size` | integer | `16777216` (16 MB) | Maximum size (in bytes) of a single I/O request.       |
+| `active`           | boolean |         N/A        | Controls whether this plugin is eligible for backend selection.|
+
+
+### 4. Multi-Threaded GDS Backend (`plugin.gds_mt`)
+
+#### Section
+
+```toml
+[plugin.gds_mt]
+```
+
+#### Description
+
+Configures the multi-threaded variant of the NVIDIA GDS backend, allowing parallel request processing using multiple CPU threads.
+
+
+
+#### Configuration Keys
+
+| Key            | Type    | Default | Description                                           |
+| -------------- | ------- | ------- | ----------------------------------------------------- |
+| `thread_count` | integer | `4`     | Number of worker threads used to submit GDS requests. |
+| `active`       | boolean |    N/A  | Controls whether this plugin is eligible for backend selection. |
+
+
+### 5. 3FS Backend (`plugin.3fs`)
+
+#### Section
+
+```toml
+[plugin.3fs]
+```
+
+#### Description
+
+Configures the 3FS (third-party filesystem) backend.
+
+**Requirements**
+
+* 3FS client library installed
+* Filesystem mounted and accessible on the host
+
+
+#### Configuration Keys
+
+| Key           | Type    | Default  | Description                        |
+| ------------- | ------- | -------- | ---------------------------------- |
+| `mount_point` | string  | *none*   | Mount point of the 3FS filesystem. |
+| `mem_config`  | string  | `"dram"` | Memory configuration mode.         |
+| `iopool_size` | integer | `64`     | Size of the I/O pool.              |
+| `active`      | boolean |   N/A    | Controls whether this plugin is eligible for backend selection.                                          |
+
+##### `mem_config` Valid Values
+
+| Value     | Description                                         |
+| --------- | --------------------------------------------------- |
+| `dram`    | Use DRAM for buffering                              |
+| `dram_zc` | Use DRAM with zero-copy support                     |
+| `auto`    | Automatically select based on platform capabilities |
+
+##### `iopool_size` Constraints
+
+* Valid range: **[2⁶, 2²⁰]**
+* Values outside this range may cause initialization failure.
+
+
+### 6. Object Storage Backend (`plugin.obj`)
+
+#### Section
+
+```toml
+[plugin.obj]
+```
+
+#### Description
+
+Configures an object storage backend compatible with S3 APIs (e.g., AWS S3, MinIO, Ceph).
+
+
+#### Configuration Keys
+
+| Key                      | Type    | Default      | Description                                    |
+| ------------------------ | ------- | ------------ | ---------------------------------------------- |
+| `num_threads`            | integer | `4`          | Number of client worker threads.               |
+| `endpoint_override`      | string  | `""`         | Custom endpoint URL (for non-AWS S3 services). |
+| `scheme`                 | string  | `"http"`     | Connection scheme (`http` or `https`).         |
+| `region`                 | string  | `""`         | Cloud region (if applicable).                  |
+| `req_checksum`           | string  | `"required"` | Request checksum behavior.                     |
+| `ca_bundle`              | string  | `""`         | Path to a custom CA bundle.                    |
+| `access_key`             | string  | `""`         | Access key credential.                         |
+| `secrete_key`            | string  | `""`         | Secret key credential.                         |
+| `session_token`          | string  | `""`         | Session token (optional).                      |
+| `use_virtual_addressing` | string  | `"true"`     | Enables virtual-hosted-style addressing.       |
+| `bucket`                 | string  | `""`         | Default bucket name.                           |
+| `active`                 | boolean |      N/A     | Controls whether this plugin is eligible for backend selection.                                          |
+
+##### `req_checksum` Valid Values
+
+| Value       | Description                                    |
+| ----------- | ---------------------------------------------- |
+| `required`  | Always include a checksum                      |
+| `supported` | Include checksum when supported by the backend |
+
+
+### 7. Notes and Best Practices
+
+* All plugin sections are optional.
+* Multiple plugins may be configured in a single file. However, it is recommended that **only one plugin** is configured `active = true`.
+* Plugins whose dependencies are unavailable will be skipped.
+* Use a TOML configuration file instead of inline JSON for:
+
+  * Multi-plugin setups
+  * Production deployments
+  * Clear validation and maintainability
+
+
+## Note
+
+This is v0 of the NIXL connector. The current implementation favors correctness and compatibility with the existing HiCache API:
+
+- file / object targets are registered per transfer
+- FILE descriptors are explicitly cleaned up after registration / transfer setup
+- MLA uses shared storage naming and backend-local write skipping on non-zero TP ranks
+- zero-copy is driven by HiCache host-memory layout rather than a separate NIXL flag
+
+Future versions will focus on further performance optimizations such as memory pre-registration (pre-allocating and registering memory buffers to reduce registration overhead during transfers) and block merging (combining related blocks as offsets within the same file to reduce file operations and improve throughput). These optimizations require changes at a higher layer, as the current HiCache API doesn't expose information like block relationships or hash patterns that would enable these optimizations.

python/sglang/srt/mem_cache/storage/nixl/hicache_nixl.py

@@ -0,0 +1,622 @@
+import logging
+import os
+import time
+import uuid
+from typing import Any, List, Optional
+
+import torch
+
+from sglang.srt.environ import envs
+from sglang.srt.mem_cache.hicache_storage import (
+    STORAGE_BATCH_SIZE,
+    HiCacheStorage,
+    HiCacheStorageConfig,
+    HiCacheStorageExtraInfo,
+)
+from sglang.srt.mem_cache.mmap_allocator import alloc_mmap
+from sglang.srt.mem_cache.pool_host import HostKVCache
+from sglang.srt.mem_cache.storage.nixl.nixl_cleaner import HiCacheL3Cleaner
+
+from .nixl_registry import NixlRegistry
+from .nixl_utils import NixlBackendConfig, NixlBackendSelection, NixlFileManager
+
+try:
+    from nixl._api import nixl_agent, nixl_agent_config, nixlBind
+except ImportError as e:
+    raise ImportError(
+        "Please install NIXL by following the instructions at "
+        "https://github.com/ai-dynamo/nixl/blob/main/README.md "
+        "to use HiCacheNixl storage backend."
+    ) from e
+
+logger = logging.getLogger(__name__)
+
+
+def _parse_storage_dirs(raw: Optional[str]) -> List[str]:
+    """Split NIXL FILE storage directory config into ordered unique paths."""
+    if not raw:
+        return []
+    candidates = [path.strip() for path in raw.split(",")]
+    candidates = [path for path in candidates if path]
+    seen: dict[str, str] = {}
+    ordered: List[str] = []
+    for path in candidates:
+        real_path = os.path.realpath(path)
+        if real_path in seen:
+            raise ValueError(
+                "SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR contains duplicate "
+                f"path {path!r} (same mount as {seen[real_path]!r})."
+            )
+        seen[real_path] = path
+        ordered.append(path)
+    return ordered
+
+
+class HiCacheNixl(HiCacheStorage):
+    """HiCacheNixl provides high-performance storage using NIXL plugins."""
+
+    def __init__(
+        self,
+        storage_config: HiCacheStorageConfig,
+        file_path: str = "/tmp/hicache_storage",
+    ):
+        """Initialize NIXL storage connector."""
+
+        # create nixlconfig from the --hicache-storage-backend-extra-config
+        nixlconfig = NixlBackendConfig(storage_config.extra_config)
+
+        # select the NIXL backend plugin from extra_config or environment variable
+        plugin = nixlconfig.get_specified_plugin()
+
+        use_direct_io = nixlconfig.get_use_direct_io()
+
+        # Might be better to be unified across HiCache backends and moved to HiCacheController
+        storage_dirs = _parse_storage_dirs(
+            envs.SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR.get() or file_path
+        )
+        self.file_manager = (
+            NixlFileManager(storage_dirs, use_direct_io=use_direct_io)
+            if plugin not in NixlBackendSelection.OBJ_PLUGINS
+            else None
+        )
+
+        tp_rank, tp_size, model_name = (
+            storage_config.tp_rank,
+            storage_config.tp_size,
+            storage_config.model_name,
+        )
+
+        self.is_mla_model = storage_config.is_mla_model
+        self.is_zero_copy = False
+        self.storage_config = storage_config
+        self.backup_skip = self.is_mla_model and storage_config.tp_rank != 0
+
+        model_name = "-".join(model_name.split("/")) if model_name else ""
+
+        if self.is_mla_model:
+            self.config_suffix = f"_{model_name}"
+        else:
+            self.config_suffix = f"_{model_name}_{tp_rank}_{tp_size}"
+
+        sync_mode = getattr(
+            nixlBind, "NIXL_THREAD_SYNC_RW", nixlBind.NIXL_THREAD_SYNC_STRICT
+        )
+        agent_config = nixl_agent_config(backends=[])
+        self.agent_name = f"hicache_nixl_{str(uuid.uuid4())}"
+        self.agent = nixl_agent(self.agent_name, agent_config)
+        bind_cfg = nixlBind.nixlAgentConfig()
+        bind_cfg.useProgThread = agent_config.enable_pthread
+        bind_cfg.useListenThread = agent_config.enable_listen
+        bind_cfg.listenPort = agent_config.port
+        bind_cfg.syncMode = sync_mode
+        bind_cfg.pthrDelay = 0
+        bind_cfg.lthrDelay = 100000
+        bind_cfg.captureTelemetry = agent_config.capture_telemetry
+        self.agent.agent = nixlBind.nixlAgent(self.agent_name, bind_cfg)
+        self.agent.plugin_list = self.agent.agent.getAvailPlugins()
+
+        self.backend_selector = NixlBackendSelection(plugin, nixlconfig)
+        if not self.backend_selector.create_backend(self.agent):
+            raise RuntimeError("Failed to create NIXL backend")
+
+        self.registry = NixlRegistry(
+            self.agent,
+            self.backend_selector.mem_type,
+            self.file_manager,
+        )
+        # O_DIRECT requires OS-page-aligned I/O buffers on all file-based backends
+        # (POSIX, GDS, GDS_MT, 3FS). OBJ backends never open files so they are exempt
+        # (file_manager is None for OBJ).
+        self.needs_page_alignment = use_direct_io and self.file_manager is not None
+        if self.needs_page_alignment:
+            logger.info(
+                "HiCacheNixl: O_DIRECT is active with a file-based backend (%s). "
+                "Page-aligned host buffers are required (needs_page_alignment=True).",
+                self.backend_selector.backend_name,
+            )
+        # Pre-registered host regions (set by register_mem_pool_host):
+        # zero-copy: one registration covering mem_pool_host.kv_buffer
+        # non-zero-copy: two registrations, one bounce buffer per direction
+        # (set/get) so the two storage threads never share slots.
+        self._host_regs: List[Any] = []
+        self._bounce_set: Optional[torch.Tensor] = None
+        self._bounce_get: Optional[torch.Tensor] = None
+        self._bounce_page_bytes: Optional[int] = None
+        cleanup_dirs = (
+            self.file_manager.iter_all_base_dirs()
+            if self.file_manager is not None
+            else []
+        )
+        cleaner_config = nixlconfig.get_l3_cleaner_config()
+        self._l3_cleaner: Optional[HiCacheL3Cleaner] = (
+            HiCacheL3Cleaner(
+                cleanup_dirs,
+                tp_rank,
+                high_watermark=cleaner_config["high_watermark"],
+                low_watermark=cleaner_config["low_watermark"],
+            )
+            if (
+                cleanup_dirs
+                and self.file_manager is not None
+                and cleaner_config["enabled"]
+            )
+            else None
+        )
+        if self._l3_cleaner is not None:
+            self._l3_cleaner.start()
+
+    def _get_suffixed_key(self, key: str) -> str:
+        return key + self.config_suffix
+
+    def _create_query_tuple(self, key: str) -> tuple:
+        """Build the NIXL query_memory tuple for a single key."""
+        if self.backend_selector.mem_type == "FILE":
+            return (0, 0, 0, self.file_manager.get_file_path(key))
+        return (0, 0, 0, key)
+
+    def _xfer_and_wait(
+        self,
+        host_descs: Any,
+        storage_descs: Any,
+        direction: str,
+    ) -> bool:
+        """Initialize and poll a NIXL transfer to completion."""
+        try:
+            xfer_req = self.agent.initialize_xfer(
+                direction, host_descs, storage_descs, self.agent_name
+            )
+        except Exception as e:
+            logger.error(f"Failed to create transfer request: {e}")
+            return False
+
+        try:
+            state = self.agent.transfer(xfer_req)
+            while state != "DONE":
+                state = self.agent.check_xfer_state(xfer_req)
+                if state == "ERR":
+                    logger.error("Transfer failed")
+                    return False
+                # Best would be to have a better notification mechanism from NIXL,
+                # but we only have polling for now.
+                time.sleep(0.0001)
+            return True
+        except Exception as e:
+            logger.error(f"Failed to execute transfer: {e}")
+            import traceback
+
+            logger.error(f"Traceback: {traceback.format_exc()}")
+            return False
+        finally:
+            self.agent.release_xfer_handle(xfer_req)
+
+    def _xfer_pre_registered(
+        self,
+        host_buffers: List[tuple],
+        keys: List[str],
+        direction: str,
+    ) -> bool:
+        """Run a transfer where the host side is already pre-registered.
+
+        ``host_buffers`` is a list of ``(addr, size)`` tuples within the
+        pre-registered host region (kv_buffer for zero-copy, bounce buffer
+        otherwise). Only the storage side is registered per transfer.
+        """
+        if len(host_buffers) != len(keys):
+            logger.error("Mismatch between number of host buffers and keys")
+            return False
+
+        host_descs = self.agent.get_xfer_descs(
+            [(addr, size, 0) for (addr, size) in host_buffers], "DRAM"
+        )
+        if host_descs is None:
+            logger.error("Failed to build host xfer descs")
+            return False
+
+        with self.registry.storage(host_buffers, keys, direction) as storage_descs:
+            if storage_descs is None:
+                return False
+            return self._xfer_and_wait(host_descs, storage_descs, direction)
+
+    def get(
+        self,
+        key: str,
+        target_location: Optional[Any] = None,
+        target_sizes: Optional[Any] = None,
+    ) -> torch.Tensor | None:
+        raise NotImplementedError("deprecated; use batch_get_v1")
+
+    def batch_get(
+        self,
+        keys: List[str],
+        target_locations: Optional[Any] = None,
+        target_sizes: Optional[Any] = None,
+    ) -> List[torch.Tensor | None]:
+        raise NotImplementedError("deprecated; use batch_get_v1")
+
+    def set(
+        self,
+        key: str,
+        value: Optional[Any] = None,
+        target_location: Optional[Any] = None,
+        target_sizes: Optional[Any] = None,
+    ) -> bool:
+        raise NotImplementedError("deprecated; use batch_set_v1")
+
+    def batch_set(
+        self,
+        keys: List[str],
+        values: Optional[Any] = None,
+        target_locations: Optional[Any] = None,
+        target_sizes: Optional[Any] = None,
+    ) -> bool:
+        raise NotImplementedError("deprecated; use batch_set_v1")
+
+    def register_mem_pool_host(self, mem_pool_host: HostKVCache):
+        super().register_mem_pool_host(mem_pool_host)
+
+        # enable zero-copy automatically if mem layout is page_first or page_first_direct
+        self.is_zero_copy = self.mem_pool_host.layout in [
+            "page_first",
+            "page_first_direct",
+        ]
+
+        if self.needs_page_alignment and self.is_zero_copy:
+            # Check that the kv_buffer base AND per-page strides are multiples of
+            # the OS page size so every pointer passed to NIXL (base + p * stride)
+            # is page-aligned. The base is whatever torch.empty() happened to give
+            # us -- it is not guaranteed to be page-aligned. Fall back to copy mode
+            # if either condition fails.
+            # 4096: O_DIRECT alignment is FS-dependent (some allow 512 B); 4 KiB
+            # is the safe lower bound all known FSes accept, and real page-sizes meet it.
+            if not self.mem_pool_host.is_stride_page_aligned(4096):
+                logger.warning(
+                    "HiCacheNixl: O_DIRECT is active but the host kv_buffer is "
+                    "not OS-page-aligned (base or per-page stride). Falling back "
+                    "to copy mode for this pool."
+                )
+                self.is_zero_copy = False
+
+        if self.is_zero_copy:
+            kv = mem_pool_host.kv_buffer
+            self._pre_register_host(
+                kv.data_ptr(), kv.numel() * kv.element_size(), "kv_buffer"
+            )
+        else:
+            # One bounce buffer per direction so set/get run lock-free across
+            # the prefetch and backup threads. Sized from get_dummy_flat_data_page()
+            # so each slot matches what the v1 path would otherwise allocate.
+            sample = mem_pool_host.get_dummy_flat_data_page()
+            page_numel = sample.numel()
+            self._bounce_page_bytes = page_numel * sample.element_size()
+            del sample
+            pin_memory = bool(getattr(mem_pool_host, "pin_memory", False))
+            self._bounce_set = self._alloc_registered(
+                page_numel, mem_pool_host.dtype, pin_memory, "bounce_set"
+            )
+            self._bounce_get = self._alloc_registered(
+                page_numel, mem_pool_host.dtype, pin_memory, "bounce_get"
+            )
+
+        logger.info(
+            f"HiCacheNixl: pre-registered host regions for "
+            f"layout={mem_pool_host.layout} zero_copy={self.is_zero_copy}"
+        )
+
+    def _alloc_registered(
+        self,
+        page_numel: int,
+        dtype: torch.dtype,
+        pin_memory: bool,
+        kind: str,
+    ) -> torch.Tensor:
+        """Allocate a ``(STORAGE_BATCH_SIZE, page_numel)`` bounce buffer and
+        pre-register it as a DRAM region with NIXL. Uses alloc_mmap so the
+        buffer is page-aligned -- required when O_DIRECT is on for any
+        file-based backend (POSIX/GDS/GDS_MT/3FS). pin_memory is currently
+        unused (alloc_mmap does not support it)."""
+        buf = alloc_mmap((STORAGE_BATCH_SIZE, page_numel), dtype)
+        self._pre_register_host(buf.data_ptr(), buf.numel() * buf.element_size(), kind)
+        return buf
+
+    def _pre_register_host(self, base_addr: int, total_size: int, kind: str) -> None:
+        """Register a single DRAM region up-front and remember the handle."""
+        reg_descs = self.agent.get_reg_descs([(base_addr, total_size, 0, "")], "DRAM")
+        if reg_descs is None:
+            raise RuntimeError(f"Failed to build reg descs for host {kind}")
+        try:
+            self._host_regs.append(self.agent.register_memory(reg_descs))
+        except Exception as e:
+            raise RuntimeError(f"Failed to pre-register host {kind} with NIXL") from e
+
+    def clear(self) -> None:
+        if self.file_manager is None:
+            return
+        self.file_manager.clear()
+
+    def close(self):
+        if self._l3_cleaner is not None:
+            self._l3_cleaner.stop()
+            self._l3_cleaner = None
+        while self._host_regs:
+            reg = self._host_regs.pop()
+            try:
+                self.agent.deregister_memory(reg)
+            except Exception as e:
+                logger.debug("deregister of pre-registered host region failed: %s", e)
+        self._bounce_set = None
+        self._bounce_get = None
+        self._bounce_page_bytes = None
+
+    def __del__(self):
+        try:
+            self.close()
+        except Exception:
+            pass
+
+    def exists(self, key: str) -> bool:
+        results = self.batch_exists([key])
+        return results > 0
+
+    def batch_exists(
+        self,
+        keys: List[str],
+        extra_info: Optional[HiCacheStorageExtraInfo] = None,
+    ) -> int:
+        if self.is_zero_copy:
+            key_list = self._get_key_list_from_meta(keys)
+            key_denominator = (
+                1 if self.is_mla_model else 2
+            )  # MLA: 1 key per page (_k only), non-MLA: 2 NIXL keys per page (_k + _v)
+        else:
+            key_list = [self._get_suffixed_key(key) for key in keys]
+            key_denominator = 1
+
+        tuples = [self._create_query_tuple(key) for key in key_list]
+
+        query_res = self.agent.query_memory(
+            tuples,
+            self.backend_selector.backend_name,
+            mem_type=self.backend_selector.mem_type,
+        )
+
+        for i in range(len(query_res)):
+            if query_res[i] is None:
+                return i // key_denominator
+        return len(query_res) // key_denominator
+
+    def _get_key_list_from_meta(self, keys: List[str]) -> List[str]:
+        # Each key maps to a `_k` entry, plus a `_v` entry on non-MLA models
+        # (MLA stores k/v interleaved in a single buffer).
+        key_list = []
+        for key in keys:
+            suffixed_key = self._get_suffixed_key(key)
+            key_list.append(f"{suffixed_key}_k")
+            if not self.is_mla_model:
+                key_list.append(f"{suffixed_key}_v")
+        return key_list
+
+    def _get_location_and_size_list_from_meta(
+        self, keys: List[str], host_indices: torch.Tensor
+    ):
+        # zero copy: mem_pool_host.get_data_page() does not work due to non-contiguous tensors, causing issues for NIXL transfer
+        ptr_list, element_size_list = self.mem_pool_host.get_page_buffer_meta(
+            host_indices
+        )
+        key_list = self._get_key_list_from_meta(keys)
+
+        if len(key_list) != len(ptr_list):
+            logger.error(
+                f"HiCacheNixl: mismatch between number of keys and number of buffer meta entries, keys: {len(keys)}, key_list: {len(key_list)}, buffer meta entries: {len(ptr_list)}"
+            )
+            return [], [], []
+
+        return key_list, ptr_list, element_size_list
+
+    def _bounce_slot_buffers(self, buf: torch.Tensor, page_num: int) -> List[tuple]:
+        """Return ``page_num`` ``(addr, size)`` tuples pointing at the first
+        ``page_num`` slots of ``buf``.
+        """
+        base = buf.data_ptr()
+        return [
+            (base + i * self._bounce_page_bytes, self._bounce_page_bytes)
+            for i in range(page_num)
+        ]
+
+    def _batch_preprocess(self, keys: List[str], host_indices: torch.Tensor, op: str):
+        """Build (key_list, host_buffers) for the v1 path.
+
+        For zero-copy: ``host_buffers`` are ``(addr, size)`` tuples inside the
+        pre-registered ``kv_buffer``.
+        For non-zero-copy: ``host_buffers`` are slots of the direction-specific
+        pre-registered bounce buffer (``_bounce_set`` for set, ``_bounce_get``
+        for get); for ``op == "set"`` we copy the host pages into those slots
+        here so the subsequent transfer reads from the bounce buffer.
+        Returns ``([], [])`` on validation failure.
+        """
+        page_size = self.mem_pool_host.page_size
+        page_num = len(host_indices) // page_size
+
+        if len(keys) == 0 or len(keys) != page_num:
+            logger.warning(
+                f"HiCacheNixl: empty keys or mismatch in keys and host_indices lengths. keys: {len(keys)}, host_indices: {len(host_indices)}, page_size: {page_size}"
+            )
+            return [], []
+
+        if self.is_zero_copy:
+            key_list, ptr_list, size_list = self._get_location_and_size_list_from_meta(
+                keys, host_indices
+            )
+            host_buffers = list(zip(ptr_list, size_list))
+            return key_list, host_buffers
+
+        if page_num > STORAGE_BATCH_SIZE:
+            logger.error(
+                f"HiCacheNixl: batch size {page_num} exceeds bounce buffer capacity {STORAGE_BATCH_SIZE}"
+            )
+            return [], []
+
+        bounce = self._bounce_set if op == "set" else self._bounce_get
+        if op == "set":
+            for i in range(page_num):
+                src = self.mem_pool_host.get_data_page(
+                    host_indices[i * page_size], flat=True
+                )
+                bounce[i].copy_(src)
+
+        host_buffers = self._bounce_slot_buffers(bounce, page_num)
+        key_list = [self._get_suffixed_key(key) for key in keys]
+        return key_list, host_buffers
+
+    def _batch_xfer(
+        self,
+        keys: List[str],
+        key_strs: List[str],
+        host_buffers: List[tuple],
+        direction: str,
+    ) -> List[bool]:
+        """Run a batch READ or WRITE for the v1 path against the pre-registered
+        host region (no per-transfer host registration).
+        """
+        if not key_strs or not host_buffers:
+            return [False] * len(keys)
+
+        if len(key_strs) != len(host_buffers):
+            logger.error("Mismatch between number of key_strs and host_buffers")
+            return [False] * len(keys)
+
+        if self.backend_selector.mem_type == "FILE":
+            file_paths = [self.file_manager.get_file_path(key) for key in key_strs]
+            success = self._xfer_pre_registered(host_buffers, file_paths, direction)
+        else:  # mem_type == "OBJ"
+            success = self._xfer_pre_registered(host_buffers, key_strs, direction)
+
+        # READ results are consumed by _batch_get_postprocess, which pairs
+        # entries 2*i / 2*i+1 for non-MLA zero-copy: it needs one bool per
+        # key_str (i.e. per `_k`/`_v` buffer). WRITE results map 1:1 to
+        # pages, i.e. to `keys`.
+        result_len = len(key_strs) if direction == "READ" else len(keys)
+        return [success] * result_len
+
+    def _batch_get_postprocess(
+        self,
+        host_indices: torch.Tensor,
+        results: List[bool],
+    ) -> List[bool]:
+        page_size = self.mem_pool_host.page_size
+        page_num = len(host_indices) // page_size
+
+        if self.is_zero_copy:
+            # zero copy: update final results based on the boolean results from NIXL transfer
+            if self.is_mla_model:
+                return results
+            return [(results[2 * i] and results[2 * i + 1]) for i in range(page_num)]
+
+        # non zero copy: copy data from the get-side bounce buffer to mem_pool_host
+        for i in range(page_num):
+            if not results[i]:
+                break
+            self.mem_pool_host.set_from_flat_data_page(
+                host_indices[i * page_size], self._bounce_get[i]
+            )
+        return results
+
+    def _log_xfer_stats(
+        self,
+        op_name: str,
+        num_keys: int,
+        host_indices: torch.Tensor,
+        buffer_sizes: List[int],
+        elapsed_ms: float,
+    ) -> None:
+        total_bytes = sum(s for s in buffer_sizes if s is not None)
+        bw = total_bytes / (elapsed_ms / 1000) / (1024 * 1024) if elapsed_ms else 0.0
+        logger.debug(
+            f"HiCacheNixl {op_name} transferred: {num_keys} keys (pages), "
+            f"{host_indices.numel()} host_indices, {total_bytes} bytes, "
+            f"total time: {elapsed_ms:.3f} ms, effective bandwidth: {bw:.2f} MB/s"
+        )
+
+    def batch_get_v1(
+        self,
+        keys: List[str],
+        host_indices: torch.Tensor,
+        extra_info: Optional[HiCacheStorageExtraInfo] = None,
+    ) -> List[bool]:
+        if not self._host_regs:
+            logger.error(
+                "HiCacheNixl batch_get_v1: register_mem_pool_host must be called first"
+            )
+            return [False] * len(keys)
+
+        key_strs, host_buffers = self._batch_preprocess(keys, host_indices, "get")
+        if not key_strs or not host_buffers:
+            return [False] * len(keys)
+
+        start_time = time.perf_counter()
+        results = self._batch_xfer(keys, key_strs, host_buffers, "READ")
+        elapsed_ms = (time.perf_counter() - start_time) * 1000
+        self._log_xfer_stats(
+            "batch_get_v1",
+            len(keys),
+            host_indices,
+            [s for _, s in host_buffers],
+            elapsed_ms,
+        )
+
+        return self._batch_get_postprocess(host_indices, results)
+
+    def batch_set_v1(
+        self,
+        keys: List[str],
+        host_indices: torch.Tensor,
+        extra_info: Optional[HiCacheStorageExtraInfo] = None,
+    ) -> List[bool]:
+        # skip on MLA backup rank
+        if self.backup_skip:
+            return [True] * len(keys)
+
+        if len(keys) == 0:
+            return []
+
+        if not self._host_regs:
+            logger.error(
+                "HiCacheNixl batch_set_v1: register_mem_pool_host must be called first"
+            )
+            return [False] * len(keys)
+
+        key_strs, host_buffers = self._batch_preprocess(keys, host_indices, "set")
+        if not key_strs or not host_buffers:
+            return [False] * len(keys)
+
+        start_time = time.perf_counter()
+        results = self._batch_xfer(keys, key_strs, host_buffers, "WRITE")
+        elapsed_ms = (time.perf_counter() - start_time) * 1000
+        self._log_xfer_stats(
+            "batch_set_v1",
+            len(keys),
+            host_indices,
+            [s for _, s in host_buffers],
+            elapsed_ms,
+        )
+
+        return results

python/sglang/srt/mem_cache/storage/nixl/nixl.config.toml.sample

@@ -0,0 +1,133 @@
+################################################################################
+# IMPORTANT
+# 1. to enable a plugin, add "active = true" in the corresponding section
+# 2. the configs inside plugin.posix (i.e., use_aio, use_uring, use_posix_aio)
+#    are mutually exclusive
+################################################################################
+
+
+########################################
+# GLOBAL NIXL HICACHE SETTINGS
+########################################
+
+# Open FILE-backend cache files with O_DIRECT when supported.
+use_direct_io = true
+
+# Built-in background cleaner for FILE-backed L3 storage. Set to false when an
+# external cleaner is responsible for eviction. OBJ plugins ignore this.
+l3_cleaner_enabled = true
+
+# Background cleaner watermarks for FILE-backed L3 storage. OBJ plugins ignore these.
+l3_cleaner_high_watermark = 80.0
+l3_cleaner_low_watermark = 70.0
+
+
+########################################
+# POSIX FILE SYSTEM BACKEND
+########################################
+[plugin.posix]
+# Configuration for the POSIX file-based backend.
+#
+# The supported backends include:
+# 1. AIO (`use_aio = "true"`)
+# 2. io_uring (`use_uring = "true"`)
+# 3. POSIX AIO (`use_posix_aio = "true"`)
+#
+# If not specified, NIXL will automatically detect and use available backends based on the following default priority: AIO > io_uring > POSIX AIO
+
+# Enable Linux io_uring for async I/O (recommended if supported)
+use_uring = "true"
+
+# Enable POSIX AIO (alternative async I/O mechanism)
+# use_posix_aio = "true"
+
+# Enable generic AIO
+# use_aio = "true"
+
+# Whether this plugin is eligible for selection
+active = true
+
+
+########################################
+# NVIDIA GDS (GPUDirect Storage) BACKEND
+########################################
+[plugin.gds]
+# Configuration for NVIDIA GPUDirect Storage
+# Requires compatible GPU, driver, and filesystem support
+
+# Number of requests per batch, default: 128
+batch_pool_size = 128
+
+# Maximum number of requests issued at once, default: 128
+batch_limit = 128
+
+# Maximum size of a single request (bytes), default: 16MB
+max_request_size = 16777216  # 16 MB
+
+
+########################################
+# MULTI-THREADED GDS BACKEND
+########################################
+[plugin.gds_mt]
+# Multi-threaded variant of the GDS backend
+
+# Number of worker threads, default: 4
+thread_count = 4
+
+
+########################################
+# 3FS (THIRD-PARTY FILE SYSTEM) BACKEND
+########################################
+[plugin.3fs]
+# Configuration for 3FS backend
+# Requires the 3FS library to be installed and mounted
+
+# Mount point of the 3FS filesystem
+mount_point = "/mnt/3fs"
+
+# Memory configuration mode:
+#   dram     - use DRAM
+#   dram_zc  - DRAM with zero-copy
+#   auto     - let backend decide
+mem_config = "dram"
+
+# Size of the I/O pool
+# Valid range: [2^6, 2^20]
+iopool_size = 64
+
+
+########################################
+# OBJECT STORAGE BACKEND (S3 / COMPATIBLE)
+########################################
+[plugin.obj]
+# Object storage backend (e.g., S3-compatible services)
+
+# Number of client worker threads, default: 4
+num_threads = 4
+
+# Override endpoint (useful for non-AWS S3 services)
+endpoint_override = ""
+
+# Connection scheme: http or https, default: http
+scheme = "http"
+
+# Cloud region (if applicable)
+region = ""
+
+# Request checksum behavior:
+#   required, supported
+req_checksum = "required"
+
+# Custom CA bundle path (if needed)
+ca_bundle = ""
+
+# Credentials
+access_key = ""
+secrete_key = ""
+session_token = ""
+
+# Use virtual-hosted-style addressing (true/false), default: true
+use_virtual_addressing = "true"
+
+# Default bucket name
+bucket = ""

python/sglang/srt/mem_cache/storage/nixl/nixl_cleaner.py

@@ -0,0 +1,270 @@
+"""Background disk cleaner for the NIXL FILE HiCache backend."""
+
+from __future__ import annotations
+
+import concurrent.futures
+import logging
+import os
+import re
+import threading
+import time
+from dataclasses import dataclass, field
+from typing import Iterable, Optional
+
+from sglang.srt.mem_cache.storage.nixl.nixl_routing import BUCKET_HEX_CHARS
+
+logger = logging.getLogger(__name__)
+
+_DEFAULT_INTERVAL_SEC = 30.0
+_DEFAULT_RECHECK_GROUPS = 50
+_DEFAULT_HIGH_WATERMARK = 80.0
+_DEFAULT_LOW_WATERMARK = 70.0
+_RANK_SUFFIX_RE = re.compile(r"_(\d+)_(\d+)$")
+_KV_SUFFIXES = ("_k", "_v")
+_BUCKET_NAME_RE = re.compile(rf"^[0-9a-f]{{{BUCKET_HEX_CHARS}}}$")
+
+
+@dataclass
+class _GroupInfo:
+    """Metadata for one logical cache key group in a cleaner tick."""
+
+    base_key: str
+    mtime: float = 0.0
+    size: int = 0
+    paths: set[str] = field(default_factory=set)
+    # Physical files in one logical group can hash to different base dirs
+    # because TP-rank and zero-copy K/V suffixes are part of the routed key.
+    base_dirs: set[str] = field(default_factory=set)
+
+
+def _parse_group_key(name: str) -> str:
+    """Return the logical cache-key group for one NIXL FILE object name.
+
+    The physical names are produced by ``HiCacheNixl._get_suffixed_key`` and
+    ``HiCacheNixl._get_key_list_from_meta``.
+    """
+    stem = name
+    for suffix in _KV_SUFFIXES:
+        if stem.endswith(suffix):
+            stem = stem[: -len(suffix)]
+            break
+
+    match = _RANK_SUFFIX_RE.search(stem)
+    if match is not None:
+        stem = stem[: match.start()]
+    return stem
+
+
+def _safe_unlink(path: str) -> tuple[bool, int]:
+    """Best-effort unlink; returns whether a file was removed and its size."""
+    try:
+        size = os.stat(path).st_size
+        os.unlink(path)
+        return True, size
+    except FileNotFoundError:
+        logger.debug("NIXL L3 file already removed before cleanup: %s", path)
+        return False, 0
+    except OSError:
+        logger.debug("Failed to unlink NIXL L3 file %s", path, exc_info=True)
+        return False, 0
+
+
+class HiCacheL3Cleaner:
+    """Delete old NIXL FILE cache entries when disk usage exceeds watermarks.
+
+    Cleanup operates on physical file names only, so it is compatible with MHA,
+    MLA, and DSA naming as long as the keys are generated by ``HiCacheNixl``.
+    """
+
+    def __init__(
+        self,
+        storage_dirs: list[str] | str,
+        tp_rank: int,
+        *,
+        high_watermark: float = _DEFAULT_HIGH_WATERMARK,
+        low_watermark: float = _DEFAULT_LOW_WATERMARK,
+        interval_sec: float = _DEFAULT_INTERVAL_SEC,
+        recheck_groups: int = _DEFAULT_RECHECK_GROUPS,
+        unlink_workers: Optional[int] = None,
+    ) -> None:
+        if isinstance(storage_dirs, str):
+            storage_dirs = [storage_dirs] if storage_dirs else []
+        self.storage_dirs = [path for path in storage_dirs if path]
+        self.tp_rank = tp_rank
+        self.high_watermark = high_watermark
+        self.low_watermark = low_watermark
+        self.interval_sec = interval_sec
+        self.recheck_groups = max(1, recheck_groups)
+        self.unlink_workers = unlink_workers or max(
+            8, 8 * max(len(self.storage_dirs), 1)
+        )
+
+        if self.low_watermark >= self.high_watermark:
+            raise ValueError(
+                "L3 cleaner low_watermark must be lower than high_watermark "
+                f"(low_watermark={self.low_watermark}, "
+                f"high_watermark={self.high_watermark})"
+            )
+
+        self._stop = threading.Event()
+        self._thread: Optional[threading.Thread] = None
+
+    def start(self) -> None:
+        """Start the cleaner thread on TP rank 0."""
+        if self.tp_rank != 0 or not self.storage_dirs:
+            return
+        if self._thread is not None and self._thread.is_alive():
+            return
+        self._thread = threading.Thread(
+            target=self._loop, name="hicache-l3-cleaner", daemon=True
+        )
+        self._thread.start()
+        logger.info(
+            "HiCacheL3Cleaner started: dirs=%s high=%.1f%% low=%.1f%% "
+            "interval=%.1fs unlink_workers=%d",
+            self.storage_dirs,
+            self.high_watermark,
+            self.low_watermark,
+            self.interval_sec,
+            self.unlink_workers,
+        )
+
+    def stop(self) -> None:
+        """Stop the cleaner thread if it was started."""
+        self._stop.set()
+        if self._thread is not None:
+            self._thread.join(timeout=5.0)
+            self._thread = None
+
+    def _disk_usage_pct(self, path: str) -> float:
+        try:
+            stat = os.statvfs(path)
+        except OSError:
+            return 0.0
+        total = stat.f_blocks * stat.f_frsize
+        if total == 0:
+            return 0.0
+        available = stat.f_bavail * stat.f_frsize
+        return 100.0 * (total - available) / total
+
+    def _loop(self) -> None:
+        while not self._stop.is_set():
+            try:
+                self._tick()
+            except Exception:
+                logger.warning("NIXL L3 cleaner tick failed", exc_info=True)
+            if self._stop.wait(self.interval_sec):
+                break
+
+    def _tick(self) -> bool:
+        initial_pcts = {path: self._disk_usage_pct(path) for path in self.storage_dirs}
+        hot_dirs = {
+            path for path, pct in initial_pcts.items() if pct >= self.high_watermark
+        }
+        if not hot_dirs:
+            return False
+
+        scan_start = time.perf_counter()
+        groups: dict[str, _GroupInfo] = {}
+        for base_dir in self.storage_dirs:
+            self._scan_base_dir(base_dir, groups)
+
+        ordered = sorted(
+            (group for group in groups.values() if group.base_dirs & hot_dirs),
+            key=lambda group: group.mtime,
+        )
+        if not ordered:
+            return False
+
+        deleted_groups = 0
+        deleted_files = 0
+        bytes_deleted = 0
+
+        with concurrent.futures.ThreadPoolExecutor(
+            max_workers=self.unlink_workers,
+            thread_name_prefix="hicache-l3-unlink",
+        ) as pool:
+            idx = 0
+            while idx < len(ordered) and not self._stop.is_set():
+                batch = ordered[idx : idx + self.recheck_groups]
+                paths = list(self._iter_group_paths(batch))
+                for removed, removed_bytes in pool.map(_safe_unlink, paths):
+                    if removed:
+                        deleted_files += 1
+                        bytes_deleted += removed_bytes
+                deleted_groups += len(batch)
+                idx += len(batch)
+
+                if all(
+                    self._disk_usage_pct(path) < self.low_watermark for path in hot_dirs
+                ):
+                    break
+
+        final_pcts = {path: self._disk_usage_pct(path) for path in self.storage_dirs}
+        logger.info(
+            "NIXL L3 cleanup: deleted %d groups / %d files (%.2f GiB) in %.1fs, "
+            "initial_hot=%s final=%s",
+            deleted_groups,
+            deleted_files,
+            bytes_deleted / (1024**3),
+            time.perf_counter() - scan_start,
+            {path: f"{initial_pcts[path]:.1f}%" for path in hot_dirs},
+            {path: f"{pct:.1f}%" for path, pct in final_pcts.items()},
+        )
+        return True
+
+    def _scan_base_dir(self, base_dir: str, groups: dict[str, _GroupInfo]) -> None:
+        if not os.path.isdir(base_dir):
+            return
+        try:
+            bucket_entries = list(os.scandir(base_dir))
+        except OSError:
+            logger.warning("NIXL L3 cleaner failed to scan %s", base_dir, exc_info=True)
+            return
+
+        for bucket_entry in bucket_entries:
+            try:
+                is_bucket_dir = bucket_entry.is_dir(follow_symlinks=False)
+            except OSError:
+                continue
+            if (
+                not is_bucket_dir
+                or _BUCKET_NAME_RE.fullmatch(bucket_entry.name) is None
+            ):
+                continue
+            self._scan_bucket(base_dir, bucket_entry.path, groups)
+
+    def _scan_bucket(
+        self, base_dir: str, bucket_path: str, groups: dict[str, _GroupInfo]
+    ) -> None:
+        try:
+            entries = list(os.scandir(bucket_path))
+        except OSError:
+            logger.debug(
+                "NIXL L3 cleaner skipped bucket %s", bucket_path, exc_info=True
+            )
+            return
+
+        for entry in entries:
+            try:
+                if not entry.is_file(follow_symlinks=False):
+                    continue
+                stat = entry.stat(follow_symlinks=False)
+            except OSError:
+                continue
+
+            group_key = _parse_group_key(entry.name)
+            group = groups.setdefault(group_key, _GroupInfo(group_key))
+            group.paths.add(entry.path)
+            group.base_dirs.add(base_dir)
+            group.size += stat.st_size
+            group.mtime = max(group.mtime, stat.st_mtime)
+
+    def _iter_group_paths(self, groups: Iterable[_GroupInfo]) -> Iterable[str]:
+        seen: set[str] = set()
+        for group in groups:
+            for path in sorted(group.paths):
+                if path in seen:
+                    continue
+                seen.add(path)
+                yield path

python/sglang/srt/mem_cache/storage/nixl/nixl_registry.py

@@ -0,0 +1,193 @@
+"""NIXL memory-registration helpers, exposed as context managers.
+
+A ``NixlRegistry`` instance bundles the agent, the memory type, and
+(optionally) the file manager.  Its ``storage(...)`` method is a context
+manager that performs the entire register-and-build-descs sequence for
+the storage side of a transfer on entry, yields the ``xfer_descs`` (or
+None on failure), and unwinds ``agent.deregister_memory`` plus any
+``os.close(fd)`` on exit.
+
+The host side is pre-registered up front by ``HiCacheNixl`` and is not
+touched per transfer.
+"""
+
+import logging
+import threading
+from contextlib import contextmanager
+from typing import List, Optional
+
+from .nixl_utils import NixlFileManager
+
+logger = logging.getLogger(__name__)
+
+
+def _buffer_sizes(buffers) -> Optional[List[int]]:
+    """Per-buffer byte sizes for ``(addr, len)`` tuple inputs."""
+    if not buffers or not isinstance(buffers[0], tuple):
+        return None
+    return [b[1] for b in buffers]
+
+
+class NixlRegistry:
+    """Owns the (agent, mem_type, file_manager) triple and provides a
+    context manager for the storage side of a transfer.
+
+    A single instance is created once per HiCacheNixl in __init__ and
+    reused for every transfer.
+    """
+
+    def __init__(
+        self,
+        agent,
+        mem_type: str,
+        file_manager: Optional[NixlFileManager] = None,
+    ):
+        self.agent = agent
+        self.mem_type = mem_type
+        self.file_manager = file_manager
+        # OBJ devIds key a process-wide map in the NIXL OBJ plugin
+        # (devIdToObjKey_) that is not protected by a lock, so concurrent
+        # OBJ registrations must use disjoint devId ranges. Allocate them
+        # from a single monotonic counter.
+        self._obj_devid_lock = threading.Lock()
+        self._obj_devid_next = 1
+        self.path_mode = mem_type == "FILE" and self._probe_path_mode()
+        if mem_type == "FILE" and self.path_mode:
+            logger.info("HiCacheNixl: path-mode FILE registration active.")
+        elif mem_type == "FILE":
+            # TODO: NIXL 1.3.0 adds path-mode support; remove this fd fallback once 1.3.0 is widely installed.
+            logger.info(
+                "HiCacheNixl: the installed NIXL build does not "
+                "support path-mode FILE registration; using legacy "
+                "fd registration."
+            )
+
+    @contextmanager
+    def _open_files(self, paths: List[str], create: bool):
+        """Open fds for ``paths``; close all of them on exit.
+
+        Yields the list of fds, or None if any open fails (already-opened
+        fds are closed before returning by the same ``finally``).
+        """
+        fds: List[int] = []
+        try:
+            for path in paths:
+                fd = self.file_manager.open_file(path, create=create)
+                if fd is None:
+                    yield None
+                    return
+                fds.append(fd)
+            yield fds
+        finally:
+            for fd in fds:
+                self.file_manager.close_file(fd)
+
+    @contextmanager
+    def _registered(self, items: List[tuple], mem_type: str):
+        """Register ``items`` with NIXL; deregister on exit.
+
+        Yields the registration handle, or None if registration fails.
+        """
+        reg = None
+        if items:
+            reg_descs = self.agent.get_reg_descs(items, mem_type)
+            if reg_descs is not None:
+                try:
+                    reg = self.agent.register_memory(reg_descs)
+                except Exception as e:
+                    logger.error(f"Failed to register memory of type {mem_type}: {e}")
+        try:
+            yield reg
+        finally:
+            if reg is not None:
+                try:
+                    self.agent.deregister_memory(reg)
+                except Exception as e:
+                    logger.debug("deregister_memory skipped: %s", e)
+
+    def _probe_path_mode(self) -> bool:
+        """Probe whether NIXL honours path-mode metaInfo.
+
+        Register a FILE_SEG with a valid path-mode string pointing at a
+        nonexistent path (no 'create' flag). A path-mode-capable NIXL tries
+        to open() the path, fails with NIXL_ERR_BACKEND, and raises. A
+        pre-path-mode NIXL ignores metaInfo and returns NIXL_SUCCESS.
+        Error from register_memory => path mode supported.
+        """
+        reg_descs = self.agent.get_reg_descs(
+            [(0, 4096, 1, "rw:/nonexistent-nixl-probe")], "FILE"
+        )
+        if reg_descs is None:
+            return False
+        try:
+            reg = self.agent.register_memory(reg_descs)
+            if reg is not None:
+                try:
+                    self.agent.deregister_memory(reg)
+                except Exception:
+                    pass
+            return False
+        except Exception:
+            return True
+
+    @contextmanager
+    def storage(self, buffers, keys, direction):
+        """Open + register the storage side; deregister and close fds on exit.
+
+        Yields the storage xfer_descs, or None on failure.  For the FILE
+        backend, files are created (O_CREAT) when ``direction == "WRITE"``.
+        """
+        sizes = _buffer_sizes(buffers)
+        if sizes is None:
+            yield None
+            return
+
+        if self.mem_type == "FILE":
+            if self.path_mode:
+                parts = ["rw", "create"] if direction == "WRITE" else ["ro"]
+                if self.file_manager.use_direct_io:
+                    parts.append("direct")
+                spec = ",".join(parts)
+                tuples = [
+                    (0, sizes[i], i + 1, f"{spec}:{keys[i]}") for i in range(len(keys))
+                ]
+                with self._registered(tuples, "FILE") as reg:
+                    if reg is None:
+                        yield None
+                        return
+                    yield reg.trim()
+            else:
+                with self._open_files(keys, create=(direction == "WRITE")) as fds:
+                    if fds is None:
+                        yield None
+                        return
+                    tuples = [(0, sizes[i], fds[i], keys[i]) for i in range(len(keys))]
+                    with self._registered(tuples, "FILE") as reg:
+                        if reg is None:
+                            yield None
+                            return
+                        yield self.agent.get_xfer_descs(
+                            [(0, sizes[i], fds[i]) for i in range(len(fds))], "FILE"
+                        )
+        else:  # OBJ
+            # Reg tuple: (addr=0, size, devId, metaInfo=key).
+            # Xfer tuple: (addr=0, size, devId). devId links each xfer desc
+            # back to its registered object's metaInfo, so devIds must be
+            # unique within the list AND globally unique across concurrent
+            # storage() calls (the OBJ plugin's devIdToObjKey_ map is shared
+            # and unlocked). NIXL's pybind layer requires position 3 to be
+            # int, hence the key goes in metaInfo (position 4).
+            n = len(keys)
+            with self._obj_devid_lock:
+                base = self._obj_devid_next
+                self._obj_devid_next += n
+            dev_ids = list(range(base, base + n))
+            tuples = [(0, sizes[i], dev_ids[i], keys[i]) for i in range(n)]
+            with self._registered(tuples, "OBJ") as reg:
+                if reg is None:
+                    yield None
+                    return
+                yield self.agent.get_xfer_descs(
+                    [(0, sizes[i], dev_ids[i]) for i in range(n)],
+                    self.mem_type,
+                )

python/sglang/srt/mem_cache/storage/nixl/nixl_routing.py

@@ -0,0 +1,29 @@
+"""Deterministic path routing for NIXL FILE-backed HiCache storage."""
+
+import hashlib
+
+BUCKET_HEX_CHARS = 2
+_BUCKET_MASK = (1 << (4 * BUCKET_HEX_CHARS)) - 1
+
+
+def stable_key_hash(key: str) -> int:
+    """Return a process-stable 64-bit hash for a NIXL storage key."""
+    return int.from_bytes(
+        hashlib.blake2b(key.encode("utf-8"), digest_size=8).digest(), "big"
+    )
+
+
+def route_key(key: str, num_disks: int) -> tuple[int, str]:
+    """Return the storage disk index and bucket directory for a storage key."""
+    if num_disks <= 0:
+        raise ValueError("num_disks must be positive")
+    key_hash = stable_key_hash(key)
+    return (
+        (key_hash >> 16) % num_disks,
+        f"{key_hash & _BUCKET_MASK:0{BUCKET_HEX_CHARS}x}",
+    )
+
+
+def route_disk(key: str, num_disks: int) -> int:
+    """Return the storage disk index for a storage key."""
+    return route_key(key, num_disks)[0]

python/sglang/srt/mem_cache/storage/nixl/nixl_utils.py

@@ -0,0 +1,315 @@
+import logging
+import os
+from typing import Optional
+
+from sglang.srt.environ import envs
+from sglang.srt.mem_cache.storage.nixl.nixl_routing import (
+    _BUCKET_MASK,
+    BUCKET_HEX_CHARS,
+    route_key,
+)
+
+logger = logging.getLogger(__name__)
+
+_SGLANG_NIXL_CONFIG_KEYS = {
+    "use_direct_io",
+    "l3_cleaner_enabled",
+    "l3_cleaner_high_watermark",
+    "l3_cleaner_low_watermark",
+}
+
+
+class NixlBackendConfig:
+    """Handles NIXL backend configurations"""
+
+    def __init__(self, config: Optional[dict[str, str]] = None):
+        """Initialize backend configuration.
+        Args:
+            config: configurations in a dictionary. This config comes from --hicache-storage-backend-extra-config
+
+            config can be in two forms:
+            1. fully qualified form (for all plugins, some of them are enabled, others not):
+                {'plugin': { 'posix': {...}, 'gds': {...}, ...}}
+            2. flat form (for a specific selected plugin), assuming all params apply to a selected plugin
+                {'param1': 'value1', 'param2': 'value2', ...}
+        """
+        self.config = config or {}
+
+    def get_use_direct_io(self) -> bool:
+        """Return True if O_DIRECT should be requested when opening files.
+
+        Checks the top-level ``use_direct_io`` key in the long-form JSON config first,
+        then falls back to the ``SGLANG_HICACHE_NIXL_USE_DIRECT_IO`` environment variable
+        (default: enabled).
+        """
+        if "use_direct_io" in self.config:
+            return bool(self.config["use_direct_io"])
+        return envs.SGLANG_HICACHE_NIXL_USE_DIRECT_IO.get()
+
+    def get_l3_cleaner_config(self) -> dict:
+        """Return typed NIXL FILE L3 cleaner options from top-level config."""
+        config = {
+            "enabled": True,
+            "high_watermark": 80.0,
+            "low_watermark": 70.0,
+        }
+        if "l3_cleaner_enabled" in self.config:
+            enabled = self.config["l3_cleaner_enabled"]
+            if not isinstance(enabled, bool):
+                raise ValueError("l3_cleaner_enabled must be a boolean")
+            config["enabled"] = enabled
+        key_map = {
+            "l3_cleaner_high_watermark": ("high_watermark", float),
+            "l3_cleaner_low_watermark": ("low_watermark", float),
+        }
+        for raw_key, (cleaner_key, parser) in key_map.items():
+            if raw_key in self.config:
+                config[cleaner_key] = parser(self.config[raw_key])
+        return config
+
+    def get_specified_plugin(self) -> str:
+        """decide which plugin to use: either config or SGLANG_HICACHE_NIXL_BACKEND_PLUGIN specifies the plugin, if not, use "auto" """
+
+        if "plugin" in self.config:
+            # fully qualified form: {'plugin': { 'posix': {...}, 'gds': {...}, ...}}
+            # choose the FIRST active plugin
+            for key, item in self.config["plugin"].items():
+                if item.get("active", False) in [True, "true", "True"]:
+                    plugin = key.upper()
+                    break
+        else:
+            # config is empty, or in flat form {'param1': 'value1', 'param2': 'value2', ...}
+            plugin = os.getenv("SGLANG_HICACHE_NIXL_BACKEND_PLUGIN", "auto")
+
+        return plugin
+
+    def get_backend_initparams(self, backend_name) -> dict:
+        """Get initialization parameters from config of NIXL backend for backend creation.
+        Args:
+            backend_name: a specific backend's name (already converted "auto" into a specific backend name)
+
+        """
+
+        initparams = {}
+
+        # config can be in two forms:
+        if "plugin" in self.config:
+            # fully qualified form: {'plugin': { 'posix': {...}, 'gds': {...}, ...}}
+            if backend_name.lower() in self.config["plugin"]:
+                config_data = self.config["plugin"][backend_name.lower()]
+            else:
+                logger.debug(
+                    f"No specific config found for plugin {backend_name} in extra_config. Use default init params."
+                )
+                config_data = {}
+        else:
+            # flat form {'param1': 'value1', 'param2': 'value2', ...}
+            config_data = self.config
+
+        for key, value in config_data.items():
+            # These keys are consumed by SGLang itself, not by NIXL plugins.
+            if key in _SGLANG_NIXL_CONFIG_KEYS:
+                continue
+            initparams[key] = str(value)
+
+        return initparams
+
+
+class NixlBackendSelection:
+    """Handles NIXL backend selection and creation."""
+
+    # Priority order for File-based plugins in case of auto selection
+    FILE_PLUGINS = ["3FS", "POSIX", "GDS_MT", "GDS"]
+    # Priority order for File-based plugins in case of auto selection (add more as needed)
+    OBJ_PLUGINS = ["OBJ"]  # Based on Amazon S3 SDK
+
+    def __init__(
+        self, plugin: str = "auto", nixlconfig: Optional[NixlBackendConfig] = None
+    ):
+        """Initialize backend selection.
+        Args:
+            plugin: Plugin to use (default "auto" selects best available).
+                   Can be a file plugin (3FS, POSIX, GDS, GDS_MT) or
+                   an object plugin (OBJ).
+        """
+        self.plugin = plugin
+        self.backend_name = None
+        self.mem_type = None
+        self.nixlconfig = nixlconfig
+
+    def create_backend(self, agent) -> bool:
+        """Create the appropriate NIXL backend based on configuration."""
+        try:
+            plugin_list = agent.get_plugin_list()
+            logger.debug(f"Available NIXL plugins: {plugin_list}")
+
+            # Handle explicit plugin selection or auto priority
+            if self.plugin == "auto":
+                # Try all file plugins first
+                for plugin in self.FILE_PLUGINS:
+                    if plugin in plugin_list:
+                        self.backend_name = plugin
+                        break
+                # If no file plugin found, try object plugins
+                if not self.backend_name:
+                    for plugin in self.OBJ_PLUGINS:
+                        if plugin in plugin_list:
+                            self.backend_name = plugin
+                            break
+            else:
+                # Use explicitly requested plugin
+                self.backend_name = self.plugin
+
+            if self.backend_name not in plugin_list:
+                logger.error(
+                    f"Backend {self.backend_name} not available in plugins: {plugin_list}"
+                )
+                return False
+
+            # obtain initparams for the backend from the NIXL config
+            initparams = (
+                self.nixlconfig.get_backend_initparams(self.backend_name)
+                if self.nixlconfig
+                else {}
+            )
+
+            # Create backend and set memory type
+            if self.backend_name in self.OBJ_PLUGINS and "bucket" not in initparams:
+                bucket = os.environ.get("AWS_DEFAULT_BUCKET")
+                if not bucket:
+                    logger.error(
+                        "AWS_DEFAULT_BUCKET environment variable must be set for object storage"
+                    )
+                    return False
+
+                initparams["bucket"] = bucket
+
+            # create backend using initialization parameters
+            agent.create_backend(self.backend_name, initparams)
+
+            logger.info(
+                f"NixlBackendSelection.create_backend: backend_name {self.backend_name} initparams {initparams} customParams {agent.get_backend_params(self.backend_name)} supported plugins {plugin_list}"
+            )
+
+            self.mem_type = "OBJ" if self.backend_name in self.OBJ_PLUGINS else "FILE"
+            logger.debug(
+                f"Created NIXL backend: {self.backend_name} with memory type: {self.mem_type}"
+            )
+            return True
+
+        except Exception as e:
+            logger.error(
+                f"Failed to create NIXL backend: {e}, backend_name {self.backend_name}, supported plugins {plugin_list} initparams {initparams}"
+            )
+            return False
+
+
+class NixlFileManager:
+    """Handles file system operations for NIXL."""
+
+    def __init__(self, base_dir: "list[str] | str", use_direct_io: bool = True):
+        """
+        Initialize file manager.
+        Args:
+            base_dir: Base directory or ordered base directories for tensor files.
+            use_direct_io: If True, open files with O_DIRECT (bypasses OS page cache).
+                Falls back to buffered I/O with a warning when O_DIRECT is unavailable.
+        """
+        if isinstance(base_dir, str):
+            self.base_dirs = [base_dir] if base_dir else []
+        else:
+            self.base_dirs = [d for d in base_dir if d]
+        self.use_direct_io = use_direct_io
+        self._created_bucket_dirs: set[str] = set()
+        if not self.base_dirs:
+            logger.debug(
+                f"Initialized file manager without a base directory. Direct I/O: {use_direct_io}"
+            )
+        else:
+            for base in self.base_dirs:
+                os.makedirs(base, exist_ok=True)
+            self.ensure_all_bucket_dirs()
+            logger.debug(
+                f"Initialized file manager with base directories: {self.base_dirs}. Direct I/O: {use_direct_io}"
+            )
+
+    def clear(self) -> None:
+        """Clear all files below every configured base directory."""
+        if not self.base_dirs:
+            logger.warning("Base directories are empty, skipping clear operation")
+            return
+
+        for base in self.base_dirs:
+            try:
+                for root, _dirs, files in os.walk(base):
+                    for file in files:
+                        file_path = os.path.join(root, file)
+                        try:
+                            os.remove(file_path)
+                        except OSError as e:
+                            logger.warning(f"Failed to remove file {file_path}: {e}")
+            except Exception as e:
+                logger.error(f"Failed to clear base directory {base}: {e}")
+        logger.debug(f"Cleared all files in base directories: {self.base_dirs}")
+
+    def ensure_all_bucket_dirs(self) -> None:
+        """Pre-create every possible bucket directory under each base dir.
+
+        Called once when path mode is active so NIXL O_CREAT writes never
+        fail due to a missing parent directory.
+        """
+        for base in self.base_dirs:
+            for i in range(_BUCKET_MASK + 1):
+                os.makedirs(
+                    os.path.join(base, f"{i:0{BUCKET_HEX_CHARS}x}"),
+                    exist_ok=True,
+                )
+
+    def iter_all_base_dirs(self) -> list[str]:
+        """Return base directories that may contain NIXL FILE cache entries."""
+        return list(self.base_dirs)
+
+    def get_file_path(self, key: str) -> str:
+        """Get full file path for a given key."""
+        if not self.base_dirs:
+            return key
+        disk_idx, bucket = route_key(key, len(self.base_dirs))
+        return os.path.join(self.base_dirs[disk_idx], bucket, key)
+
+    def open_file(self, file_path: str, create: bool = False) -> Optional[int]:
+        """Open a file and return its file descriptor.
+
+        If ``create`` is True, the file is created if it does not exist
+        (mode 0o644, no truncation). When ``self.use_direct_io`` is True,
+        the file is opened with ``O_DIRECT`` (bypasses the OS page cache);
+        falls back to buffered I/O with a warning if ``O_DIRECT`` is
+        unavailable on this platform.
+        """
+        flags = os.O_RDWR | os.O_CREAT if create else os.O_RDWR
+        if self.use_direct_io:
+            if hasattr(os, "O_DIRECT"):
+                flags |= os.O_DIRECT
+            else:
+                logger.warning(
+                    "use_direct_io is True, but O_DIRECT is not available on "
+                    "this system. Falling back to buffered I/O."
+                )
+        try:
+            if create:
+                parent = os.path.dirname(file_path)
+                if parent and parent not in self._created_bucket_dirs:
+                    os.makedirs(parent, exist_ok=True)
+                    self._created_bucket_dirs.add(parent)
+            return os.open(file_path, flags, 0o644)
+        except Exception as e:
+            logger.error(f"Failed to open file {file_path}: {e}")
+            return None
+
+    def close_file(self, fd: int) -> bool:
+        """Close a file descriptor."""
+        try:
+            os.close(fd)
+            return True
+        except Exception as e:
+            logger.error(f"Failed to close file descriptor {fd}: {e}")
+            return False