wehub-resource-sync
155 lines
5.6 KiB
ReStructuredText
155 lines
5.6 KiB
ReStructuredText
S3 Backend
|
|
==========
|
|
|
|
Example Configurations
|
|
----------------------
|
|
|
|
Basic S3 Configuration
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 256
|
|
local_cpu: False
|
|
save_unfull_chunk: False
|
|
remote_url: "s3://your-bucket-name"
|
|
remote_serde: "naive"
|
|
blocking_timeout_secs: 10
|
|
extra_config:
|
|
s3_region: "us-east-1"
|
|
s3_num_io_threads: 64
|
|
save_chunk_meta: False
|
|
|
|
S3 Express One Zone
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 256
|
|
local_cpu: False
|
|
save_unfull_chunk: False
|
|
remote_url: "s3://{BUCKET_NAME}.s3express-{AZ_ID}.{REGION}.amazonaws.com"
|
|
remote_serde: "naive"
|
|
blocking_timeout_secs: 10
|
|
extra_config:
|
|
save_chunk_meta: False
|
|
s3_num_io_threads: 64
|
|
s3_prefer_http2: True
|
|
s3_region: "{REGION}"
|
|
s3_enable_s3express: True
|
|
|
|
CoreWeave (S3-compatible)
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 256
|
|
local_cpu: False
|
|
max_local_cpu_size: 50
|
|
save_unfull_chunk: False
|
|
enable_async_loading: True
|
|
remote_url: "s3://test-127.cwlota.com"
|
|
remote_serde: "naive"
|
|
blocking_timeout_secs: 10
|
|
extra_config:
|
|
s3_num_io_threads: 320
|
|
s3_prefer_http2: False
|
|
s3_region: "US-WEST-04A"
|
|
s3_enable_s3express: False
|
|
save_chunk_meta: False
|
|
disable_tls: True
|
|
aws_access_key_id: "your-access-key-id"
|
|
aws_secret_access_key: "your-secret-access-key"
|
|
|
|
**Note**: `cwlota.com` is CoreWeave's S3-compatible Cloud Storage that caches for GPU locality. You can set `disable_tls: True` for non-AWS services.
|
|
|
|
Check out the blog post between LMCache, Cohere, and CoreWeave: https://blog.lmcache.ai/en/2025/10/29/breaking-the-memory-barrier-how-lmcache-and-coreweave-power-efficient-llm-inference-for-cohere/
|
|
|
|
|
|
Configuration Parameters
|
|
------------------------
|
|
|
|
* **remote_url**: S3 bucket URL (`s3://bucket-name`)
|
|
* **save_unfull_chunk**: Save partial chunks (default: False, **must be False for S3**)
|
|
* **enable_async_loading**: Async loading (default: False)
|
|
* **blocking_timeout_secs**: Timeout seconds (default: 10)
|
|
|
|
S3-Specific (in extra_config)
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
* **s3_region**: AWS region for S3 client (required)
|
|
* **s3_num_io_threads**: Number of IO threads for the AWS CRT client to spawn. Benefits taper out after exceeding the number of CPU cores. This is also a way to restrict the number of outgoing requests in case your S3-compatible object store has a rate-limiting gateway.
|
|
* **s3_prefer_http2**: Enable HTTP/2 with ALPN negotiation (["h2", "http/1.1"])
|
|
* **s3_enable_s3express**: Enable S3 Express One Zone support in AWS CRT client
|
|
* **save_chunk_meta**: Whether to save chunk metadata in the object store along with your data (False required for S3)
|
|
* **aws_access_key_id**: AWS access key ID (or log in with `aws configure` in your environment)
|
|
* **aws_secret_access_key**: AWS secret access key (or log in with `aws configure` in your environment)
|
|
|
|
**Tips:**::
|
|
|
|
- Use same region for compute and S3
|
|
|
|
- Consider S3 Express One Zone for less redundancy but better performance
|
|
|
|
|
|
MP Mode Configuration
|
|
---------------------
|
|
|
|
In multi-process (MP) mode, S3 is configured as an L2 adapter via a JSON
|
|
spec passed to the LMCache server, rather than through ``remote_url`` +
|
|
``extra_config``. Each ``--l2-adapter`` argument takes a JSON object
|
|
whose ``"type": "s3"`` field selects the S3 adapter.
|
|
|
|
.. code-block:: json
|
|
|
|
{
|
|
"type": "s3",
|
|
"s3_endpoint": "s3://my-bucket",
|
|
"s3_region": "us-east-1",
|
|
"s3_num_io_threads": 64,
|
|
"s3_prefer_http2": true,
|
|
"s3_enable_s3express": false,
|
|
"disable_tls": false,
|
|
"max_capacity_gb": 500,
|
|
"eviction": {
|
|
"eviction_policy": "LRU",
|
|
"trigger_watermark": 0.85,
|
|
"eviction_ratio": 0.2
|
|
}
|
|
}
|
|
|
|
S3 L2 Adapter Fields
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
* **type** (required): must be ``"s3"``.
|
|
* **s3_endpoint** (required): bucket URL. Accepts either ``s3://bucket`` or
|
|
the bare host form (e.g. ``bucket.s3.us-east-1.amazonaws.com``).
|
|
* **s3_region** (required): AWS region for the S3 client.
|
|
* **s3_num_io_threads**: number of CRT IO threads (default ``64``).
|
|
* **s3_prefer_http2**: attempt HTTP/2 via ALPN negotiation (default ``true``).
|
|
* **s3_enable_s3express**: enable S3 Express One Zone signing (default ``false``).
|
|
* **disable_tls**: bypass TLS, for non-AWS HTTP endpoints (default ``false``).
|
|
* **aws_access_key_id**, **aws_secret_access_key**: optional static credentials.
|
|
When omitted the adapter uses the AWS default credentials chain
|
|
(``aws configure``, environment variables, IRSA, etc.).
|
|
* **max_capacity_gb**: capacity used by ``get_usage()`` for watermark-based
|
|
L2 eviction. Set to ``0`` (default) to disable usage tracking — ``get_usage()``
|
|
then returns ``(-1.0, -1.0)`` and no automatic eviction is triggered.
|
|
* **eviction**: optional sub-dict enabling the L2 eviction controller for
|
|
this adapter. See :class:`L2AdapterConfigBase` ``_parse_eviction_config``
|
|
for the full schema. When present, keys that are currently being loaded
|
|
(reference-counted by the lookup-and-lock path) are skipped by
|
|
``delete()``.
|
|
|
|
Differences vs Non-MP S3
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
* The MP adapter honors first-class eviction: it implements ``delete()``
|
|
(real S3 ``DeleteObject``), refcounted ``submit_unlock``, and
|
|
``get_usage()`` driven by ``max_capacity_gb``.
|
|
* Keys are identified by ``ObjectKey`` (``model_name`` + ``kv_rank`` +
|
|
``chunk_hash``) rather than ``CacheEngineKey``. The wire-format object
|
|
name is ``<model>@<kv_rank_hex>@<chunk_hash_hex>``, which is **not**
|
|
compatible with the non-MP naming. A bucket populated by non-MP LMCache
|
|
cannot be read directly by MP LMCache and vice versa.
|
|
* Unfull chunks are rejected (same constraint as non-MP). |