wehub-resource-sync
138 lines
6.6 KiB
Markdown
138 lines
6.6 KiB
Markdown
# Anonymous Usage Telemetry (`lmcache/usage_telemetry/`)
|
|
|
|
Phone-home usage statistics: what LMCache deployments look like and how much
|
|
caching they do. This is **not** operator observability — Prometheus/OTel
|
|
metrics for operators live in `lmcache/v1/mp_observability/` (MP mode) and
|
|
`lmcache/observability.py` (single-process mode). The two systems share event
|
|
sources in places but have different consumers (LMCache maintainers vs.
|
|
deployment operators), different transports, and independent opt-outs.
|
|
|
|
## Package layout
|
|
|
|
| Module | Contents |
|
|
|---|---|
|
|
| `messages.py` | **Wire schema — single source of truth.** Every message dataclass, `USAGE_SCHEMA_VERSION`, per-message `ENDPOINT` |
|
|
| `guard.py` | `swallow_telemetry_errors` decorator — the no-throw guarantee |
|
|
| `identity.py` | Opt-out gate (`is_usage_tracking_enabled`), `UsageIdentity` (session/machine ids) |
|
|
| `transport.py` | `UsageMessageSender` (HTTP), `build_usage_payload` (identity + schema stamping) |
|
|
| `env_probe.py` | Hardware/platform/cloud detection feeding `EnvMessage` |
|
|
| `context.py` | `UsageContextBase`, single-process `UsageContext`, `InitializeUsageContext` — the `/context` snapshot reporters |
|
|
| `continuous.py` | `ContinuousUsageContext` interval counters and lifespan histogram |
|
|
| `mp.py` | MP server `MPUsageContext`, `InitializeMPUsageContext` |
|
|
|
|
`lmcache/usage_context.py` remains as a backward-compatibility shim
|
|
re-exporting the pre-package public names.
|
|
|
|
## Message schema contract
|
|
|
|
`messages.py` is the protobuf-style registry: every message LMCache can
|
|
phone home is a dataclass there, and nowhere else. The contract:
|
|
|
|
- The **class name** is the wire `message_type` discriminator —
|
|
`build_usage_payload` derives it, so senders cannot disagree with the
|
|
schema.
|
|
- Each message declares the stats-server **`ENDPOINT`** it POSTs to as a
|
|
class attribute; senders route by it.
|
|
- Fields are **flat and JSON-serializable** (the backend stores flat
|
|
key-value pairs; lists are joined into delimited strings).
|
|
- `session_id`, `machine_id`, `schema_version`, and `deployment_mode` are
|
|
stamped onto every payload by `build_usage_payload`, not declared per
|
|
message.
|
|
|
|
## Reporting paths
|
|
|
|
| Path | Trigger | Messages |
|
|
|---|---|---|
|
|
| Single-process one-shot | `LMCacheEngine.__init__` → `InitializeUsageContext` | `EnvMessage`, `EngineMessage`, `MetadataMessage` |
|
|
| MP server one-shot | `run_cache_server` → `InitializeMPUsageContext` | `EnvMessage`, `MPServerMessage` |
|
|
| Continuous (single-process) | `LMCacheStatsLogger.log_worker`, every `LMCACHE_USAGE_TRACK_INTERVAL` s (default 600) | `ContinuousContextMessage`, `CacheLifespanMessage` |
|
|
|
|
Model and KV-layout information only become visible to the MP server when
|
|
a serving engine registers its KV caches; a registration-time report
|
|
(`MPInstanceMessage`) is planned for a follow-up PR, together with MP-mode
|
|
continuous reporting (an EventBus subscriber feeding the same endpoints).
|
|
|
|
## Failure isolation (no-throw guarantee)
|
|
|
|
A failure anywhere in telemetry must never affect caching or serving. The
|
|
mechanisms, outermost first:
|
|
|
|
- Every entry point called from serving code — `InitializeUsageContext`,
|
|
`InitializeMPUsageContext`, `ContinuousUsageContext.incr_or_send_stats`,
|
|
`report_once` — is decorated with
|
|
`guard.swallow_telemetry_errors`: exceptions are logged at debug level
|
|
and swallowed.
|
|
- One-shot sends run on a daemon thread; startup never blocks on the
|
|
stats server.
|
|
- `UsageMessageSender.send` additionally swallows all transport errors
|
|
with a 5 s timeout.
|
|
- `ContinuousUsageContext.__init__` degrades (kv-bytes-per-token = 0)
|
|
rather than raising on nonstandard kv shapes.
|
|
|
|
Tests for this contract live in `TestFailureIsolation`
|
|
(`tests/test_usage_telemetry.py`): a transport that raises must not break
|
|
the startup report or the stats-logger flush.
|
|
|
|
## Identity and association contract
|
|
|
|
Every payload — one-shot or continuous — is stamped with:
|
|
|
|
- `session_id`: random UUID minted once per process. The join key: the
|
|
`/context` rows keyed by a `session_id` describe the deployment that
|
|
produced all other rows with that `session_id`.
|
|
- `machine_id`: random UUID persisted at `~/.config/lmcache/machine_id`.
|
|
Groups sessions from the same machine across restarts (deployment-level
|
|
dedup, restart cadence). Empty string when the file cannot be created.
|
|
- `schema_version`: bump `USAGE_SCHEMA_VERSION` whenever a field changes
|
|
meaning so backend analysis can partition by schema.
|
|
- `deployment_mode`: `single_process` or `mp_server` — which deployment
|
|
mode produced the payload. Needed because `EnvMessage` and the continuous
|
|
messages are shared between modes and are otherwise ambiguous.
|
|
|
|
Continuous messages additionally carry `sequence_number` (monotonic per
|
|
session). Send failures drop the interval's data rather than retrying —
|
|
telemetry must never accumulate unbounded state — so sequence gaps are the
|
|
backend's signal that intervals were lost rather than idle.
|
|
|
|
Privacy rules:
|
|
|
|
- Identifiers are random UUIDs, never derived from hardware (no MAC address,
|
|
hostname, or `/etc/machine-id`).
|
|
- The MP server's `instance_id` is deliberately excluded from payloads: it is
|
|
operator-settable (`--instance-id`) and may contain identifying strings.
|
|
It stays in the operator-facing OTel `service.instance.id` only.
|
|
- No prompts, keys, or KV-cache contents are ever sent. Model names are
|
|
reported (they identify public model architectures, not deployments).
|
|
|
|
## Opt-out semantics
|
|
|
|
`is_usage_tracking_enabled()` is the single gate, checked by every path:
|
|
|
|
1. `LMCACHE_TRACK_USAGE=false` (LMCache-specific),
|
|
2. `DO_NOT_TRACK` in `1`/`true`/`yes` (cross-tool convention).
|
|
|
|
When disabled, the factory functions return `None`, the continuous reporter
|
|
no-ops, and no state files (`machine_id`) are created.
|
|
|
|
## Transport and testing
|
|
|
|
`UsageMessageSender` is the injectable transport boundary. Tests inject a
|
|
recording stub into the usage contexts (see `tests/test_usage_telemetry.py`);
|
|
no test may hit the network. The test suite globally disables tracking via an
|
|
autouse fixture in `tests/conftest.py` so no other test phones home.
|
|
|
|
## Extension guide
|
|
|
|
To add a new message type:
|
|
|
|
1. Define a dataclass in `messages.py` extending `UsageMessage`, with flat
|
|
JSON-serializable fields and an `ENDPOINT` (the class name becomes the
|
|
wire `message_type`).
|
|
2. Send it through `build_usage_payload` so identity and schema stamping
|
|
stay uniform; wrap any new serving-code entry point with
|
|
`swallow_telemetry_errors`.
|
|
3. Coordinate with the stats-server owner: unknown `message_type` values are
|
|
dropped silently on the backend.
|
|
4. Bump `USAGE_SCHEMA_VERSION` only when an *existing* field changes meaning;
|
|
adding a message type or field is not a schema bump.
|