lmcache--lmcache/docs/design/v1/mp_observability/errors.md

Observability-aware errors (errors.py)

lmcache/v1/mp_observability/errors.py defines LMCacheTimeoutError, a timeout exception that reports itself to the MP observability EventBus when it is constructed. It is the single place timeouts in lmcache/ should be raised from; the ban-raw-timeout-error pre-commit hook enforces this by rejecting any bare raise TimeoutError(...) / raise asyncio.TimeoutError(...) under lmcache/.

Why

Timeouts are the most common "something is stuck" failure in MP mode (MQ round-trips, GPU transfer waits, adapter drains, NIXL handshakes). Before this, each site raised the built-in TimeoutError, which left no trace in the observability stack — operators only learned of a timeout if the surrounding code happened to log it. Routing every timeout through one class makes them uniformly observable (counter + log + trace) without touching each call site beyond the class swap.

Contract

LMCacheTimeoutError(message: str, *, session_id: str = "")

• Subclass of the built-in TimeoutError. Every existing except TimeoutError handler continues to catch it unchanged, so swapping a raw raise TimeoutError(...) for raise LMCacheTimeoutError(...) is behaviour-preserving. (On Python 3.11+ asyncio.TimeoutError is the same type, so except asyncio.TimeoutError catches it too; on 3.10 it does not — do not rely on the async alias for catching these raises.)
• Emits on construction, not on raise. __init__ publishes one EventType.TIMEOUT_RAISED event (see EVENTS.md) to the global EventBus via get_event_bus().
• Zero-cost when observability is off. The emit path is guarded by is_observability_enabled(), which is only True inside the MP server process. In single-process / CLI mode the constructor does nothing beyond super().__init__(message) (one boolean check) — no event, no stack-trace capture, no OTel dependency exercised.
• Never raises from __init__. Any failure to publish is swallowed and logged at debug level: observability must never break error handling.
• session_id is forwarded onto the event so the timeout span can nest under the originating request's root span. Pass it where the raise site has a request/IPCCacheServerKey.request_id in scope (e.g. shm.prepare_store); leave it empty otherwise.

What gets recorded

The emitted TIMEOUT_RAISED event carries message, exception_type, and a captured stacktrace (the construction stack minus the __init__ frame, following the OTel exception.* semantic conventions). Three subscribers consume it, registered by init_observability under the usual metrics/logging/tracing toggles:

Subscriber	Output	Default
TimeoutMetricsSubscriber	lmcache_mp.timeouts counter, tagged exception_type	on (metrics_enabled)
TimeoutLoggingSubscriber	WARNING log with message + stack trace	on (logging_enabled)
TimeoutTracingSubscriber	zero-duration timeout span with an exception event + ERROR status	on when tracing enabled

The tracing subscriber records the exception the same way OTel's Span.record_exception would (an exception span event with exception.type / exception.message / exception.stacktrace plus ERROR status), driven from the EventBus drain thread where the original exception object is no longer available.

Extending

To make another timeout observable, raise LMCacheTimeoutError instead of the built-in. To add a new observable error family, define a sibling subclass of the relevant built-in here, add a matching EventType, and add subscriber(s) that consume it — mirroring the timeout wiring.