wehub-resource-sync
181 lines
4.0 KiB
ReStructuredText
181 lines
4.0 KiB
ReStructuredText
.. _observability_periodic_thread_api:
|
|
|
|
Periodic Thread Monitoring API
|
|
==============================
|
|
|
|
.. warning::
|
|
|
|
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
|
|
|
|
|
This document describes the Periodic Thread Monitoring API feature introduced
|
|
in LMCache.
|
|
|
|
Overview
|
|
--------
|
|
|
|
The Periodic Thread Monitoring API provides HTTP endpoints to monitor and
|
|
inspect the status of all periodic background threads running in the LMCache
|
|
system. This is useful for debugging, health checking, and operational
|
|
monitoring.
|
|
|
|
API Endpoints
|
|
-------------
|
|
|
|
Three API endpoints are available under the internal API server:
|
|
|
|
GET /periodic-threads
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Returns information about all registered periodic threads.
|
|
|
|
**Query Parameters:**
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 20 15 65
|
|
|
|
* - Parameter
|
|
- Default
|
|
- Description
|
|
* - ``level``
|
|
- (none)
|
|
- Filter by thread level (``critical``, ``high``, ``medium``, ``low``)
|
|
* - ``running_only``
|
|
- ``false``
|
|
- Only show running threads
|
|
* - ``active_only``
|
|
- ``false``
|
|
- Only show active threads
|
|
|
|
**Response Example:**
|
|
|
|
.. code-block:: json
|
|
|
|
{
|
|
"summary": {
|
|
"total_count": 5,
|
|
"running_count": 3,
|
|
"active_count": 3,
|
|
"by_level": {
|
|
"critical": {"total": 1, "running": 1, "active": 1},
|
|
"high": {"total": 2, "running": 1, "active": 1},
|
|
"medium": {"total": 1, "running": 1, "active": 1},
|
|
"low": {"total": 1, "running": 0, "active": 0}
|
|
}
|
|
},
|
|
"threads": [
|
|
{
|
|
"name": "pin_monitor",
|
|
"level": "high",
|
|
"is_running": true,
|
|
"is_active": true,
|
|
"last_run_time": 1706000000.0,
|
|
"last_run_ago": "2.5s"
|
|
}
|
|
]
|
|
}
|
|
|
|
GET /periodic-threads/{thread_name}
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Returns detailed information about a specific periodic thread.
|
|
|
|
**Response Example:**
|
|
|
|
.. code-block:: json
|
|
|
|
{
|
|
"name": "pin_monitor",
|
|
"level": "high",
|
|
"is_running": true,
|
|
"is_active": true,
|
|
"last_run_time": 1706000000.0,
|
|
"last_run_ago": "2.5s",
|
|
"interval": 1.0,
|
|
"total_runs": 100,
|
|
"failed_runs": 0
|
|
}
|
|
|
|
GET /periodic-threads-health
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Quick health check for periodic threads. Returns whether all critical
|
|
and high-level threads are active.
|
|
|
|
**Response Example:**
|
|
|
|
.. code-block:: json
|
|
|
|
{
|
|
"healthy": true,
|
|
"unhealthy_count": 0,
|
|
"unhealthy_threads": []
|
|
}
|
|
|
|
Thread Levels
|
|
-------------
|
|
|
|
Periodic threads are categorized by importance level:
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 20 80
|
|
|
|
* - Level
|
|
- Description
|
|
* - ``critical``
|
|
- Essential for system operation
|
|
* - ``high``
|
|
- Important for performance
|
|
* - ``medium``
|
|
- Standard background tasks
|
|
* - ``low``
|
|
- Optional/auxiliary tasks
|
|
|
|
The health check endpoint specifically monitors ``critical`` and ``high``
|
|
level threads to determine system health.
|
|
|
|
IrrecoverableException Handling
|
|
-------------------------------
|
|
|
|
Periodic threads now properly handle ``IrrecoverableException``. When such an
|
|
exception is raised during thread execution:
|
|
|
|
- The exception is logged with full traceback
|
|
- The thread run is marked as failed
|
|
- The thread **stops its execution loop** instead of continuing
|
|
|
|
This prevents threads from endlessly retrying operations that cannot succeed.
|
|
|
|
Usage Examples
|
|
--------------
|
|
|
|
Check Overall Thread Health
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: bash
|
|
|
|
curl http://localhost:8080/periodic-threads-health
|
|
|
|
List All Running Threads
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: bash
|
|
|
|
curl "http://localhost:8080/periodic-threads?running_only=true"
|
|
|
|
Get Critical Threads Only
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: bash
|
|
|
|
curl "http://localhost:8080/periodic-threads?level=critical"
|
|
|
|
Check Specific Thread Status
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. code-block:: bash
|
|
|
|
curl http://localhost:8080/periodic-threads/pin_monitor
|