mirrors/sgl-project--sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
PR Test (NPU) / check-changes (push) Has been cancelled
Details
PR Test (NPU) / pr-gate (push) Has been cancelled
Details
PR Test (NPU) / set-image-config (push) Has been cancelled
Details
PR Test (NPU) / stage-b-test-1-npu-a2 (0) (push) Has been cancelled
Details
PR Test (NPU) / stage-b-test-1-npu-a2 (1) (push) Has been cancelled
Details
PR Test (NPU) / stage-b-test-2-npu-a2 (0) (push) Has been cancelled
Details
PR Test (NPU) / stage-b-test-2-npu-a2 (1) (push) Has been cancelled
Details
PR Test (NPU) / stage-b-test-4-npu-a3 (push) Has been cancelled
Details
PR Test (NPU) / stage-b-test-16-npu-a3 (push) Has been cancelled
Details
PR Test (NPU) / multimodal-gen-test-1-npu-a3 (push) Has been cancelled
Details
PR Test (NPU) / multimodal-gen-test-2-npu-a3 (push) Has been cancelled
Details
PR Test (Arm64) / pr-gate (push) Has been cancelled
Details
PR Test (Arm64) / check-changes (push) Has been cancelled
Details
PR Test (Arm64) / build-test (push) Has been cancelled
Details
PR Test (sgl-router) / gate (push) Has been cancelled
Details
PR Test (sgl-router) / tier-1 — lint (push) Has been cancelled
Details
PR Test (sgl-router) / tier-2 — build + test (push) Has been cancelled
Details
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Has been cancelled
Details
PR Test (sgl-router) / tier-3 — k8s integration (push) Has been cancelled
Details
PR Test (sgl-router) / tier-3 — e2e (push) Has been cancelled
Details
PR Test (sgl-router) / finish (push) Has been cancelled
Details
PR Test (NPU) / single-node-poc (map[name:qwen3_6_27b_w8a8_1p_in64k_out1k_50ms runner:linux-aarch64-a3-2 test_case:test/registered/ascend/performance/qwen3_6_27b/test_npu_qwen3_6_27b_w8a8_1p_in64k_out1k_50ms.py test_type:perf]) (push) Has been cancelled
Details
PR Test (NPU) / pr-test-npu-finish (push) Has been cancelled
Details
PR Test (Xeon) / pr-gate (push) Has been cancelled
Details
PR Test (Xeon) / check-changes (push) Has been cancelled
Details
PR Test (Xeon) / build-test (, xeon-gnr, base-b-test-cpu) (push) Has been cancelled
Details
PR Test (XPU) / check-changes (push) Has been cancelled
Details
PR Test (XPU) / pr-gate (push) Has been cancelled
Details
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Has been cancelled
Details
PR Test (XPU) / wait-for-stage-a (push) Has been cancelled
Details
PR Test (XPU) / stage-b-test-1-gpu-xpu (push) Has been cancelled
Details
PR Test (XPU) / finish (push) Has been cancelled
Details
CI Model Inventory / build-inventory (push) Has been cancelled
Details
Lint / lint (push) Has been cancelled
Details
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Has been cancelled
Details
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Has been cancelled
Details
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Has been cancelled
Details
PR Benchmark (SMG Components) / Benchmark Summary (push) Has been cancelled
Details
PR Test (SMG) / build-wheel (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on windows (x86_64 - auto) (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on macos (x86_64 - auto) (push) Has been cancelled
Details
PR Test (SMG) / python-unit-tests (push) Has been cancelled
Details
PR Test (SMG) / unit-tests (push) Has been cancelled
Details
PR Test (SMG) / benchmarks (push) Has been cancelled
Details
PR Test (SMG) / chat-completions (push) Has been cancelled
Details
PR Test (SMG) / chat-completions-4gpu (push) Has been cancelled
Details
PR Test (SMG) / e2e (push) Has been cancelled
Details
PR Test (SMG) / docker-build-test (push) Has been cancelled
Details
PR Test (SMG) / k8s-integration (push) Has been cancelled
Details
PR Test (SMG) / finish (push) Has been cancelled
Details
PR Test (SMG) / summarize-benchmarks (push) Has been cancelled
Details
Release SGLang Model Gateway Docker Image / publish (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on macos (aarch64 - auto) (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on linux (aarch64 - auto) (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on linux (x86_64 - auto) (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on linux (aarch64 - musllinux_1_1) (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / build on linux (x86_64 - musllinux_1_1) (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / Build SDist (push) Has been cancelled
Details
Release SGLang Model Gateway to PyPI / Upload to PyPI (push) Has been cancelled
Details
Release SGLang Kernels / build-cu129-matrix (aarch64, 12.9, 3.10, arm-kernel-build-node) (push) Has been cancelled
Details
Release SGLang Kernels / build-cu129-matrix (x86_64, 12.9, 3.10, x64-kernel-build-node) (push) Has been cancelled
Details
Release SGLang Kernels / release-cu129 (push) Has been cancelled
Details
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.10, arm-kernel-build-node) (push) Has been cancelled
Details
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.10, x64-kernel-build-node) (push) Has been cancelled
Details
Release SGLang Kernels / release-cu130 (push) Has been cancelled
Details
Release SGLang Kernels / build-rocm-matrix (3.10, 700) (push) Has been cancelled
Details
Release SGLang Kernels / build-rocm-matrix (3.10, 720) (push) Has been cancelled
Details
Release SGLang Kernels / release-rocm700 (push) Has been cancelled
Details
Release SGLang Kernels / release-rocm720 (push) Has been cancelled
Details
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Has been cancelled
Details
Release SGLang Kernels / release-musa43 (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:38:16 +08:00
commit 94057c3d3e
7152 changed files with 2120455 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/monitoring/README.md
+76
View File
@@ -0,0 +1,76 @@
# SGLang Monitoring Setup
This directory contains a ready-to-use monitoring setup for SGLang using Prometheus and Grafana.
## Prerequisites
- Docker and Docker Compose installed
- SGLang server running with metrics enabled
## Usage
1. Start your SGLang server with metrics enabled:
```bash
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000 --enable-metrics
```
By default, the metrics server will run on `127.0.0.1:30000`.
2. Start the monitoring stack:
```bash
cd examples/monitoring
docker compose up
```
3. Access the monitoring interfaces:
- Grafana: [http://localhost:3000](http://localhost:3000)
- Prometheus: [http://localhost:9090](http://localhost:9090)
Default Grafana login credentials:
- Username: `admin`
- Password: `admin`
You'll be prompted to change the password on first login.
4. The SGLang dashboard will be automatically available in the "SGLang Monitoring" folder.
## Troubleshooting
### Port Conflicts
If you see errors like "port is already allocated":
1. Check if you already have Prometheus or Grafana running:
```bash
docker ps | grep -E 'prometheus|grafana'
```
2. Stop any conflicting containers:
```bash
docker stop <container_id>
```
3. Ensure no other services are using ports 9090 and 3000:
```bash
lsof -i :9090
lsof -i :3000
```
### Connection Issues
If Grafana cannot connect to Prometheus:
1. Check that both services are running
2. Verify the datasource configuration in Grafana
3. Check that your SGLang server is properly exposing metrics
## Configuration
- Prometheus configuration: `prometheus.yaml`
- Docker Compose configuration: `docker-compose.yaml`
- Grafana datasource: `grafana/datasources/datasource.yaml`
- Grafana dashboard configuration: `grafana/dashboards/config/dashboard.yaml`
- SGLang dashboard JSON: `grafana/dashboards/json/sglang-dashboard.json`
## Customization
You can customize the monitoring setup by modifying the configuration files as needed.
examples/monitoring/docker-compose.yaml
+28
View File
@@ -0,0 +1,28 @@
version: '3'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
network_mode: host
volumes:
- ./prometheus.yaml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
grafana:
image: grafana/grafana:latest
container_name: grafana
network_mode: host
volumes:
- ./grafana/datasources:/etc/grafana/provisioning/datasources
- ./grafana/dashboards/config:/etc/grafana/provisioning/dashboards
- ./grafana/dashboards/json:/var/lib/grafana/dashboards
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
- GF_AUTH_BASIC_ENABLED=false
- GF_USERS_ALLOW_SIGN_UP=false
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/sglang-dashboard.json
depends_on:
- prometheus
examples/monitoring/grafana/dashboards/config/dashboard.yaml
+11
View File
@@ -0,0 +1,11 @@
apiVersion: 1
providers:
- name: 'SGLang'
orgId: 1
folder: 'SGLang Monitoring'
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: false
options:
path: /var/lib/grafana/dashboards
examples/monitoring/grafana/dashboards/json/sglang-dashboard.json
+984
View File
@@ -0,0 +1,984 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 8,
"links": [],
"panels": [
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 14,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (le) (rate(sglang_e2e_request_latency_seconds_bucket[$__rate_interval])))\r\n",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "P99",
"range": true,
"refId": "A",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "histogram_quantile(0.9, sum by (le) (rate(sglang_e2e_request_latency_seconds_bucket[$__rate_interval])))\r\n",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "P90",
"range": true,
"refId": "B",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "histogram_quantile(0.5, sum by (le) (rate(sglang_e2e_request_latency_seconds_bucket[$__rate_interval])))\r\n",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "P50",
"range": true,
"refId": "C",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "avg(rate(sglang_e2e_request_latency_seconds_sum[$__rate_interval]) / rate(sglang_e2e_request_latency_seconds_count[$__rate_interval]))\r\n",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "Avg",
"range": true,
"refId": "D",
"useBackend": false
}
],
"title": "End-to-End Request Latency",
"type": "timeseries"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"scaleDistribution": {
"type": "linear"
}
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 17,
"maxDataPoints": 30,
"options": {
"calculate": false,
"calculation": {
"yBuckets": {
"scale": {
"type": "linear"
}
}
},
"cellGap": 1,
"cellValues": {},
"color": {
"exponent": 0.5,
"fill": "dark-orange",
"mode": "scheme",
"reverse": false,
"scale": "exponential",
"scheme": "Spectral",
"steps": 64
},
"exemplars": {
"color": "rgba(255,0,255,0.7)"
},
"filterValues": {
"le": 1e-9
},
"legend": {
"show": true
},
"rowsFrame": {
"layout": "auto"
},
"tooltip": {
"mode": "single",
"showColorScale": true,
"yHistogram": false
},
"yAxis": {
"axisPlacement": "left",
"reverse": false,
"unit": "secs"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "sum(increase(sglang_e2e_request_latency_seconds_bucket{model_name=~\"$model_name\"}[$__rate_interval])) by (le)\r\n",
"format": "heatmap",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "{{le}}",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "End-to-End Request Latency(s) Heatmap",
"type": "heatmap"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 20,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum by (le) (rate(sglang_time_to_first_token_seconds_bucket[$__rate_interval])))\r\n",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "P99",
"range": true,
"refId": "A",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "histogram_quantile(0.9, sum by (le) (rate(sglang_time_to_first_token_seconds_bucket[$__rate_interval])))\r\n",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "P90",
"range": true,
"refId": "B",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "histogram_quantile(0.5, sum by (le) (rate(sglang_time_to_first_token_seconds_bucket[$__rate_interval])))\r\n",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "P50",
"range": true,
"refId": "C",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "avg(rate(sglang_time_to_first_token_seconds_sum[$__rate_interval]) / rate(sglang_time_to_first_token_seconds_count[$__rate_interval]))\r\n",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "Avg",
"range": true,
"refId": "D",
"useBackend": false
}
],
"title": "Time-To-First-Token Latency",
"type": "timeseries"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"scaleDistribution": {
"type": "linear"
}
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 19,
"maxDataPoints": 30,
"options": {
"calculate": false,
"calculation": {
"xBuckets": {
"value": ""
},
"yBuckets": {
"mode": "size",
"scale": {
"type": "linear"
},
"value": ""
}
},
"cellGap": 1,
"color": {
"exponent": 0.5,
"fill": "dark-orange",
"mode": "scheme",
"reverse": false,
"scale": "exponential",
"scheme": "Spectral",
"steps": 64
},
"exemplars": {
"color": "rgba(255,0,255,0.7)"
},
"filterValues": {
"le": 1e-9
},
"legend": {
"show": true
},
"rowsFrame": {
"layout": "auto"
},
"tooltip": {
"mode": "single",
"showColorScale": true,
"yHistogram": false
},
"yAxis": {
"axisPlacement": "left",
"reverse": false
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "sum by(le) (increase(sglang_time_to_first_token_seconds_bucket{model_name=~\"$model_name\"}[$__rate_interval]))",
"format": "heatmap",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"interval": "",
"legendFormat": "{{le}}",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "Time-To-First-Token Seconds Heatmap",
"type": "heatmap"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"id": 7,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sglang_num_running_reqs",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "Num Running Requests",
"type": "timeseries"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"id": 18,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"editorMode": "code",
"expr": "sglang_gen_throughput",
"instant": false,
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Token Generation Throughput (Tokens / S)",
"type": "timeseries"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 24
},
"id": 11,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sglang_cache_hit_rate",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "{{instance}}",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "Cache Hit Rate",
"type": "timeseries"
},
{
"datasource": {
"default": true,
"type": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"id": 8,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "ddyfngn31dg5cf"
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sglang_num_queue_reqs",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
"legendFormat": "{{instance}}",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "Number Queued Requests",
"type": "timeseries"
}
],
"preload": false,
"refresh": "5s",
"schemaVersion": 41,
"tags": [],
"templating": {
"list": [
{
"current": {
"text": "127.0.0.1:30000",
"value": "127.0.0.1:30000"
},
"datasource": {
"type": "prometheus"
},
"definition": "label_values(instance)",
"includeAll": false,
"label": "instance",
"name": "instance",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(instance)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": "meta-llama/Llama-3.1-8B-Instruct",
"value": "meta-llama/Llama-3.1-8B-Instruct"
},
"datasource": {
"type": "prometheus"
},
"definition": "label_values(model_name)",
"includeAll": false,
"label": "model name",
"name": "model_name",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(model_name)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
}
]
},
"time": {
"from": "now-30m",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "SGLang Dashboard",
"uid": "sglang-dashboard",
"version": 11
}
examples/monitoring/grafana/datasources/datasource.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://localhost:9090
isDefault: true
editable: false
examples/monitoring/opentelemetry.yaml
+38
View File
@@ -0,0 +1,38 @@
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
otlp:
endpoint: jaeger:4317
tls:
insecure: true
file:
path: /tmp/otel_trace.json
extensions:
health_check:
pprof:
zpages:
service:
extensions: [health_check, pprof, zpages]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp, file]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
examples/monitoring/prometheus.yaml
+10
View File
@@ -0,0 +1,10 @@
# prometheus.yaml
global:
scrape_interval: 5s
evaluation_interval: 30s
scrape_configs:
- job_name: sglang
static_configs:
- targets:
- '127.0.0.1:30000'
examples/monitoring/tracing_compose.yaml
+21
View File
@@ -0,0 +1,21 @@
services:
otel-collector:
image: docker.io/otel/opentelemetry-collector
volumes:
- ./opentelemetry.yaml:/etc/otelcol/config.yaml
- /tmp:/tmp
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
depends_on:
- jaeger
restart: unless-stopped
jaeger:
image: jaegertracing/all-in-one
container_name: jaeger
ports:
- "16686:16686"
environment:
- COLLECTOR_OTLP_ENABLED=true
restart: unless-stopped