Files
comet-ml--opik/sdks/python/examples/dashboard_example.ipynb
T
wehub-resource-sync 5a558eb09e
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
chore: import upstream snapshot with attribution
2026-07-13 13:25:44 +08:00

656 lines
22 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"cells": [
{
"cell_type": "markdown",
"id": "cell0",
"metadata": {},
"source": "# Opik Dashboards — Python SDK\n\nA comprehensive walkthrough of the **dashboard** API on the `opik.Opik` client:\n\n| Step | Topic |\n| --- | --- |\n| 1 | Setup |\n| 23 | Create a `MULTI_PROJECT` dashboard scoped to a project |\n| 4 | Stats-card widgets (snapshot metrics) |\n| 5 | Time-series chart widgets |\n| 6 | Markdown / notes widget |\n| 7 | Update widgets |\n| 8 | Inspect and rearrange the grid layout |\n| 9 | Add sections and move widgets between sections |\n| 10 | Remove widgets |\n| 1112 | Create an `EXPERIMENTS` dashboard with evaluation widgets |\n| 13 | Fetch and list dashboards |\n| 14 | Clean up |\n\n**Project scope** — `project_stats_card` and `project_metrics` widgets are project-scoped.\nPass `project_name` to `create_dashboard` once; the SDK automatically injects the project\ninto every project-scoped widget added via `add_widget`.\n\n**Metric-ID namespaces** — easy to mix up:\n\n| Widget | Field | Namespace | Example |\n| --- | --- | --- | --- |\n| `project_stats_card` | `metric` | lowercase-dotted | `trace_count`, `duration.p50` |\n| `project_metrics` | `metric_type` | ALL-CAPS | `TRACE_COUNT`, `DURATION` |"
},
{
"cell_type": "markdown",
"id": "cell1",
"metadata": {},
"source": [
"## 1 · Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell2",
"metadata": {},
"outputs": [],
"source": [
"%pip install opik --quiet"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell3",
"metadata": {},
"outputs": [],
"source": [
"import copy\n",
"\n",
"import opik\n",
"from opik import dashboard\n",
"\n",
"client = opik.Opik()\n",
"\n",
"PROJECT_NAME = \"Default Project\""
]
},
{
"cell_type": "markdown",
"id": "cell4",
"metadata": {},
"source": [
"## 2 · Create a MULTI_PROJECT dashboard\n",
"\n",
"`MULTI_PROJECT` dashboards support `project_stats_card`, `project_metrics`, and `text_markdown`\n",
"widgets. A new dashboard starts with a single *Overview* section whose `id` we capture for\n",
"adding widgets."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell5",
"metadata": {},
"outputs": [],
"source": [
"mp_dash = client.create_dashboard(\n",
" name=\"SDK comprehensive demo\",\n",
" type=dashboard.DashboardType.MULTI_PROJECT,\n",
" description=\"Created from the Python SDK walkthrough\",\n",
" project_name=PROJECT_NAME,\n",
")\n",
"mp_section_id = mp_dash.sections[0].id\n",
"print(f\"Dashboard id : {mp_dash.id}\")\n",
"print(f\"Type : {mp_dash.type}\")\n",
"print(f\"Scope : {mp_dash.scope}\")\n",
"print(f\"Section id : {mp_section_id}\")"
]
},
{
"cell_type": "markdown",
"id": "cell6",
"metadata": {},
"source": "## 3 · Project scope\n\nThe `project_name` passed to `create_dashboard` links the dashboard to a project.\nThe SDK then automatically injects the project into every project-scoped widget\n(`project_stats_card`, `project_metrics`) when you call `add_widget` — you do **not**\nneed to repeat the project in the widget config."
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell7",
"metadata": {},
"outputs": [],
"source": [
"print(f\"Dashboard linked to project: {PROJECT_NAME!r}\")"
]
},
{
"cell_type": "markdown",
"id": "cell8",
"metadata": {},
"source": [
"## 4 · Stats-card widgets\n",
"\n",
"`project_stats_card` shows a **single current-value metric** for a project.\n",
"The `metric` field uses the **lowercase-dotted** namespace — see `dashboard.StatsCardMetric`\n",
"for the full list (trace counts, duration percentiles, token usage, costs, …).\n",
"\n",
"`source` selects whether the metric is computed over traces or spans."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell9",
"metadata": {},
"outputs": [],
"source": [
"# Total trace count\n",
"sc_trace_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
" title=\"Traces\",\n",
" config=dashboard.ProjectStatsCardConfig(\n",
" source=dashboard.TraceDataType.TRACES,\n",
" metric=dashboard.StatsCardMetric.TRACE_COUNT,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Estimated total cost\n",
"sc_cost_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
" title=\"Total cost\",\n",
" config=dashboard.ProjectStatsCardConfig(\n",
" source=dashboard.TraceDataType.TRACES,\n",
" metric=dashboard.StatsCardMetric.TOTAL_ESTIMATED_COST_SUM,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Median latency (p50)\n",
"sc_p50_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
" title=\"Latency p50\",\n",
" config=dashboard.ProjectStatsCardConfig(\n",
" source=dashboard.TraceDataType.TRACES,\n",
" metric=dashboard.StatsCardMetric.DURATION_P50,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# LLM span count (source=SPANS to query span-level metrics)\n",
"sc_llm_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
" title=\"LLM calls\",\n",
" config=dashboard.ProjectStatsCardConfig(\n",
" source=dashboard.TraceDataType.SPANS,\n",
" metric=dashboard.StatsCardMetric.LLM_SPAN_COUNT,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"print(f\"Stats cards added, total widgets: {len(mp_dash.sections[0].widgets)}\")"
]
},
{
"cell_type": "markdown",
"id": "cell10",
"metadata": {},
"source": [
"## 5 · Time-series chart widgets\n",
"\n",
"`project_metrics` renders a time-series for an aggregate metric.\n",
"The `metric_type` field uses the **ALL-CAPS** namespace — see `dashboard.ProjectMetricType`.\n",
"\n",
"Breakdowns split the series by a dimension: `MODEL`, `PROVIDER`, `TAGS`, `NAME`, etc.\n",
"Available chart types: `LINE` (default), `BAR`, `RADAR`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell11",
"metadata": {},
"outputs": [],
"source": [
"# Line chart: duration over time, broken down by model\n",
"chart_duration_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
" title=\"Duration by model\",\n",
" config=dashboard.ProjectMetricsConfig(\n",
" metric_type=dashboard.ProjectMetricType.DURATION,\n",
" chart_type=dashboard.ChartType.LINE,\n",
" breakdown=dashboard.BreakdownConfig(field=dashboard.BreakdownField.MODEL),\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Bar chart: token usage over time\n",
"chart_tokens_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
" title=\"Token usage\",\n",
" config=dashboard.ProjectMetricsConfig(\n",
" metric_type=dashboard.ProjectMetricType.TOKEN_USAGE,\n",
" chart_type=dashboard.ChartType.BAR,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Line chart: trace count broken down by tag\n",
"chart_count_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
" title=\"Trace count by tag\",\n",
" config=dashboard.ProjectMetricsConfig(\n",
" metric_type=dashboard.ProjectMetricType.TRACE_COUNT,\n",
" chart_type=dashboard.ChartType.LINE,\n",
" breakdown=dashboard.BreakdownConfig(field=dashboard.BreakdownField.TAGS),\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Line chart: estimated cost broken down by provider\n",
"chart_cost_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
" title=\"Cost by provider\",\n",
" config=dashboard.ProjectMetricsConfig(\n",
" metric_type=dashboard.ProjectMetricType.COST,\n",
" chart_type=dashboard.ChartType.LINE,\n",
" breakdown=dashboard.BreakdownConfig(\n",
" field=dashboard.BreakdownField.PROVIDER\n",
" ),\n",
" ),\n",
" ),\n",
")\n",
"\n",
"print(f\"Total widgets: {len(mp_dash.sections[0].widgets)}\")"
]
},
{
"cell_type": "markdown",
"id": "cell12",
"metadata": {},
"source": [
"## 6 · Markdown / notes widget\n",
"\n",
"`text_markdown` renders freeform Markdown — useful for section headers, runbook links,\n",
"or context notes. It is valid in **both** `MULTI_PROJECT` and `EXPERIMENTS` dashboards.\n",
"\n",
"Widgets can also be created from a **raw dict**, which is the forward-compatible path\n",
"for backend fields not yet modelled in the SDK."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell13",
"metadata": {},
"outputs": [],
"source": [
"# Typed config\n",
"notes_id = mp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.TEXT_MARKDOWN,\n",
" title=\"\",\n",
" config=dashboard.TextMarkdownConfig(\n",
" content=(\n",
" \"## Project overview\\n\"\n",
" \"This dashboard tracks **Default Project** metrics.\\n\\n\"\n",
" \"- Duration p50 / p90\\n\"\n",
" \"- Token costs by provider\\n\"\n",
" \"- Error rate\"\n",
" )\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Raw-dict style: forward-compatible with new backend fields\n",
"raw_id = mp_dash.add_widget(\n",
" {\n",
" \"type\": dashboard.WidgetType.TEXT_MARKDOWN.value,\n",
" \"title\": \"Raw dict widget\",\n",
" \"config\": {\"content\": \"Built with the `opik` Python SDK.\"},\n",
" },\n",
")\n",
"\n",
"print(f\"Total widgets: {len(mp_dash.sections[0].widgets)}\")\n",
"print(f\"Notes id : {notes_id}\")\n",
"print(f\"Raw id : {raw_id}\")"
]
},
{
"cell_type": "markdown",
"id": "cell14",
"metadata": {},
"source": [
"## 7 · Update widgets\n",
"\n",
"`update_widget` patches **only the fields you pass** — omitted kwargs are left unchanged.\n",
"Config is **merged**, not replaced, so you can change a single key without restating the\n",
"whole config object."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell15",
"metadata": {},
"outputs": [],
"source": [
"# Change the chart title\n",
"mp_dash.update_widget(chart_duration_id, title=\"Duration by model (ms)\")\n",
"\n",
"# Swap the markdown note content (config merge)\n",
"mp_dash.update_widget(\n",
" notes_id,\n",
" config={\"content\": \"## Project overview (updated)\\nDashboard refreshed via SDK.\"},\n",
")\n",
"\n",
"# Add a subtitle to the trace-count stats card\n",
"mp_dash.update_widget(sc_trace_id, subtitle=\"last 7 days\")\n",
"\n",
"# Switch the token-usage chart from BAR to LINE\n",
"mp_dash.update_widget(\n",
" chart_tokens_id,\n",
" config=dashboard.ProjectMetricsConfig(\n",
" metric_type=dashboard.ProjectMetricType.TOKEN_USAGE,\n",
" chart_type=dashboard.ChartType.LINE,\n",
" ),\n",
")\n",
"\n",
"# Rename the dashboard and update its description\n",
"mp_dash.rename(\"SDK comprehensive demo (v2)\")\n",
"mp_dash.set_description(\"Updated via the Python SDK.\")\n",
"print(\"Name:\", mp_dash.name)"
]
},
{
"cell_type": "markdown",
"id": "cell16",
"metadata": {},
"source": [
"## 8 · Inspect and rearrange the grid layout\n",
"\n",
"The grid is **6 columns wide** with unlimited rows. Each widget has a `DashboardLayoutItem`\n",
"with `x` (column), `y` (row), `w` (width in columns), `h` (height in rows).\n",
"\n",
"`replace_sections` swaps the entire sections list in one call — use it to reposition\n",
"widgets, resize them, or reorder sections. All other mutators persist immediately after\n",
"each individual call."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell17",
"metadata": {},
"outputs": [],
"source": [
"section = mp_dash.sections[0]\n",
"by_id = {w.id: w for w in section.widgets}\n",
"\n",
"print(f\"{'title':35s} x y w h\")\n",
"print(\"-\" * 50)\n",
"for li in section.layout:\n",
" title = by_id[li.id].title or \"(no title)\"\n",
" print(f\"{title:35s} {li.x} {li.y} {li.w} {li.h}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell18",
"metadata": {},
"outputs": [],
"source": [
"# Rearrange: full-width notes banner at the top (row 0),\n",
"# four stats cards side-by-side below (row 2),\n",
"# charts below that (rows 4+).\n",
"new_section = copy.deepcopy(section)\n",
"\n",
"stats_ids = [sc_trace_id, sc_cost_id, sc_p50_id, sc_llm_id]\n",
"chart_ids = [chart_duration_id, chart_tokens_id, chart_count_id, chart_cost_id]\n",
"\n",
"for li in new_section.layout:\n",
" if li.id == notes_id:\n",
" # Full-width banner spanning all 6 columns\n",
" li.x, li.y, li.w, li.h = 0, 0, 6, 2\n",
" elif li.id == raw_id:\n",
" # Small note pinned to the top-right\n",
" li.x, li.y, li.w, li.h = 4, 2, 2, 2\n",
" elif li.id in stats_ids:\n",
" col = stats_ids.index(li.id)\n",
" li.x, li.y, li.w, li.h = col, 2, 1, 2\n",
" elif li.id in chart_ids:\n",
" col = chart_ids.index(li.id)\n",
" li.x, li.y, li.w, li.h = (col % 3) * 2, 4 + (col // 3) * 4, 2, 4\n",
"\n",
"mp_dash.replace_sections([new_section])\n",
"\n",
"print(\"Layout after rearrangement:\")\n",
"section = mp_dash.sections[0]\n",
"by_id = {w.id: w for w in section.widgets}\n",
"print(f\"{'title':35s} x y w h\")\n",
"print(\"-\" * 50)\n",
"for li in section.layout:\n",
" title = by_id[li.id].title or \"(no title)\"\n",
" print(f\"{title:35s} {li.x} {li.y} {li.w} {li.h}\")"
]
},
{
"cell_type": "markdown",
"id": "cell19",
"metadata": {},
"source": [
"## 9 · Add sections and move widgets\n",
"\n",
"`add_section` appends a new empty section. \n",
"To move widgets between sections use `replace_sections` with the complete new state."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell20",
"metadata": {},
"outputs": [],
"source": [
"analytics_section_id = mp_dash.add_section(\"Analytics\")\n",
"print(\"Sections:\", [s.title for s in mp_dash.sections])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell21",
"metadata": {},
"outputs": [],
"source": [
"# Move the four chart widgets from Overview into the new Analytics section.\n",
"new_sections = [copy.deepcopy(s) for s in mp_dash.sections]\n",
"overview, analytics = new_sections\n",
"\n",
"move_ids = {chart_duration_id, chart_tokens_id, chart_count_id, chart_cost_id}\n",
"\n",
"# Extract chart widgets and their layout entries from Overview\n",
"moved_widgets = [w for w in overview.widgets if w.id in move_ids]\n",
"moved_layout = [li for li in overview.layout if li.id in move_ids]\n",
"\n",
"overview.widgets = [w for w in overview.widgets if w.id not in move_ids]\n",
"overview.layout = [li for li in overview.layout if li.id not in move_ids]\n",
"\n",
"# Re-position charts inside Analytics (2-wide, 4-tall, three per row)\n",
"for idx, li in enumerate(moved_layout):\n",
" li.x, li.y, li.w, li.h = (idx % 3) * 2, (idx // 3) * 4, 2, 4\n",
"\n",
"analytics.widgets.extend(moved_widgets)\n",
"analytics.layout.extend(moved_layout)\n",
"\n",
"mp_dash.replace_sections(new_sections)\n",
"\n",
"for s in mp_dash.sections:\n",
" print(f\" [{s.title}] {len(s.widgets)} widget(s)\")"
]
},
{
"cell_type": "markdown",
"id": "cell22",
"metadata": {},
"source": [
"## 10 · Remove widgets\n",
"\n",
"`remove_widget` removes a widget and its layout entry from whichever section contains it.\n",
"Raises `DashboardValidationError` if the ID is not found."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell23",
"metadata": {},
"outputs": [],
"source": [
"# Remove the raw-dict markdown widget\n",
"mp_dash.remove_widget(raw_id)\n",
"\n",
"total = sum(len(s.widgets) for s in mp_dash.sections)\n",
"print(f\"Widgets after removal: {total}\")"
]
},
{
"cell_type": "markdown",
"id": "cell24",
"metadata": {},
"source": [
"## 11 · EXPERIMENTS dashboard\n",
"\n",
"`EXPERIMENTS` dashboards target evaluation results rather than live traces.\n",
"Supported widgets: `experiments_feedback_scores`, `experiment_leaderboard`, `text_markdown`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell25",
"metadata": {},
"outputs": [],
"source": [
"exp_dash = client.create_dashboard(\n",
" name=\"SDK experiments demo\",\n",
" type=dashboard.DashboardType.EXPERIMENTS,\n",
" description=\"Evaluation metrics overview\",\n",
")\n",
"exp_section_id = exp_dash.sections[0].id\n",
"print(f\"Experiments dashboard id: {exp_dash.id}\")"
]
},
{
"cell_type": "markdown",
"id": "cell26",
"metadata": {},
"source": [
"## 12 · Experiments evaluation widgets\n",
"\n",
"`experiments_feedback_scores` plots feedback score distributions across experiments. \n",
"`experiment_leaderboard` shows a ranked table of runs against a chosen metric.\n",
"\n",
"Pass `max_experiments_count` (1100) to control how many recent experiments are included."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell27",
"metadata": {},
"outputs": [],
"source": [
"# Bar chart: feedback scores across the last 10 experiments\n",
"fb_bar_id = exp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.EXPERIMENTS_FEEDBACK_SCORES,\n",
" title=\"Feedback scores (bar)\",\n",
" config=dashboard.ExperimentsFeedbackScoresConfig(\n",
" chart_type=dashboard.ChartType.BAR,\n",
" max_experiments_count=10,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Radar chart: quality shape across the last 5 experiments\n",
"fb_radar_id = exp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.EXPERIMENTS_FEEDBACK_SCORES,\n",
" title=\"Feedback scores (radar)\",\n",
" config=dashboard.ExperimentsFeedbackScoresConfig(\n",
" chart_type=dashboard.ChartType.RADAR,\n",
" max_experiments_count=5,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Leaderboard with ranking enabled by a specific feedback-score metric\n",
"leaderboard_id = exp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.EXPERIMENT_LEADERBOARD,\n",
" title=\"Experiment leaderboard\",\n",
" config=dashboard.ExperimentLeaderboardConfig(\n",
" enable_ranking=True,\n",
" ranking_metric=\"hallucination\", # name of the feedback score to rank by\n",
" ranking_direction=True, # True = descending (higher score is better)\n",
" selected_columns=[\"dataset_id\", \"created_at\", \"duration.p50\", \"pass_rate\"],\n",
" max_rows=20,\n",
" ),\n",
" ),\n",
")\n",
"\n",
"# Context note\n",
"exp_dash.add_widget(\n",
" dashboard.DashboardWidget(\n",
" type=dashboard.WidgetType.TEXT_MARKDOWN,\n",
" title=\"\",\n",
" config=dashboard.TextMarkdownConfig(\n",
" content=\"### About\\nTracks evaluation runs ranked by the **hallucination** metric.\"\n",
" ),\n",
" ),\n",
")\n",
"\n",
"print(f\"Experiments dashboard widgets: {len(exp_dash.sections[0].widgets)}\")"
]
},
{
"cell_type": "markdown",
"id": "cell28",
"metadata": {},
"source": [
"## 13 · Fetch and list dashboards\n",
"\n",
"`get_dashboard` retrieves a single dashboard by ID (re-fetches from the backend). \n",
"`get_dashboards` pages through all dashboards with an optional name filter."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell29",
"metadata": {},
"outputs": [],
"source": [
"# Fetch the multi-project dashboard by id\n",
"fetched_mp = client.get_dashboard(mp_dash.id)\n",
"print(f\"Fetched: {fetched_mp.name!r} ({len(fetched_mp.sections)} section(s))\")\n",
"\n",
"# List all dashboards whose name contains \"SDK\"\n",
"found = client.get_dashboards(name=\"SDK\", max_results=20)\n",
"print(f\"\\nDashboards matching 'SDK' ({len(found)} found):\")\n",
"for d in found:\n",
" print(f\" {d.id[:8]}… {d.type:15s} {d.name!r}\")"
]
},
{
"cell_type": "markdown",
"id": "cell32",
"metadata": {},
"source": [
"## 14 · Clean up"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell33",
"metadata": {},
"outputs": [],
"source": [
"mp_dash.delete()\n",
"exp_dash.delete()\n",
"print(\"Both dashboards deleted.\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.19"
}
},
"nbformat": 4,
"nbformat_minor": 5
}