5a558eb09e
Backend Tests / discover-tests (push) Waiting to run
Backend Tests / ${{ matrix.name }} (push) Blocked by required conditions
Build and Publish SDK / build-and-publish (push) Waiting to run
Build Opik Docker Images / set-version (push) Waiting to run
Build Opik Docker Images / build-backend (push) Blocked by required conditions
Build Opik Docker Images / build-sandbox-executor-python (push) Blocked by required conditions
Build Opik Docker Images / build-python-backend (push) Blocked by required conditions
Build Opik Docker Images / build-frontend (push) Blocked by required conditions
Build Opik Docker Images / create-git-tag (push) Blocked by required conditions
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Waiting to run
Docs - Publish / run (push) Waiting to run
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Waiting to run
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Blocked by required conditions
Frontend Unit Tests / Test on Node 20 (push) Waiting to run
Guardrails E2E Tests / Select Python version matrix (push) Waiting to run
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Blocked by required conditions
Guardrails E2E Tests / 📢 Slack Notification (push) Blocked by required conditions
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Waiting to run
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Blocked by required conditions
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Waiting to run
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Waiting to run
Lint Opik Helm Chart / unittest-helm-chart (push) Waiting to run
Lint Opik Helm Chart / render-equality (push) Waiting to run
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Waiting to run
Python BE E2E Tests / Python BE E2E (push) Waiting to run
Python Backend Tests / run-python-backend-tests (push) Waiting to run
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Waiting to run
Release Drafter / update_release_draft (push) Waiting to run
SDK E2E Libraries Integration Tests / Check Secrets (push) Waiting to run
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Blocked by required conditions
SDK E2E Libraries Integration Tests / build-opik (push) Blocked by required conditions
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Blocked by required conditions
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Waiting to run
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Waiting to run
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Waiting to run
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Waiting to run
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Waiting to run
TypeScript SDK Build & Publish / build-and-publish (push) Waiting to run
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Waiting to run
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
656 lines
22 KiB
Plaintext
656 lines
22 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell0",
|
||
"metadata": {},
|
||
"source": "# Opik Dashboards — Python SDK\n\nA comprehensive walkthrough of the **dashboard** API on the `opik.Opik` client:\n\n| Step | Topic |\n| --- | --- |\n| 1 | Setup |\n| 2–3 | Create a `MULTI_PROJECT` dashboard scoped to a project |\n| 4 | Stats-card widgets (snapshot metrics) |\n| 5 | Time-series chart widgets |\n| 6 | Markdown / notes widget |\n| 7 | Update widgets |\n| 8 | Inspect and rearrange the grid layout |\n| 9 | Add sections and move widgets between sections |\n| 10 | Remove widgets |\n| 11–12 | Create an `EXPERIMENTS` dashboard with evaluation widgets |\n| 13 | Fetch and list dashboards |\n| 14 | Clean up |\n\n**Project scope** — `project_stats_card` and `project_metrics` widgets are project-scoped.\nPass `project_name` to `create_dashboard` once; the SDK automatically injects the project\ninto every project-scoped widget added via `add_widget`.\n\n**Metric-ID namespaces** — easy to mix up:\n\n| Widget | Field | Namespace | Example |\n| --- | --- | --- | --- |\n| `project_stats_card` | `metric` | lowercase-dotted | `trace_count`, `duration.p50` |\n| `project_metrics` | `metric_type` | ALL-CAPS | `TRACE_COUNT`, `DURATION` |"
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell1",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1 · Setup"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell2",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install opik --quiet"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell3",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import copy\n",
|
||
"\n",
|
||
"import opik\n",
|
||
"from opik import dashboard\n",
|
||
"\n",
|
||
"client = opik.Opik()\n",
|
||
"\n",
|
||
"PROJECT_NAME = \"Default Project\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell4",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2 · Create a MULTI_PROJECT dashboard\n",
|
||
"\n",
|
||
"`MULTI_PROJECT` dashboards support `project_stats_card`, `project_metrics`, and `text_markdown`\n",
|
||
"widgets. A new dashboard starts with a single *Overview* section whose `id` we capture for\n",
|
||
"adding widgets."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell5",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"mp_dash = client.create_dashboard(\n",
|
||
" name=\"SDK comprehensive demo\",\n",
|
||
" type=dashboard.DashboardType.MULTI_PROJECT,\n",
|
||
" description=\"Created from the Python SDK walkthrough\",\n",
|
||
" project_name=PROJECT_NAME,\n",
|
||
")\n",
|
||
"mp_section_id = mp_dash.sections[0].id\n",
|
||
"print(f\"Dashboard id : {mp_dash.id}\")\n",
|
||
"print(f\"Type : {mp_dash.type}\")\n",
|
||
"print(f\"Scope : {mp_dash.scope}\")\n",
|
||
"print(f\"Section id : {mp_section_id}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell6",
|
||
"metadata": {},
|
||
"source": "## 3 · Project scope\n\nThe `project_name` passed to `create_dashboard` links the dashboard to a project.\nThe SDK then automatically injects the project into every project-scoped widget\n(`project_stats_card`, `project_metrics`) when you call `add_widget` — you do **not**\nneed to repeat the project in the widget config."
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(f\"Dashboard linked to project: {PROJECT_NAME!r}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell8",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4 · Stats-card widgets\n",
|
||
"\n",
|
||
"`project_stats_card` shows a **single current-value metric** for a project.\n",
|
||
"The `metric` field uses the **lowercase-dotted** namespace — see `dashboard.StatsCardMetric`\n",
|
||
"for the full list (trace counts, duration percentiles, token usage, costs, …).\n",
|
||
"\n",
|
||
"`source` selects whether the metric is computed over traces or spans."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell9",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Total trace count\n",
|
||
"sc_trace_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
|
||
" title=\"Traces\",\n",
|
||
" config=dashboard.ProjectStatsCardConfig(\n",
|
||
" source=dashboard.TraceDataType.TRACES,\n",
|
||
" metric=dashboard.StatsCardMetric.TRACE_COUNT,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Estimated total cost\n",
|
||
"sc_cost_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
|
||
" title=\"Total cost\",\n",
|
||
" config=dashboard.ProjectStatsCardConfig(\n",
|
||
" source=dashboard.TraceDataType.TRACES,\n",
|
||
" metric=dashboard.StatsCardMetric.TOTAL_ESTIMATED_COST_SUM,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Median latency (p50)\n",
|
||
"sc_p50_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
|
||
" title=\"Latency p50\",\n",
|
||
" config=dashboard.ProjectStatsCardConfig(\n",
|
||
" source=dashboard.TraceDataType.TRACES,\n",
|
||
" metric=dashboard.StatsCardMetric.DURATION_P50,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# LLM span count (source=SPANS to query span-level metrics)\n",
|
||
"sc_llm_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_STATS_CARD,\n",
|
||
" title=\"LLM calls\",\n",
|
||
" config=dashboard.ProjectStatsCardConfig(\n",
|
||
" source=dashboard.TraceDataType.SPANS,\n",
|
||
" metric=dashboard.StatsCardMetric.LLM_SPAN_COUNT,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Stats cards added, total widgets: {len(mp_dash.sections[0].widgets)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell10",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5 · Time-series chart widgets\n",
|
||
"\n",
|
||
"`project_metrics` renders a time-series for an aggregate metric.\n",
|
||
"The `metric_type` field uses the **ALL-CAPS** namespace — see `dashboard.ProjectMetricType`.\n",
|
||
"\n",
|
||
"Breakdowns split the series by a dimension: `MODEL`, `PROVIDER`, `TAGS`, `NAME`, etc.\n",
|
||
"Available chart types: `LINE` (default), `BAR`, `RADAR`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell11",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Line chart: duration over time, broken down by model\n",
|
||
"chart_duration_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
|
||
" title=\"Duration by model\",\n",
|
||
" config=dashboard.ProjectMetricsConfig(\n",
|
||
" metric_type=dashboard.ProjectMetricType.DURATION,\n",
|
||
" chart_type=dashboard.ChartType.LINE,\n",
|
||
" breakdown=dashboard.BreakdownConfig(field=dashboard.BreakdownField.MODEL),\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Bar chart: token usage over time\n",
|
||
"chart_tokens_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
|
||
" title=\"Token usage\",\n",
|
||
" config=dashboard.ProjectMetricsConfig(\n",
|
||
" metric_type=dashboard.ProjectMetricType.TOKEN_USAGE,\n",
|
||
" chart_type=dashboard.ChartType.BAR,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Line chart: trace count broken down by tag\n",
|
||
"chart_count_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
|
||
" title=\"Trace count by tag\",\n",
|
||
" config=dashboard.ProjectMetricsConfig(\n",
|
||
" metric_type=dashboard.ProjectMetricType.TRACE_COUNT,\n",
|
||
" chart_type=dashboard.ChartType.LINE,\n",
|
||
" breakdown=dashboard.BreakdownConfig(field=dashboard.BreakdownField.TAGS),\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Line chart: estimated cost broken down by provider\n",
|
||
"chart_cost_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.PROJECT_METRICS,\n",
|
||
" title=\"Cost by provider\",\n",
|
||
" config=dashboard.ProjectMetricsConfig(\n",
|
||
" metric_type=dashboard.ProjectMetricType.COST,\n",
|
||
" chart_type=dashboard.ChartType.LINE,\n",
|
||
" breakdown=dashboard.BreakdownConfig(\n",
|
||
" field=dashboard.BreakdownField.PROVIDER\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Total widgets: {len(mp_dash.sections[0].widgets)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell12",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 6 · Markdown / notes widget\n",
|
||
"\n",
|
||
"`text_markdown` renders freeform Markdown — useful for section headers, runbook links,\n",
|
||
"or context notes. It is valid in **both** `MULTI_PROJECT` and `EXPERIMENTS` dashboards.\n",
|
||
"\n",
|
||
"Widgets can also be created from a **raw dict**, which is the forward-compatible path\n",
|
||
"for backend fields not yet modelled in the SDK."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell13",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Typed config\n",
|
||
"notes_id = mp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.TEXT_MARKDOWN,\n",
|
||
" title=\"\",\n",
|
||
" config=dashboard.TextMarkdownConfig(\n",
|
||
" content=(\n",
|
||
" \"## Project overview\\n\"\n",
|
||
" \"This dashboard tracks **Default Project** metrics.\\n\\n\"\n",
|
||
" \"- Duration p50 / p90\\n\"\n",
|
||
" \"- Token costs by provider\\n\"\n",
|
||
" \"- Error rate\"\n",
|
||
" )\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Raw-dict style: forward-compatible with new backend fields\n",
|
||
"raw_id = mp_dash.add_widget(\n",
|
||
" {\n",
|
||
" \"type\": dashboard.WidgetType.TEXT_MARKDOWN.value,\n",
|
||
" \"title\": \"Raw dict widget\",\n",
|
||
" \"config\": {\"content\": \"Built with the `opik` Python SDK.\"},\n",
|
||
" },\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Total widgets: {len(mp_dash.sections[0].widgets)}\")\n",
|
||
"print(f\"Notes id : {notes_id}\")\n",
|
||
"print(f\"Raw id : {raw_id}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell14",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 7 · Update widgets\n",
|
||
"\n",
|
||
"`update_widget` patches **only the fields you pass** — omitted kwargs are left unchanged.\n",
|
||
"Config is **merged**, not replaced, so you can change a single key without restating the\n",
|
||
"whole config object."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell15",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Change the chart title\n",
|
||
"mp_dash.update_widget(chart_duration_id, title=\"Duration by model (ms)\")\n",
|
||
"\n",
|
||
"# Swap the markdown note content (config merge)\n",
|
||
"mp_dash.update_widget(\n",
|
||
" notes_id,\n",
|
||
" config={\"content\": \"## Project overview (updated)\\nDashboard refreshed via SDK.\"},\n",
|
||
")\n",
|
||
"\n",
|
||
"# Add a subtitle to the trace-count stats card\n",
|
||
"mp_dash.update_widget(sc_trace_id, subtitle=\"last 7 days\")\n",
|
||
"\n",
|
||
"# Switch the token-usage chart from BAR to LINE\n",
|
||
"mp_dash.update_widget(\n",
|
||
" chart_tokens_id,\n",
|
||
" config=dashboard.ProjectMetricsConfig(\n",
|
||
" metric_type=dashboard.ProjectMetricType.TOKEN_USAGE,\n",
|
||
" chart_type=dashboard.ChartType.LINE,\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Rename the dashboard and update its description\n",
|
||
"mp_dash.rename(\"SDK comprehensive demo (v2)\")\n",
|
||
"mp_dash.set_description(\"Updated via the Python SDK.\")\n",
|
||
"print(\"Name:\", mp_dash.name)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell16",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 8 · Inspect and rearrange the grid layout\n",
|
||
"\n",
|
||
"The grid is **6 columns wide** with unlimited rows. Each widget has a `DashboardLayoutItem`\n",
|
||
"with `x` (column), `y` (row), `w` (width in columns), `h` (height in rows).\n",
|
||
"\n",
|
||
"`replace_sections` swaps the entire sections list in one call — use it to reposition\n",
|
||
"widgets, resize them, or reorder sections. All other mutators persist immediately after\n",
|
||
"each individual call."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell17",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"section = mp_dash.sections[0]\n",
|
||
"by_id = {w.id: w for w in section.widgets}\n",
|
||
"\n",
|
||
"print(f\"{'title':35s} x y w h\")\n",
|
||
"print(\"-\" * 50)\n",
|
||
"for li in section.layout:\n",
|
||
" title = by_id[li.id].title or \"(no title)\"\n",
|
||
" print(f\"{title:35s} {li.x} {li.y} {li.w} {li.h}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell18",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Rearrange: full-width notes banner at the top (row 0),\n",
|
||
"# four stats cards side-by-side below (row 2),\n",
|
||
"# charts below that (rows 4+).\n",
|
||
"new_section = copy.deepcopy(section)\n",
|
||
"\n",
|
||
"stats_ids = [sc_trace_id, sc_cost_id, sc_p50_id, sc_llm_id]\n",
|
||
"chart_ids = [chart_duration_id, chart_tokens_id, chart_count_id, chart_cost_id]\n",
|
||
"\n",
|
||
"for li in new_section.layout:\n",
|
||
" if li.id == notes_id:\n",
|
||
" # Full-width banner spanning all 6 columns\n",
|
||
" li.x, li.y, li.w, li.h = 0, 0, 6, 2\n",
|
||
" elif li.id == raw_id:\n",
|
||
" # Small note pinned to the top-right\n",
|
||
" li.x, li.y, li.w, li.h = 4, 2, 2, 2\n",
|
||
" elif li.id in stats_ids:\n",
|
||
" col = stats_ids.index(li.id)\n",
|
||
" li.x, li.y, li.w, li.h = col, 2, 1, 2\n",
|
||
" elif li.id in chart_ids:\n",
|
||
" col = chart_ids.index(li.id)\n",
|
||
" li.x, li.y, li.w, li.h = (col % 3) * 2, 4 + (col // 3) * 4, 2, 4\n",
|
||
"\n",
|
||
"mp_dash.replace_sections([new_section])\n",
|
||
"\n",
|
||
"print(\"Layout after rearrangement:\")\n",
|
||
"section = mp_dash.sections[0]\n",
|
||
"by_id = {w.id: w for w in section.widgets}\n",
|
||
"print(f\"{'title':35s} x y w h\")\n",
|
||
"print(\"-\" * 50)\n",
|
||
"for li in section.layout:\n",
|
||
" title = by_id[li.id].title or \"(no title)\"\n",
|
||
" print(f\"{title:35s} {li.x} {li.y} {li.w} {li.h}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell19",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 9 · Add sections and move widgets\n",
|
||
"\n",
|
||
"`add_section` appends a new empty section. \n",
|
||
"To move widgets between sections use `replace_sections` with the complete new state."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell20",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"analytics_section_id = mp_dash.add_section(\"Analytics\")\n",
|
||
"print(\"Sections:\", [s.title for s in mp_dash.sections])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell21",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Move the four chart widgets from Overview into the new Analytics section.\n",
|
||
"new_sections = [copy.deepcopy(s) for s in mp_dash.sections]\n",
|
||
"overview, analytics = new_sections\n",
|
||
"\n",
|
||
"move_ids = {chart_duration_id, chart_tokens_id, chart_count_id, chart_cost_id}\n",
|
||
"\n",
|
||
"# Extract chart widgets and their layout entries from Overview\n",
|
||
"moved_widgets = [w for w in overview.widgets if w.id in move_ids]\n",
|
||
"moved_layout = [li for li in overview.layout if li.id in move_ids]\n",
|
||
"\n",
|
||
"overview.widgets = [w for w in overview.widgets if w.id not in move_ids]\n",
|
||
"overview.layout = [li for li in overview.layout if li.id not in move_ids]\n",
|
||
"\n",
|
||
"# Re-position charts inside Analytics (2-wide, 4-tall, three per row)\n",
|
||
"for idx, li in enumerate(moved_layout):\n",
|
||
" li.x, li.y, li.w, li.h = (idx % 3) * 2, (idx // 3) * 4, 2, 4\n",
|
||
"\n",
|
||
"analytics.widgets.extend(moved_widgets)\n",
|
||
"analytics.layout.extend(moved_layout)\n",
|
||
"\n",
|
||
"mp_dash.replace_sections(new_sections)\n",
|
||
"\n",
|
||
"for s in mp_dash.sections:\n",
|
||
" print(f\" [{s.title}] {len(s.widgets)} widget(s)\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell22",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 10 · Remove widgets\n",
|
||
"\n",
|
||
"`remove_widget` removes a widget and its layout entry from whichever section contains it.\n",
|
||
"Raises `DashboardValidationError` if the ID is not found."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell23",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Remove the raw-dict markdown widget\n",
|
||
"mp_dash.remove_widget(raw_id)\n",
|
||
"\n",
|
||
"total = sum(len(s.widgets) for s in mp_dash.sections)\n",
|
||
"print(f\"Widgets after removal: {total}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell24",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 11 · EXPERIMENTS dashboard\n",
|
||
"\n",
|
||
"`EXPERIMENTS` dashboards target evaluation results rather than live traces.\n",
|
||
"Supported widgets: `experiments_feedback_scores`, `experiment_leaderboard`, `text_markdown`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell25",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"exp_dash = client.create_dashboard(\n",
|
||
" name=\"SDK experiments demo\",\n",
|
||
" type=dashboard.DashboardType.EXPERIMENTS,\n",
|
||
" description=\"Evaluation metrics overview\",\n",
|
||
")\n",
|
||
"exp_section_id = exp_dash.sections[0].id\n",
|
||
"print(f\"Experiments dashboard id: {exp_dash.id}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell26",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 12 · Experiments evaluation widgets\n",
|
||
"\n",
|
||
"`experiments_feedback_scores` plots feedback score distributions across experiments. \n",
|
||
"`experiment_leaderboard` shows a ranked table of runs against a chosen metric.\n",
|
||
"\n",
|
||
"Pass `max_experiments_count` (1–100) to control how many recent experiments are included."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell27",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Bar chart: feedback scores across the last 10 experiments\n",
|
||
"fb_bar_id = exp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.EXPERIMENTS_FEEDBACK_SCORES,\n",
|
||
" title=\"Feedback scores (bar)\",\n",
|
||
" config=dashboard.ExperimentsFeedbackScoresConfig(\n",
|
||
" chart_type=dashboard.ChartType.BAR,\n",
|
||
" max_experiments_count=10,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Radar chart: quality shape across the last 5 experiments\n",
|
||
"fb_radar_id = exp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.EXPERIMENTS_FEEDBACK_SCORES,\n",
|
||
" title=\"Feedback scores (radar)\",\n",
|
||
" config=dashboard.ExperimentsFeedbackScoresConfig(\n",
|
||
" chart_type=dashboard.ChartType.RADAR,\n",
|
||
" max_experiments_count=5,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Leaderboard with ranking enabled by a specific feedback-score metric\n",
|
||
"leaderboard_id = exp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.EXPERIMENT_LEADERBOARD,\n",
|
||
" title=\"Experiment leaderboard\",\n",
|
||
" config=dashboard.ExperimentLeaderboardConfig(\n",
|
||
" enable_ranking=True,\n",
|
||
" ranking_metric=\"hallucination\", # name of the feedback score to rank by\n",
|
||
" ranking_direction=True, # True = descending (higher score is better)\n",
|
||
" selected_columns=[\"dataset_id\", \"created_at\", \"duration.p50\", \"pass_rate\"],\n",
|
||
" max_rows=20,\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Context note\n",
|
||
"exp_dash.add_widget(\n",
|
||
" dashboard.DashboardWidget(\n",
|
||
" type=dashboard.WidgetType.TEXT_MARKDOWN,\n",
|
||
" title=\"\",\n",
|
||
" config=dashboard.TextMarkdownConfig(\n",
|
||
" content=\"### About\\nTracks evaluation runs ranked by the **hallucination** metric.\"\n",
|
||
" ),\n",
|
||
" ),\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Experiments dashboard widgets: {len(exp_dash.sections[0].widgets)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell28",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 13 · Fetch and list dashboards\n",
|
||
"\n",
|
||
"`get_dashboard` retrieves a single dashboard by ID (re-fetches from the backend). \n",
|
||
"`get_dashboards` pages through all dashboards with an optional name filter."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell29",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Fetch the multi-project dashboard by id\n",
|
||
"fetched_mp = client.get_dashboard(mp_dash.id)\n",
|
||
"print(f\"Fetched: {fetched_mp.name!r} ({len(fetched_mp.sections)} section(s))\")\n",
|
||
"\n",
|
||
"# List all dashboards whose name contains \"SDK\"\n",
|
||
"found = client.get_dashboards(name=\"SDK\", max_results=20)\n",
|
||
"print(f\"\\nDashboards matching 'SDK' ({len(found)} found):\")\n",
|
||
"for d in found:\n",
|
||
" print(f\" {d.id[:8]}… {d.type:15s} {d.name!r}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cell32",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 14 · Clean up"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cell33",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"mp_dash.delete()\n",
|
||
"exp_dash.delete()\n",
|
||
"print(\"Both dashboards deleted.\")"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python",
|
||
"version": "3.10.19"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|