{ "cells": [ { "cell_type": "markdown", "id": "cell0", "metadata": {}, "source": "# Opik Dashboards — Python SDK\n\nA comprehensive walkthrough of the **dashboard** API on the `opik.Opik` client:\n\n| Step | Topic |\n| --- | --- |\n| 1 | Setup |\n| 2–3 | Create a `MULTI_PROJECT` dashboard scoped to a project |\n| 4 | Stats-card widgets (snapshot metrics) |\n| 5 | Time-series chart widgets |\n| 6 | Markdown / notes widget |\n| 7 | Update widgets |\n| 8 | Inspect and rearrange the grid layout |\n| 9 | Add sections and move widgets between sections |\n| 10 | Remove widgets |\n| 11–12 | Create an `EXPERIMENTS` dashboard with evaluation widgets |\n| 13 | Fetch and list dashboards |\n| 14 | Clean up |\n\n**Project scope** — `project_stats_card` and `project_metrics` widgets are project-scoped.\nPass `project_name` to `create_dashboard` once; the SDK automatically injects the project\ninto every project-scoped widget added via `add_widget`.\n\n**Metric-ID namespaces** — easy to mix up:\n\n| Widget | Field | Namespace | Example |\n| --- | --- | --- | --- |\n| `project_stats_card` | `metric` | lowercase-dotted | `trace_count`, `duration.p50` |\n| `project_metrics` | `metric_type` | ALL-CAPS | `TRACE_COUNT`, `DURATION` |" }, { "cell_type": "markdown", "id": "cell1", "metadata": {}, "source": [ "## 1 · Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "cell2", "metadata": {}, "outputs": [], "source": [ "%pip install opik --quiet" ] }, { "cell_type": "code", "execution_count": null, "id": "cell3", "metadata": {}, "outputs": [], "source": [ "import copy\n", "\n", "import opik\n", "from opik import dashboard\n", "\n", "client = opik.Opik()\n", "\n", "PROJECT_NAME = \"Default Project\"" ] }, { "cell_type": "markdown", "id": "cell4", "metadata": {}, "source": [ "## 2 · Create a MULTI_PROJECT dashboard\n", "\n", "`MULTI_PROJECT` dashboards support `project_stats_card`, `project_metrics`, and `text_markdown`\n", "widgets. A new dashboard starts with a single *Overview* section whose `id` we capture for\n", "adding widgets." ] }, { "cell_type": "code", "execution_count": null, "id": "cell5", "metadata": {}, "outputs": [], "source": [ "mp_dash = client.create_dashboard(\n", " name=\"SDK comprehensive demo\",\n", " type=dashboard.DashboardType.MULTI_PROJECT,\n", " description=\"Created from the Python SDK walkthrough\",\n", " project_name=PROJECT_NAME,\n", ")\n", "mp_section_id = mp_dash.sections[0].id\n", "print(f\"Dashboard id : {mp_dash.id}\")\n", "print(f\"Type : {mp_dash.type}\")\n", "print(f\"Scope : {mp_dash.scope}\")\n", "print(f\"Section id : {mp_section_id}\")" ] }, { "cell_type": "markdown", "id": "cell6", "metadata": {}, "source": "## 3 · Project scope\n\nThe `project_name` passed to `create_dashboard` links the dashboard to a project.\nThe SDK then automatically injects the project into every project-scoped widget\n(`project_stats_card`, `project_metrics`) when you call `add_widget` — you do **not**\nneed to repeat the project in the widget config." }, { "cell_type": "code", "execution_count": null, "id": "cell7", "metadata": {}, "outputs": [], "source": [ "print(f\"Dashboard linked to project: {PROJECT_NAME!r}\")" ] }, { "cell_type": "markdown", "id": "cell8", "metadata": {}, "source": [ "## 4 · Stats-card widgets\n", "\n", "`project_stats_card` shows a **single current-value metric** for a project.\n", "The `metric` field uses the **lowercase-dotted** namespace — see `dashboard.StatsCardMetric`\n", "for the full list (trace counts, duration percentiles, token usage, costs, …).\n", "\n", "`source` selects whether the metric is computed over traces or spans." ] }, { "cell_type": "code", "execution_count": null, "id": "cell9", "metadata": {}, "outputs": [], "source": [ "# Total trace count\n", "sc_trace_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_STATS_CARD,\n", " title=\"Traces\",\n", " config=dashboard.ProjectStatsCardConfig(\n", " source=dashboard.TraceDataType.TRACES,\n", " metric=dashboard.StatsCardMetric.TRACE_COUNT,\n", " ),\n", " ),\n", ")\n", "\n", "# Estimated total cost\n", "sc_cost_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_STATS_CARD,\n", " title=\"Total cost\",\n", " config=dashboard.ProjectStatsCardConfig(\n", " source=dashboard.TraceDataType.TRACES,\n", " metric=dashboard.StatsCardMetric.TOTAL_ESTIMATED_COST_SUM,\n", " ),\n", " ),\n", ")\n", "\n", "# Median latency (p50)\n", "sc_p50_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_STATS_CARD,\n", " title=\"Latency p50\",\n", " config=dashboard.ProjectStatsCardConfig(\n", " source=dashboard.TraceDataType.TRACES,\n", " metric=dashboard.StatsCardMetric.DURATION_P50,\n", " ),\n", " ),\n", ")\n", "\n", "# LLM span count (source=SPANS to query span-level metrics)\n", "sc_llm_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_STATS_CARD,\n", " title=\"LLM calls\",\n", " config=dashboard.ProjectStatsCardConfig(\n", " source=dashboard.TraceDataType.SPANS,\n", " metric=dashboard.StatsCardMetric.LLM_SPAN_COUNT,\n", " ),\n", " ),\n", ")\n", "\n", "print(f\"Stats cards added, total widgets: {len(mp_dash.sections[0].widgets)}\")" ] }, { "cell_type": "markdown", "id": "cell10", "metadata": {}, "source": [ "## 5 · Time-series chart widgets\n", "\n", "`project_metrics` renders a time-series for an aggregate metric.\n", "The `metric_type` field uses the **ALL-CAPS** namespace — see `dashboard.ProjectMetricType`.\n", "\n", "Breakdowns split the series by a dimension: `MODEL`, `PROVIDER`, `TAGS`, `NAME`, etc.\n", "Available chart types: `LINE` (default), `BAR`, `RADAR`." ] }, { "cell_type": "code", "execution_count": null, "id": "cell11", "metadata": {}, "outputs": [], "source": [ "# Line chart: duration over time, broken down by model\n", "chart_duration_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_METRICS,\n", " title=\"Duration by model\",\n", " config=dashboard.ProjectMetricsConfig(\n", " metric_type=dashboard.ProjectMetricType.DURATION,\n", " chart_type=dashboard.ChartType.LINE,\n", " breakdown=dashboard.BreakdownConfig(field=dashboard.BreakdownField.MODEL),\n", " ),\n", " ),\n", ")\n", "\n", "# Bar chart: token usage over time\n", "chart_tokens_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_METRICS,\n", " title=\"Token usage\",\n", " config=dashboard.ProjectMetricsConfig(\n", " metric_type=dashboard.ProjectMetricType.TOKEN_USAGE,\n", " chart_type=dashboard.ChartType.BAR,\n", " ),\n", " ),\n", ")\n", "\n", "# Line chart: trace count broken down by tag\n", "chart_count_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_METRICS,\n", " title=\"Trace count by tag\",\n", " config=dashboard.ProjectMetricsConfig(\n", " metric_type=dashboard.ProjectMetricType.TRACE_COUNT,\n", " chart_type=dashboard.ChartType.LINE,\n", " breakdown=dashboard.BreakdownConfig(field=dashboard.BreakdownField.TAGS),\n", " ),\n", " ),\n", ")\n", "\n", "# Line chart: estimated cost broken down by provider\n", "chart_cost_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.PROJECT_METRICS,\n", " title=\"Cost by provider\",\n", " config=dashboard.ProjectMetricsConfig(\n", " metric_type=dashboard.ProjectMetricType.COST,\n", " chart_type=dashboard.ChartType.LINE,\n", " breakdown=dashboard.BreakdownConfig(\n", " field=dashboard.BreakdownField.PROVIDER\n", " ),\n", " ),\n", " ),\n", ")\n", "\n", "print(f\"Total widgets: {len(mp_dash.sections[0].widgets)}\")" ] }, { "cell_type": "markdown", "id": "cell12", "metadata": {}, "source": [ "## 6 · Markdown / notes widget\n", "\n", "`text_markdown` renders freeform Markdown — useful for section headers, runbook links,\n", "or context notes. It is valid in **both** `MULTI_PROJECT` and `EXPERIMENTS` dashboards.\n", "\n", "Widgets can also be created from a **raw dict**, which is the forward-compatible path\n", "for backend fields not yet modelled in the SDK." ] }, { "cell_type": "code", "execution_count": null, "id": "cell13", "metadata": {}, "outputs": [], "source": [ "# Typed config\n", "notes_id = mp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.TEXT_MARKDOWN,\n", " title=\"\",\n", " config=dashboard.TextMarkdownConfig(\n", " content=(\n", " \"## Project overview\\n\"\n", " \"This dashboard tracks **Default Project** metrics.\\n\\n\"\n", " \"- Duration p50 / p90\\n\"\n", " \"- Token costs by provider\\n\"\n", " \"- Error rate\"\n", " )\n", " ),\n", " ),\n", ")\n", "\n", "# Raw-dict style: forward-compatible with new backend fields\n", "raw_id = mp_dash.add_widget(\n", " {\n", " \"type\": dashboard.WidgetType.TEXT_MARKDOWN.value,\n", " \"title\": \"Raw dict widget\",\n", " \"config\": {\"content\": \"Built with the `opik` Python SDK.\"},\n", " },\n", ")\n", "\n", "print(f\"Total widgets: {len(mp_dash.sections[0].widgets)}\")\n", "print(f\"Notes id : {notes_id}\")\n", "print(f\"Raw id : {raw_id}\")" ] }, { "cell_type": "markdown", "id": "cell14", "metadata": {}, "source": [ "## 7 · Update widgets\n", "\n", "`update_widget` patches **only the fields you pass** — omitted kwargs are left unchanged.\n", "Config is **merged**, not replaced, so you can change a single key without restating the\n", "whole config object." ] }, { "cell_type": "code", "execution_count": null, "id": "cell15", "metadata": {}, "outputs": [], "source": [ "# Change the chart title\n", "mp_dash.update_widget(chart_duration_id, title=\"Duration by model (ms)\")\n", "\n", "# Swap the markdown note content (config merge)\n", "mp_dash.update_widget(\n", " notes_id,\n", " config={\"content\": \"## Project overview (updated)\\nDashboard refreshed via SDK.\"},\n", ")\n", "\n", "# Add a subtitle to the trace-count stats card\n", "mp_dash.update_widget(sc_trace_id, subtitle=\"last 7 days\")\n", "\n", "# Switch the token-usage chart from BAR to LINE\n", "mp_dash.update_widget(\n", " chart_tokens_id,\n", " config=dashboard.ProjectMetricsConfig(\n", " metric_type=dashboard.ProjectMetricType.TOKEN_USAGE,\n", " chart_type=dashboard.ChartType.LINE,\n", " ),\n", ")\n", "\n", "# Rename the dashboard and update its description\n", "mp_dash.rename(\"SDK comprehensive demo (v2)\")\n", "mp_dash.set_description(\"Updated via the Python SDK.\")\n", "print(\"Name:\", mp_dash.name)" ] }, { "cell_type": "markdown", "id": "cell16", "metadata": {}, "source": [ "## 8 · Inspect and rearrange the grid layout\n", "\n", "The grid is **6 columns wide** with unlimited rows. Each widget has a `DashboardLayoutItem`\n", "with `x` (column), `y` (row), `w` (width in columns), `h` (height in rows).\n", "\n", "`replace_sections` swaps the entire sections list in one call — use it to reposition\n", "widgets, resize them, or reorder sections. All other mutators persist immediately after\n", "each individual call." ] }, { "cell_type": "code", "execution_count": null, "id": "cell17", "metadata": {}, "outputs": [], "source": [ "section = mp_dash.sections[0]\n", "by_id = {w.id: w for w in section.widgets}\n", "\n", "print(f\"{'title':35s} x y w h\")\n", "print(\"-\" * 50)\n", "for li in section.layout:\n", " title = by_id[li.id].title or \"(no title)\"\n", " print(f\"{title:35s} {li.x} {li.y} {li.w} {li.h}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell18", "metadata": {}, "outputs": [], "source": [ "# Rearrange: full-width notes banner at the top (row 0),\n", "# four stats cards side-by-side below (row 2),\n", "# charts below that (rows 4+).\n", "new_section = copy.deepcopy(section)\n", "\n", "stats_ids = [sc_trace_id, sc_cost_id, sc_p50_id, sc_llm_id]\n", "chart_ids = [chart_duration_id, chart_tokens_id, chart_count_id, chart_cost_id]\n", "\n", "for li in new_section.layout:\n", " if li.id == notes_id:\n", " # Full-width banner spanning all 6 columns\n", " li.x, li.y, li.w, li.h = 0, 0, 6, 2\n", " elif li.id == raw_id:\n", " # Small note pinned to the top-right\n", " li.x, li.y, li.w, li.h = 4, 2, 2, 2\n", " elif li.id in stats_ids:\n", " col = stats_ids.index(li.id)\n", " li.x, li.y, li.w, li.h = col, 2, 1, 2\n", " elif li.id in chart_ids:\n", " col = chart_ids.index(li.id)\n", " li.x, li.y, li.w, li.h = (col % 3) * 2, 4 + (col // 3) * 4, 2, 4\n", "\n", "mp_dash.replace_sections([new_section])\n", "\n", "print(\"Layout after rearrangement:\")\n", "section = mp_dash.sections[0]\n", "by_id = {w.id: w for w in section.widgets}\n", "print(f\"{'title':35s} x y w h\")\n", "print(\"-\" * 50)\n", "for li in section.layout:\n", " title = by_id[li.id].title or \"(no title)\"\n", " print(f\"{title:35s} {li.x} {li.y} {li.w} {li.h}\")" ] }, { "cell_type": "markdown", "id": "cell19", "metadata": {}, "source": [ "## 9 · Add sections and move widgets\n", "\n", "`add_section` appends a new empty section. \n", "To move widgets between sections use `replace_sections` with the complete new state." ] }, { "cell_type": "code", "execution_count": null, "id": "cell20", "metadata": {}, "outputs": [], "source": [ "analytics_section_id = mp_dash.add_section(\"Analytics\")\n", "print(\"Sections:\", [s.title for s in mp_dash.sections])" ] }, { "cell_type": "code", "execution_count": null, "id": "cell21", "metadata": {}, "outputs": [], "source": [ "# Move the four chart widgets from Overview into the new Analytics section.\n", "new_sections = [copy.deepcopy(s) for s in mp_dash.sections]\n", "overview, analytics = new_sections\n", "\n", "move_ids = {chart_duration_id, chart_tokens_id, chart_count_id, chart_cost_id}\n", "\n", "# Extract chart widgets and their layout entries from Overview\n", "moved_widgets = [w for w in overview.widgets if w.id in move_ids]\n", "moved_layout = [li for li in overview.layout if li.id in move_ids]\n", "\n", "overview.widgets = [w for w in overview.widgets if w.id not in move_ids]\n", "overview.layout = [li for li in overview.layout if li.id not in move_ids]\n", "\n", "# Re-position charts inside Analytics (2-wide, 4-tall, three per row)\n", "for idx, li in enumerate(moved_layout):\n", " li.x, li.y, li.w, li.h = (idx % 3) * 2, (idx // 3) * 4, 2, 4\n", "\n", "analytics.widgets.extend(moved_widgets)\n", "analytics.layout.extend(moved_layout)\n", "\n", "mp_dash.replace_sections(new_sections)\n", "\n", "for s in mp_dash.sections:\n", " print(f\" [{s.title}] {len(s.widgets)} widget(s)\")" ] }, { "cell_type": "markdown", "id": "cell22", "metadata": {}, "source": [ "## 10 · Remove widgets\n", "\n", "`remove_widget` removes a widget and its layout entry from whichever section contains it.\n", "Raises `DashboardValidationError` if the ID is not found." ] }, { "cell_type": "code", "execution_count": null, "id": "cell23", "metadata": {}, "outputs": [], "source": [ "# Remove the raw-dict markdown widget\n", "mp_dash.remove_widget(raw_id)\n", "\n", "total = sum(len(s.widgets) for s in mp_dash.sections)\n", "print(f\"Widgets after removal: {total}\")" ] }, { "cell_type": "markdown", "id": "cell24", "metadata": {}, "source": [ "## 11 · EXPERIMENTS dashboard\n", "\n", "`EXPERIMENTS` dashboards target evaluation results rather than live traces.\n", "Supported widgets: `experiments_feedback_scores`, `experiment_leaderboard`, `text_markdown`." ] }, { "cell_type": "code", "execution_count": null, "id": "cell25", "metadata": {}, "outputs": [], "source": [ "exp_dash = client.create_dashboard(\n", " name=\"SDK experiments demo\",\n", " type=dashboard.DashboardType.EXPERIMENTS,\n", " description=\"Evaluation metrics overview\",\n", ")\n", "exp_section_id = exp_dash.sections[0].id\n", "print(f\"Experiments dashboard id: {exp_dash.id}\")" ] }, { "cell_type": "markdown", "id": "cell26", "metadata": {}, "source": [ "## 12 · Experiments evaluation widgets\n", "\n", "`experiments_feedback_scores` plots feedback score distributions across experiments. \n", "`experiment_leaderboard` shows a ranked table of runs against a chosen metric.\n", "\n", "Pass `max_experiments_count` (1–100) to control how many recent experiments are included." ] }, { "cell_type": "code", "execution_count": null, "id": "cell27", "metadata": {}, "outputs": [], "source": [ "# Bar chart: feedback scores across the last 10 experiments\n", "fb_bar_id = exp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.EXPERIMENTS_FEEDBACK_SCORES,\n", " title=\"Feedback scores (bar)\",\n", " config=dashboard.ExperimentsFeedbackScoresConfig(\n", " chart_type=dashboard.ChartType.BAR,\n", " max_experiments_count=10,\n", " ),\n", " ),\n", ")\n", "\n", "# Radar chart: quality shape across the last 5 experiments\n", "fb_radar_id = exp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.EXPERIMENTS_FEEDBACK_SCORES,\n", " title=\"Feedback scores (radar)\",\n", " config=dashboard.ExperimentsFeedbackScoresConfig(\n", " chart_type=dashboard.ChartType.RADAR,\n", " max_experiments_count=5,\n", " ),\n", " ),\n", ")\n", "\n", "# Leaderboard with ranking enabled by a specific feedback-score metric\n", "leaderboard_id = exp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.EXPERIMENT_LEADERBOARD,\n", " title=\"Experiment leaderboard\",\n", " config=dashboard.ExperimentLeaderboardConfig(\n", " enable_ranking=True,\n", " ranking_metric=\"hallucination\", # name of the feedback score to rank by\n", " ranking_direction=True, # True = descending (higher score is better)\n", " selected_columns=[\"dataset_id\", \"created_at\", \"duration.p50\", \"pass_rate\"],\n", " max_rows=20,\n", " ),\n", " ),\n", ")\n", "\n", "# Context note\n", "exp_dash.add_widget(\n", " dashboard.DashboardWidget(\n", " type=dashboard.WidgetType.TEXT_MARKDOWN,\n", " title=\"\",\n", " config=dashboard.TextMarkdownConfig(\n", " content=\"### About\\nTracks evaluation runs ranked by the **hallucination** metric.\"\n", " ),\n", " ),\n", ")\n", "\n", "print(f\"Experiments dashboard widgets: {len(exp_dash.sections[0].widgets)}\")" ] }, { "cell_type": "markdown", "id": "cell28", "metadata": {}, "source": [ "## 13 · Fetch and list dashboards\n", "\n", "`get_dashboard` retrieves a single dashboard by ID (re-fetches from the backend). \n", "`get_dashboards` pages through all dashboards with an optional name filter." ] }, { "cell_type": "code", "execution_count": null, "id": "cell29", "metadata": {}, "outputs": [], "source": [ "# Fetch the multi-project dashboard by id\n", "fetched_mp = client.get_dashboard(mp_dash.id)\n", "print(f\"Fetched: {fetched_mp.name!r} ({len(fetched_mp.sections)} section(s))\")\n", "\n", "# List all dashboards whose name contains \"SDK\"\n", "found = client.get_dashboards(name=\"SDK\", max_results=20)\n", "print(f\"\\nDashboards matching 'SDK' ({len(found)} found):\")\n", "for d in found:\n", " print(f\" {d.id[:8]}… {d.type:15s} {d.name!r}\")" ] }, { "cell_type": "markdown", "id": "cell32", "metadata": {}, "source": [ "## 14 · Clean up" ] }, { "cell_type": "code", "execution_count": null, "id": "cell33", "metadata": {}, "outputs": [], "source": [ "mp_dash.delete()\n", "exp_dash.delete()\n", "print(\"Both dashboards deleted.\")" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.19" } }, "nbformat": 4, "nbformat_minor": 5 }