chore: import upstream snapshot with attribution
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:44 +08:00
commit 5a558eb09e
11579 changed files with 1795921 additions and 0 deletions
+1
View File
@@ -0,0 +1 @@
"""Unit tests for CLI functionality."""
@@ -0,0 +1,196 @@
"""Shared mock helpers for ``opik migrate`` test modules.
Used by both ``test_migrate_dataset_exclude_versions.py`` (Slice 1 paths) and
``test_migrate_dataset_version_replay.py`` (Slice 2 paths). Lives as a plain
module (not conftest.py) because these are helper classes, not pytest
fixtures.
"""
from __future__ import annotations
from typing import Dict, List, Optional
from unittest.mock import MagicMock
class _DatasetRow:
def __init__(
self,
id: str,
name: str,
description: Optional[str] = None,
items: int = 0,
type: Optional[str] = "dataset",
visibility: Optional[str] = "private",
tags: Optional[List[str]] = None,
# ``project_id=None`` represents a workspace-scoped dataset (V1
# entity, or anything left at workspace scope after auto-migration).
# Tests that want a project-scoped source pass an explicit id.
project_id: Optional[str] = None,
) -> None:
self.id = id
self.name = name
self.description = description
self.dataset_items_count = items
self.type = type
self.visibility = visibility
self.tags = tags
self.project_id = project_id
class _Page:
def __init__(self, content: List[_DatasetRow]) -> None:
self.content = content
def _named(name: str) -> MagicMock:
obj = MagicMock()
obj.name = name
return obj
def _planner_rest_client(
find_side_effects: List[_Page],
*,
target_project_exists: bool = True,
workspace_project_names: Optional[List[str]] = None,
) -> MagicMock:
"""Build a rest_client mock for direct planner unit tests."""
rest_client = MagicMock()
if target_project_exists:
target_project = MagicMock()
target_project.id = "target-project-id"
rest_client.projects.retrieve_project.return_value = target_project
else:
from opik.rest_api.core.api_error import ApiError
rest_client.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={}
)
# Suggestion lookup queries find_projects on the 404 path.
candidates = [_named(name) for name in (workspace_project_names or [])]
rest_client.projects.find_projects.return_value = _Page(candidates)
rest_client.datasets.find_datasets.side_effect = find_side_effects
return rest_client
def _planner_client(rest_client: MagicMock) -> MagicMock:
"""Wrap a planner rest_client mock as an ``opik.Opik``-shaped client.
The planner now takes the high-level client (so the resolver can route
``get_project_by_id`` through ``client.get_project`` instead of the
Fern surface). Tests still build the low-level rest_client mock and
drive its side_effects, then wrap with this helper to satisfy the new
planner signature.
``client.get_project(id=...)`` delegates to
``rest_client.projects.get_project_by_id(id=...)`` so any per-test
stub on the rest_client side still flows through unchanged.
"""
client = MagicMock()
client.rest_client = rest_client
client.get_project = MagicMock(
side_effect=lambda id: rest_client.projects.get_project_by_id(id=id)
)
return client
def _build_fake_client(
*,
source_rows: List[_DatasetRow],
destination_rows: List[_DatasetRow],
items: List[Dict[str, object]],
target_project_exists: bool = True,
stale_temp_rows: Optional[List[_DatasetRow]] = None,
) -> MagicMock:
"""Construct an opik.Opik mock matching the executor's call surface.
The planner makes three workspace ``find_datasets`` lookups in a fixed
order: (1) source resolution, (2) the ``<name>_v1`` rename-target
collision pre-flight, and (3) the ``<name>__migrating`` stale-temp
lookup (OPIK-7162). A side-effect list drives all three deterministically.
``destination_rows`` feeds the ``_v1`` check; the stale-temp lookup
returns empty by default (no leftover temp) unless ``stale_temp_rows``
is supplied.
``items`` is a list of dicts; we materialize them as DatasetItem
dataclasses for the streaming mock (matches the real `__internal_api__
stream_items_as_dataclasses__` shape) so per-item fidelity assertions
have somewhere to land.
"""
from opik.api_objects.dataset import dataset_item
rest_client = MagicMock()
if target_project_exists:
target_project = MagicMock()
target_project.id = "target-project-id"
rest_client.projects.retrieve_project.return_value = target_project
else:
from opik.rest_api.core.api_error import ApiError
rest_client.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={}
)
rest_client.datasets.find_datasets.side_effect = [
_Page(source_rows),
_Page(destination_rows),
_Page(stale_temp_rows or []),
]
rest_client.datasets.update_dataset = MagicMock()
rest_client.datasets.create_dataset = MagicMock()
rest_client.datasets.delete_dataset = MagicMock()
client = MagicMock()
client.rest_client = rest_client
client._workspace = "default"
# Build DatasetItem dataclasses from the provided dicts so the executor's
# dataclass-form stream returns a realistic shape. Top-level fields like
# `description` / `source` / `trace_id` / `span_id` can be passed via the
# dict (other keys get stuffed into `data`/extra).
top_level = {
"id",
"trace_id",
"span_id",
"source",
"description",
"evaluators",
"execution_policy",
}
source_items: List[dataset_item.DatasetItem] = []
for raw in items:
kwargs = {k: v for k, v in raw.items() if k in top_level and k != "id"}
data = {k: v for k, v in raw.items() if k not in top_level}
ds_item = dataset_item.DatasetItem(**kwargs, **data)
if "id" in raw:
ds_item.id = raw["id"] # type: ignore[assignment]
source_items.append(ds_item)
# MagicMock treats dunder-prefixed names as magic and blocks them by
# default; pre-attach plain MagicMocks so attribute access works.
source_dataset = MagicMock()
stream_mock = MagicMock(return_value=iter(source_items))
source_dataset.__internal_api__stream_items_as_dataclasses__ = stream_mock
dest_dataset = MagicMock()
insert_mock = MagicMock()
dest_dataset.__internal_api__insert_items_as_dataclasses__ = insert_mock
# Under the OPIK-7162 ordering the destination is written under the temp
# name ``<orig>__migrating`` for the whole copy, then promoted to the
# original name at the end. The executor resolves the destination by name
# in ``_replay_versions`` (temp name) and again in ``PromoteDestination``
# (temp name -> id). The source is never resolved via ``get_dataset`` any
# more (item reads stream by name on the rest_client), so route the temp
# name (and anything else) to the destination dataset.
source_orig_name = source_rows[0].name if source_rows else ""
temp_name = f"{source_orig_name}__migrating"
def _get_dataset(name: str, project_name: Optional[str] = None) -> MagicMock:
if name == temp_name:
return dest_dataset
return source_dataset
client.get_dataset.side_effect = _get_dataset
client.create_dataset = MagicMock()
client.delete_dataset = MagicMock()
return client, source_dataset, dest_dataset
@@ -0,0 +1,125 @@
"""Shared mock helpers for ``opik migrate prompt`` test modules.
Used by ``test_migrate_prompt_planner.py``,
``test_migrate_prompt_executor.py``, and
``test_migrate_prompt_version_replay.py``. Lives as a plain module (not
a conftest.py) because these are helper classes, not pytest fixtures.
"""
from __future__ import annotations
from typing import Any, List, Optional
from unittest.mock import MagicMock
class _PromptRow:
"""Minimal stand-in for ``PromptPublic`` rows returned by ``get_prompts``."""
def __init__(
self,
id: str,
name: str,
*,
description: Optional[str] = None,
tags: Optional[List[str]] = None,
# None = workspace-scoped (legacy v1 prompts that never had a
# project). Tests that want a project-scoped source pass an id.
project_id: Optional[str] = None,
template_structure: Optional[str] = "text",
) -> None:
self.id = id
self.name = name
self.description = description
self.tags = tags
self.project_id = project_id
self.template_structure = template_structure
class _PromptVersionRow:
"""Stand-in for ``PromptVersionPublic`` rows from ``get_prompt_versions``.
Field set mirrors the Fern model: ``id``, ``prompt_id``, ``commit``,
``template`` (required), plus optional metadata / type / change_description
/ tags / environments / template_structure.
"""
def __init__(
self,
id: str,
prompt_id: str,
commit: str,
template: str,
*,
metadata: Optional[Any] = None,
type: Optional[str] = None,
change_description: Optional[str] = None,
tags: Optional[List[str]] = None,
environments: Optional[List[str]] = None,
template_structure: Optional[str] = "text",
) -> None:
self.id = id
self.prompt_id = prompt_id
self.commit = commit
self.template = template
self.metadata = metadata
self.type = type
self.change_description = change_description
self.tags = tags
self.environments = environments
self.template_structure = template_structure
class _Page:
def __init__(self, content: List[Any]) -> None:
self.content = content
def _named(name: str) -> MagicMock:
obj = MagicMock()
obj.name = name
return obj
def _planner_rest_client(
find_prompts_side_effects: List[_Page],
*,
target_project_exists: bool = True,
workspace_project_names: Optional[List[str]] = None,
) -> MagicMock:
"""Build a rest_client mock for direct prompt-planner unit tests.
``find_prompts_side_effects`` is the side-effect list for
``rest_client.prompts.get_prompts``; the planner calls it twice (once
for source resolution, once for the rename-collision preflight) so
callers pass two pages.
"""
rest_client = MagicMock()
if target_project_exists:
target_project = MagicMock()
target_project.id = "target-project-id"
rest_client.projects.retrieve_project.return_value = target_project
else:
from opik.rest_api.core.api_error import ApiError
rest_client.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={}
)
candidates = [_named(n) for n in (workspace_project_names or [])]
rest_client.projects.find_projects.return_value = _Page(candidates)
rest_client.prompts.get_prompts.side_effect = find_prompts_side_effects
return rest_client
def _planner_client(rest_client: MagicMock) -> MagicMock:
"""Wrap a rest_client mock as an ``opik.Opik``-shaped client.
The resolver routes project-id lookups through ``client.get_project``
(high-level surface) so the project-name resolution still benefits
from any test-side stubs on ``rest_client.projects.get_project_by_id``.
"""
client = MagicMock()
client.rest_client = rest_client
client.get_project = MagicMock(
side_effect=lambda id: rest_client.projects.get_project_by_id(id=id)
)
return client
@@ -0,0 +1,313 @@
"""Tests for opik.cli.local_runner.stop and the `opik <type> stop` CLI surface."""
import json
import os
import signal
from pathlib import Path
from typing import Optional
from unittest.mock import patch
import pytest
from click.testing import CliRunner
from opik.cli.local_runner import stop as stop_module
from opik.cli.local_runner.pairing import RunnerType
from opik.cli.local_runner.stop import do_stop
from opik.cli.main import cli
from opik.runner import pid_file
from opik.runner.pid_file import RunnerInfo
@pytest.fixture
def tmp_runners_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
runners = tmp_path / "runners"
monkeypatch.setattr(pid_file, "_RUNNERS_DIR", runners)
return runners
def _write_runner_file(
dir: Path,
runner_id: str,
runner_type: str,
project_name: str,
pid: int,
workspace: Optional[str] = None,
) -> Path:
dir.mkdir(parents=True, exist_ok=True)
path = dir / f"{runner_type}-{runner_id}.json"
path.write_text(
json.dumps(
{
"pid": pid,
"runner_id": runner_id,
"runner_type": runner_type,
"project_name": project_name,
"workspace": workspace,
"started_at": 1.0,
}
)
)
return path
class TestDoStopFilters:
def test_do_stop__no_filter__usage_error(self, tmp_runners_dir: Path) -> None:
# No --project / --runner / --all → usage error before any signaling.
with pytest.raises(Exception) as exc_info:
do_stop(
runner_type=RunnerType.CONNECT,
project_name=None,
all_flag=False,
runner_id_filter=None,
)
# click.UsageError prints "Error: ..." and is a Click exception.
assert "specify" in str(exc_info.value).lower()
def test_do_stop__no_match__prints_no_match_message(
self, tmp_runners_dir: Path, capsys: pytest.CaptureFixture
) -> None:
do_stop(
runner_type=RunnerType.CONNECT,
project_name="missing",
all_flag=False,
runner_id_filter=None,
)
out = capsys.readouterr().out
assert "No local 'connect' runners found" in out
def test_do_stop__project_filter__signals_matching_only(
self, tmp_runners_dir: Path
) -> None:
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
_write_runner_file(tmp_runners_dir, "r-2", "connect", "beta", os.getpid())
seen = []
def fake_signal(info: RunnerInfo):
seen.append(info.runner_id)
return True, ""
with patch.object(stop_module, "_signal_until_gone", side_effect=fake_signal):
do_stop(
runner_type=RunnerType.CONNECT,
project_name="alpha",
all_flag=False,
runner_id_filter=None,
)
assert seen == ["r-1"]
# The matched runner's pid file is cleaned up.
assert not (tmp_runners_dir / "connect-r-1.json").exists()
# Non-matched is left alone.
assert (tmp_runners_dir / "connect-r-2.json").exists()
def test_do_stop__all_flag__signals_only_target_runner_type(
self, tmp_runners_dir: Path
) -> None:
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
_write_runner_file(tmp_runners_dir, "r-2", "connect", "beta", os.getpid())
# An endpoint runner must NOT be touched by `connect stop --all`.
_write_runner_file(tmp_runners_dir, "r-3", "endpoint", "alpha", os.getpid())
seen = []
with patch.object(
stop_module,
"_signal_until_gone",
lambda info: (seen.append(info.runner_id) or (True, "")),
):
do_stop(
runner_type=RunnerType.CONNECT,
project_name=None,
all_flag=True,
runner_id_filter=None,
)
assert sorted(seen) == ["r-1", "r-2"]
assert (tmp_runners_dir / "endpoint-r-3.json").exists()
def test_do_stop__runner_id_filter__signals_matching_only(
self, tmp_runners_dir: Path
) -> None:
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
_write_runner_file(tmp_runners_dir, "r-2", "connect", "alpha", os.getpid())
seen = []
with patch.object(
stop_module,
"_signal_until_gone",
lambda info: (seen.append(info.runner_id) or (True, "")),
):
do_stop(
runner_type=RunnerType.CONNECT,
project_name=None,
all_flag=False,
runner_id_filter="r-2",
)
assert seen == ["r-2"]
class TestSignalUntilGone:
"""Logic tests for the SIGTERM → wait → SIGKILL flow.
Real-subprocess kill is exercised by `tests/unit/runner/test_pid_file.py::TestSignalRoundtrip`;
those cases call `proc.wait()` to reap zombies, which pytest's parent-child
relationship requires. Here we mock the alive-check + signal so we can assert
the escalation order without fighting zombie state.
"""
def _make_info(self, tmp_runners_dir: Path, pid: int = 12345) -> RunnerInfo:
return RunnerInfo(
pid=pid,
runner_id="r-1",
runner_type="connect",
project_name="p",
workspace=None,
started_at=0.0,
path=tmp_runners_dir / "connect-r-1.json",
)
def test_signal_until_gone__target_exits_after_sigterm__returns_ok(
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(stop_module, "_POLL_INTERVAL_SECONDS", 0.01)
sent_signals = []
monkeypatch.setattr(
stop_module.os, "kill", lambda pid, sig: sent_signals.append((pid, sig))
)
# First poll says alive, next says gone.
alive_seq = iter([True, False])
monkeypatch.setattr(
stop_module.pid_file, "is_pid_alive", lambda _pid: next(alive_seq)
)
ok, _ = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
assert ok
assert sent_signals == [(12345, signal.SIGTERM)]
def test_signal_until_gone__sigterm_ignored__escalates_to_sigkill(
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(stop_module, "_SIGTERM_GRACE_SECONDS", 0.05)
monkeypatch.setattr(stop_module, "_POLL_INTERVAL_SECONDS", 0.01)
sent_signals = []
state = {"alive": True}
def fake_kill(pid: int, sig: int) -> None:
sent_signals.append((pid, sig))
if sig == signal.SIGKILL:
state["alive"] = False
monkeypatch.setattr(stop_module.os, "kill", fake_kill)
monkeypatch.setattr(
stop_module.pid_file, "is_pid_alive", lambda _pid: state["alive"]
)
ok, _ = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
assert ok
assert [sig for _, sig in sent_signals] == [signal.SIGTERM, signal.SIGKILL]
def test_signal_until_gone__already_dead_pid__returns_ok(
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
def fake_kill(_pid: int, _sig: int) -> None:
raise ProcessLookupError()
monkeypatch.setattr(stop_module.os, "kill", fake_kill)
ok, reason = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
assert ok
assert "exited" in reason
def test_signal_until_gone__permission_error__returns_failure(
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
def fake_kill(_pid: int, _sig: int) -> None:
raise PermissionError("not yours")
monkeypatch.setattr(stop_module.os, "kill", fake_kill)
ok, reason = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
assert not ok
assert "permission" in reason.lower()
class TestStopExitCode:
def test_do_stop__signal_failed__raises_systemexit_1(
self, tmp_runners_dir: Path
) -> None:
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
with patch.object(
stop_module, "_signal_until_gone", return_value=(False, "denied")
):
with pytest.raises(SystemExit) as exc_info:
do_stop(
runner_type=RunnerType.CONNECT,
project_name="alpha",
all_flag=False,
runner_id_filter=None,
)
assert exc_info.value.code == 1
class TestConnectStopCli:
def test_cli_connect_stop__no_filter__exits_with_usage_error(self) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["connect", "stop"])
assert result.exit_code == 2
assert "Specify" in result.output or "specify" in result.output
def test_cli_connect_stop__with_project__routes_to_do_stop(
self, tmp_runners_dir: Path
) -> None:
runner = CliRunner()
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
result = runner.invoke(cli, ["connect", "stop", "--project", "p"])
assert result.exit_code == 0, result.output
mock_stop.assert_called_once()
assert mock_stop.call_args.kwargs == dict(
runner_type=RunnerType.CONNECT,
project_name="p",
all_flag=False,
runner_id_filter=None,
)
def test_cli_connect_stop__all_flag__sets_all_flag_kwarg(
self, tmp_runners_dir: Path
) -> None:
runner = CliRunner()
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
result = runner.invoke(cli, ["connect", "stop", "--all"])
assert result.exit_code == 0, result.output
assert mock_stop.call_args.kwargs["all_flag"] is True
def test_cli_connect_stop__runner_flag__sets_runner_id_filter(
self, tmp_runners_dir: Path
) -> None:
runner = CliRunner()
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
result = runner.invoke(cli, ["connect", "stop", "--runner", "r-1"])
assert result.exit_code == 0, result.output
assert mock_stop.call_args.kwargs["runner_id_filter"] == "r-1"
class TestEndpointStopCli:
def test_cli_endpoint_stop__with_project__routes_to_do_stop(
self, tmp_runners_dir: Path
) -> None:
runner = CliRunner()
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
result = runner.invoke(cli, ["endpoint", "stop", "--project", "p"])
assert result.exit_code == 0, result.output
mock_stop.assert_called_once()
assert mock_stop.call_args.kwargs == dict(
runner_type=RunnerType.ENDPOINT,
project_name="p",
all_flag=False,
runner_id_filter=None,
)
def test_cli_endpoint_stop__all_flag__sets_all_flag_kwarg(
self, tmp_runners_dir: Path
) -> None:
runner = CliRunner()
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
result = runner.invoke(cli, ["endpoint", "stop", "--all"])
assert result.exit_code == 0, result.output
assert mock_stop.call_args.kwargs["all_flag"] is True
File diff suppressed because it is too large Load Diff
+445
View File
@@ -0,0 +1,445 @@
"""Tests for CLI commands."""
from pathlib import Path
from unittest.mock import MagicMock, patch
from contextlib import contextmanager
from click.testing import CliRunner
from opik.cli import cli
class TestDownloadCommand:
"""Test the download CLI command."""
def test_export_group_help(self):
"""Test that the export group shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["export", "--help"])
assert result.exit_code == 0
assert "Export data from an Opik project" in result.output
assert "dataset" in result.output
assert "traces" in result.output
assert "experiment" in result.output
def test_export_dataset_help(self):
"""Test that the export dataset command shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["export", "default", "proj", "dataset", "--help"])
assert result.exit_code == 0
assert "Export a dataset by exact name" in result.output
assert "--force" in result.output
def test_export_traces_help(self):
"""Test that the export traces command shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["export", "default", "proj", "traces", "--help"])
assert result.exit_code == 0
assert "Export the project's traces" in result.output
def test_export_experiment_help(self):
"""Test that the export experiment command shows help."""
runner = CliRunner()
result = runner.invoke(
cli, ["export", "default", "proj", "experiment", "--help"]
)
assert result.exit_code == 0
assert "Export an experiment by exact name" in result.output
class TestUploadCommand:
"""Test the upload CLI command."""
def test_import_group_help(self):
"""Test that the import group shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["import", "--help"])
assert result.exit_code == 0
assert "Import data into an Opik project" in result.output
assert "dataset" in result.output
assert "traces" in result.output
assert "experiment" in result.output
def test_import_dataset_help(self):
"""Test that the import dataset command shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["import", "default", "proj", "dataset", "--help"])
assert result.exit_code == 0
assert "Import datasets from" in result.output
assert "--dry-run" in result.output
def test_import_traces_help(self):
"""Test that the import traces command shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["import", "default", "proj", "traces", "--help"])
assert result.exit_code == 0
assert "Import the project's traces" in result.output
assert "--dry-run" in result.output
def test_import_experiment_help(self):
"""Test that the import experiment command shows help."""
runner = CliRunner()
result = runner.invoke(
cli, ["import", "default", "proj", "experiment", "--help"]
)
assert result.exit_code == 0
assert "Import experiments from" in result.output
assert "--dry-run" in result.output
class TestSmokeTestCommand:
"""Test the smoke-test functionality via healthcheck command."""
def test_smoke_test_help(self):
"""Test that the healthcheck --smoke-test command shows help."""
runner = CliRunner()
result = runner.invoke(cli, ["healthcheck", "--help"])
assert result.exit_code == 0
assert "--smoke-test" in result.output
assert "--project-name" in result.output
assert "Project name for the smoke test" in result.output
assert "WORKSPACE" in result.output
assert "Run a smoke test to verify Opik integration" in result.output
def test_smoke_test_minimal_args_parsing(self):
"""Test that healthcheck --smoke-test command requires workspace value."""
runner = CliRunner()
# Test that help shows workspace is required
result = runner.invoke(cli, ["healthcheck", "--help"])
assert result.exit_code == 0
assert "WORKSPACE" in result.output
# Test that missing workspace value causes error
result = runner.invoke(cli, ["healthcheck", "--smoke-test"])
assert result.exit_code != 0
assert "Error" in result.output or "Missing" in result.output
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.smoke_test.opik.Opik")
@patch("opik.cli.healthcheck.smoke_test.opik.start_as_current_trace")
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_trace")
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_span")
@patch("opik.cli.healthcheck.smoke_test.create_opik_logo_image")
@patch("opik.cli.healthcheck.smoke_test.track")
def test_smoke_test_with_workspace_and_project_name(
self,
mock_track,
mock_create_logo,
mock_update_span,
mock_update_trace,
mock_start_trace,
mock_opik_class,
mock_healthcheck_run,
):
"""Test healthcheck --smoke-test command with workspace and --project-name arguments."""
# Setup mocks
mock_client = MagicMock()
# Mock search_traces to raise an exception immediately to skip verification polling
# This prevents the verification from running and triggering real client creation
mock_client.search_traces = MagicMock(
side_effect=AttributeError("Mock client - search_traces not available")
)
mock_opik_class.return_value = mock_client
# Mock the context manager for start_as_current_trace
@contextmanager
def mock_trace_context(*args, **kwargs):
yield MagicMock()
mock_start_trace.return_value = mock_trace_context()
# Mock create_opik_logo_image to return a fake path
mock_logo_path = Path("/tmp/fake_logo.png")
mock_create_logo.return_value = mock_logo_path
# Mock the track decorator to return the function unchanged
def track_decorator(*args, **kwargs):
def decorator(func):
return func
return decorator
mock_track.side_effect = track_decorator
# Run the command
runner = CliRunner()
result = runner.invoke(
cli,
[
"healthcheck",
"--smoke-test",
"test-workspace",
"--project-name",
"test-project",
],
catch_exceptions=False,
)
# Assertions
assert result.exit_code == 0
# Verify client was created with correct arguments
mock_opik_class.assert_called_once()
call_kwargs = mock_opik_class.call_args[1]
assert call_kwargs["workspace"] == "test-workspace"
assert call_kwargs["project_name"] == "test-project"
# Verify trace was started
mock_start_trace.assert_called_once()
# Verify explicit client was flushed and ended
mock_client.flush.assert_called_once()
mock_client.end.assert_called_once()
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.smoke_test.opik.Opik")
@patch("opik.cli.healthcheck.smoke_test.opik.start_as_current_trace")
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_trace")
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_span")
@patch("opik.cli.healthcheck.smoke_test.create_opik_logo_image")
@patch("opik.cli.healthcheck.smoke_test.track")
def test_smoke_test_with_default_project_name(
self,
mock_track,
mock_create_logo,
mock_update_span,
mock_update_trace,
mock_start_trace,
mock_opik_class,
mock_healthcheck_run,
):
"""Test healthcheck --smoke-test command with workspace only (uses default project name)."""
# Setup mocks
mock_client = MagicMock()
# Mock search_traces to raise an exception immediately to skip verification polling
# This prevents the verification from running and triggering real client creation
mock_client.search_traces = MagicMock(
side_effect=AttributeError("Mock client - search_traces not available")
)
mock_opik_class.return_value = mock_client
# Mock the context manager for start_as_current_trace
@contextmanager
def mock_trace_context(*args, **kwargs):
yield MagicMock()
mock_start_trace.return_value = mock_trace_context()
# Mock create_opik_logo_image to return a fake path
mock_logo_path = Path("/tmp/fake_logo.png")
mock_create_logo.return_value = mock_logo_path
# Mock the track decorator to return the function unchanged
def track_decorator(*args, **kwargs):
def decorator(func):
return func
return decorator
mock_track.side_effect = track_decorator
# Run the command without --project-name (should use default)
runner = CliRunner()
result = runner.invoke(
cli,
["healthcheck", "--smoke-test", "test-workspace"],
catch_exceptions=False,
)
# Assertions
assert result.exit_code == 0
# Verify client was created with default project name
mock_opik_class.assert_called_once()
call_kwargs = mock_opik_class.call_args[1]
assert call_kwargs["workspace"] == "test-workspace"
assert call_kwargs["project_name"] == "smoke-test-project" # Default value
# Verify trace was started with default project name
mock_start_trace.assert_called_once()
trace_call_kwargs = mock_start_trace.call_args[1]
assert trace_call_kwargs["project_name"] == "smoke-test-project"
class TestCheckPermissionsCommand:
"""Test the --check-permissions functionality via healthcheck command."""
def test_check_permissions_option__in_healthcheck_help__shown(self):
"""Test that --check-permissions option is shown in healthcheck help."""
runner = CliRunner()
result = runner.invoke(cli, ["healthcheck", "--help"])
assert result.exit_code == 0
assert "--check-permissions" in result.output
assert "WORKSPACE" in result.output
def test_check_permissions__missing_workspace__raises_error(self):
"""Test that --check-permissions without a workspace value raises an error."""
runner = CliRunner()
result = runner.invoke(cli, ["healthcheck", "--check-permissions"])
assert result.exit_code != 0
assert "Error" in result.output or "Missing" in result.output
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
def test_check_permissions__workspace_provided__calls_run_with_workspace(
self,
mock_permissions_run,
mock_standard_run,
):
"""Test that --check-permissions pass the workspace to check_user_permissions.run."""
runner = CliRunner()
result = runner.invoke(
cli,
["healthcheck", "--check-permissions", "my-workspace"],
catch_exceptions=False,
)
assert result.exit_code == 0
mock_permissions_run.assert_called_once_with(
api_key=None, workspace="my-workspace"
)
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
def test_check_permissions__api_key_in_cli_context__passes_api_key_to_run(
self,
mock_permissions_run,
mock_standard_run,
):
"""Test that --check-permissions forward the api_key from CLI context."""
from opik.cli import cli as opik_cli
runner = CliRunner()
result = runner.invoke(
opik_cli,
[
"--api-key",
"test-api-key",
"healthcheck",
"--check-permissions",
"my-workspace",
],
catch_exceptions=False,
)
assert result.exit_code == 0
mock_permissions_run.assert_called_once_with(
api_key="test-api-key", workspace="my-workspace"
)
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
def test_check_permissions__empty_workspace__raises_bad_parameter(
self,
mock_permissions_run,
mock_standard_run,
):
"""Test that --check-permissions with an empty workspace string raises BadParameter."""
runner = CliRunner()
result = runner.invoke(
cli,
["healthcheck", "--check-permissions", ""],
)
assert result.exit_code != 0
assert (
"Error" in result.output
or "requires a non-empty workspace name" in result.output
)
mock_permissions_run.assert_not_called()
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
def test_check_permissions__no_flag__not_called(
self,
mock_permissions_run,
mock_standard_run,
):
"""Test that check_user_permissions.run is not called when --check-permissions is absent."""
runner = CliRunner()
result = runner.invoke(cli, ["healthcheck"], catch_exceptions=False)
assert result.exit_code == 0
mock_permissions_run.assert_not_called()
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.check_user_permissions.get_user_permissions")
@patch("opik.cli.healthcheck.check_user_permissions.config.OpikConfig")
def test_check_permissions__api_returns_user_and_workspace__displays_info(
self,
mock_opik_config,
mock_get_permissions,
mock_standard_run,
):
"""Test that user and workspace info from API response are printed."""
mock_config = MagicMock()
mock_config.api_key = "test-api-key"
mock_config.workspace = "my-workspace"
mock_config.url_override = "https://opik.example.com"
mock_opik_config.return_value = mock_config
mock_get_permissions.return_value = {
"user_name": "alice",
"workspace_name": "my-workspace",
"permissions": [
{"permission_name": "read", "permission_value": True},
{"permission_name": "write", "permission_value": False},
],
}
runner = CliRunner()
result = runner.invoke(
cli,
["healthcheck", "--check-permissions", "my-workspace"],
catch_exceptions=False,
)
assert result.exit_code == 0
assert "alice" in result.output
assert "my-workspace" in result.output
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.check_user_permissions.get_user_permissions")
@patch("opik.cli.healthcheck.check_user_permissions.config.OpikConfig")
def test_check_permissions__connection_error__prints_error(
self,
mock_opik_config,
mock_get_permissions,
mock_standard_run,
):
"""Test that a ConnectionError from the API prints an error message."""
mock_config = MagicMock()
mock_config.api_key = "test-api-key"
mock_config.workspace = "my-workspace"
mock_config.url_override = "https://opik.example.com"
mock_opik_config.return_value = mock_config
mock_get_permissions.side_effect = ConnectionError("Network error: timeout")
runner = CliRunner()
result = runner.invoke(
cli,
["healthcheck", "--check-permissions", "my-workspace"],
catch_exceptions=False,
)
assert result.exit_code == 0
assert "Failed to fetch user permissions" in result.output
@patch("opik.cli.healthcheck.cli.standard_check.run")
@patch("opik.cli.healthcheck.check_user_permissions.config.OpikConfig")
def test_check_permissions__missing_api_key__raises_value_error(
self,
mock_opik_config,
mock_standard_run,
):
"""Test that missing api_key (not in CLI context and not in config) raises ValueError."""
mock_config = MagicMock()
mock_config.api_key = None
mock_config.workspace = None
mock_config.url_override = "https://opik.example.com"
mock_opik_config.return_value = mock_config
runner = CliRunner()
result = runner.invoke(
cli,
["healthcheck", "--check-permissions", "my-workspace"],
)
assert result.exit_code != 0
assert isinstance(result.exception, ValueError)
assert "API key is required" in str(result.exception)
@@ -0,0 +1,54 @@
"""Tests for the ``opik configure`` command group."""
import pathlib
from unittest import mock
from click.testing import CliRunner
from opik.cli import cli
from opik.cli import configure as configure_cli
def test_configure_status__prints_config_summary():
runner = CliRunner()
config = mock.Mock(
config_file_exists=True,
config_file_fullpath=pathlib.Path("/home/u/.opik.config"),
url_override="https://dev.comet.com/opik/api/",
workspace="my-ws",
)
with mock.patch.object(
configure_cli.opik_config, "OpikConfig", return_value=config
):
result = runner.invoke(cli, ["configure", "status"])
assert result.exit_code == 0
assert "Your Opik configuration" in result.output
assert "https://dev.comet.com/opik/api/" in result.output
assert "my-ws" in result.output
def test_configure_status__not_configured__points_to_configure():
runner = CliRunner()
config = mock.Mock(
config_file_exists=False,
config_file_fullpath=pathlib.Path("/home/u/.opik.config"),
)
with mock.patch.object(
configure_cli.opik_config, "OpikConfig", return_value=config
):
result = runner.invoke(cli, ["configure", "status"])
assert result.exit_code == 0
assert "not found" in result.output
assert "opik configure" in result.output
def test_configure_no_subcommand__runs_configurator():
runner = CliRunner()
with mock.patch.object(configure_cli, "run_interactive_configure") as spy:
result = runner.invoke(cli, ["configure", "--use-local"])
assert result.exit_code == 0
spy.assert_called_once()
assert spy.call_args.kwargs["use_local"] is True
+230
View File
@@ -0,0 +1,230 @@
from unittest.mock import MagicMock, patch
import httpx
from click.testing import CliRunner
from opik.cli.main import cli
from opik.cli.local_runner.pairing import PairingResult, RunnerType
from opik.rest_api.core.api_error import ApiError
class TestConnect:
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__with_project__calls_pairing_and_supervisor(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-abc",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(cli, ["connect", "--project", "my-proj"])
assert result.exit_code == 0
mock_run_pairing.assert_called_once()
call_kwargs = mock_run_pairing.call_args[1]
assert call_kwargs["project_name"] == "my-proj"
assert call_kwargs["runner_type"] == RunnerType.CONNECT
mock_launch.assert_called_once()
launch_kwargs = mock_launch.call_args[1]
assert launch_kwargs["command"] is None
def test_connect__no_project__shows_error(self):
runner = CliRunner()
# Any arg flips the `_EndpointGroup`-style fallback into the hidden _run
# subcommand, where `--project` is enforced as required by Click. Bare
# `opik connect` prints group help instead; covered by the next test.
result = runner.invoke(cli, ["connect", "--name", "test-runner"])
assert result.exit_code == 2
assert "--project" in result.output
def test_connect__no_args__shows_help(self):
runner = CliRunner()
result = runner.invoke(cli, ["connect"])
assert result.exit_code == 0
# Help body lists `stop` and the docstring mentions `--project`.
assert "stop" in result.output
assert "--project" in result.output
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__network_failure__shows_clean_error(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.side_effect = httpx.ConnectError("Connection refused")
runner = CliRunner()
result = runner.invoke(cli, ["connect", "--project", "my-proj"])
assert result.exit_code != 0
assert "Could not connect to Opik backend" in result.output
assert "https://api.test/" in result.output
assert "Run: opik configure" in result.output
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__api_error__shows_error_body(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.side_effect = ApiError(status_code=409, body="conflict")
runner = CliRunner()
result = runner.invoke(cli, ["connect", "--project", "my-proj"])
assert result.exit_code != 0
assert "conflict" in result.output
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__tui_stopped_on_pairing_failure(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.side_effect = ApiError(status_code=500, body="boom")
runner = CliRunner()
runner.invoke(cli, ["connect", "--project", "my-proj"])
tui_instance = mock_tui_cls.return_value
tui_instance.stop.assert_called_once()
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__workspace_and_api_key_passed__forwarded_to_opik_constructor(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
client.config.config_file_exists = True
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-abc",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli,
[
"connect",
"--project",
"my-proj",
"--workspace",
"my-ws",
"--api-key",
"my-key",
],
)
assert result.exit_code == 0
mock_opik_cls.assert_called_once_with(
project_name="my-proj",
api_key="my-key",
workspace="my-ws",
_show_misconfiguration_message=False,
)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__local_api_key_overrides_global(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
client.config.config_file_exists = True
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-abc",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli,
[
"--api-key",
"global-key",
"connect",
"--project",
"my-proj",
"--api-key",
"local-key",
],
)
assert result.exit_code == 0
mock_opik_cls.assert_called_once_with(
project_name="my-proj",
api_key="local-key",
workspace=None,
_show_misconfiguration_message=False,
)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_connect__no_local_api_key__falls_back_to_global(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
):
client = MagicMock()
client.config.url_override = "https://api.test/"
client.config.config_file_exists = True
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-abc",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli,
["--api-key", "global-key", "connect", "--project", "my-proj"],
)
assert result.exit_code == 0
mock_opik_cls.assert_called_once_with(
project_name="my-proj",
api_key="global-key",
workspace=None,
_show_misconfiguration_message=False,
)
+240
View File
@@ -0,0 +1,240 @@
from unittest.mock import MagicMock, patch
import httpx
from click.testing import CliRunner
from opik.cli.main import cli
from opik.cli.local_runner.pairing import PairingResult, RunnerType
from opik.rest_api.core.api_error import ApiError
class TestEndpoint:
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_endpoint__with_command__calls_pairing_and_supervisor(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-xyz",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli, ["endpoint", "--project", "my-proj", "--", "echo", "hello"]
)
assert result.exit_code == 0
mock_run_pairing.assert_called_once()
call_kwargs = mock_run_pairing.call_args[1]
assert call_kwargs["project_name"] == "my-proj"
assert call_kwargs["runner_type"] == RunnerType.ENDPOINT
mock_launch.assert_called_once()
launch_kwargs = mock_launch.call_args[1]
assert launch_kwargs["command"] == ["echo", "hello"]
def test_endpoint__no_project__shows_error(self):
runner = CliRunner()
result = runner.invoke(cli, ["endpoint", "--", "echo", "hello"])
assert result.exit_code == 2
assert "--project" in result.output
def test_endpoint__no_command__shows_error(self):
runner = CliRunner()
result = runner.invoke(cli, ["endpoint", "--project", "my-proj"])
assert result.exit_code == 2
def test_endpoint__nonexistent_binary__shows_error(self):
runner = CliRunner()
result = runner.invoke(
cli,
["endpoint", "--project", "my-proj", "--", "nonexistent-binary-xyz-12345"],
)
assert result.exit_code == 2
assert "not found" in result.output.lower()
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_endpoint__network_failure__shows_clean_error(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.side_effect = httpx.ConnectError("Connection refused")
runner = CliRunner()
result = runner.invoke(
cli, ["endpoint", "--project", "my-proj", "--", "echo", "hello"]
)
assert result.exit_code != 0
assert "Could not connect to Opik backend" in result.output
assert "https://api.test/" in result.output
assert "Run: opik configure" in result.output
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_endpoint__tui_stopped_on_pairing_failure(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
):
client = MagicMock()
client.config.url_override = "https://api.test/"
mock_opik_cls.return_value = client
mock_run_pairing.side_effect = ApiError(status_code=500, body="boom")
runner = CliRunner()
runner.invoke(cli, ["endpoint", "--project", "my-proj", "--", "echo", "hello"])
tui_instance = mock_tui_cls.return_value
tui_instance.stop.assert_called_once()
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_endpoint__workspace_and_api_key_passed__forwarded_to_opik_constructor(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
):
client = MagicMock()
client.config.url_override = "https://api.test/"
client.config.config_file_exists = True
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-xyz",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli,
[
"endpoint",
"--project",
"my-proj",
"--workspace",
"my-ws",
"--api-key",
"my-key",
"--",
"echo",
"hello",
],
)
assert result.exit_code == 0
mock_opik_cls.assert_called_once_with(
project_name="my-proj",
api_key="my-key",
workspace="my-ws",
_show_misconfiguration_message=False,
)
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_endpoint__local_api_key_overrides_global(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
):
client = MagicMock()
client.config.url_override = "https://api.test/"
client.config.config_file_exists = True
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-xyz",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli,
[
"--api-key",
"global-key",
"endpoint",
"--project",
"my-proj",
"--api-key",
"local-key",
"--",
"echo",
"hello",
],
)
assert result.exit_code == 0
mock_opik_cls.assert_called_once_with(
project_name="my-proj",
api_key="local-key",
workspace=None,
_show_misconfiguration_message=False,
)
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
@patch("opik.cli.local_runner._run.RunnerTUI")
@patch("opik.cli.local_runner.pairing.launch_supervisor")
@patch("opik.cli.local_runner.pairing.run_pairing")
@patch("opik.cli.local_runner._run.Opik")
def test_endpoint__no_local_api_key__falls_back_to_global(
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
):
client = MagicMock()
client.config.url_override = "https://api.test/"
client.config.config_file_exists = True
mock_opik_cls.return_value = client
mock_run_pairing.return_value = PairingResult(
runner_id="r-xyz",
project_name="my-proj",
project_id="p-123",
bridge_key=b"\x00" * 32,
)
runner = CliRunner()
result = runner.invoke(
cli,
[
"--api-key",
"global-key",
"endpoint",
"--project",
"my-proj",
"--",
"echo",
"hello",
],
)
assert result.exit_code == 0
mock_opik_cls.assert_called_once_with(
project_name="my-proj",
api_key="global-key",
workspace=None,
_show_misconfiguration_message=False,
)
@@ -0,0 +1,282 @@
"""Unit tests for export error-handling fixes.
Covers three HIGH-priority fixes to prevent silent data loss during export:
HIGH-1 experiment.export_traces_by_ids: when get_trace_content raises 429,
the internal _fetch_trace_data retry decorator must kick in so the
trace is eventually exported, not silently dropped. Only genuine 404s
should be skipped without retrying.
HIGH-2 experiment.export_experiment_by_id: must re-raise on error so that
the all.py caller can set had_errors=True (previously it swallowed
all exceptions and returned a zero-stats tuple).
HIGH-3 all._fetch_experiments_page_raw: 429 must trigger a retry via the
_export_rest_retry decorator instead of silently returning ([], 0)
and losing an entire page of experiments.
"""
from unittest.mock import MagicMock, patch
import pytest
import httpx
from opik.rest_api.core.api_error import ApiError
from opik import exceptions as opik_exceptions
_EXPERIMENT_MODULE = "opik.cli.exports.experiment"
_ALL_MODULE = "opik.cli.exports.all"
# ---------------------------------------------------------------------------
# HIGH-1: export_traces_by_ids retries 429 rather than dropping the trace
# ---------------------------------------------------------------------------
class TestExportTracesByIdsRetry:
def _make_mock_trace(self, trace_id: str) -> MagicMock:
t = MagicMock()
t.model_dump.return_value = {"id": trace_id}
return t
def test_export_traces_by_ids__429_is_retried_not_dropped(self, tmp_path):
"""A 429 ApiError must trigger the retry decorator so the trace is
exported, not silently dropped."""
from opik.cli.exports.experiment import export_traces_by_ids
call_count = 0
def get_trace_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
raise ApiError(status_code=429, headers={})
return self._make_mock_trace("t1")
client = MagicMock()
client.get_trace_content.side_effect = get_trace_side_effect
client.search_spans.return_value = []
with patch(f"{_EXPERIMENT_MODULE}._fetch_trace_data.retry.sleep"):
exported, skipped = export_traces_by_ids(
client, ["t1"], tmp_path, "proj", None, "json", False, False
)
assert exported == 1, "429 must not silently drop the trace"
assert call_count == 2, "429 must trigger a retry"
def test_export_traces_by_ids__rate_limited_exception_is_retried(self, tmp_path):
"""OpikCloudRequestsRateLimited must also be retried."""
from opik.cli.exports.experiment import export_traces_by_ids
call_count = 0
def get_trace_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
raise opik_exceptions.OpikCloudRequestsRateLimited(
headers={}, retry_after=1.0
)
return self._make_mock_trace("t2")
client = MagicMock()
client.get_trace_content.side_effect = get_trace_side_effect
client.search_spans.return_value = []
with patch(f"{_EXPERIMENT_MODULE}._fetch_trace_data.retry.sleep"):
exported, skipped = export_traces_by_ids(
client, ["t2"], tmp_path, "proj", None, "json", False, False
)
assert exported == 1, "rate-limit exception must not silently drop the trace"
assert call_count == 2
def test_export_traces_by_ids__404_is_skipped_not_retried(self, tmp_path):
"""A genuine 404 (trace not found) should be skipped cleanly without retrying."""
from opik.cli.exports.experiment import export_traces_by_ids
call_count = 0
def get_trace_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
raise ApiError(status_code=404, headers={})
client = MagicMock()
client.get_trace_content.side_effect = get_trace_side_effect
exported, skipped = export_traces_by_ids(
client, ["missing-id"], tmp_path, "proj", None, "json", False, False
)
assert exported == 0, "404 trace must not be exported"
assert call_count == 1, "404 must not trigger a retry"
def test_export_traces_by_ids__success__exports_trace(self, tmp_path):
"""Happy path: trace is written to disk and counted as exported."""
from opik.cli.exports.experiment import export_traces_by_ids
client = MagicMock()
client.get_trace_content.return_value = self._make_mock_trace("t3")
client.search_spans.return_value = []
exported, skipped = export_traces_by_ids(
client, ["t3"], tmp_path, "proj", None, "json", False, False
)
assert exported == 1
# ---------------------------------------------------------------------------
# HIGH-2: export_experiment_by_id propagates errors instead of swallowing them
# ---------------------------------------------------------------------------
class TestExportExperimentByIdPropagatesErrors:
def test_export_experiment_by_id__api_error_propagates(self, tmp_path):
"""An unexpected error must propagate so the caller can set had_errors=True."""
from opik.cli.exports.experiment import export_experiment_by_id
client = MagicMock()
# Simulate a non-retriable API error (e.g. 403 Forbidden)
client.get_experiment_by_id.side_effect = ApiError(status_code=403, headers={})
with pytest.raises(ApiError):
export_experiment_by_id(
client=client,
output_dir=tmp_path,
project_name="proj",
experiment_id="exp-1",
max_traces=None,
force=True,
debug=False,
format="json",
)
def test_export_all_experiments__experiment_failure_sets_had_errors(self, tmp_path):
"""When export_experiment_by_id raises, _export_all_experiments must set
had_errors=True rather than silently counting it as skipped."""
from opik.cli.exports.all import _export_all_experiments
client = MagicMock()
experiments_dir = tmp_path / "experiments"
experiments_dir.mkdir()
# Two experiments: first succeeds, second raises.
exp1 = MagicMock()
exp1.id = "e1"
exp1.name = "exp-one"
exp2 = MagicMock()
exp2.id = "e2"
exp2.name = "exp-two"
def export_side_effect(
client, output_dir, project_name, exp_id, *args, **kwargs
):
if exp_id == "e2":
raise RuntimeError("transient failure")
m = MagicMock()
m.get_all_trace_ids.return_value = []
return {}, 1, m
with (
patch(
f"{_ALL_MODULE}._paginate_experiments",
return_value=[exp1, exp2],
),
patch(
f"{_ALL_MODULE}.export_experiment_by_id",
side_effect=export_side_effect,
),
patch(f"{_ALL_MODULE}.export_collected_trace_ids", return_value=(0, 0)),
):
_, _, _, _, had_errors = _export_all_experiments(
client=client,
project_dir=tmp_path,
project_name="proj",
project_id=None,
experiments_dir=experiments_dir,
max_results=None,
force=False,
debug=False,
format="json",
)
assert had_errors is True, "a failed experiment must set had_errors=True"
# ---------------------------------------------------------------------------
# HIGH-3: _fetch_experiments_page_raw retries 429 instead of returning [], 0
# ---------------------------------------------------------------------------
class TestFetchExperimentsPageRawRetry:
def _make_httpx_response(self, status_code: int, headers=None, json_body=None):
"""Build a minimal httpx.Response for raise_for_status() tests."""
headers = headers or {}
body = json_body or {"content": [], "total": 0}
import json as _json
# httpx.Response.raise_for_status() requires a request to be attached.
request = httpx.Request("GET", "http://test/v1/private/experiments")
response = httpx.Response(
status_code=status_code,
headers=headers,
content=_json.dumps(body).encode(),
request=request,
)
return response
def test_fetch_experiments_page_raw__429_raises_rate_limited_not_empty_list(
self,
):
"""A 429 response must raise OpikCloudRequestsRateLimited (for the retry
decorator), NOT silently return ([], 0)."""
from opik.cli.exports.all import _fetch_experiments_page_raw
client = MagicMock()
httpx_mock = MagicMock()
client.rest_client.experiments._raw_client._client_wrapper.httpx_client = (
httpx_mock
)
httpx_mock.request.return_value = self._make_httpx_response(
429, headers={"Retry-After": "5"}
)
with pytest.raises(opik_exceptions.OpikCloudRequestsRateLimited):
_fetch_experiments_page_raw.__wrapped__(client, 1, None)
def test_fetch_experiments_page_raw__429_is_retried_by_decorator(self):
"""When the underlying call raises 429, the decorator retries rather
than letting a silent ([], 0) escape to the caller."""
from opik.cli.exports.all import _fetch_experiments_page_raw
call_count = 0
def request_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
return self._make_httpx_response(429, headers={"Retry-After": "1"})
return self._make_httpx_response(
200,
json_body={
"content": [{"id": "exp-1", "name": "my-exp"}],
"total": 1,
},
)
client = MagicMock()
httpx_mock = MagicMock()
client.rest_client.experiments._raw_client._client_wrapper.httpx_client = (
httpx_mock
)
httpx_mock.request.side_effect = request_side_effect
with patch(f"{_ALL_MODULE}._fetch_experiments_page_raw.retry.sleep"):
items, total = _fetch_experiments_page_raw(client, 1, None)
assert call_count == 2, "429 must trigger a retry"
assert len(items) == 1, "successful retry must return the page contents"
assert total == 1
@@ -0,0 +1,788 @@
"""Tests for rate-limiting resilience in project export.
Covers the changes introduced to handle server-side 429 throttling:
Retry wait behaviour (via export_traces public API)
- 429 with Retry-After header → sleep honours header value
- 429 with Retry-After exceeding cap → sleep falls back to 30 s
- 429 with no Retry-After → sleep is 30 s
- non-429 transient error → sleep is in exponential backoff range [2, 60] s
export_traces (page-fetch failures)
- 429 on page N → page is skipped, remaining pages still fetched, had_errors=True
- all pages succeed → had_errors=False, all traces exported
- middle page failure → subsequent pages still exported
export_traces (span-fetch failures)
- bulk span fetch raises repeatedly → traces exported with empty spans, had_errors=True
export_traces (inter-page delay)
- time.sleep NOT called between successful full pages (inter-page sleep removed)
- time.sleep not called after partial (terminal) page
export_single_project (had_errors propagation)
- returns 4-tuple; had_errors=False on clean run, True when errors occurred
- first element is 0 when no traces exported or skipped, 1 otherwise
"""
import tempfile
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from opik.rest_api.core.api_error import ApiError
_MODULE = "opik.cli.exports.project"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_mock_trace(trace_id: str) -> MagicMock:
t = MagicMock()
t.id = trace_id
t.name = f"trace-{trace_id}"
t.model_dump.return_value = {
"id": trace_id,
"name": f"trace-{trace_id}",
"start_time": None,
"end_time": None,
"input": {},
"output": {},
"metadata": {},
"tags": [],
"feedback_scores": [],
"error_info": None,
"thread_id": None,
"created_at": None,
"created_by": None,
"last_updated_at": None,
"last_updated_by": None,
"visibility_mode": None,
"ttft": None,
"project_name": "test-project",
}
return t
def _make_page(traces, total=None):
page = MagicMock()
page.content = traces
page.total = total if total is not None else len(traces)
return page
def _make_full_page(n=500, offset=0):
"""Return a page with exactly *n* mock traces (matches the internal page_size=500).
Use *offset* to generate distinct trace IDs across pages.
"""
return _make_page([_make_mock_trace(f"bulk-{offset + i}") for i in range(n)])
def _uuid7(n: int) -> str:
"""Deterministic, valid UUIDv7 string for trace index *n*.
Version nibble is 7 and the variant nibble is RFC-4122 (8), so
``uuid.UUID(...).version == 7`` — this is what makes the exporter build a
bounded ``trace_id`` range filter instead of falling back to a full scan.
"""
high = f"{0x0190A1B2C3D4 + n:012x}"
return f"{high[:8]}-{high[8:12]}-7000-8000-{n:012x}"
def _make_full_page_uuid7(n=500, offset=0):
"""Like ``_make_full_page`` but with UUIDv7 trace ids (bounded-scan path)."""
return _make_page([_make_mock_trace(_uuid7(offset + i)) for i in range(n)])
# ---------------------------------------------------------------------------
# Retry wait behaviour — tested via export_traces (public API)
# ---------------------------------------------------------------------------
class TestExportTracesRetryWaitBehaviour:
"""Verify that the export-specific retry decorator applies the correct wait
durations when the underlying page-fetch raises errors. Tests go through
the public export_traces entrypoint so the actual retry decorator fires;
time.sleep is patched to capture what tenacity passes to it.
"""
def _run_with_first_call_raising(self, exc, tmp_path):
"""Run export_traces where the first page fetch raises *exc* then succeeds."""
from opik.cli.exports.project import export_traces
trace = _make_mock_trace("t1")
call_count = 0
def get_traces_side_effect(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
raise exc
return _make_page([trace])
mock_client = MagicMock()
mock_client.rest_client.traces.get_traces_by_project.side_effect = (
get_traces_side_effect
)
with patch(f"{_MODULE}._fetch_traces_page.retry.sleep") as mock_sleep:
export_traces(
client=mock_client,
project_name="proj",
project_dir=tmp_path,
max_results=None,
filter_string=None,
show_progress=False,
)
return [c.args[0] for c in mock_sleep.call_args_list]
def test_export_traces__page_fetch_429_with_retry_after_header__sleep_honours_header(
self, tmp_path
):
exc = ApiError(status_code=429, headers={"retry-after": "45"})
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
# Jitter adds 05 s on top of the header value.
assert any(45.0 <= s <= 50.0 for s in sleep_calls)
def test_export_traces__page_fetch_429_retry_after_exceeds_cap__sleep_clamped_to_max(
self, tmp_path
):
# 200 s > _EXPORT_MAX_RETRY_AFTER_SECONDS (120 s) → clamped to 120 s
exc = ApiError(status_code=429, headers={"retry-after": "200"})
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
# Jitter adds 05 s on top of the 120 s cap.
assert any(120.0 <= s <= 125.0 for s in sleep_calls)
def test_export_traces__page_fetch_429_retry_after_http_date__sleep_honours_header(
self, tmp_path
):
import email.utils
import time
# Build an HTTP-date Retry-After 60 seconds in the future.
future = email.utils.formatdate(timeval=time.time() + 60, usegmt=True)
exc = ApiError(status_code=429, headers={"retry-after": future})
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
# Should sleep approximately 60 s (allow ±2 s for execution) plus 05 s jitter.
assert any(58.0 <= s <= 67.0 for s in sleep_calls)
def test_export_traces__page_fetch_429_no_retry_after_header__sleep_is_30s(
self, tmp_path
):
exc = ApiError(status_code=429, headers={})
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
# Jitter adds 05 s on top of the 30 s fallback.
assert any(30.0 <= s <= 35.0 for s in sleep_calls)
def test_export_traces__page_fetch_non_429_transient_error__sleep_is_exponential_backoff(
self, tmp_path
):
exc = ApiError(status_code=503)
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
# Backoff is exponential starting at 2 s, capped at 60 s.
assert any(2.0 <= s <= 60.0 for s in sleep_calls)
# ---------------------------------------------------------------------------
# export_traces — page-fetch failures
# ---------------------------------------------------------------------------
class TestExportTracesPageFetchFailures:
def test_export_traces__first_page_429__skips_page_and_continues(self):
from opik.cli.exports.project import export_traces
trace_p2 = _make_mock_trace("t-p2")
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
mock_fetch.side_effect = [
ApiError(status_code=429, body="rate limited"),
_make_page([trace_p2]),
_make_page([]),
]
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
)
assert had_errors is True
assert exported == 1
assert skipped == 0
def test_export_traces__all_pages_succeed__happyflow(self):
from opik.cli.exports.project import export_traces
traces = [_make_mock_trace("t1"), _make_mock_trace("t2")]
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
mock_fetch.side_effect = [
_make_page(traces),
_make_page([]),
]
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
)
assert had_errors is False
assert exported == 2
def test_export_traces__middle_page_fails__subsequent_pages_still_exported(self):
"""A failed page in the middle must not prevent later pages from being fetched."""
from opik.cli.exports.project import export_traces
# Use page_size=2 so pages with 2 traces are "full" (2 == page_size).
# Keeps file-write count tiny (4 files) while still exercising the
# pagination loop — the loop only breaks early on a *short* page.
page1 = _make_full_page(2, offset=0)
page3 = _make_full_page(2, offset=2)
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
mock_fetch.side_effect = [
page1, # page 1: OK, 2 traces
ApiError(status_code=429), # page 2: rate limited
page3, # page 3: OK, 2 traces
_make_page([]), # page 4: end
]
with patch(f"{_MODULE}._fetch_spans_page", return_value=_make_page([])):
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
page_size=2,
)
assert had_errors is True
assert exported == 4 # page 1 and page 3 traces both exported
def test_export_traces__consecutive_failures_reach_cap__export_aborted(self):
"""After MAX_CONSECUTIVE_PAGE_FAILURES pages in a row fail, the loop aborts."""
from opik.cli.exports.project import (
export_traces,
MAX_CONSECUTIVE_PAGE_FAILURES,
)
mock_client = MagicMock()
# Every page raises a transient error
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
mock_fetch.side_effect = ApiError(status_code=503)
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
)
assert had_errors is True
assert exported == 0
# Should have stopped after MAX_CONSECUTIVE_PAGE_FAILURES attempts
assert mock_fetch.call_count == MAX_CONSECUTIVE_PAGE_FAILURES
def test_export_traces__permanent_api_error__raises_instead_of_skipping(self):
"""A 400 Bad Request must not be silently skipped — it should propagate."""
from opik.cli.exports.project import export_traces
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
mock_fetch.side_effect = ApiError(status_code=400, body="bad request")
with patch(f"{_MODULE}.time.sleep"):
with pytest.raises(ApiError) as exc_info:
export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
)
assert exc_info.value.status_code == 400
# Should have raised immediately, not retried
assert mock_fetch.call_count == 1
# ---------------------------------------------------------------------------
# export_traces — span-fetch failures
# ---------------------------------------------------------------------------
class TestExportTracesSpanFetchFailures:
def test_export_traces__bulk_span_fetch_fails__traces_exported_with_empty_spans(
self,
):
"""When the bulk span page fetch keeps failing, traces are still written
(with empty span lists) and had_errors is set to True."""
from opik.cli.exports.project import export_traces
t1 = _make_mock_trace("t1")
t2 = _make_mock_trace("t2")
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_trace_fetch:
mock_trace_fetch.side_effect = [
_make_page([t1, t2]),
_make_page([]),
]
with patch(f"{_MODULE}._fetch_spans_page") as mock_span_fetch:
mock_span_fetch.side_effect = ApiError(status_code=503)
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
)
assert had_errors is True
assert exported == 2 # both trace files written, just without spans
assert skipped == 0
# ---------------------------------------------------------------------------
# export_traces — bounded-chunk (memory) behaviour
# ---------------------------------------------------------------------------
class TestExportTracesChunking:
def test_export_traces__large_project_flushes_in_bounded_chunks(self):
"""A multi-page export writes traces incrementally, one bounded chunk at
a time, instead of buffering every trace and span before the first write.
Incremental flushing is what keeps peak memory from scaling with the
whole project (the previous all-at-once design OOM'd on large projects).
The export is driven purely through the public ``export_traces`` API,
mocking only the public REST client boundary
(``traces.get_traces_by_project`` / ``spans.get_spans_by_project``) — no
private helpers are patched. Correctness is asserted on the returned
tuple and the final ``trace_*.json`` artifacts. The bounded-chunk
property is asserted through robust invariants observed at the public
span endpoint (files already on disk when each span fetch fires), not a
brittle exact-call-order snapshot: the assertions hold regardless of the
exact chunk boundary the implementation picks.
"""
from opik.cli.exports.project import export_traces
# UUIDv7 ids so each chunk builds a bounded trace_id range filter and
# scans spans per chunk (the common, memory-bounded path).
page1 = _make_full_page_uuid7(2, offset=0)
page2 = _make_full_page_uuid7(2, offset=2)
page3 = _make_full_page_uuid7(2, offset=4)
mock_client = MagicMock()
expected_ids = [_uuid7(i) for i in range(6)]
with tempfile.TemporaryDirectory() as tmp:
project_dir = Path(tmp)
files_on_disk_at_each_span_fetch = []
def span_fetch_side_effect(*args, **kwargs):
# How many trace files already exist when a span fetch begins.
# A growing, sub-total count proves earlier chunks were flushed
# to disk before later chunks are processed (incremental), rather
# than one all-at-once sweep after buffering everything.
files_on_disk_at_each_span_fetch.append(
len(list(project_dir.glob("trace_*.json")))
)
return _make_page([])
mock_client.rest_client.traces.get_traces_by_project.side_effect = [
page1,
page2,
page3,
_make_page([]),
]
mock_client.rest_client.spans.get_spans_by_project.side_effect = (
span_fetch_side_effect
)
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=project_dir,
max_results=None,
filter_string=None,
show_progress=False,
page_size=2,
)
# Public result contract.
assert (exported, skipped, had_errors) == (6, 0, False)
# Final artifacts: exactly the six expected trace files were written.
written_ids = sorted(
p.name[len("trace_") : -len(".json")]
for p in project_dir.glob("trace_*.json")
)
assert written_ids == sorted(expected_ids)
# Bounded-chunk invariants (robust to the exact chunk boundary):
# - more than one span-fetch session → work is chunked, not one sweep;
# - counts never decrease → files accumulate as chunks flush;
# - a later fetch sees files already written → earlier chunks were
# flushed to disk before the last chunk was processed, so only ~one
# chunk's worth of traces is held in memory at a time.
assert len(files_on_disk_at_each_span_fetch) > 1
assert files_on_disk_at_each_span_fetch == sorted(
files_on_disk_at_each_span_fetch
)
assert files_on_disk_at_each_span_fetch[0] == 0
assert files_on_disk_at_each_span_fetch[-1] > 0
def test_export_traces__non_uuid7_ids__span_scan_bounded_per_chunk(self):
"""When trace ids aren't UUIDv7 the span scan can't be bounded by a
trace_id range, so the backend streams back every span in the project.
Rather than cache that whole-project span set — which would reintroduce
the unbounded memory peak this chunked export exists to prevent — each
chunk runs its own scan and keeps only its own traces' spans. Peak
memory therefore stays bounded to one chunk regardless of project size;
the trade-off is one unfiltered scan per chunk on this rare fallback.
"""
from opik.cli.exports.project import export_traces
# Non-UUIDv7 ids ("bulk-N") force the unfiltered-scan fallback.
page1 = _make_full_page(2, offset=0)
page2 = _make_full_page(2, offset=2)
page3 = _make_full_page(2, offset=4)
mock_client = MagicMock()
expected_ids = [f"bulk-{i}" for i in range(6)]
# Each per-chunk scan pages GET /spans starting at page 1; a fresh page-1
# request marks a new scan session.
scan_sessions = []
def span_fetch_side_effect(*args, **kwargs):
page = kwargs.get("page")
if page == 1:
scan_sessions.append([])
scan_sessions[-1].append(page)
return _make_page([])
mock_client.rest_client.traces.get_traces_by_project.side_effect = [
page1,
page2,
page3,
_make_page([]),
]
mock_client.rest_client.spans.get_spans_by_project.side_effect = (
span_fetch_side_effect
)
with tempfile.TemporaryDirectory() as tmp:
project_dir = Path(tmp)
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=project_dir,
max_results=None,
filter_string=None,
show_progress=False,
page_size=2,
)
assert (exported, skipped, had_errors) == (6, 0, False)
written_ids = sorted(
p.name[len("trace_") : -len(".json")]
for p in project_dir.glob("trace_*.json")
)
assert written_ids == sorted(expected_ids)
# One unfiltered scan per flushed chunk (3) — not a single cached
# whole-project scan — which is what keeps peak memory bounded to a chunk.
assert len(scan_sessions) == 3
def test_export_traces__span_scan_error_isolated_to_its_chunk(self):
"""A transient span-scan failure is confined to the chunk that hit it.
With per-chunk scanning there is no shared span cache to poison: the
failed chunk is exported with partial spans and had_errors is set, while
later chunks scan independently and export normally. The failure is
injected at the public REST boundary (``spans.get_spans_by_project``),
per the SDK testing guidelines, not at a private transport helper.
"""
from opik.cli.exports.project import export_traces
# Non-UUIDv7 ids force the unfiltered per-chunk scan fallback.
page1 = _make_full_page(2, offset=0)
page2 = _make_full_page(2, offset=2)
page3 = _make_full_page(2, offset=4)
mock_client = MagicMock()
expected_ids = [f"bulk-{i}" for i in range(6)]
with tempfile.TemporaryDirectory() as tmp:
project_dir = Path(tmp)
def get_spans_side_effect(*args, **kwargs):
# Fail every span page while the first chunk is being scanned
# (nothing flushed yet); once earlier chunks have written their
# files, succeed — proving the error didn't leak across chunks.
if not any(project_dir.glob("trace_*.json")):
raise ApiError(status_code=503)
return _make_page([])
mock_client.rest_client.traces.get_traces_by_project.side_effect = [
page1,
page2,
page3,
_make_page([]),
]
mock_client.rest_client.spans.get_spans_by_project.side_effect = (
get_spans_side_effect
)
# Neutralise the retry decorator's sleep (public boundary triggers
# the real 8-attempt retry) and the inter-page delay.
with patch(f"{_MODULE}._fetch_spans_page.retry.sleep"):
with patch(f"{_MODULE}.time.sleep"):
exported, skipped, had_errors = export_traces(
client=mock_client,
project_name="proj",
project_dir=project_dir,
max_results=None,
filter_string=None,
show_progress=False,
page_size=2,
)
# The failed chunk still wrote its traces (with partial spans) and the
# failure is flagged; later chunks were unaffected, so all 6 landed.
assert (exported, skipped) == (6, 0)
assert had_errors is True
written_ids = sorted(
p.name[len("trace_") : -len(".json")]
for p in project_dir.glob("trace_*.json")
)
assert written_ids == sorted(expected_ids)
# ---------------------------------------------------------------------------
# export_traces — inter-page delay
# ---------------------------------------------------------------------------
class TestExportTracesInterPageDelay:
def test_export_traces__multiple_full_pages__sleep_not_called_on_success(self):
"""sleep is NOT called between successful pages — the inter-page delay was
removed because server-side 429s are already handled by the retry decorator."""
from opik.cli.exports.project import export_traces, _PAGE_FETCH_DELAY_SECONDS
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
mock_fetch.side_effect = [
_make_full_page(2, offset=0), # page 1: full, IDs 01
_make_full_page(2, offset=2), # page 2: full, IDs 23
_make_page([]), # page 3: empty — end
]
with patch(f"{_MODULE}._fetch_spans_page", return_value=_make_page([])):
with patch(f"{_MODULE}.time.sleep") as mock_sleep:
export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
page_size=2,
)
from unittest.mock import call as mock_call
delay_calls = mock_sleep.call_args_list.count(
mock_call(_PAGE_FETCH_DELAY_SECONDS)
)
assert delay_calls == 0
def test_export_traces__single_partial_page__sleep_not_called(self):
"""A partial (< 100 trace) page means we're at the end — no sleep needed."""
from opik.cli.exports.project import export_traces, _PAGE_FETCH_DELAY_SECONDS
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
# Single partial page — loop breaks immediately on short-page check
mock_fetch.side_effect = [
_make_page([_make_mock_trace("t1")]),
]
with patch(f"{_MODULE}.time.sleep") as mock_sleep:
export_traces(
client=mock_client,
project_name="proj",
project_dir=Path(tmp),
max_results=None,
filter_string=None,
show_progress=False,
)
from unittest.mock import call as mock_call
delay_calls = mock_sleep.call_args_list.count(
mock_call(_PAGE_FETCH_DELAY_SECONDS)
)
assert delay_calls == 0
# ---------------------------------------------------------------------------
# export_single_project — had_errors and project_exported propagation
# ---------------------------------------------------------------------------
class TestExportSingleProjectReturnValues:
def _make_project(self, name="test-proj"):
p = MagicMock()
p.name = name
p.id = "proj-id-1"
return p
def test_export_single_project__clean_run__had_errors_false(self):
from opik.cli.exports.project import export_single_project
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
projects_dir = Path(tmp) / "projects"
projects_dir.mkdir()
with (
patch(f"{_MODULE}.export_traces") as mock_export,
patch(f"{_MODULE}.time.sleep"),
):
mock_export.return_value = (3, 0, False)
result = export_single_project(
client=mock_client,
project_name="test-proj",
project_dir=projects_dir,
filter_string=None,
max_results=None,
force=False,
debug=False,
format="json",
)
assert len(result) == 4
proj_count, _exported, _skipped, had_errors = result
assert had_errors is False
assert proj_count == 1 # traces were exported
def test_export_single_project__run_with_errors__had_errors_true(self):
from opik.cli.exports.project import export_single_project
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
projects_dir = Path(tmp) / "projects"
projects_dir.mkdir()
with (
patch(f"{_MODULE}.export_traces") as mock_export,
patch(f"{_MODULE}.time.sleep"),
):
mock_export.return_value = (2, 1, True)
result = export_single_project(
client=mock_client,
project_name="test-proj",
project_dir=projects_dir,
filter_string=None,
max_results=None,
force=False,
debug=False,
format="json",
)
assert len(result) == 4
_proj_count, _exported, _skipped, had_errors = result
assert had_errors is True
def test_export_single_project__no_traces_found__project_count_is_zero(self):
"""When there are no traces at all, the project-exported flag must be 0."""
from opik.cli.exports.project import export_single_project
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
projects_dir = Path(tmp) / "projects"
projects_dir.mkdir()
with (
patch(f"{_MODULE}.export_traces") as mock_export,
patch(f"{_MODULE}.time.sleep"),
):
mock_export.return_value = (0, 0, False) # nothing exported or skipped
result = export_single_project(
client=mock_client,
project_name="test-proj",
project_dir=projects_dir,
filter_string=None,
max_results=None,
force=False,
debug=False,
format="json",
)
proj_count, _exported, _skipped, _had_errors = result
assert proj_count == 0
def test_export_single_project__exception_during_export__returns_had_errors_true(
self,
):
from opik.cli.exports.project import export_single_project
mock_client = MagicMock()
with tempfile.TemporaryDirectory() as tmp:
projects_dir = Path(tmp) / "projects"
projects_dir.mkdir()
with patch(f"{_MODULE}.export_traces") as mock_export:
mock_export.side_effect = RuntimeError("unexpected failure")
result = export_single_project(
client=mock_client,
project_name="test-proj",
project_dir=projects_dir,
filter_string=None,
max_results=None,
force=False,
debug=False,
format="json",
)
assert result == (0, 0, 0, True)
@@ -0,0 +1,900 @@
"""Unit tests for experiment import functionality."""
import json
import sys
import types
from pathlib import Path
from typing import Dict, Any
from unittest.mock import Mock, MagicMock, patch
import pytest
# Mock the problematic imports before importing
# Mock the prompt import that's causing issues
sys.modules["opik.api_objects.prompt.prompt"] = MagicMock()
# Now we can import normally
from opik.cli.imports.experiment import ( # noqa: E402
ExperimentData,
load_experiment_data,
recreate_experiment,
_import_traces_for_project,
)
from opik.cli.imports.utils import ( # noqa: E402
translate_trace_id as utils_translate_trace_id,
sort_spans_topologically,
)
class TestExperimentData:
"""Test ExperimentData dataclass."""
def test_experiment_data_from_dict(self) -> None:
"""Test creating ExperimentData from dictionary."""
data = {
"experiment": {
"id": "exp-123",
"name": "test-experiment",
"dataset_name": "test-dataset",
},
"items": [
{"id": "item-1", "trace_id": "trace-1"},
{"id": "item-2", "trace_id": "trace-2"},
],
"downloaded_at": "2024-01-01T00:00:00",
}
exp_data = ExperimentData.from_dict(data)
assert exp_data.experiment["id"] == "exp-123"
assert exp_data.experiment["name"] == "test-experiment"
assert len(exp_data.items) == 2
assert exp_data.downloaded_at == "2024-01-01T00:00:00"
def test_experiment_data_from_dict_minimal(self) -> None:
"""Test creating ExperimentData with minimal data."""
data = {
"experiment": {"id": "exp-123", "dataset_name": "test-dataset"},
"items": [],
}
exp_data = ExperimentData.from_dict(data)
assert exp_data.experiment["id"] == "exp-123"
assert exp_data.items == []
assert exp_data.downloaded_at is None
def test_load_experiment_data_from_file(self, tmp_path: Path) -> None:
"""Test loading experiment data from JSON file."""
experiment_file = tmp_path / "experiment_test.json"
data = {
"experiment": {
"id": "exp-123",
"name": "test-experiment",
"dataset_name": "test-dataset",
},
"items": [{"id": "item-1"}],
}
with open(experiment_file, "w") as f:
json.dump(data, f)
exp_data = load_experiment_data(experiment_file)
assert isinstance(exp_data, ExperimentData)
assert exp_data.experiment["id"] == "exp-123"
assert len(exp_data.items) == 1
class TestTranslateTraceId:
"""Test translate_trace_id function."""
def test_translate_trace_id_found(self) -> None:
"""Test translating trace ID when mapping exists."""
trace_id_map = {"old-trace-1": "new-trace-1", "old-trace-2": "new-trace-2"}
result = utils_translate_trace_id("old-trace-1", trace_id_map)
assert result == "new-trace-1"
def test_translate_trace_id_not_found(self) -> None:
"""Test translating trace ID when mapping doesn't exist."""
trace_id_map = {"old-trace-1": "new-trace-1"}
result = utils_translate_trace_id("old-trace-2", trace_id_map)
assert result is None
def test_translate_trace_id_empty_map(self) -> None:
"""Test translating trace ID with empty map."""
trace_id_map: Dict[str, str] = {}
result = utils_translate_trace_id("old-trace-1", trace_id_map)
assert result is None
def test_translate_trace_id_requires_dict(self) -> None:
"""Test that translate_trace_id requires Dict, not Optional."""
# This test verifies the type signature is correct
# If someone tries to pass None, type checker should catch it
trace_id_map: Dict[str, str] = {} # Required, not Optional
result = utils_translate_trace_id("trace-1", trace_id_map)
assert result is None
class TestRecreateExperiment:
"""Test recreate_experiment function."""
@staticmethod
def _extract_items_arg_from_call_args(call_args: Any) -> Any:
"""Helper to extract the items argument from call_args.
Handles both positional and keyword arguments.
"""
if hasattr(call_args, "args") and call_args.args:
return call_args.args[0]
if hasattr(call_args, "kwargs"):
if "items" in call_args.kwargs:
return call_args.kwargs["items"]
for value in call_args.kwargs.values():
if isinstance(value, list) and len(value) > 0:
return value
return None
@pytest.fixture
def mock_client(self) -> Mock:
"""Create a mock Opik client."""
client = Mock()
# Ensure flush returns True to indicate success
client.flush = Mock(return_value=True)
# Mock dataset
mock_dataset = Mock()
mock_dataset.name = "test-dataset"
mock_dataset.__internal_api__insert_items_as_dataclasses__ = Mock()
# Mock experiment
mock_experiment = Mock()
mock_experiment.insert = Mock()
mock_experiment.id = "exp-123"
client.get_or_create_dataset = Mock(return_value=mock_dataset)
client.create_experiment = Mock(return_value=mock_experiment)
# Mock REST client for experiment items creation
client._rest_client = Mock()
client._rest_client.experiments = Mock()
client._rest_client.experiments.create_experiment_items = Mock()
return client
@pytest.fixture
def experiment_data(self) -> ExperimentData:
"""Create sample experiment data."""
return ExperimentData(
experiment={
"id": "exp-123",
"name": "test-experiment",
"dataset_name": "test-dataset",
"type": "regular",
},
items=[
{
"trace_id": "trace-1",
"dataset_item_id": "ds-item-1",
"dataset_item_data": {
"input": "test input",
"expected_output": "test output",
},
},
{
"trace_id": "trace-2",
"dataset_item_id": "ds-item-2",
"dataset_item_data": {"input": "test input 2"},
},
],
)
def test_recreate_experiment_requires_trace_id_map(
self, mock_client: Mock, experiment_data: ExperimentData
) -> None:
"""Test that recreate_experiment requires trace_id_map (not Optional)."""
# This test verifies the type signature
trace_id_map: Dict[str, str] = {
"trace-1": "new-trace-1",
"trace-2": "new-trace-2",
}
dataset_item_id_map: Dict[str, str] = {
"ds-item-1": "new-ds-item-1",
"ds-item-2": "new-ds-item-2",
}
# Should not accept None - type checker would catch this
# We test that it works with a dict
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
mock_id_helpers.generate_id = Mock(return_value="generated-id")
recreate_experiment(
mock_client,
experiment_data,
"test-project",
trace_id_map, # Required, not Optional
dataset_item_id_map, # Required for mapping dataset items
dry_run=False,
debug=False,
)
# Verify experiment items were created via REST API
assert mock_client._rest_client.experiments.create_experiment_items.called
def test_recreate_experiment_batches_dataset_items(
self, mock_client: Mock, experiment_data: ExperimentData
) -> None:
"""Test that experiment items are created in batch, not one at a time."""
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
mock_id_helpers.generate_id = Mock(side_effect=["exp-item-1", "exp-item-2"])
trace_id_map = {"trace-1": "new-trace-1", "trace-2": "new-trace-2"}
dataset_item_id_map = {
"ds-item-1": "new-ds-item-1",
"ds-item-2": "new-ds-item-2",
}
recreate_experiment(
mock_client,
experiment_data,
"test-project",
trace_id_map,
dataset_item_id_map,
dry_run=False,
debug=False,
)
# Verify batch insert was called ONCE with all items
assert (
mock_client._rest_client.experiments.create_experiment_items.call_count
== 1
)
# Verify it was called with a list of items (batch)
call_args = (
mock_client._rest_client.experiments.create_experiment_items.call_args
)
assert call_args is not None
# Extract experiment_items argument from call_args
experiment_items_arg = None
if hasattr(call_args, "kwargs") and "experiment_items" in call_args.kwargs:
experiment_items_arg = call_args.kwargs["experiment_items"]
elif hasattr(call_args, "args") and call_args.args:
experiment_items_arg = call_args.args[0]
assert experiment_items_arg is not None, (
f"Could not find experiment_items in call_args: {call_args}"
)
assert len(experiment_items_arg) == 2, (
f"Expected 2 items in batch, got {len(experiment_items_arg)}"
)
def test_recreate_experiment_uses_module_names_correctly(
self, mock_client: Mock, experiment_data: ExperimentData
) -> None:
"""Test that module names (id_helpers_module) are used correctly."""
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
mock_id_helpers.generate_id = Mock(return_value="generated-id")
trace_id_map = {"trace-1": "new-trace-1"}
dataset_item_id_map = {"ds-item-1": "new-ds-item-1"}
recreate_experiment(
mock_client,
experiment_data,
"test-project",
trace_id_map,
dataset_item_id_map,
dry_run=False,
debug=False,
)
# Verify id_helpers module is used (not checked for None)
assert mock_id_helpers.generate_id.called
def test_recreate_experiment_handles_empty_trace_id_map(
self, mock_client: Mock, experiment_data: ExperimentData
) -> None:
"""Test that empty trace_id_map is handled correctly."""
trace_id_map: Dict[str, str] = {} # Empty but valid
dataset_item_id_map: Dict[str, str] = {} # Empty but valid
recreate_experiment(
mock_client,
experiment_data,
"test-project",
trace_id_map,
dataset_item_id_map,
dry_run=False,
debug=False,
)
# Should still create experiment and dataset, but skip items
assert mock_client.get_or_create_dataset.called
assert mock_client.create_experiment.called
def test_recreate_experiment_dry_run(
self, mock_client: Mock, experiment_data: ExperimentData
) -> None:
"""Test dry run mode."""
trace_id_map = {"trace-1": "new-trace-1"}
dataset_item_id_map = {"ds-item-1": "new-ds-item-1"}
result = recreate_experiment(
mock_client,
experiment_data,
"test-project",
trace_id_map,
dataset_item_id_map,
dry_run=True,
debug=False,
)
assert result is True
# Should not create anything in dry run
assert not mock_client.get_or_create_dataset.called
assert not mock_client.create_experiment.called
def test_recreate_experiment_chunks_items_within_be_cap(
self, mock_client: Mock
) -> None:
# BE rejects a single ``create_experiment_items`` POST whose item
# count exceeds ``ExperimentItemsBatch``'s ``@Size(max=…)``. The
# actual cap value is BE-configured and may change over time, so
# we read it from the module-level constant the chunker uses and
# assert behavior against that, not against a hardcoded literal.
# Contract: every batch ``<= cap``, total across batches ==
# full item count, multiple batches when items > cap.
from opik.cli.imports.experiment import _EXPERIMENT_ITEMS_INSERT_BATCH_SIZE
cap = _EXPERIMENT_ITEMS_INSERT_BATCH_SIZE
# 2.5 × cap exercises the multi-batch path with a partial-last-
# batch remainder; any value above ``cap`` works for the contract.
n_items = cap * 5 // 2
experiment_data = ExperimentData(
experiment={
"id": "exp-big",
"name": "big-experiment",
"dataset_name": "test-dataset",
"type": "regular",
},
items=[
{
"trace_id": f"trace-{i}",
"dataset_item_id": f"ds-item-{i}",
"dataset_item_data": {"input": f"test input {i}"},
}
for i in range(n_items)
],
)
trace_id_map = {f"trace-{i}": f"new-trace-{i}" for i in range(n_items)}
dataset_item_id_map = {
f"ds-item-{i}": f"new-ds-item-{i}" for i in range(n_items)
}
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
mock_id_helpers.generate_id = Mock(
side_effect=[f"exp-item-{i}" for i in range(n_items)]
)
recreate_experiment(
mock_client,
experiment_data,
"test-project",
trace_id_map,
dataset_item_id_map,
dry_run=False,
debug=False,
)
create_calls = mock_client._rest_client.experiments.create_experiment_items.call_args_list
# ceil(n_items / cap) batched POSTs.
expected_batches = (n_items + cap - 1) // cap
assert len(create_calls) == expected_batches, (
f"expected {expected_batches} chunked create_experiment_items "
f"calls for {n_items} items at cap={cap}, got {len(create_calls)}"
)
# Every batch must respect the BE cap; total must equal the
# full item count (nothing dropped, nothing duplicated).
batch_sizes = [
len(call.kwargs.get("experiment_items", []))
if call.kwargs.get("experiment_items") is not None
else len(call.args[0])
for call in create_calls
]
assert all(size <= cap for size in batch_sizes), (
f"every batch must respect the BE cap (={cap}), got {batch_sizes}"
)
assert sum(batch_sizes) == n_items, (
f"all {n_items} items must end up across the chunked batches, "
f"got sum={sum(batch_sizes)}"
)
class TestImportTracesWithSpans:
"""Test trace import with span parent_span_id preservation."""
@pytest.fixture
def mock_client(self) -> Mock:
"""Create a mock Opik client."""
client = Mock()
client.flush = Mock()
# Mock trace creation
mock_trace = Mock()
mock_trace.id = "new-trace-1"
client.trace = Mock(return_value=mock_trace)
# Mock span creation
mock_spans = []
for i in range(3):
mock_span = Mock()
mock_span.id = f"new-span-{i + 1}"
mock_spans.append(mock_span)
client.span = Mock(side_effect=mock_spans)
return client
def test_import_traces_preserves_span_hierarchy(
self, mock_client: Mock, tmp_path: Path
) -> None:
"""Test that span parent_span_id relationships are preserved."""
# Create test trace file with spans
projects_dir = tmp_path / "projects" / "test-project"
projects_dir.mkdir(parents=True)
trace_data = {
"trace": {
"id": "original-trace-1",
"name": "test-trace",
"input": {},
"output": {},
},
"spans": [
{
"id": "span-1",
"name": "root-span",
"parent_span_id": None, # Root span
"input": {},
"output": {},
},
{
"id": "span-2",
"name": "child-span",
"parent_span_id": "span-1", # Child of span-1
"input": {},
"output": {},
},
{
"id": "span-3",
"name": "grandchild-span",
"parent_span_id": "span-2", # Child of span-2
"input": {},
"output": {},
},
],
}
trace_file = projects_dir / "trace_original-trace-1.json"
with open(trace_file, "w") as f:
json.dump(trace_data, f)
# Import traces
trace_id_map, _ = _import_traces_for_project(
mock_client, projects_dir, "test-project", dry_run=False, debug=False
)
# Verify spans were created
assert mock_client.span.call_count == 3
# Verify spans were created in correct order (root first, then children)
span_calls = mock_client.span.call_args_list
# First span should be root (no parent_span_id)
first_call = span_calls[0]
assert first_call.kwargs.get("parent_span_id") is None
# Second span should have parent_span_id set to first span's new ID
# Note: We can't easily verify the exact ID mapping without more complex mocking,
# but we can verify that parent_span_id is being passed
second_call = span_calls[1]
# The parent_span_id should be set (not None) since span-1 was created first
# and its new ID should be in span_id_map
assert "parent_span_id" in second_call.kwargs
# Third span (grandchild) should have parent_span_id set to second span's new ID
third_call = span_calls[2]
assert "parent_span_id" in third_call.kwargs
assert third_call.kwargs.get("parent_span_id") is not None
# Verify trace was created
assert mock_client.trace.called
assert "original-trace-1" in trace_id_map
def test_import_traces_preserves_deep_hierarchy(
self, mock_client: Mock, tmp_path: Path
) -> None:
"""Test that deep hierarchies (4+ levels) are preserved correctly."""
projects_dir = tmp_path / "projects" / "test-project"
projects_dir.mkdir(parents=True)
# Create a 4-level hierarchy: root -> child -> grandchild -> great-grandchild
# Spans are intentionally in wrong order to test sorting
trace_data = {
"trace": {
"id": "original-trace-1",
"name": "test-trace",
"input": {},
"output": {},
},
"spans": [
{
"id": "span-4",
"name": "great-grandchild",
"parent_span_id": "span-3",
"input": {},
"output": {},
},
{
"id": "span-2",
"name": "child",
"parent_span_id": "span-1",
"input": {},
"output": {},
},
{
"id": "span-1",
"name": "root",
"parent_span_id": None,
"input": {},
"output": {},
},
{
"id": "span-3",
"name": "grandchild",
"parent_span_id": "span-2",
"input": {},
"output": {},
},
],
}
trace_file = projects_dir / "trace_original-trace-1.json"
with open(trace_file, "w") as f:
json.dump(trace_data, f)
# Import traces
trace_id_map, _ = _import_traces_for_project(
mock_client, projects_dir, "test-project", dry_run=False, debug=False
)
# Verify all spans were created
assert mock_client.span.call_count == 4
span_calls = mock_client.span.call_args_list
# Verify order: root -> child -> grandchild -> great-grandchild
# First span should be root (no parent)
assert span_calls[0].kwargs.get("parent_span_id") is None
# Second span should be child (has parent)
assert span_calls[1].kwargs.get("parent_span_id") is not None
# Third span should be grandchild (has parent)
assert span_calls[2].kwargs.get("parent_span_id") is not None
# Fourth span should be great-grandchild (has parent)
assert span_calls[3].kwargs.get("parent_span_id") is not None
# Verify trace was created
assert mock_client.trace.called
assert "original-trace-1" in trace_id_map
def test_import_traces_sorts_spans_correctly(
self, mock_client: Mock, tmp_path: Path
) -> None:
"""Test that spans are sorted (root spans first, then children)."""
projects_dir = tmp_path / "projects" / "test-project"
projects_dir.mkdir(parents=True)
trace_data = {
"trace": {
"id": "original-trace-1",
"name": "test-trace",
"input": {},
"output": {},
},
"spans": [
{
"id": "span-2",
"name": "child-span",
"parent_span_id": "span-1", # Child - should come after root
"input": {},
"output": {},
},
{
"id": "span-1",
"name": "root-span",
"parent_span_id": None, # Root - should come first
"input": {},
"output": {},
},
],
}
trace_file = projects_dir / "trace_original-trace-1.json"
with open(trace_file, "w") as f:
json.dump(trace_data, f)
# Import traces
_, _ = _import_traces_for_project(
mock_client, projects_dir, "test-project", dry_run=False, debug=False
) # Returns (trace_id_map, stats), but we don't need them for this test
# Verify spans were created in correct order
span_calls = mock_client.span.call_args_list
# First span should be root (no parent)
assert span_calls[0].kwargs.get("parent_span_id") is None
# Second span should have parent_span_id
assert span_calls[1].kwargs.get("parent_span_id") is not None
class TestTopologicalSort:
"""Test the topological sort function for spans."""
def test_sort_spans_simple_hierarchy(self) -> None:
"""Test sorting with a simple 2-level hierarchy."""
spans = [
{"id": "span-2", "name": "child", "parent_span_id": "span-1"},
{"id": "span-1", "name": "root", "parent_span_id": None},
]
sorted_spans = sort_spans_topologically(spans)
# Root should come first
assert sorted_spans[0]["id"] == "span-1"
assert sorted_spans[0]["parent_span_id"] is None
# Child should come second
assert sorted_spans[1]["id"] == "span-2"
assert sorted_spans[1]["parent_span_id"] == "span-1"
def test_sort_spans_multi_level_hierarchy(self) -> None:
"""Test sorting with a 4-level hierarchy (root -> child -> grandchild -> great-grandchild)."""
spans = [
{"id": "span-4", "name": "great-grandchild", "parent_span_id": "span-3"},
{"id": "span-2", "name": "child", "parent_span_id": "span-1"},
{"id": "span-1", "name": "root", "parent_span_id": None},
{"id": "span-3", "name": "grandchild", "parent_span_id": "span-2"},
]
sorted_spans = sort_spans_topologically(spans)
# Verify order: root -> child -> grandchild -> great-grandchild
assert sorted_spans[0]["id"] == "span-1"
assert sorted_spans[0]["parent_span_id"] is None
assert sorted_spans[1]["id"] == "span-2"
assert sorted_spans[1]["parent_span_id"] == "span-1"
assert sorted_spans[2]["id"] == "span-3"
assert sorted_spans[2]["parent_span_id"] == "span-2"
assert sorted_spans[3]["id"] == "span-4"
assert sorted_spans[3]["parent_span_id"] == "span-3"
def test_sort_spans_multiple_roots(self) -> None:
"""Test sorting with multiple root spans."""
spans = [
{"id": "span-3", "name": "child-of-2", "parent_span_id": "span-2"},
{"id": "span-1", "name": "root-1", "parent_span_id": None},
{"id": "span-2", "name": "root-2", "parent_span_id": None},
]
sorted_spans = sort_spans_topologically(spans)
# Both roots should come before the child
root_ids = {sorted_spans[0]["id"], sorted_spans[1]["id"]}
assert root_ids == {"span-1", "span-2"}
assert sorted_spans[0]["parent_span_id"] is None
assert sorted_spans[1]["parent_span_id"] is None
# Child should come last
assert sorted_spans[2]["id"] == "span-3"
assert sorted_spans[2]["parent_span_id"] == "span-2"
def test_sort_spans_multiple_children(self) -> None:
"""Test sorting with a root that has multiple children."""
spans = [
{"id": "span-3", "name": "child-2", "parent_span_id": "span-1"},
{"id": "span-1", "name": "root", "parent_span_id": None},
{"id": "span-2", "name": "child-1", "parent_span_id": "span-1"},
]
sorted_spans = sort_spans_topologically(spans)
# Root should come first
assert sorted_spans[0]["id"] == "span-1"
assert sorted_spans[0]["parent_span_id"] is None
# Both children should come after root (order doesn't matter for siblings)
child_ids = {sorted_spans[1]["id"], sorted_spans[2]["id"]}
assert child_ids == {"span-2", "span-3"}
assert sorted_spans[1]["parent_span_id"] == "span-1"
assert sorted_spans[2]["parent_span_id"] == "span-1"
def test_sort_spans_missing_parent(self) -> None:
"""Test sorting when a span references a non-existent parent."""
spans = [
{"id": "span-1", "name": "root", "parent_span_id": None},
{"id": "span-2", "name": "orphan", "parent_span_id": "nonexistent"},
]
sorted_spans = sort_spans_topologically(spans)
# Both should be treated as roots (orphan becomes root)
assert len(sorted_spans) == 2
# Both should have no parent or invalid parent
for span in sorted_spans:
assert (
span["parent_span_id"] is None
or span["parent_span_id"] == "nonexistent"
)
def test_sort_spans_empty_list(self) -> None:
"""Test sorting with empty list."""
spans: list = []
sorted_spans = sort_spans_topologically(spans)
assert sorted_spans == []
def test_sort_spans_single_root(self) -> None:
"""Test sorting with single root span."""
spans = [{"id": "span-1", "name": "root", "parent_span_id": None}]
sorted_spans = sort_spans_topologically(spans)
assert len(sorted_spans) == 1
assert sorted_spans[0]["id"] == "span-1"
assert sorted_spans[0]["parent_span_id"] is None
def test_sort_spans_all_have_parents(self) -> None:
"""Test sorting when all spans have parents (no explicit root).
This tests the fix for the bug where empty root_spans would cause
the function to return an empty list, silently dropping all spans.
"""
spans = [
{"id": "span-1", "name": "child-1", "parent_span_id": "span-2"},
{"id": "span-2", "name": "child-2", "parent_span_id": "span-1"},
]
sorted_spans = sort_spans_topologically(spans)
# Should not return empty list - all spans should be included
assert len(sorted_spans) == 2
# Verify all spans are present
span_ids = {span["id"] for span in sorted_spans}
assert span_ids == {"span-1", "span-2"}
def test_sort_spans_cycle(self) -> None:
"""Test sorting with a cycle in the span graph.
This tests that cycles don't cause infinite loops and all spans
are still included in the result.
"""
spans = [
{"id": "span-1", "name": "span-1", "parent_span_id": "span-2"},
{"id": "span-2", "name": "span-2", "parent_span_id": "span-3"},
{"id": "span-3", "name": "span-3", "parent_span_id": "span-1"},
]
sorted_spans = sort_spans_topologically(spans)
# Should not return empty list - all spans should be included
assert len(sorted_spans) == 3
# Verify all spans are present
span_ids = {span["id"] for span in sorted_spans}
assert span_ids == {"span-1", "span-2", "span-3"}
def test_sort_spans_disconnected_components(self) -> None:
"""Test sorting with disconnected components (multiple separate graphs).
This tests that spans not reachable from root spans are still included.
"""
spans = [
{"id": "span-1", "name": "root-1", "parent_span_id": None},
{"id": "span-2", "name": "child-1", "parent_span_id": "span-1"},
{"id": "span-3", "name": "disconnected-1", "parent_span_id": "span-4"},
{"id": "span-4", "name": "disconnected-2", "parent_span_id": "span-3"},
]
sorted_spans = sort_spans_topologically(spans)
# All spans should be included
assert len(sorted_spans) == 4
# Verify all spans are present
span_ids = {span["id"] for span in sorted_spans}
assert span_ids == {"span-1", "span-2", "span-3", "span-4"}
# Root span should come first
assert sorted_spans[0]["id"] == "span-1"
assert sorted_spans[0]["parent_span_id"] is None
class TestModuleNameUsage:
"""Test that module names are used correctly (not checked for None)."""
def test_module_names_are_modules_not_variables(self) -> None:
"""Test that dataset_item_module and id_helpers_module are modules."""
from opik.cli.imports.experiment import dataset_item_module, id_helpers_module
# Modules should exist and be importable
assert dataset_item_module is not None
assert id_helpers_module is not None
# They should be modules, not None
assert isinstance(dataset_item_module, types.ModuleType)
assert isinstance(id_helpers_module, types.ModuleType)
class TestProjectTraceImport:
"""Test importing a project's traces from its project directory."""
@pytest.fixture
def mock_client(self) -> Mock:
"""Create a minimal mock Opik client."""
client = Mock()
client.flush = Mock()
mock_trace = Mock()
mock_trace.id = "new-trace-id"
client.trace = Mock(return_value=mock_trace)
client.span = Mock()
return client
def test_trace_files_in_project_dir_are_imported(
self, mock_client: Mock, tmp_path: Path
) -> None:
"""trace_{id}.json files directly under the project dir are imported and
appear in the returned trace_id_map, and the trace is created in the
named project."""
project_dir = tmp_path / "projects" / "my-project"
project_dir.mkdir(parents=True)
trace_id = "abc123"
trace_data: Dict[str, Any] = {
"trace": {"id": trace_id, "name": "t", "input": {}, "output": {}},
"spans": [],
}
trace_file = project_dir / f"trace_{trace_id}.json"
with open(trace_file, "w") as f:
json.dump(trace_data, f)
trace_id_map, _ = _import_traces_for_project(
mock_client, project_dir, "my-project", dry_run=False, debug=False
)
# The original trace ID must appear in the returned map
assert trace_id in trace_id_map
# The trace must be created in the named project (no "default" fallback)
assert mock_client.trace.call_args.kwargs.get("project_name") == "my-project"
def test_empty_project_dir_returns_empty_map(
self, mock_client: Mock, tmp_path: Path
) -> None:
"""A project dir with no trace files yields an empty map without error."""
project_dir = tmp_path / "projects" / "empty-project"
project_dir.mkdir(parents=True)
trace_id_map, stats = _import_traces_for_project(
mock_client, project_dir, "empty-project", dry_run=False, debug=False
)
assert trace_id_map == {}
assert stats["traces"] == 0
assert not mock_client.trace.called
+208
View File
@@ -0,0 +1,208 @@
"""Tests for the ``opik mcp configure`` command."""
import pathlib
from unittest.mock import patch
from click.testing import CliRunner
from opik.cli import cli
from opik.cli import mcp as mcp_cli
from opik.config import OpikConfig
def _config(**overrides) -> OpikConfig:
values = dict(url_override="https://www.comet.com/opik/api/", workspace="acme-ai")
values.update(overrides)
return OpikConfig(**values)
class TestResolveSetupParams:
def test_cloud__no_url_flags(self):
params = mcp_cli._resolve_setup_params(
_config(api_key="key", url_override="https://www.comet.com/opik/api/")
)
assert params["use_local"] is False
assert params["self_hosted_comet"] is False
assert params["api_url"] == "https://www.comet.com/opik/api/"
def test_self_hosted_comet__detected_from_opik_api_path(self):
params = mcp_cli._resolve_setup_params(
_config(api_key="key", url_override="https://opik.acme.com/opik/api/")
)
assert params["self_hosted_comet"] is True
assert params["use_local"] is False
assert params["base_url"] == "https://opik.acme.com/"
def test_localhost__is_use_local(self):
params = mcp_cli._resolve_setup_params(
_config(api_key=None, url_override="http://localhost:5173/api/")
)
assert params["use_local"] is True
assert params["self_hosted_comet"] is False
def test_self_hosted_oss__non_opik_path_is_use_local(self):
params = mcp_cli._resolve_setup_params(
_config(api_key=None, url_override="https://opik.acme.com/api/")
)
assert params["use_local"] is True
assert params["self_hosted_comet"] is False
class TestInstallCommand:
def test_install__reads_config_and_calls_setup(self):
runner = CliRunner()
with (
patch.object(
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
),
patch.object(
mcp_cli.interactive_helpers, "is_interactive", return_value=True
),
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
):
result = runner.invoke(cli, ["mcp", "configure"])
assert result.exit_code == 0
setup_spy.assert_called_once()
assert setup_spy.call_args.kwargs["api_key"] == "key"
assert setup_spy.call_args.kwargs["workspace"] == "acme-ai"
assert setup_spy.call_args.kwargs["force_local_server"] is False
def test_install__local_server_flag__forces_local(self):
runner = CliRunner()
with (
patch.object(
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
),
patch.object(
mcp_cli.interactive_helpers, "is_interactive", return_value=True
),
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
):
result = runner.invoke(cli, ["mcp", "configure", "--local-server"])
assert result.exit_code == 0
setup_spy.assert_called_once()
assert setup_spy.call_args.kwargs["force_local_server"] is True
def test_install__non_interactive__errors(self):
runner = CliRunner()
with (
patch.object(
mcp_cli.interactive_helpers, "is_interactive", return_value=False
),
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
):
result = runner.invoke(cli, ["mcp", "configure"])
assert result.exit_code != 0
assert "interactive terminal" in result.output
setup_spy.assert_not_called()
def test_install__no_config_user_declines__errors(self):
runner = CliRunner()
with (
patch.object(
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key=None)
),
patch.object(
mcp_cli.interactive_helpers, "is_interactive", return_value=True
),
patch.object(
mcp_cli.configure_cli, "run_interactive_configure"
) as configure_spy,
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
):
result = runner.invoke(cli, ["mcp", "configure"], input="n\n")
assert result.exit_code != 0
assert "opik configure" in result.output
configure_spy.assert_not_called()
setup_spy.assert_not_called()
def test_install__no_config_user_accepts__runs_configure_then_installs(self):
runner = CliRunner()
configs = iter([_config(api_key=None), _config(api_key="new-key")])
with (
patch.object(
mcp_cli.opik_config, "OpikConfig", side_effect=lambda: next(configs)
),
patch.object(
mcp_cli.interactive_helpers, "is_interactive", return_value=True
),
patch.object(
mcp_cli.configure_cli, "run_interactive_configure"
) as configure_spy,
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
):
result = runner.invoke(cli, ["mcp", "configure"], input="y\n")
assert result.exit_code == 0
configure_spy.assert_called_once_with(install_mcp=False)
setup_spy.assert_called_once()
assert setup_spy.call_args.kwargs["api_key"] == "new-key"
def test_status__lists_sdk_env_and_host_drift(self):
runner = CliRunner()
host = mcp_cli.mcp_status.HostStatus(
display_name="Claude Code",
config_path=pathlib.Path("/home/u/.claude.json"),
detected=True,
registered=True,
transport=mcp_cli.mcp_status.TRANSPORT_LOCAL,
points_to="http://localhost:5173/api/",
workspace="default",
in_sync=False,
)
with (
patch.object(
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
),
patch.object(
mcp_cli.mcp_status, "collect_host_statuses", return_value=[host]
),
):
result = runner.invoke(cli, ["mcp", "status"])
assert result.exit_code == 0
assert "Your Opik configuration" in result.output
assert "configured for 1 AI assistant" in result.output
assert "Claude Code" in result.output
assert "OUT OF SYNC with your Opik configuration" in result.output
assert "http://localhost:5173/api/" in result.output
assert "default" in result.output
def test_status__none_configured__suggests_configure(self):
runner = CliRunner()
with (
patch.object(
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
),
patch.object(mcp_cli.mcp_status, "collect_host_statuses", return_value=[]),
):
result = runner.invoke(cli, ["mcp", "status"])
assert result.exit_code == 0
assert "not configured for any AI assistant" in result.output
assert "opik mcp configure" in result.output
def test_install__local_without_api_key__proceeds(self):
runner = CliRunner()
with (
patch.object(
mcp_cli.opik_config,
"OpikConfig",
return_value=_config(
api_key=None, url_override="http://localhost:5173/api/"
),
),
patch.object(
mcp_cli.interactive_helpers, "is_interactive", return_value=True
),
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
):
result = runner.invoke(cli, ["mcp", "configure"])
assert result.exit_code == 0
setup_spy.assert_called_once()
assert setup_spy.call_args.kwargs["use_local"] is True
@@ -0,0 +1,553 @@
"""Tests for ``opik migrate dataset`` checkpoint/resume (OPIK-7168).
Two layers:
* ``TestMigrationCheckpoint`` -- the checkpoint store in isolation
(``opik.cli.migrate.checkpoint``): key derivation, atomic round-trip,
in-flight tracking, corrupt/foreign-schema tolerance, delete lifecycle.
* ``TestCascadeResume`` -- the cascade's resume behaviour
(``cascade_experiments`` with a ``checkpoint``): skip already-completed
experiments, re-migrate an interrupted experiment after deleting its partial
destination data, and seed the progress callback so a resumed run reports the
right completed count instead of starting at 0.
The cascade tests reuse the elaborate REST/client fakes from
``test_migrate_dataset_experiments_cascade`` so the resume behaviour is
exercised against the same call surface the real cascade drives.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any, List, Tuple
from unittest.mock import MagicMock
import pytest
from opik.cli.migrate import checkpoint as checkpoint_module
from opik.cli.migrate.checkpoint import (
SCHEMA_VERSION,
MigrationCheckpoint,
checkpoint_key,
checkpoint_path,
load_or_create,
)
from opik.cli.migrate.datasets.experiments import cascade_experiments
from .test_migrate_dataset_experiments_cascade import (
_Experiment,
_ExperimentItem,
_Trace,
_audit,
_cascade_rest_client,
_client_with_recreate_capture,
)
# ---------------------------------------------------------------------------
# Checkpoint store (unit)
# ---------------------------------------------------------------------------
@pytest.fixture(autouse=True)
def _isolate_checkpoint_dir(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""Point the checkpoint store at a tmp dir instead of the real ~/.opik.
The checkpoint now lives at a fixed per-user path (``checkpoint_dir``),
independent of cwd/--audit-log. Redirecting it here keeps the unit tests
hermetic and off the developer's home directory.
"""
cp_dir = tmp_path / "migrate-checkpoints"
cp_dir.mkdir(parents=True, exist_ok=True)
monkeypatch.setattr(checkpoint_module, "checkpoint_dir", lambda: cp_dir)
class TestMigrationCheckpoint:
def test_checkpoint_key__stable_and_distinct_per_tuple(self) -> None:
key = checkpoint_key("ws", "proj", "ds")
# Deterministic: same tuple -> same key across calls.
assert key == checkpoint_key("ws", "proj", "ds")
# Distinct tuples -> distinct keys, including the ambiguous shift of a
# separator across the three components (the \x00 join prevents
# "a"+"bc" colliding with "ab"+"c").
assert checkpoint_key("a", "bc", "d") != checkpoint_key("ab", "c", "d")
def test_checkpoint_path__lives_in_checkpoint_dir_keyed_by_hash(self) -> None:
key = checkpoint_key("ws", "proj", "ds")
path = checkpoint_path(key)
assert path.parent == checkpoint_module.checkpoint_dir()
assert path.name == f"opik-migrate-checkpoint-{key}.json"
def test_load_or_create__no_file__starts_fresh(self, tmp_path: Path) -> None:
cp = load_or_create(
workspace="ws",
project="proj",
dataset="ds",
)
assert cp.completed_count == 0
assert cp.in_flight is None
assert not cp.path.exists()
def test_flush_then_load__round_trips_completed_and_in_flight(
self, tmp_path: Path
) -> None:
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
cp.total_experiments = 3
cp.mark_in_flight(
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-ds"
)
cp.record_dest_trace_ids(["dest-trace-a", "dest-trace-b"])
cp.mark_completed("src-exp-1")
cp.flush()
reloaded = load_or_create(workspace="ws", project="proj", dataset="ds")
assert reloaded.total_experiments == 3
assert reloaded.completed_experiment_ids == {"src-exp-1"}
# ``mark_completed`` clears in_flight, so only the completed set survives.
assert reloaded.in_flight is None
def test_flush_preserves_in_flight_when_not_completed(self, tmp_path: Path) -> None:
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
cp.mark_in_flight(
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-ds"
)
cp.record_dest_trace_ids(["dest-trace-a"])
cp.flush()
reloaded = load_or_create(workspace="ws", project="proj", dataset="ds")
assert reloaded.in_flight is not None
assert reloaded.in_flight.source_experiment_id == "src-exp-2"
assert reloaded.in_flight.experiment_name == "exp-2"
assert reloaded.in_flight.dest_dataset_id == "dest-ds"
assert reloaded.in_flight.dest_trace_ids == ["dest-trace-a"]
def test_load_or_create__corrupt_file__starts_fresh(self, tmp_path: Path) -> None:
# A truncated / unparseable checkpoint (e.g. an interrupted write on a
# filesystem without atomic replace) must be treated as "no progress",
# not crash the migration.
key = checkpoint_key("ws", "proj", "ds")
checkpoint_path(key).write_text('{"schema_version": 1, "comp')
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
assert cp.completed_count == 0
assert cp.in_flight is None
def test_load_or_create__non_bool_dataset_phase_done__starts_fresh(
self, tmp_path: Path
) -> None:
# ``dataset_phase_done`` gates resume-vs-full-plan, so a truthy non-bool
# (the string "false", a non-empty list) must NOT be coerced to True and
# force a resume that skips rename/create/replay. A non-bool is
# malformed -> fresh.
key = checkpoint_key("ws", "proj", "ds")
for bad in ('"false"', "[1]", "1"):
# Use the CURRENT schema so the load reaches the non-bool guard
# rather than short-circuiting on a schema-version mismatch.
checkpoint_path(key).write_text(
f'{{"schema_version": {SCHEMA_VERSION}, "dataset_phase_done": {bad}}}'
)
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
assert cp.dataset_phase_done is False
def test_load_or_create__malformed_in_flight__starts_fresh(
self, tmp_path: Path
) -> None:
# A checkpoint with the right schema_version but a wrong-shaped
# ``in_flight`` (here a bare string, or a dict missing the required
# ``source_experiment_id``) must fall back to fresh rather than crash
# the CLI with a TypeError/KeyError.
key = checkpoint_key("ws", "proj", "ds")
for bad_in_flight in ('"not-a-dict"', '{"experiment_name": "x"}'):
checkpoint_path(key).write_text(
'{"schema_version": 1, "completed_experiment_ids": ["e1"], '
f'"in_flight": {bad_in_flight}}}'
)
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
assert cp.completed_count == 0
assert cp.in_flight is None
def test_load_or_create__corrupt_completed_ids__starts_fresh(
self, tmp_path: Path
) -> None:
# ``set("abc")`` / ``set([1, 2])`` don't raise, so a corrupt
# ``completed_experiment_ids`` (a bare string, or a list with non-string
# entries) would silently seed the wrong completed set and make the
# cascade skip/re-run the wrong experiments. It must fall back to fresh.
key = checkpoint_key("ws", "proj", "ds")
for bad in ('"abc"', "[1, 2, 3]", "{}"):
checkpoint_path(key).write_text(
f'{{"schema_version": 1, "completed_experiment_ids": {bad}}}'
)
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
assert cp.completed_experiment_ids == set()
def test_load_or_create__corrupt_dest_trace_ids__starts_fresh(
self, tmp_path: Path
) -> None:
# Same silent-corruption guard for the in-flight trace-id list.
key = checkpoint_key("ws", "proj", "ds")
checkpoint_path(key).write_text(
'{"schema_version": 1, "in_flight": '
'{"source_experiment_id": "e2", "dest_trace_ids": "trace-1"}}'
)
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
assert cp.in_flight is None
assert cp.completed_count == 0
def test_load_or_create__non_string_resume_names__starts_fresh(
self, tmp_path: Path
) -> None:
# OPIK-7162: source_dataset_id / source_name / temp_dest_name are the
# resume handles that flow unvalidated into client.get_dataset(name=...)
# and stream_dataset_items(dataset_name=...). A hand-edited non-string
# would otherwise reach the API; it must fall back to fresh, matching
# the same corrupt-recovery contract as the id collections. Uses the
# CURRENT schema so the load reaches the field guard (not the schema
# short-circuit).
key = checkpoint_key("ws", "proj", "ds")
for field in ("source_dataset_id", "source_name", "temp_dest_name"):
for bad in ("[1]", "{}", "5"):
checkpoint_path(key).write_text(
f'{{"schema_version": {SCHEMA_VERSION}, '
f'"dataset_phase_done": true, "{field}": {bad}}}'
)
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
# Fell back to fresh: no phase-done carried over from the bad file.
assert cp.dataset_phase_done is False
assert getattr(cp, field) is None
def test_load_or_create__unresolvable_home__returns_none(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
# A homeless environment (Path.home() raising, as in some CI/containers)
# must NOT crash the migration before it starts -- load_or_create
# returns None and the caller runs without resume support.
def _boom() -> Path:
raise RuntimeError("Could not determine home directory")
monkeypatch.setattr(checkpoint_module, "checkpoint_dir", _boom, raising=True)
assert load_or_create(workspace="ws", project="proj", dataset="ds") is None
def test_flush__write_failure__swallowed_not_raised(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
# A checkpoint is only a resume aid: a read-only / full disk on flush
# must be logged and swallowed, never abort an otherwise-healthy
# migration.
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
assert cp is not None
def _readonly(*_a: Any, **_k: Any) -> None:
raise OSError("Read-only file system")
monkeypatch.setattr(Path, "mkdir", _readonly, raising=True)
cp.flush() # must not raise
def test_flush_then_load__round_trips_dest_experiment_id(
self, tmp_path: Path
) -> None:
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
cp.mark_in_flight(
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-ds"
)
cp.record_dest_experiment_id("dest-exp-99")
cp.flush()
reloaded = load_or_create(workspace="ws", project="proj", dataset="ds")
assert reloaded.in_flight is not None
assert reloaded.in_flight.dest_experiment_id == "dest-exp-99"
def test_load_or_create__foreign_schema_version__starts_fresh(
self, tmp_path: Path
) -> None:
key = checkpoint_key("ws", "proj", "ds")
checkpoint_path(key).write_text(
'{"schema_version": 999, "completed_experiment_ids": ["src-exp-1"]}'
)
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
# A newer schema we can't interpret is ignored -> fresh start.
assert cp.completed_count == 0
def test_delete__removes_file_and_is_idempotent(self, tmp_path: Path) -> None:
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
cp.flush()
assert cp.path.exists()
cp.delete()
assert not cp.path.exists()
# Idempotent: deleting again is a no-op, not an error.
cp.delete()
def test_flush__atomic__no_stray_tmp_file_left(self, tmp_path: Path) -> None:
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
cp.flush()
# The temp file used for the atomic write must have been renamed away.
tmp_files = list(tmp_path.glob("*.tmp"))
assert tmp_files == []
# ---------------------------------------------------------------------------
# Cascade resume behaviour
# ---------------------------------------------------------------------------
def _two_experiment_rig() -> Tuple[Any, Any]:
"""Two source experiments, each with one item/trace, wired end-to-end.
Returns ``(rest_client, client)`` ready to pass to ``cascade_experiments``.
"""
exp1 = _Experiment(id="src-exp-1", name="exp-1", dataset_version_id="src-v-1")
exp2 = _Experiment(id="src-exp-2", name="exp-2", dataset_version_id="src-v-1")
item1 = _ExperimentItem(
id="i1",
experiment_id="src-exp-1",
trace_id="src-trace-1",
dataset_item_id="src-ds-item-1",
)
item2 = _ExperimentItem(
id="i2",
experiment_id="src-exp-2",
trace_id="src-trace-2",
dataset_item_id="src-ds-item-1",
)
rest_client = _cascade_rest_client(
experiments_by_dataset={"src-dataset-1": [exp1, exp2]},
items_by_experiment={"exp-1": [item1], "exp-2": [item2]},
traces_by_id={
"src-trace-1": _Trace(id="src-trace-1"),
"src-trace-2": _Trace(id="src-trace-2"),
},
spans_by_trace={"src-trace-1": [], "src-trace-2": []},
)
client = _client_with_recreate_capture(rest_client)
return rest_client, client
def _run_cascade(
rest_client: Any, client: Any, checkpoint: MigrationCheckpoint, **overrides: Any
) -> Tuple[Any, List[Tuple[int, int, str]]]:
"""Run ``cascade_experiments`` with a progress-capturing callback."""
progress_calls: List[Tuple[int, int, str]] = []
kwargs: dict = dict(
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
target_dataset_id="dest-dataset-1",
version_remap={"src-v-1": "dest-v-1"},
item_id_remap={"src-ds-item-1": "dest-ds-item-1"},
audit=_audit(),
checkpoint=checkpoint,
progress_callback=lambda c, t, label: progress_calls.append((c, t, label)),
)
kwargs.update(overrides)
result = cascade_experiments(client, rest_client, **kwargs)
return result, progress_calls
class TestCascadeResume:
def test_skip_completed__does_not_recreate_already_done_experiment(
self, tmp_path: Path
) -> None:
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
# Pretend exp-1 already migrated on a prior run.
cp.mark_completed("src-exp-1")
result, _ = _run_cascade(rest_client, client, cp)
# Only the not-yet-done experiment is recreated.
assert result.experiments_migrated == 1
assert client.create_experiment.call_count == 1
created_names = [
call.kwargs["name"] for call in client.create_experiment.call_args_list
]
assert created_names == ["exp-2"]
def test_skip_completed__all_done__no_recreation(self, tmp_path: Path) -> None:
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
cp.mark_completed("src-exp-1")
cp.mark_completed("src-exp-2")
result, _ = _run_cascade(rest_client, client, cp)
assert result.experiments_migrated == 0
client.create_experiment.assert_not_called()
def test_happyflow__marks_each_experiment_completed_and_flushes(
self, tmp_path: Path
) -> None:
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
result, _ = _run_cascade(rest_client, client, cp)
assert result.experiments_migrated == 2
assert cp.completed_experiment_ids == {"src-exp-1", "src-exp-2"}
# in_flight is cleared once the last experiment completes.
assert cp.in_flight is None
# Progress was flushed to disk (a re-run would see both done).
reloaded = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
assert reloaded.completed_experiment_ids == {"src-exp-1", "src-exp-2"}
def test_re_migrate_incomplete__deletes_partial_dest_data_before_rerun(
self, tmp_path: Path
) -> None:
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
# exp-1 finished on the prior run; exp-2 was interrupted mid-flight with
# two destination traces and its destination experiment row already
# created (id recorded before creation).
cp.mark_completed("src-exp-1")
cp.mark_in_flight(
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-dataset-1"
)
cp.record_dest_trace_ids(["stale-dest-trace-1", "stale-dest-trace-2"])
cp.record_dest_experiment_id("stale-dest-exp")
_run_cascade(rest_client, client, cp)
# Partial traces were deleted (spans cascade on the BE), so they don't
# duplicate on the re-run.
rest_client.traces.delete_traces.assert_called_once()
deleted_trace_ids = rest_client.traces.delete_traces.call_args.kwargs["ids"]
assert set(deleted_trace_ids) == {"stale-dest-trace-1", "stale-dest-trace-2"}
# The exact recorded destination experiment row was deleted BY ID --
# cleanup never does a name lookup, so no same-named peer can be hit.
rest_client.experiments.delete_experiments_by_id.assert_called_once()
deleted_exp_ids = (
rest_client.experiments.delete_experiments_by_id.call_args.kwargs["ids"]
)
assert deleted_exp_ids == ["stale-dest-exp"]
# in_flight is cleared after cleanup so a further crash won't re-delete.
assert cp.in_flight is None
def test_re_migrate_incomplete__no_experiment_row_yet__only_traces_deleted(
self, tmp_path: Path
) -> None:
# Interruption landed after traces were flushed but before the
# experiment row was created: dest_experiment_id is None, so cleanup
# deletes the traces and skips the experiment delete entirely (no
# name lookup that could hit a peer).
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
cp.mark_in_flight(
"src-exp-1", experiment_name="exp-1", dest_dataset_id="dest-dataset-1"
)
cp.record_dest_trace_ids(["stale-dest-trace-1"])
# no record_dest_experiment_id -> dest_experiment_id stays None
_run_cascade(rest_client, client, cp)
rest_client.traces.delete_traces.assert_called_once()
rest_client.experiments.delete_experiments_by_id.assert_not_called()
def test_re_migrate_incomplete__records_dest_ids_before_backend_write(
self, tmp_path: Path
) -> None:
# OPIK-7168 (#533/#536): the destination trace ids and experiment id
# must be flushed to the checkpoint BEFORE the backend write that
# persists them, so a crash in that window still leaves them recorded
# for the next run's cleanup. We assert the checkpoint file on disk
# already carries this experiment's dest trace ids by the time the
# trace-flush happens.
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
flushed_state: dict = {}
original_flush = client.flush
def _capture_on_first_flush() -> None:
# On the first client.flush() (the trace flush), read back the
# checkpoint from disk and snapshot its in-flight trace ids.
if "dest_trace_ids" not in flushed_state:
reloaded = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
flushed_state["dest_trace_ids"] = (
list(reloaded.in_flight.dest_trace_ids)
if reloaded.in_flight
else []
)
return original_flush()
client.flush = MagicMock(side_effect=_capture_on_first_flush)
_run_cascade(rest_client, client, cp)
# By the time the backend trace flush ran, the checkpoint on disk had
# already recorded this experiment's destination trace id.
assert len(flushed_state["dest_trace_ids"]) == 1
def test_progress_seeded__resumed_run_starts_at_completed_count(
self, tmp_path: Path
) -> None:
rest_client, client = _two_experiment_rig()
cp = load_or_create(
workspace="ws",
project="DestProject",
dataset="MyDataset",
)
cp.mark_completed("src-exp-1")
_, progress_calls = _run_cascade(rest_client, client, cp)
# First progress tick reports 1 already-completed of 2 total (not 0),
# so the bar opens at 50% rather than restarting.
assert progress_calls[0] == (1, 2, "exp-2")
# Final tick snaps to total/total "done".
assert progress_calls[-1] == (2, 2, "done")
def test_no_checkpoint__cascade_unchanged(self) -> None:
# Without a checkpoint the cascade behaves exactly as before: both
# experiments migrate, no delete calls, progress counts from 0.
rest_client, client = _two_experiment_rig()
progress_calls: List[Tuple[int, int, str]] = []
result = cascade_experiments(
client,
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
version_remap={"src-v-1": "dest-v-1"},
item_id_remap={"src-ds-item-1": "dest-ds-item-1"},
audit=_audit(),
progress_callback=lambda c, t, label: progress_calls.append((c, t, label)),
)
assert result.experiments_migrated == 2
rest_client.traces.delete_traces.assert_not_called()
assert progress_calls[0] == (0, 2, "exp-1")
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,496 @@
"""Tests for ``opik migrate dataset`` Slice 5 — optimization cascade.
Slice 5 sits between Slice 2 (``ReplayVersions``) and Slice 3
(``CascadeExperiments``). It recreates every optimization referencing
the source dataset under the destination project and populates
``plan.optimization_id_remap`` so the experiment cascade can re-point
each experiment's ``optimization_id`` FK.
Scope this module covers (ticket AC for OPIK-6532):
* Optimization fidelity round-trip (name / objective_name / status /
metadata / studio_config carried verbatim)
* READ-ONLY aggregates (num_trials / feedback_scores / baseline_* /
best_* / total_optimization_cost) not forwarded
* Fresh client-side UUID minted per source optimization; ``id_remap``
populated with source -> destination
* Empty optimization (zero trials) round-trips and produces a remap entry
* Zero-optimization datasets are a no-op (returns empty result, no
REST call)
* Per-optimization ``migrate_optimization`` audit record emitted
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
from unittest.mock import MagicMock
import pytest
from opik.cli.migrate.audit import AuditLog
from opik.cli.migrate.datasets import optimizations as optimizations_module
from opik.cli.migrate.datasets.optimizations import cascade_optimizations
# ---------------------------------------------------------------------------
# Stand-in wire shapes for the cascade's reads.
#
# The cascade reads ``OptimizationPublic`` from ``find_optimizations`` and
# writes via ``create_optimization``. Constructing the real pydantic
# models would impose ~20 fields per row; the cascade only inspects a
# handful, so we use plain objects with the fields it actually reads.
# ---------------------------------------------------------------------------
class _Optimization:
def __init__(
self,
*,
id: str,
name: Optional[str] = "opt",
objective_name: str = "accuracy",
status: str = "completed",
dataset_name: str = "MyDataset",
dataset_id: Optional[str] = "src-dataset-1",
metadata: Optional[Any] = None,
studio_config: Optional[Any] = None,
last_updated_at: Optional[Any] = None,
# READ-ONLY aggregates -- attached so we can assert the cascade
# does NOT forward them on the create call.
num_trials: Optional[int] = None,
feedback_scores: Optional[List[Any]] = None,
experiment_scores: Optional[List[Any]] = None,
baseline_objective_score: Optional[float] = None,
best_objective_score: Optional[float] = None,
baseline_duration: Optional[float] = None,
best_duration: Optional[float] = None,
baseline_cost: Optional[float] = None,
best_cost: Optional[float] = None,
total_optimization_cost: Optional[float] = None,
) -> None:
self.id = id
self.name = name
self.objective_name = objective_name
self.status = status
self.dataset_name = dataset_name
self.dataset_id = dataset_id
self.metadata = metadata
self.studio_config = studio_config
self.last_updated_at = last_updated_at
self.num_trials = num_trials
self.feedback_scores = feedback_scores
self.experiment_scores = experiment_scores
self.baseline_objective_score = baseline_objective_score
self.best_objective_score = best_objective_score
self.baseline_duration = baseline_duration
self.best_duration = best_duration
self.baseline_cost = baseline_cost
self.best_cost = best_cost
self.total_optimization_cost = total_optimization_cost
class _Page:
def __init__(self, content: List[_Optimization]) -> None:
self.content = content
def _audit() -> AuditLog:
"""Real AuditLog so the cascade's per-optimization ``record`` calls
land in a list we can assert on. The audit log carries no side
effects until ``write`` is called, so using the real type in tests
is cheaper than building a coupled mock."""
return AuditLog(command="migrate dataset", args={})
def _cascade_rest_client(
pages: List[_Page],
*,
create_side_effect: Optional[Any] = None,
) -> MagicMock:
"""Build a rest_client mock for the optimization cascade.
``find_optimizations`` is called once per page until a short page
breaks the loop. ``create_optimization`` is captured so individual
tests can assert on its kwargs.
"""
rest_client = MagicMock()
rest_client.optimizations.find_optimizations.side_effect = pages
if create_side_effect is not None:
rest_client.optimizations.create_optimization.side_effect = create_side_effect
else:
rest_client.optimizations.create_optimization.return_value = None
return rest_client
# ---------------------------------------------------------------------------
# Cascade tests
# ---------------------------------------------------------------------------
class TestCascadeOptimizations:
def test_no_source_optimizations__noop_no_remap_no_create_calls(self) -> None:
# Empty result page -> no create_optimization calls, empty
# remap, counters at zero. Slice 5 always emits the action; the
# cascade handles the zero case quietly.
rest_client = _cascade_rest_client([_Page([])])
result = cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
assert result.id_remap == {}
assert result.optimizations_total == 0
assert result.optimizations_migrated == 0
assert result.optimizations_skipped == 0
rest_client.optimizations.create_optimization.assert_not_called()
def test_one_optimization__fidelity_fields_forwarded_and_remap_populated(
self,
) -> None:
# Full fidelity copy of the fields the BE round-trips:
# name, objective_name, status, metadata, studio_config,
# last_updated_at. Dest dataset_name + project_name are taken
# from the action, NOT from the source row (the source name was
# renamed to <name>_v1 by the time the cascade runs).
opt = _Optimization(
id="src-opt-1",
name="overnight-tune",
objective_name="f1",
status="completed",
metadata={"trial_count": 12, "user_note": "kept"},
studio_config=None,
num_trials=12,
baseline_objective_score=0.5,
best_objective_score=0.81,
total_optimization_cost=4.20,
)
rest_client = _cascade_rest_client([_Page([opt])])
result = cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
rest_client.optimizations.create_optimization.assert_called_once()
kwargs = rest_client.optimizations.create_optimization.call_args.kwargs
# Fresh destination id -- not the source id.
assert kwargs["id"] != "src-opt-1"
assert kwargs["id"] == result.id_remap["src-opt-1"]
# Fidelity fields forwarded verbatim.
assert kwargs["name"] == "overnight-tune"
assert kwargs["objective_name"] == "f1"
assert kwargs["status"] == "completed"
assert kwargs["metadata"] == {"trial_count": 12, "user_note": "kept"}
# Destination scoping comes from the action, not the source row.
assert kwargs["dataset_name"] == "MyDataset"
assert kwargs["project_name"] == "DestProject"
def test_readonly_aggregates__not_forwarded_on_create(self) -> None:
# ``num_trials`` / ``feedback_scores`` / ``baseline_*`` / ``best_*``
# / ``total_optimization_cost`` are recomputed by the BE at read
# time from constituent experiments. The cascade must NOT forward
# them -- if it did, they'd be silently dropped on write today
# but might persist (with stale numbers) under a future BE
# schema change.
opt = _Optimization(
id="src-opt-1",
num_trials=99,
feedback_scores=[MagicMock()],
experiment_scores=[MagicMock()],
baseline_objective_score=0.1,
best_objective_score=0.9,
baseline_duration=10.0,
best_duration=2.0,
baseline_cost=1.0,
best_cost=0.5,
total_optimization_cost=42.0,
)
rest_client = _cascade_rest_client([_Page([opt])])
cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
kwargs = rest_client.optimizations.create_optimization.call_args.kwargs
for forbidden in (
"num_trials",
"feedback_scores",
"experiment_scores",
"baseline_objective_score",
"best_objective_score",
"baseline_duration",
"best_duration",
"baseline_cost",
"best_cost",
"total_optimization_cost",
):
assert forbidden not in kwargs, (
f"{forbidden} is BE-computed; cascade must not forward it"
)
def test_empty_optimization__zero_trials__still_migrated(self) -> None:
# Optimization with zero constituent experiments is still a
# user-visible row in the Optimization Studio UI; migrating it
# preserves the user's setup even if no trials ran yet.
opt = _Optimization(id="src-opt-1", num_trials=0, name="never-ran")
rest_client = _cascade_rest_client([_Page([opt])])
result = cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
assert result.optimizations_migrated == 1
assert "src-opt-1" in result.id_remap
rest_client.optimizations.create_optimization.assert_called_once()
def test_multiple_optimizations__each_gets_distinct_destination_id(
self,
) -> None:
opt1 = _Optimization(id="src-opt-1", name="run-a")
opt2 = _Optimization(id="src-opt-2", name="run-b")
opt3 = _Optimization(id="src-opt-3", name="run-c")
rest_client = _cascade_rest_client([_Page([opt1, opt2, opt3])])
result = cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
assert result.optimizations_migrated == 3
assert set(result.id_remap.keys()) == {
"src-opt-1",
"src-opt-2",
"src-opt-3",
}
# Destination ids must be unique (fresh UUIDs).
assert len(set(result.id_remap.values())) == 3
def test_audit__per_optimization_record_emitted(self) -> None:
# Each successful migration emits a ``migrate_optimization`` audit
# record carrying source_id -> destination_id mapping plus
# name / objective_name / status so users can trace what moved.
opt = _Optimization(
id="src-opt-1",
name="overnight",
objective_name="accuracy",
status="completed",
)
rest_client = _cascade_rest_client([_Page([opt])])
audit = _audit()
cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=audit,
)
migrate_records = [
r for r in audit.actions if r["type"] == "migrate_optimization"
]
assert len(migrate_records) == 1
record = migrate_records[0]
# Audit-level status conveys action success/failure -- must not
# be shadowed by the optimization's own status.
assert record["status"] == "ok"
assert record["source_id"] == "src-opt-1"
assert record["destination_id"] != "src-opt-1"
assert record["name"] == "overnight"
assert record["objective_name"] == "accuracy"
# Optimization's own status lives under a non-shadowing key.
assert record["optimization_status"] == "completed"
def test_pagination__exhausts_pages_until_short(self) -> None:
# find_optimizations is paginated; the cascade walks until a
# short page (< page_size). With page_size=100 in production,
# two pages of 100 and a final short page of 50 should fire
# three reads in order.
page_size = optimizations_module._OPTIMIZATION_PAGE_SIZE
page_a = _Page([_Optimization(id=f"src-{i}") for i in range(page_size)])
page_b = _Page(
[_Optimization(id=f"src-{page_size + i}") for i in range(page_size)]
)
page_c = _Page([_Optimization(id=f"src-{2 * page_size + i}") for i in range(3)])
rest_client = _cascade_rest_client([page_a, page_b, page_c])
result = cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
assert result.optimizations_total == 2 * page_size + 3
# 3 reads + N writes.
assert rest_client.optimizations.find_optimizations.call_count == 3
def test_create_optimization_raises__exception_propagates(self) -> None:
# The BE returns 409 on id conflict (we mint client-side UUIDs
# so within-run collisions are infinitesimal but possible across
# retries). Surface as a cascade error rather than silently
# continuing; the audit log's surrounding ``failed`` entry will
# capture partial progress via the umbrella action.
opt = _Optimization(id="src-opt-1")
rest_client = _cascade_rest_client(
[_Page([opt])],
create_side_effect=RuntimeError("409 conflict"),
)
with pytest.raises(RuntimeError, match="409 conflict"):
cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
def test_source_missing_id__skipped_not_remapped(self) -> None:
# Defensive path: BE returns an optimization without an id.
# Skip rather than fail -- other optimizations are still
# recoverable, and a non-zero ``optimizations_skipped`` counter
# in the audit surfaces the malformed row for investigation.
ok = _Optimization(id="src-opt-1")
bad = _Optimization(id="")
rest_client = _cascade_rest_client([_Page([ok, bad])])
result = cascade_optimizations(
rest_client,
source_dataset_id="src-dataset-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
assert result.optimizations_migrated == 1
assert result.optimizations_skipped == 1
assert list(result.id_remap.keys()) == ["src-opt-1"]
class TestCascadeOptimizationsWithExperiments:
"""End-to-end planner-shape assertion: the optimization remap built
by Slice 5 is observable to Slice 3's experiment cascade via the
shared ``MigrationPlan``. The actual experiment-cascade FK forwarding
is asserted in ``test_migrate_dataset_experiments_cascade``; this
test only pins the wiring (action ordering + remap propagation).
"""
def test_plan_remap__populated_by_cascade__readable_by_subsequent_actions(
self,
) -> None:
from opik.cli.migrate.datasets.planner import MigrationPlan
from opik.cli.migrate.datasets.resolver import ResolvedDataset
opt = _Optimization(id="src-opt-1", name="trained")
rest_client = _cascade_rest_client([_Page([opt])])
plan = MigrationPlan(
source=ResolvedDataset(
id="src-1",
name="MyDataset",
project_name=None,
description=None,
visibility=None,
tags=None,
type="dataset",
),
target_name="MyDataset",
to_project="DestProject",
)
result = cascade_optimizations(
rest_client,
source_dataset_id="src-1",
target_dataset_name="MyDataset",
target_project_name="DestProject",
audit=_audit(),
)
plan.optimization_id_remap.update(result.id_remap)
# Subsequent CascadeExperiments would read plan.optimization_id_remap;
# the wiring works as long as result.id_remap propagates here.
assert plan.optimization_id_remap == result.id_remap
assert "src-opt-1" in plan.optimization_id_remap
# ---------------------------------------------------------------------------
# Studio config Read -> Write reconstruction
# ---------------------------------------------------------------------------
class TestStudioConfigReconstruction:
"""The Fern-generated Read and Write variants of
``OptimizationStudioConfig`` are structurally identical but
nominally distinct; the cascade has to reconstruct the Write
variant from the Read variant so the typed ``create_optimization``
parameter is satisfied. Pin the round-trip so any future Fern
regeneration that changes the shape on one side gets caught.
"""
def test_none_in_none_out(self) -> None:
from opik.cli.migrate.datasets.optimizations import (
_studio_config_public_to_write,
)
assert _studio_config_public_to_write(None) is None
def test_round_trip__fields_preserved(self) -> None:
from opik.cli.migrate.datasets.optimizations import (
_studio_config_public_to_write,
)
from opik.rest_api.types.optimization_studio_config_public import (
OptimizationStudioConfigPublic,
)
from opik.rest_api.types.studio_evaluation_public import (
StudioEvaluationPublic,
)
from opik.rest_api.types.studio_llm_model_public import (
StudioLlmModelPublic,
)
from opik.rest_api.types.studio_message_public import StudioMessagePublic
from opik.rest_api.types.studio_metric_public import StudioMetricPublic
from opik.rest_api.types.studio_optimizer_public import (
StudioOptimizerPublic,
)
from opik.rest_api.types.studio_prompt_public import StudioPromptPublic
public = OptimizationStudioConfigPublic(
dataset_name="MyDataset",
prompt=StudioPromptPublic(
messages=[StudioMessagePublic(role="system", content="you are helpful")]
),
llm_model=StudioLlmModelPublic(
model="gpt-4o", parameters={"temperature": 0.2}
),
evaluation=StudioEvaluationPublic(
metrics=[StudioMetricPublic(type="accuracy")]
),
optimizer=StudioOptimizerPublic(type="grid", parameters={"steps": 5}),
)
write = _studio_config_public_to_write(public)
assert write is not None
# Round-trip via model_dump -- the typed Write variant accepts
# extra fields so this also covers any forward-compatible BE
# additions on the Public variant.
write_dump: Dict[str, Any] = write.model_dump()
public_dump: Dict[str, Any] = public.model_dump()
assert write_dump == public_dump
@@ -0,0 +1,950 @@
"""Planner + CLI-help tests for ``opik migrate dataset``.
The planner cases (conflict, project-not-found, default flow ordering)
live here; the meaty version-replay tests live in
``test_migrate_dataset_version_replay.py`` and the cascade tests live in
``test_migrate_dataset_experiments_cascade.py``.
Shared helpers (``_DatasetRow``, ``_Page``, ``_planner_rest_client``)
come from ``_migrate_helpers``.
"""
from __future__ import annotations
import json
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from click.testing import CliRunner
from opik.cli import cli
from opik.cli.migrate.audit import AuditLog
from opik.cli.migrate.checkpoint import MigrationCheckpoint
from opik.cli.migrate.datasets import planner as planner_module
from opik.cli.migrate.datasets.planner import TEMP_MIGRATION_MARKER_TAG
from opik.cli.migrate.datasets.resume import ReconstructedRemaps
from opik.cli.migrate.errors import (
ConflictError,
DatasetNotFoundError,
MigrationError,
ProjectNotFoundError,
)
from ._migrate_helpers import (
_build_fake_client,
_DatasetRow,
_Page,
_planner_client,
_planner_rest_client,
)
# ---------------------------------------------------------------------------
# Elapsed-time formatter
# ---------------------------------------------------------------------------
class TestFormatElapsed:
"""Pin the wall-clock duration renderer used in the migrate success /
failure lines. Sub-minute → one decimal of seconds; past a minute →
integer ``Mm Ss`` or ``Hh Mm Ss`` (no fractional seconds).
"""
def test_sub_minute__one_decimal_seconds(self) -> None:
from opik.cli.migrate.main import _format_elapsed
assert _format_elapsed(0.0) == "0.0s"
assert _format_elapsed(12.34) == "12.3s"
assert _format_elapsed(59.99) == "60.0s"
def test_minute_range__integer_m_s(self) -> None:
from opik.cli.migrate.main import _format_elapsed
assert _format_elapsed(60.0) == "1m 0s"
assert _format_elapsed(125.7) == "2m 5s"
def test_hour_range__integer_h_m_s(self) -> None:
from opik.cli.migrate.main import _format_elapsed
assert _format_elapsed(3600.0) == "1h 0m 0s"
assert _format_elapsed(3725.0) == "1h 2m 5s"
# ---------------------------------------------------------------------------
# OPIK-6599: loud-fail on skipped items
#
# When the cascade emits any ``skip`` audit record, the migrate must:
# 1. Finalize the audit log to ``failed`` (not ``ok``)
# 2. Print a SKIP_SUMMARY line to stderr (not stdout) so CI gates can
# grep without parsing the JSON
# 3. Exit non-zero
#
# Tests below cover ``_finalize_with_skips_or_ok`` directly so they don't
# need a fully-wired Opik client + REST server stub. The CLI is only the
# wrapper around this helper.
# ---------------------------------------------------------------------------
class TestFinalizeWithSkipsOrOk:
def _make_audit_with_skips(self) -> AuditLog:
audit = AuditLog(command="opik migrate dataset", args={})
audit.record(
type="skip",
status="skipped",
details={
"reason": "items_missing_dataset_item_remap",
"experiment_id": "src-exp-1",
"experiment_name": "exp-1",
"count": 2500,
"sample_source_ids": ["src-ds-item-1", "src-ds-item-2"],
},
)
return audit
def test_skips_present__finalizes_failed_exits_1_stderr_summary(
self, tmp_path, capsys
) -> None:
from opik.cli.migrate.main import _finalize_with_skips_or_ok
audit = self._make_audit_with_skips()
audit_path = tmp_path / "audit.json"
with pytest.raises(SystemExit) as exc:
_finalize_with_skips_or_ok(
audit,
audit_path,
name="MyDataset",
target_label="MyDataset",
target_project="DestProject",
elapsed_seconds=12.3,
)
# AC 1: non-zero exit code
assert exc.value.code == 1
captured = capsys.readouterr()
# AC 3: skip message on stderr (not stdout), with the
# machine-parseable SKIP_SUMMARY suffix
assert "SKIP_SUMMARY:" in captured.err
assert "items_skipped_missing_item=2500" in captured.err
assert "experiments_skipped=0" in captured.err
assert "items_skipped_missing_trace=0" in captured.err
assert "NOT rolled back" in captured.err
# Rollback hint names the entities the operator must remove,
# the destination project, and the rename-back step on the source.
assert "roll back manually" in captured.err
assert "DestProject" in captured.err
assert "MyDataset_v1" in captured.err
# AC 2: audit finalized to failed with skip record intact
on_disk = json.loads(audit_path.read_text())
assert on_disk["status"] == "failed"
assert any(a.get("status") == "skipped" for a in on_disk["actions"])
def test_no_skips__finalizes_ok_no_exit_no_stderr(self, tmp_path, capsys) -> None:
from opik.cli.migrate.main import _finalize_with_skips_or_ok
audit = AuditLog(command="opik migrate dataset", args={})
audit.record(type="rename_source", status="ok", details={})
audit_path = tmp_path / "audit.json"
# Happy path returns without raising; happy-path message goes to
# stdout, stderr stays clean.
_finalize_with_skips_or_ok(
audit,
audit_path,
name="MyDataset",
target_label="MyDataset",
target_project="DestProject",
elapsed_seconds=5.0,
)
captured = capsys.readouterr()
assert "SKIP_SUMMARY:" not in captured.err
on_disk = json.loads(audit_path.read_text())
assert on_disk["status"] == "ok"
def test_multiple_skip_records__totals_aggregated_by_reason(
self, tmp_path, capsys
) -> None:
from opik.cli.migrate.main import _finalize_with_skips_or_ok
# Two experiments, each contributing skips for both reasons.
audit = AuditLog(command="opik migrate dataset", args={})
for exp_id in ("src-exp-1", "src-exp-2"):
audit.record(
type="skip",
status="skipped",
details={
"reason": "items_missing_trace_remap",
"experiment_id": exp_id,
"experiment_name": exp_id,
"count": 7,
"sample_source_ids": [],
},
)
audit.record(
type="skip",
status="skipped",
details={
"reason": "items_missing_dataset_item_remap",
"experiment_id": exp_id,
"experiment_name": exp_id,
"count": 3,
"sample_source_ids": [],
},
)
with pytest.raises(SystemExit):
_finalize_with_skips_or_ok(
audit,
tmp_path / "audit.json",
name="MyDataset",
target_label="MyDataset",
target_project="DestProject",
elapsed_seconds=1.0,
)
captured = capsys.readouterr()
# 7 + 7 = 14 trace skips, 3 + 3 = 6 dataset-item skips
assert "items_skipped_missing_trace=14" in captured.err
assert "items_skipped_missing_item=6" in captured.err
# ---------------------------------------------------------------------------
# Help text
# ---------------------------------------------------------------------------
class TestMigrateHelp:
def test_migrate_group__help_invoked__lists_subcommands(self) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["migrate", "--help"])
assert result.exit_code == 0
assert "Migrate Opik entities" in result.output
assert "dataset" in result.output
def test_migrate_dataset__help_invoked__lists_required_flags(self) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["migrate", "dataset", "--help"])
assert result.exit_code == 0
assert "--to-project" in result.output
assert "--from-project" in result.output
assert "--dry-run" in result.output
def test_migrate_dataset__help_invoked__lists_exclude_experiments(self) -> None:
# OPIK-7161 AC: the opt-out flag must be discoverable in --help.
runner = CliRunner()
result = runner.invoke(cli, ["migrate", "dataset", "--help"])
assert result.exit_code == 0
assert "--exclude-experiments" in result.output
def test_migrate_dataset__exclude_experiments__cli_run_skips_and_reports(
self, tmp_path
) -> None:
# OPIK-7161: exercise the flag through the public Click entrypoint,
# not just the finalize helper, so the option -> build_dataset_plan
# -> finalize plumbing in migrate_dataset_command is covered end to
# end (per .agents/skills/python-sdk/testing.md: test the public API).
# The fake client mocks the whole rename/create/replay surface; with
# --exclude-experiments the plan carries no cascade actions, so the
# command reaches the success finalize with zero experiment work.
client, _, _ = _build_fake_client(
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
destination_rows=[],
items=[{"id": "item-a", "input": "hello"}],
)
audit_path = tmp_path / "audit.json"
runner = CliRunner()
with patch("opik.cli.migrate.main._build_client", return_value=client):
result = runner.invoke(
cli,
[
"migrate",
"dataset",
"MyDataset",
"--to-project",
"B",
"--exclude-experiments",
"--audit-log",
str(audit_path),
],
)
assert result.exit_code == 0, result.output
# User-facing output makes the intentional skip clear.
assert "--exclude-experiments" in result.output
assert "skipped" in result.output
# No experiment cascade ran: the source dataset was never queried for
# experiments (find_experiments belongs only to the cascade path).
assert client.rest_client.experiments.find_experiments.call_count == 0
# Audit log finalized ok and recorded the flag in its args.
on_disk = json.loads(audit_path.read_text())
assert on_disk["status"] == "ok"
assert on_disk["args"]["exclude_experiments"] is True
cascade_types = {a.get("type") for a in on_disk["actions"]}
assert "cascade_experiments" not in cascade_types
assert "cascade_optimizations" not in cascade_types
class TestTempDestRenameOnSuccess:
"""OPIK-7162 acceptance criteria, exercised through the public Click
entrypoint: the source keeps its name until the copy succeeds, then the
handoff runs (source -> _v1, temp -> original). A mid-run failure leaves
the source name untouched, and a re-run after failure is safe/idempotent.
All cases use ``--exclude-experiments`` so the plan is the minimal
Create -> Replay -> Rename -> Promote shape and the assertions stay
focused on the handoff, not the cascade.
"""
def _run(self, client, tmp_path, extra_args=()):
audit_path = tmp_path / "audit.json"
runner = CliRunner()
with patch("opik.cli.migrate.main._build_client", return_value=client):
result = runner.invoke(
cli,
[
"migrate",
"dataset",
"MyDataset",
"--to-project",
"B",
"--exclude-experiments",
"--audit-log",
str(audit_path),
*extra_args,
],
)
return result, audit_path
def test_success__source_renamed_to_v1_and_dest_promoted_to_original(
self, tmp_path
) -> None:
# AC: a successful migration leaves the destination under the
# original name and the source under the _v1 suffix. The two renames
# happen via update_dataset PUTs; the destination is created under
# the temp name first.
client, _, _ = _build_fake_client(
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
destination_rows=[],
items=[{"id": "item-a", "input": "hello"}],
)
result, audit_path = self._run(client, tmp_path)
assert result.exit_code == 0, result.output
rest = client.rest_client
# Destination created under the temp name (not the final name), and
# stamped with the migration marker tag so a future re-run can prove
# it's ours before discarding it.
create_kwargs = rest.datasets.create_dataset.call_args.kwargs
assert create_kwargs["name"] == "MyDataset__migrating"
assert TEMP_MIGRATION_MARKER_TAG in (create_kwargs["tags"] or [])
# Two rename PUTs: source -> _v1, then temp -> original. The promote PUT
# re-passes the source's ORIGINAL tags (marker stripped).
rename_calls = [c.kwargs for c in rest.datasets.update_dataset.call_args_list]
source_rename = next(c for c in rename_calls if c["id"] == "src-1")
assert source_rename["name"] == "MyDataset_v1"
promote = next(c for c in rename_calls if c.get("name") == "MyDataset")
assert promote["name"] == "MyDataset"
# The promote PUT must pass tags as an EXPLICIT list (never None), so the
# BE actually overwrites and drops the marker. A live backend treats
# ``tags=None`` as "leave unchanged", which would strand the marker on a
# source with no tags — so assert the concrete list, not just marker
# absence. Source here has no tags -> promote clears with [].
assert promote["tags"] == []
assert TEMP_MIGRATION_MARKER_TAG not in promote["tags"]
# Audit ends ok and records the handoff actions in order.
on_disk = json.loads(audit_path.read_text())
assert on_disk["status"] == "ok"
ok_types = [a["type"] for a in on_disk["actions"] if a.get("status") == "ok"]
assert ok_types.index("rename_source") < ok_types.index("promote_destination")
assert ok_types.index("create_destination") < ok_types.index("rename_source")
def test_success__source_tags_preserved_marker_stripped(self, tmp_path) -> None:
# When the source has real tags, the temp create adds the marker
# alongside them, and the promote re-passes exactly the source's
# originals (marker dropped, real tags kept).
client, _, _ = _build_fake_client(
source_rows=[
_DatasetRow(id="src-1", name="MyDataset", tags=["team-a", "prod"])
],
destination_rows=[],
items=[{"id": "item-a", "input": "hello"}],
)
result, _ = self._run(client, tmp_path)
assert result.exit_code == 0, result.output
rest = client.rest_client
create_tags = rest.datasets.create_dataset.call_args.kwargs["tags"]
assert set(create_tags) == {"team-a", "prod", TEMP_MIGRATION_MARKER_TAG}
promote = next(
c.kwargs
for c in rest.datasets.update_dataset.call_args_list
if c.kwargs.get("name") == "MyDataset"
)
assert promote["tags"] == ["team-a", "prod"]
assert TEMP_MIGRATION_MARKER_TAG not in promote["tags"]
def test_midrun_failure__source_name_untouched(self, tmp_path) -> None:
# AC: a migration interrupted mid-run leaves the source name
# untouched. Blow up the destination create (the first copy action);
# the source-rename PUT must never fire.
client, _, _ = _build_fake_client(
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
destination_rows=[],
items=[{"id": "item-a", "input": "hello"}],
)
client.rest_client.datasets.create_dataset.side_effect = RuntimeError(
"boom mid-copy"
)
result, audit_path = self._run(client, tmp_path)
assert result.exit_code == 1
# No update_dataset PUT touched the source id -> its name is intact.
source_touched = [
c
for c in client.rest_client.datasets.update_dataset.call_args_list
if c.kwargs.get("id") == "src-1"
]
assert source_touched == []
on_disk = json.loads(audit_path.read_text())
assert on_disk["status"] == "failed"
# The handoff actions never reached ``ok``.
ok_types = {a["type"] for a in on_disk["actions"] if a.get("status") == "ok"}
assert "rename_source" not in ok_types
assert "promote_destination" not in ok_types
def test_rerun_after_failure__discards_stale_temp_then_completes(
self, tmp_path
) -> None:
# AC: re-running after an interrupted run completes with no manual
# cleanup. A stale ``MyDataset__migrating`` from the prior failed run —
# carrying the migration marker tag that proves it's ours — is
# discovered and deleted before the destination is recreated.
client, _, _ = _build_fake_client(
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
destination_rows=[],
items=[{"id": "item-a", "input": "hello"}],
stale_temp_rows=[
_DatasetRow(
id="stale-1",
name="MyDataset__migrating",
tags=[TEMP_MIGRATION_MARKER_TAG],
)
],
)
result, audit_path = self._run(client, tmp_path)
assert result.exit_code == 0, result.output
# The stale temp was deleted by id before recreate.
client.rest_client.datasets.delete_dataset.assert_called_once_with(id="stale-1")
on_disk = json.loads(audit_path.read_text())
assert on_disk["status"] == "ok"
action_types = [
a["type"] for a in on_disk["actions"] if a.get("status") == "ok"
]
assert action_types.index("discard_stale_temp") < action_types.index(
"create_destination"
)
# ---------------------------------------------------------------------------
# Planner unit tests (no Click invocation)
# ---------------------------------------------------------------------------
class TestPlanBuilding:
def test_build_dataset_plan__default_flow__orders_create_replay_cascades_then_handoff(
self,
) -> None:
# OPIK-7162: the plan builds the destination under a temp name FIRST
# (source keeps its name), runs the copy + cascades, then does the
# name handoff LAST: rename source -> <name>_v1, promote temp ->
# <name>. The order is load-bearing on two axes:
# * CascadeOptimizations before CascadeExperiments (opt-id remap).
# * RenameSource before PromoteDestination (source-away then
# destination-in, so <name> is never held by two rows at once).
# Three find_datasets pages: source resolve, _v1 collision check,
# __migrating stale-temp lookup.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset", description="d")]),
_Page([]),
_Page([]),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
types = [type(a).__name__ for a in plan.actions]
assert types == [
"CreateDestination",
"ReplayVersions",
"CascadeOptimizations",
"CascadeExperiments",
"RenameSource",
"PromoteDestination",
]
# Destination is created under the temp name, not the final name.
create = plan.actions[0]
assert create.name == "MyDataset__migrating"
replay = plan.actions[1]
assert replay.source_name == "MyDataset"
assert replay.dest_name == "MyDataset__migrating"
# Handoff: source away first, destination in second.
rename = plan.actions[4]
assert rename.from_name == "MyDataset"
assert rename.to_name == "MyDataset_v1"
promote = plan.actions[5]
assert promote.from_name == "MyDataset__migrating"
assert promote.to_name == "MyDataset"
assert plan.target_name == "MyDataset"
# New remap dict starts empty; _cascade_optimizations populates it.
assert plan.optimization_id_remap == {}
def test_build_dataset_plan__exclude_experiments__omits_both_cascades(
self,
) -> None:
# OPIK-7161: --exclude-experiments drops the experiment stage AND
# the optimization stage (optimizations are containers for the
# skipped experiments). No cascade actions, but the name handoff
# (rename + promote) still runs after the dataset + versions copy.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset", description="d")]),
_Page([]),
_Page([]),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
exclude_experiments=True,
)
types = [type(a).__name__ for a in plan.actions]
assert types == [
"CreateDestination",
"ReplayVersions",
"RenameSource",
"PromoteDestination",
]
assert not any(
isinstance(a, planner_module.CascadeExperiments) for a in plan.actions
)
assert not any(
isinstance(a, planner_module.CascadeOptimizations) for a in plan.actions
)
def test_build_dataset_plan__exclude_experiments_default_false__keeps_cascades(
self,
) -> None:
# Default (flag off) is unchanged: both cascades still emitted.
# Guards the opt-out default so a plain migrate never silently
# starts skipping experiments.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
_Page([]),
_Page([]),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
types = [type(a).__name__ for a in plan.actions]
assert types == [
"CreateDestination",
"ReplayVersions",
"CascadeOptimizations",
"CascadeExperiments",
"RenameSource",
"PromoteDestination",
]
def test_build_dataset_plan__test_suite__type_forwarded_to_destination(
self,
) -> None:
# Test suites flow through the same plan shape as plain datasets;
# the only difference is ``CreateDestination.type`` being forwarded
# so the target accepts suite-level evaluators + execution_policy
# via ``ReplayVersions``.
rest_client = _planner_rest_client(
[
_Page(
[_DatasetRow(id="src-1", name="MySuite", type="evaluation_suite")]
),
_Page([]),
_Page([]),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MySuite",
to_project="B",
)
types = [type(a).__name__ for a in plan.actions]
assert types == [
"CreateDestination",
"ReplayVersions",
"CascadeOptimizations",
"CascadeExperiments",
"RenameSource",
"PromoteDestination",
]
replay = plan.actions[1]
assert replay.is_test_suite is True
def test_build_dataset_plan__rename_target_collides_workspace_wide__raises_conflict(
self,
) -> None:
# The eventual source-rename target "<source>_v1" collides with
# another dataset in the workspace — caught up-front so a doomed run
# never does any copy work.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
_Page([_DatasetRow(id="other-1", name="MyDataset_v1")]),
_Page([]),
]
)
with pytest.raises(ConflictError) as exc_info:
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
assert "MyDataset_v1" in str(exc_info.value)
def test_build_dataset_plan__rename_target_match_is_source_itself__no_conflict(
self,
) -> None:
# When find_datasets returns the source itself for the _v1 check, we
# must not treat that as a collision — it's about to be renamed.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
_Page([_DatasetRow(id="src-1", name="MyDataset_v1")]),
_Page([]),
]
)
# Should NOT raise: the only "match" is the source dataset itself.
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
assert plan.target_name == "MyDataset"
def test_build_dataset_plan__marked_stale_temp__prepends_discard_action(
self,
) -> None:
# OPIK-7162 safe re-run: a leftover "<name>__migrating" carrying the
# migration marker tag (proof it's ours) is detected and a
# DiscardStaleTemp action is prepended so the re-run starts clean.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
_Page([]),
_Page(
[
_DatasetRow(
id="stale-1",
name="MyDataset__migrating",
tags=[TEMP_MIGRATION_MARKER_TAG],
)
]
),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
types = [type(a).__name__ for a in plan.actions]
assert types == [
"DiscardStaleTemp",
"CreateDestination",
"ReplayVersions",
"CascadeOptimizations",
"CascadeExperiments",
"RenameSource",
"PromoteDestination",
]
discard = plan.actions[0]
assert discard.temp_id == "stale-1"
assert discard.temp_name == "MyDataset__migrating"
def test_build_dataset_plan__unmarked_name_collision__raises_conflict(
self,
) -> None:
# A dataset named "<name>__migrating" WITHOUT the migration marker is a
# real user dataset that merely shares the name — it must NOT be
# deleted. The planner aborts with ConflictError instead.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
_Page([]),
_Page([_DatasetRow(id="user-1", name="MyDataset__migrating")]),
]
)
with pytest.raises(ConflictError) as exc_info:
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
assert "MyDataset__migrating" in str(exc_info.value)
assert TEMP_MIGRATION_MARKER_TAG in str(exc_info.value)
def test_build_dataset_plan__same_from_and_to_project_flag__raises_conflict(
self,
) -> None:
# Cheap early-out: user literally passed --from-project A --to-project A.
# Rejected before any lookup.
rest_client = _planner_rest_client(
[_Page([_DatasetRow(id="src-1", name="MyDataset")])]
)
with pytest.raises(ConflictError, match="same project"):
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="A",
from_project="A",
)
def test_build_dataset_plan__omitted_flag_source_in_dest_project__raises_conflict(
self,
) -> None:
# The gap the flag-only check missed: --from-project is OMITTED, but the
# source actually lives in the destination project. resolve_source
# populates source.project_name from the row's project_id, so the
# authoritative post-resolve guard still catches it.
source_row = _DatasetRow(id="src-1", name="MyDataset", project_id="proj-A")
rest_client = _planner_rest_client([_Page([source_row])])
# project_name_for_row -> client.get_project(id="proj-A").name == "A".
proj = MagicMock()
proj.name = "A"
rest_client.projects.get_project_by_id.return_value = proj
with pytest.raises(ConflictError, match="same project"):
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="A",
)
def test_build_dataset_plan__workspace_scoped_source__no_same_project_abort(
self,
) -> None:
# A workspace-scoped source (no project_id -> project_name is None) has
# no single project to collide with --to-project, so a workspace-scoped
# -> project migrate is legitimate and must NOT be blocked.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset", project_id=None)]),
_Page([]),
_Page([]),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="A",
)
assert plan.target_name == "MyDataset"
def test_build_dataset_plan__no_stale_temp__no_discard_action(self) -> None:
# The common case: no leftover temp, so no DiscardStaleTemp emitted.
rest_client = _planner_rest_client(
[
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
_Page([]),
_Page([]),
]
)
plan = planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
assert not any(
isinstance(a, planner_module.DiscardStaleTemp) for a in plan.actions
)
def test_build_dataset_plan__source_name_not_found__raises_dataset_not_found(
self,
) -> None:
rest_client = _planner_rest_client([_Page([])])
with pytest.raises(DatasetNotFoundError) as exc_info:
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="Missing",
to_project="B",
)
assert "Missing" in str(exc_info.value)
def test_build_dataset_plan__source_name_resolves_to_many__raises_conflict(
self,
) -> None:
# Workspace uniqueness is enforced by the BE (UNIQUE
# (workspace_id, name)); if the BE invariant is somehow
# violated, surface it as ConflictError rather than silently
# picking a row.
rest_client = _planner_rest_client(
[
_Page(
[
_DatasetRow(id="a", name="MyDataset"),
_DatasetRow(id="b", name="MyDataset"),
]
)
]
)
with pytest.raises(ConflictError):
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="B",
)
def test_build_dataset_plan__destination_project_missing__raises_project_not_found(
self,
) -> None:
rest_client = _planner_rest_client(
find_side_effects=[],
target_project_exists=False,
)
with pytest.raises(ProjectNotFoundError) as exc_info:
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="DoesNotExist",
)
assert "DoesNotExist" in str(exc_info.value)
def test_build_dataset_plan__destination_project_missing__suggests_similar_names(
self,
) -> None:
rest_client = _planner_rest_client(
find_side_effects=[],
target_project_exists=False,
workspace_project_names=["production", "staging", "Beat", "Best"],
)
with pytest.raises(ProjectNotFoundError) as exc_info:
planner_module.build_dataset_plan(
client=_planner_client(rest_client),
name="MyDataset",
to_project="Beta",
)
message = str(exc_info.value)
assert "Beta" in message
# difflib should surface the close one-letter neighbours.
assert "Did you mean" in message
assert "Beat" in message or "Best" in message
# ---------------------------------------------------------------------------
# OPIK-7162 + OPIK-7168 integration: resume reuses the temp destination and
# finishes the pending handoff (rename source -> _v1, promote temp -> original).
# ---------------------------------------------------------------------------
class TestBuildResumePlanTempDest:
def _checkpoint(self) -> MigrationCheckpoint:
return MigrationCheckpoint(
key="k",
workspace="ws",
project="B",
dataset="MyDataset",
path=Path("/tmp/does-not-matter.json"),
dataset_phase_done=True,
source_dataset_id="src-1",
source_name="MyDataset",
temp_dest_name="MyDataset__migrating",
)
def _resume_client(self, source_id: str = "src-1") -> MagicMock:
# resolve_source(MyDataset) -> the still-unrenamed source;
# get_dataset(MyDataset__migrating) -> the temp destination.
rest_client = _planner_rest_client(
[_Page([_DatasetRow(id=source_id, name="MyDataset")])]
)
client = _planner_client(rest_client)
dest = MagicMock()
dest.id = "temp-dest-1"
client.get_dataset = MagicMock(return_value=dest)
return client
def test_resume__reuses_temp_and_appends_cascade_then_handoff(self) -> None:
# A dataset_phase_done checkpoint means create-temp/replay/optimizations
# already ran into MyDataset__migrating and the source still holds its
# original name. The resume plan must NOT re-create or re-replay; it
# resolves the temp destination, then emits the pending tail:
# CascadeExperiments -> RenameSource -> PromoteDestination.
with patch.object(
planner_module, "reconstruct_remaps", return_value=ReconstructedRemaps()
):
plan = planner_module.build_dataset_plan(
client=self._resume_client(),
name="MyDataset",
to_project="B",
resume_checkpoint=self._checkpoint(),
)
types = [type(a).__name__ for a in plan.actions]
assert types == ["CascadeExperiments", "RenameSource", "PromoteDestination"]
assert plan.is_resume is True
# The cascade + promote target the TEMP destination (promote hasn't run
# yet); the source rename moves the original name to _v1.
cascade = plan.actions[0]
assert cascade.dest_name == "MyDataset__migrating"
rename = plan.actions[1]
assert rename.from_name == "MyDataset"
assert rename.to_name == "MyDataset_v1"
promote = plan.actions[2]
assert promote.from_name == "MyDataset__migrating"
assert promote.to_name == "MyDataset"
def test_resume__source_id_mismatch__raises(self) -> None:
# If the user-supplied name now resolves to a DIFFERENT dataset than the
# interrupted run's source, resume must refuse rather than migrate the
# wrong dataset.
with patch.object(
planner_module, "reconstruct_remaps", return_value=ReconstructedRemaps()
):
with pytest.raises(MigrationError, match="different dataset"):
planner_module.build_dataset_plan(
client=self._resume_client(source_id="DIFFERENT-id"),
name="MyDataset",
to_project="B",
resume_checkpoint=self._checkpoint(),
)
@@ -0,0 +1,241 @@
"""Unit tests for read-only remap reconstruction on resume (OPIK-7168).
``reconstruct_remaps`` rebuilds version_remap / item_id_remap /
optimization_id_remap from an already-migrated destination without writing
anything. These tests mock the REST reads (versions, per-version items,
optimizations) and assert the pairing/identity/fallback logic and the
fail-loud guard on a version-count mismatch.
"""
from __future__ import annotations
from typing import Any, Dict, List, Optional
from unittest.mock import MagicMock
import pytest
from opik.cli.migrate.datasets.resume import (
ResumeReconstructionError,
reconstruct_remaps,
)
class _Version:
def __init__(self, id: str, version_hash: str) -> None:
self.id = id
self.version_hash = version_hash
class _Item:
def __init__(
self,
id: str,
data: Optional[Dict[str, Any]] = None,
description: Optional[str] = None,
tags: Optional[List[str]] = None,
evaluators: Optional[List[Any]] = None,
execution_policy: Optional[Any] = None,
trace_id: Optional[str] = None,
span_id: Optional[str] = None,
source: Optional[str] = None,
) -> None:
self.id = id
self.data = data or {"q": id}
self.description = description
self.tags = tags
self.evaluators = evaluators
self.execution_policy = execution_policy
self.trace_id = trace_id
self.span_id = span_id
self.source = source
class _Optimization:
def __init__(self, id: str, name: str) -> None:
self.id = id
self.name = name
def _page(content: List[Any]) -> MagicMock:
return MagicMock(content=content)
def _rest_client(
*,
source_versions: List[_Version],
dest_versions: List[_Version],
items_by_version_hash: Dict[str, List[_Item]],
source_optimizations: Optional[List[_Optimization]] = None,
dest_optimizations: Optional[List[_Optimization]] = None,
) -> MagicMock:
rest = MagicMock()
def _list_versions(id: str, page: int, size: int) -> MagicMock:
# reconstruct calls _iter_source_versions_chronological which lists
# NEWEST-first then reverses; return newest-first here.
if page != 1:
return _page([])
if id == "src-ds":
return _page(list(reversed(source_versions)))
return _page(list(reversed(dest_versions)))
rest.datasets.list_dataset_versions.side_effect = _list_versions
def _stream(dataset_name: str, steam_limit: int, dataset_version: str, **kw: Any):
# _stream_version_items_raw parses a byte stream via rest_stream_parser;
# tests patch that helper directly (see the patch_stream fixture), so
# this raw endpoint should never be hit.
raise AssertionError("stream_dataset_items should be patched in tests")
rest.datasets.stream_dataset_items.side_effect = _stream
def _find_opts(dataset_id: str, page: int, size: int) -> MagicMock:
if page != 1:
return _page([])
if dataset_id == "src-ds":
return _page(source_optimizations or [])
return _page(dest_optimizations or [])
rest.optimizations.find_optimizations.side_effect = _find_opts
return rest
@pytest.fixture
def patch_stream(monkeypatch: pytest.MonkeyPatch):
"""Patch ``_stream_version_items_raw`` (byte-stream parser) with a direct
lookup keyed by version_hash, so tests supply plain ``_Item`` lists."""
registry: Dict[str, List[_Item]] = {}
def _fake_stream(
rest_client: Any,
*,
dataset_name: str,
project_name: Optional[str],
version_hash: Optional[str],
) -> List[_Item]:
return registry.get(version_hash or "", [])
monkeypatch.setattr(
"opik.cli.migrate.datasets.resume._stream_version_items_raw", _fake_stream
)
return registry
def test_reconstruct__paired_versions_and_identity_items__remaps_built(
patch_stream: Dict[str, List[_Item]],
) -> None:
source_versions = [_Version("sv1", "h1"), _Version("sv2", "h2")]
# Destination hashes DIFFER from source (BE recomputes them) — pairing must
# be positional, not by hash.
dest_versions = [_Version("dv1", "dh1"), _Version("dv2", "dh2")]
# Identity: dest item ids equal source item ids (copy_from preserves them).
patch_stream["h1"] = [_Item("i1"), _Item("i2")]
patch_stream["dh1"] = [_Item("i1"), _Item("i2")]
patch_stream["h2"] = [_Item("i3")]
patch_stream["dh2"] = [_Item("i3")]
rest = _rest_client(
source_versions=source_versions,
dest_versions=dest_versions,
items_by_version_hash={},
)
remaps = reconstruct_remaps(
rest,
source_dataset_id="src-ds",
source_name="ds",
source_project_name=None,
dest_dataset_id="dst-ds",
dest_name="ds",
dest_project_name="proj",
)
assert remaps.version_remap == {"sv1": "dv1", "sv2": "dv2"}
assert remaps.item_id_remap == {"i1": "i1", "i2": "i2", "i3": "i3"}
def test_reconstruct__dest_ids_differ__content_hash_fallback_maps_them(
patch_stream: Dict[str, List[_Item]],
) -> None:
# First-version read-back case: dest item id was minted fresh, so identity
# match fails and we fall back to content-hash matching.
source_versions = [_Version("sv1", "h1")]
dest_versions = [_Version("dv1", "dh1")]
patch_stream["h1"] = [_Item("src-i1", data={"q": "same"})]
patch_stream["dh1"] = [_Item("dst-i1", data={"q": "same"})]
rest = _rest_client(
source_versions=source_versions,
dest_versions=dest_versions,
items_by_version_hash={},
)
remaps = reconstruct_remaps(
rest,
source_dataset_id="src-ds",
source_name="ds",
source_project_name=None,
dest_dataset_id="dst-ds",
dest_name="ds",
dest_project_name="proj",
)
assert remaps.item_id_remap == {"src-i1": "dst-i1"}
def test_reconstruct__version_count_mismatch__raises_reconstruction_error(
patch_stream: Dict[str, List[_Item]],
) -> None:
# Destination has fewer versions than source (e.g. crash mid-replay) — must
# fail loud rather than mis-pair.
rest = _rest_client(
source_versions=[_Version("sv1", "h1"), _Version("sv2", "h2")],
dest_versions=[_Version("dv1", "dh1")],
items_by_version_hash={},
)
with pytest.raises(ResumeReconstructionError, match="Delete the destination"):
reconstruct_remaps(
rest,
source_dataset_id="src-ds",
source_name="ds",
source_project_name=None,
dest_dataset_id="dst-ds",
dest_name="ds",
dest_project_name="proj",
)
def test_reconstruct__optimizations__matched_by_name_into_remap(
patch_stream: Dict[str, List[_Item]],
) -> None:
source_versions = [_Version("sv1", "h1")]
dest_versions = [_Version("dv1", "dh1")]
patch_stream["h1"] = []
patch_stream["dh1"] = []
rest = _rest_client(
source_versions=source_versions,
dest_versions=dest_versions,
items_by_version_hash={},
source_optimizations=[
_Optimization("so1", "opt-a"),
_Optimization("so2", "opt-b"),
],
dest_optimizations=[
_Optimization("do1", "opt-a"),
_Optimization("do2", "opt-b"),
],
)
remaps = reconstruct_remaps(
rest,
source_dataset_id="src-ds",
source_name="ds",
source_project_name=None,
dest_dataset_id="dst-ds",
dest_name="ds",
dest_project_name="proj",
)
assert remaps.optimization_id_remap == {"so1": "do1", "so2": "do2"}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,305 @@
"""Executor tests for ``opik migrate prompt``.
Drives ``execute_plan`` against fully-mocked rest_client surfaces and
asserts that each action dispatches to the right Fern call with the
right kwargs. The verbatim-commit replay path and per-version audit
shape are exercised here (the dataset-side analogue lives in
``test_migrate_dataset_version_replay.py``).
"""
from __future__ import annotations
from typing import Any, List
from unittest.mock import MagicMock
from opik.cli.migrate.audit import AuditLog
from opik.cli.migrate.prompts import executor as executor_module
from opik.cli.migrate.prompts.planner import (
CreateDestination,
MigrationPlan,
RenameSource,
ReplayVersions,
)
from opik.cli.migrate.prompts.resolver import ResolvedPrompt
from opik.rest_api.types.prompt_version_detail import PromptVersionDetail
from ._migrate_prompt_helpers import _Page, _PromptVersionRow
def _client_with_rest_mock(version_pages: List[_Page]) -> Any:
rest_client = MagicMock()
rest_client.prompts.update_prompt = MagicMock()
rest_client.prompts.create_prompt = MagicMock()
rest_client.prompts.create_prompt_version = MagicMock(
side_effect=lambda **kw: MagicMock(
id=f"dest-version-{kw['version'].commit}",
commit=kw["version"].commit,
environments=kw["version"].environments,
)
)
rest_client.prompts.get_prompt_versions.side_effect = version_pages
client = MagicMock()
client.rest_client = rest_client
return client, rest_client
def _make_plan() -> MigrationPlan:
source = ResolvedPrompt(
id="src-1",
name="MyPrompt",
project_id=None,
project_name=None,
description="orig",
tags=["a"],
template_structure="text",
)
plan = MigrationPlan(source=source, target_name="MyPrompt", to_project="B")
plan.actions = [
RenameSource(
source_id="src-1",
from_name="MyPrompt",
to_name="MyPrompt_v1",
description="orig",
tags=["a"],
),
CreateDestination(
name="MyPrompt",
project_name="B",
description="orig",
tags=["a"],
template_structure="text",
),
ReplayVersions(
source_prompt_id="src-1",
source_name_after_rename="MyPrompt_v1",
source_project_name=None,
dest_name="MyPrompt",
dest_project_name="B",
template_structure="text",
),
]
return plan
class TestExecuteActions:
def test_rename_source__re_passes_description_and_tags(self) -> None:
# The BE rename PUT wipes description (no COALESCE in the UPDATE);
# the executor must re-pass it so the source description survives.
client, rest_client = _client_with_rest_mock(
[_Page([])], # no versions
)
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
rest_client.prompts.update_prompt.assert_called_once_with(
id="src-1",
name="MyPrompt_v1",
description="orig",
tags=["a"],
)
def test_create_destination__omits_template_so_be_does_not_auto_mint_v1(
self,
) -> None:
# If we passed a template here, the BE would auto-create a v1
# with its own commit hash, defeating the verbatim-replay
# guarantee. The executor must NOT include template in the
# create_prompt call.
client, rest_client = _client_with_rest_mock([_Page([])])
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
rest_client.prompts.create_prompt.assert_called_once()
kwargs = rest_client.prompts.create_prompt.call_args.kwargs
assert kwargs["name"] == "MyPrompt"
assert kwargs["project_name"] == "B"
assert kwargs["description"] == "orig"
assert kwargs["tags"] == ["a"]
assert kwargs["template_structure"] == "text"
# Critical: no template, no type, no metadata at the container
# level — those are version-level fields handled by the replay.
assert "template" not in kwargs
assert "type" not in kwargs
assert "metadata" not in kwargs
def test_replay_versions__preserves_commit_hashes_verbatim(self) -> None:
# The verbatim-commit invariant is the architectural reason this
# slice exists: source versions are minted on the destination
# with the source's exact commit hash, made possible by the
# destination prompt's fresh id (so (workspace_id, prompt_id,
# commit) never collides).
v1 = _PromptVersionRow(
id="src-v-1",
prompt_id="src-1",
commit="aaaaaaaa",
template="hello {{name}}",
type="mustache",
)
v2 = _PromptVersionRow(
id="src-v-2",
prompt_id="src-1",
commit="bbbbbbbb",
template="hello {{name}} v2",
type="mustache",
)
# BE returns newest-first (pv.id DESC); the replay loop reverses
# to oldest-first. Pass them in newest-first order to mirror the
# BE response shape.
client, rest_client = _client_with_rest_mock([_Page([v2, v1])])
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
# Two POST /v1/private/prompts/versions calls, oldest-first.
assert rest_client.prompts.create_prompt_version.call_count == 2
first_call = rest_client.prompts.create_prompt_version.call_args_list[0]
second_call = rest_client.prompts.create_prompt_version.call_args_list[1]
first_version = first_call.kwargs["version"]
second_version = second_call.kwargs["version"]
assert isinstance(first_version, PromptVersionDetail)
assert isinstance(second_version, PromptVersionDetail)
assert first_version.commit == "aaaaaaaa"
assert second_version.commit == "bbbbbbbb"
assert first_version.template == "hello {{name}}"
assert second_version.template == "hello {{name}} v2"
def test_replay_versions__carries_environments_verbatim(self) -> None:
# Environment ownership is per-version data the BE accepts inline on
# create_prompt_version. The history mixes an env-less version, a
# non-latest version owning a single env, and a latest version owning
# multiple envs — proving single + multi env work and that a
# non-latest version's env is preserved (not just the newest one's).
v1 = _PromptVersionRow(
id="src-v-1",
prompt_id="src-1",
commit="aaaaaaaa",
template="hello",
)
v2 = _PromptVersionRow(
id="src-v-2",
prompt_id="src-1",
commit="bbbbbbbb",
template="hi",
environments=["development"],
)
v3 = _PromptVersionRow(
id="src-v-3",
prompt_id="src-1",
commit="cccccccc",
template="hey",
environments=["production", "staging"],
)
# BE returns newest-first; the loop reverses to oldest-first.
client, rest_client = _client_with_rest_mock([_Page([v3, v2, v1])])
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
calls = rest_client.prompts.create_prompt_version.call_args_list
assert calls[0].kwargs["version"].environments is None
assert calls[1].kwargs["version"].environments == ["development"]
assert calls[2].kwargs["version"].environments == ["production", "staging"]
def test_replay_versions__records_environments_in_audit(self) -> None:
v1 = _PromptVersionRow(
id="src-v-1",
prompt_id="src-1",
commit="aaaaaaaa",
template="hello",
environments=["staging", "production"],
)
client, _ = _client_with_rest_mock([_Page([v1])])
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
record = next(a for a in audit.actions if a["type"] == "replay_prompt_version")
# Environments are recorded verbatim (order is irrelevant), so compare
# order-insensitively.
assert sorted(record["source_environments"]) == ["production", "staging"]
assert sorted(record["target_environments"]) == ["production", "staging"]
def test_replay_versions__populates_prompt_version_id_remap(self) -> None:
# Slice 7 (OPIK-6575) reads ``plan.prompt_version_id_remap`` to
# remap experiment FK references. The executor must populate it.
v1 = _PromptVersionRow(
id="src-v-1",
prompt_id="src-1",
commit="aaaaaaaa",
template="hello",
)
v2 = _PromptVersionRow(
id="src-v-2",
prompt_id="src-1",
commit="bbbbbbbb",
template="hi",
)
client, rest_client = _client_with_rest_mock([_Page([v2, v1])])
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
assert plan.prompt_version_id_remap == {
"src-v-1": "dest-version-aaaaaaaa",
"src-v-2": "dest-version-bbbbbbbb",
}
def test_replay_versions__records_one_audit_entry_per_version(self) -> None:
v1 = _PromptVersionRow(
id="src-v-1",
prompt_id="src-1",
commit="aaaaaaaa",
template="hello",
)
v2 = _PromptVersionRow(
id="src-v-2",
prompt_id="src-1",
commit="bbbbbbbb",
template="hi",
)
client, _ = _client_with_rest_mock([_Page([v2, v1])])
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.execute_plan(client, plan, audit)
per_version_records = [
a for a in audit.actions if a["type"] == "replay_prompt_version"
]
assert len(per_version_records) == 2
# All per-version records are recorded as ``ok`` (the per-version
# audit happens after a successful create_prompt_version).
assert all(a["status"] == "ok" for a in per_version_records)
assert {a["source_commit"] for a in per_version_records} == {
"aaaaaaaa",
"bbbbbbbb",
}
class TestRecordPlanned:
def test_record_planned__emits_one_planned_entry_per_action(self) -> None:
# Dry-run path: every action gets a ``planned`` record, no REST
# calls fire.
plan = _make_plan()
audit = AuditLog(command="opik migrate prompt", args={})
executor_module.record_planned(plan, audit)
statuses = [a["status"] for a in audit.actions]
types = [a["type"] for a in audit.actions]
assert statuses == ["planned", "planned", "planned"]
assert types == [
"rename_source",
"create_destination",
"replay_versions",
]
@@ -0,0 +1,244 @@
"""Planner tests for ``opik migrate prompt``.
The dataset planner tests are next door; this file tracks the same
patterns (happy-path action ordering, conflict detection, project-not-
found, source-not-found) for the prompt slice.
"""
from __future__ import annotations
import pytest
from click.testing import CliRunner
from opik.cli import cli
from opik.cli.migrate.errors import (
ConflictError,
ProjectNotFoundError,
PromptNotFoundError,
)
from opik.cli.migrate.prompts import planner as prompt_planner
from ._migrate_prompt_helpers import (
_Page,
_PromptRow,
_planner_client,
_planner_rest_client,
)
class TestMigratePromptHelp:
def test_migrate_group_help__lists_prompt_subcommand(self) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["migrate", "--help"])
assert result.exit_code == 0
assert "prompt" in result.output
def test_migrate_prompt_help__lists_required_flags(self) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["migrate", "prompt", "--help"])
assert result.exit_code == 0
assert "--to-project" in result.output
assert "--from-project" in result.output
assert "--dry-run" in result.output
class TestPlanBuilding:
def test_build_prompt_plan__default_flow__orders_rename_create_replay(
self,
) -> None:
# Action ordering invariant: the rename frees the workspace-unique
# name BEFORE the destination claims it, and the replay loop runs
# AFTER the destination container exists.
rest_client = _planner_rest_client(
[
_Page([_PromptRow(id="src-1", name="MyPrompt", description="d")]),
_Page([]), # rename-collision preflight finds nothing
]
)
plan = prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="B",
)
types = [type(a).__name__ for a in plan.actions]
assert types == ["RenameSource", "CreateDestination", "ReplayVersions"]
def test_build_prompt_plan__rename_re_passes_description_and_tags(
self,
) -> None:
# The BE rename PUT wipes description (UPDATE has no COALESCE on
# description); the planner forwards the source's description and
# tags into the RenameSource record so the executor re-passes them.
rest_client = _planner_rest_client(
[
_Page(
[
_PromptRow(
id="src-1",
name="MyPrompt",
description="orig description",
tags=["a", "b"],
)
]
),
_Page([]),
]
)
plan = prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="B",
)
rename = plan.actions[0]
assert isinstance(rename, prompt_planner.RenameSource)
assert rename.from_name == "MyPrompt"
assert rename.to_name == "MyPrompt_v1"
assert rename.description == "orig description"
assert rename.tags == ["a", "b"]
def test_build_prompt_plan__create_destination_carries_metadata(
self,
) -> None:
# CreateDestination forwards container-level metadata so the
# destination inherits description / tags / template_structure
# from the source.
rest_client = _planner_rest_client(
[
_Page(
[
_PromptRow(
id="src-1",
name="MyPrompt",
description="orig",
tags=["a"],
template_structure="chat",
)
]
),
_Page([]),
]
)
plan = prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="B",
)
create = plan.actions[1]
assert isinstance(create, prompt_planner.CreateDestination)
assert create.name == "MyPrompt"
assert create.project_name == "B"
assert create.description == "orig"
assert create.tags == ["a"]
assert create.template_structure == "chat"
def test_build_prompt_plan__replay_versions_carries_source_id_and_template_structure(
self,
) -> None:
rest_client = _planner_rest_client(
[
_Page(
[
_PromptRow(
id="src-1",
name="MyPrompt",
template_structure="chat",
)
]
),
_Page([]),
]
)
plan = prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="B",
)
replay = plan.actions[2]
assert isinstance(replay, prompt_planner.ReplayVersions)
assert replay.source_prompt_id == "src-1"
assert replay.source_name_after_rename == "MyPrompt_v1"
assert replay.dest_name == "MyPrompt"
assert replay.dest_project_name == "B"
assert replay.template_structure == "chat"
class TestPreflightErrors:
def test_build_prompt_plan__destination_project_missing__raises(
self,
) -> None:
rest_client = _planner_rest_client(
[], # no prompt lookups happen if project preflight fails
target_project_exists=False,
)
with pytest.raises(ProjectNotFoundError) as excinfo:
prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="DoesNotExist",
)
assert "DoesNotExist" in str(excinfo.value)
def test_build_prompt_plan__source_prompt_missing__raises(self) -> None:
rest_client = _planner_rest_client(
[_Page([])] # source lookup returns no matches
)
with pytest.raises(PromptNotFoundError) as excinfo:
prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="Missing",
to_project="B",
)
assert "Missing" in str(excinfo.value)
assert "workspace" in str(excinfo.value).lower()
def test_build_prompt_plan__from_project_set_and_missing__raises_with_project_scope(
self,
) -> None:
# When --from-project is passed and the source prompt is not in that
# project, the not-found error should name the project (not "the
# workspace") so the user knows where we looked.
rest_client = _planner_rest_client(
[_Page([])] # source lookup returns no matches in the project
)
with pytest.raises(PromptNotFoundError) as excinfo:
prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="B",
from_project="A",
)
assert "MyPrompt" in str(excinfo.value)
assert "project 'A'" in str(excinfo.value)
def test_build_prompt_plan__rename_collision__raises(self) -> None:
# Source resolves, but the rename target ``MyPrompt_v1`` is already
# taken by another prompt -> ConflictError.
rest_client = _planner_rest_client(
[
_Page([_PromptRow(id="src-1", name="MyPrompt")]),
_Page([_PromptRow(id="other-1", name="MyPrompt_v1")]),
]
)
with pytest.raises(ConflictError) as excinfo:
prompt_planner.build_prompt_plan(
client=_planner_client(rest_client),
name="MyPrompt",
to_project="B",
)
assert "MyPrompt_v1" in str(excinfo.value)
@@ -0,0 +1,358 @@
"""Tests for the migration manifest used by opik import for resumable migrations."""
import sqlite3
from pathlib import Path
import pytest
from opik.cli.migration_manifest import MigrationManifest, MANIFEST_FILENAME
@pytest.fixture
def tmp_base(tmp_path: Path) -> Path:
return tmp_path
class TestMigrationManifestLifecycle:
def test_manifest__fresh_instance__status_is_not_started(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
assert manifest.status == "not_started"
assert not manifest.is_in_progress
assert not manifest.is_completed
def test_start__fresh_manifest__status_becomes_in_progress_and_file_written(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.start()
assert manifest.is_in_progress
assert (tmp_base / MANIFEST_FILENAME).exists()
def test_complete__after_start__status_becomes_completed(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.start()
manifest.complete()
assert manifest.is_completed
assert not manifest.is_in_progress
def test_reset__after_start_with_data__all_state_cleared(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.start()
trace_file = tmp_base / "projects" / "p1" / "trace_abc.json"
trace_file.parent.mkdir(parents=True)
trace_file.touch()
manifest.mark_file_completed(trace_file)
manifest.add_trace_mapping("src-1", "dest-1")
manifest.reset()
assert manifest.status == "not_started"
assert manifest.completed_count() == 0
assert manifest.get_trace_id_map() == {}
def test_exists__before_any_save__returns_false(self, tmp_base: Path) -> None:
assert not MigrationManifest.exists(tmp_base)
def test_exists__after_start__returns_true(self, tmp_base: Path) -> None:
MigrationManifest(tmp_base).start()
assert MigrationManifest.exists(tmp_base)
def test_start__called_twice__started_at_preserved(self, tmp_base: Path) -> None:
manifest = MigrationManifest(tmp_base)
manifest.start()
conn = sqlite3.connect(str(tmp_base / MANIFEST_FILENAME))
first_started_at = conn.execute(
"SELECT value FROM status WHERE key = 'started_at'"
).fetchone()[0]
conn.close()
manifest.start() # second call must not overwrite started_at
conn = sqlite3.connect(str(tmp_base / MANIFEST_FILENAME))
second_started_at = conn.execute(
"SELECT value FROM status WHERE key = 'started_at'"
).fetchone()[0]
conn.close()
assert second_started_at == first_started_at
class TestFileTracking:
def _make_file(self, base: Path, relative: str) -> Path:
p = base / relative
p.parent.mkdir(parents=True, exist_ok=True)
p.touch()
return p
def test_is_file_completed__fresh_manifest__returns_false(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
f = self._make_file(tmp_base, "datasets/dataset_foo.json")
assert not manifest.is_file_completed(f)
def test_mark_file_completed__new_file__recorded_and_count_incremented(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
f = self._make_file(tmp_base, "datasets/dataset_foo.json")
manifest.mark_file_completed(f)
assert manifest.is_file_completed(f)
assert manifest.completed_count() == 1
def test_mark_file_completed__same_file_twice__count_remains_one(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
f = self._make_file(tmp_base, "datasets/dataset_foo.json")
manifest.mark_file_completed(f)
manifest.mark_file_completed(f)
assert manifest.completed_count() == 1
def test_mark_file_failed__new_file__recorded_with_error(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
f = self._make_file(tmp_base, "projects/p1/trace_abc.json")
manifest.mark_file_failed(f, "timeout")
assert manifest.failed_count() == 1
assert manifest.get_failed_files()[manifest.relative_path(f)] == "timeout"
def test_mark_file_completed__previously_failed_file__removed_from_failed(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
f = self._make_file(tmp_base, "projects/p1/trace_abc.json")
manifest.mark_file_failed(f, "timeout")
manifest.mark_file_completed(f)
assert manifest.failed_count() == 0
assert manifest.is_file_completed(f)
def test_relative_path__nested_file__returns_path_relative_to_base(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
f = self._make_file(tmp_base, "projects/my-project/trace_xyz.json")
assert manifest.relative_path(f) == str(
Path("projects") / "my-project" / "trace_xyz.json"
)
class TestTraceIdMapping:
def test_add_trace_mapping__single_entry__retrievable_via_get_map(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.add_trace_mapping("src-111", "dest-222")
manifest.save()
assert manifest.get_trace_id_map() == {"src-111": "dest-222"}
def test_trace_id_map__save_then_new_instance__mapping_survives(
self, tmp_base: Path
) -> None:
m1 = MigrationManifest(tmp_base)
m1.start()
m1.add_trace_mapping("src-aaa", "dest-bbb")
m1.save()
m2 = MigrationManifest(tmp_base)
assert m2.get_trace_id_map() == {"src-aaa": "dest-bbb"}
def test_get_trace_id_map__mutate_returned_dict__original_unaffected(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.add_trace_mapping("src-1", "dest-1")
manifest.save()
copy = manifest.get_trace_id_map()
copy["src-2"] = "dest-2" # mutate the copy
assert "src-2" not in manifest.get_trace_id_map()
def test_add_trace_mapping__same_src_id_twice__last_dest_wins(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.add_trace_mapping("src-1", "dest-old")
manifest.add_trace_mapping("src-1", "dest-new")
manifest.save()
assert manifest.get_trace_id_map()["src-1"] == "dest-new"
class TestDatabaseIntegrity:
def test_manifest_file__after_start_and_save__is_valid_sqlite_with_expected_tables(
self, tmp_base: Path
) -> None:
manifest = MigrationManifest(tmp_base)
manifest.start()
manifest.add_trace_mapping("a", "b")
manifest.save()
conn = sqlite3.connect(str(tmp_base / MANIFEST_FILENAME))
tables = {
row[0]
for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()
}
conn.close()
assert {"status", "completed_files", "failed_files", "trace_id_map"} <= tables
def test_save__after_start__no_tmp_file_created(self, tmp_base: Path) -> None:
manifest = MigrationManifest(tmp_base)
manifest.start()
assert not (tmp_base / "migration_manifest.tmp").exists()
def test_mark_file_completed__duplicate_write__count_remains_one(
self, tmp_base: Path
) -> None:
"""INSERT OR IGNORE means re-flushing the same path never duplicates rows."""
manifest = MigrationManifest(tmp_base, batch_size=1)
f = tmp_base / "projects" / "p" / "trace_x.json"
f.parent.mkdir(parents=True)
f.touch()
manifest.mark_file_completed(f)
manifest.mark_file_completed(f)
assert manifest.completed_count() == 1
class TestBatchingBehavior:
"""Document and verify the crash trade-off introduced by batched writes."""
def _make_trace(self, base: Path, name: str) -> Path:
p = base / "projects" / "proj" / name
p.parent.mkdir(parents=True, exist_ok=True)
p.touch()
return p
def test_mark_file_completed__within_batch__visible_via_api_before_flush(
self, tmp_base: Path
) -> None:
"""Buffered completions are visible through the public API (which flushes
before querying) even before the batch threshold is reached."""
manifest = MigrationManifest(tmp_base, batch_size=50)
f = self._make_trace(tmp_base, "trace_001.json")
manifest.mark_file_completed(f)
# completed_count() flushes first, so the buffered write IS visible.
assert manifest.completed_count() == 1
assert manifest.is_file_completed(f)
def test_mark_file_completed__crash_before_flush__unflushed_data_lost(
self, tmp_base: Path
) -> None:
"""Simulates a crash (new instance, no save()) with batch_size > pending count.
This is the documented trade-off: up to batch_size-1 completions may be
absent from the manifest after a crash. Those files will simply be
re-imported on resume.
"""
f = self._make_trace(tmp_base, "trace_001.json")
m1 = MigrationManifest(tmp_base, batch_size=50)
m1.start() # always flushes immediately
m1.mark_file_completed(f) # buffered — NOT yet on disk
# Simulate crash: m1 is abandoned without save() or complete().
# __del__ is NOT called here because m1 is still in scope when m2 is created.
m2 = MigrationManifest(tmp_base, batch_size=50)
# The completion is not on disk — resume will re-process this file.
assert not m2.is_file_completed(f)
assert m2.completed_count() == 0
def test_mark_file_completed__batch_size_one__each_write_immediately_durable(
self, tmp_base: Path
) -> None:
"""With batch_size=1 every completion is immediately committed to disk."""
f = self._make_trace(tmp_base, "trace_001.json")
m1 = MigrationManifest(tmp_base, batch_size=1)
m1.start()
m1.mark_file_completed(f) # auto-flushes (batch_size=1)
# New instance reads from disk — must see the completion.
m2 = MigrationManifest(tmp_base, batch_size=1)
assert m2.is_file_completed(f)
assert m2.completed_count() == 1
def test_mark_file_completed__batch_threshold_reached__auto_flushed_to_disk(
self, tmp_base: Path
) -> None:
"""When pending count hits batch_size the buffer is auto-flushed."""
batch_size = 3
files = [self._make_trace(tmp_base, f"trace_{i:03d}.json") for i in range(3)]
m1 = MigrationManifest(tmp_base, batch_size=batch_size)
m1.start()
for f in files:
m1.mark_file_completed(f) # third call triggers auto-flush
# New instance must see all three completions.
m2 = MigrationManifest(tmp_base, batch_size=batch_size)
assert m2.completed_count() == 3
for f in files:
assert m2.is_file_completed(f)
class TestResumeScenario:
"""Simulate a real interrupted + resumed migration."""
def test_resume__interrupted_import__completed_files_skipped_and_id_map_available(
self, tmp_base: Path
) -> None:
"""After an interrupted run the second run skips already-flushed files.
batch_size=1 so every mark_file_completed is immediately durable —
this models the boundary between two process invocations where only
flushed data survives.
"""
trace_files = []
for i in range(3):
p = tmp_base / "projects" / "proj" / f"trace_{i:03d}.json"
p.parent.mkdir(parents=True, exist_ok=True)
p.touch()
trace_files.append(p)
m1 = MigrationManifest(tmp_base, batch_size=1)
m1.start()
m1.add_trace_mapping("src-0", "dest-0")
m1.mark_file_completed(trace_files[0])
m1.add_trace_mapping("src-1", "dest-1")
m1.mark_file_completed(trace_files[1])
# Process crashes before trace_files[2] — no complete() call.
m2 = MigrationManifest(tmp_base)
assert m2.is_in_progress
assert m2.completed_count() == 2
assert m2.is_file_completed(trace_files[0])
assert m2.is_file_completed(trace_files[1])
assert not m2.is_file_completed(trace_files[2])
id_map = m2.get_trace_id_map()
assert id_map["src-0"] == "dest-0"
assert id_map["src-1"] == "dest-1"
assert "src-2" not in id_map
def test_reset__after_completed_files__all_state_cleared(
self, tmp_base: Path
) -> None:
trace_file = tmp_base / "projects" / "p" / "trace_abc.json"
trace_file.parent.mkdir(parents=True)
trace_file.touch()
m1 = MigrationManifest(tmp_base)
m1.start()
m1.mark_file_completed(trace_file)
m2 = MigrationManifest(tmp_base)
m2.reset()
assert m2.completed_count() == 0
assert not m2.is_file_completed(trace_file)
assert m2.status == "not_started"
+723
View File
@@ -0,0 +1,723 @@
import base64
import uuid
from unittest.mock import MagicMock, patch
import click
import pytest
from opik.cli.local_runner.pairing import (
PairingResult,
RunnerType,
build_pairing_link,
generate_runner_name,
hkdf_sha256,
resolve_project_id,
run_headless,
run_pairing,
validate_runner_name,
)
from opik.rest_api.errors.not_found_error import NotFoundError
# Helpers — fill in the keyword-only args that production callers always supply
# so tests don't have to repeat them. Tests override what they actually care
# about via **overrides.
def _resolve(api, project_name, **overrides):
kwargs = dict(
create_if_missing=False,
workspace=None,
base_url=None,
config_file_exists=True,
)
kwargs.update(overrides)
return resolve_project_id(api, project_name, **kwargs)
def _headless(api, **overrides):
kwargs = dict(
api=api,
project_name="my-proj",
runner_name="r-1",
runner_type=RunnerType.ENDPOINT,
create_if_missing=False,
workspace=None,
base_url=None,
config_file_exists=True,
)
kwargs.update(overrides)
return run_headless(**kwargs)
def _link(**overrides):
kwargs = dict(
base_url="https://www.comet.com/opik/api/",
session_id="550e8400-e29b-41d4-a716-446655440000",
activation_key=b"\x00" * 32,
project_id="660e8400-e29b-41d4-a716-446655440000",
project_name="my-proj",
runner_name="r",
runner_type=RunnerType.CONNECT,
workspace=None,
)
kwargs.update(overrides)
return build_pairing_link(**kwargs)
def _pair(api, **overrides):
kwargs = dict(
api=api,
project_name="my-proj",
runner_name="r-1",
runner_type=RunnerType.ENDPOINT,
base_url="http://localhost:5173/api/",
workspace=None,
tui=None,
create_if_missing=False,
config_file_exists=True,
)
kwargs.update(overrides)
return run_pairing(**kwargs)
class TestHKDF:
def test_hkdf__pinned_vector__matches_expected(self):
ikm = bytes(range(32))
salt = uuid.UUID("00000000-0000-0000-0000-000000000001").bytes
info = b"opik-bridge-v1"
okm = hkdf_sha256(ikm=ikm, salt=salt, info=info)
assert (
okm.hex()
== "9647b7959765ecad68dfef02f31cfca7f7901a9076c15ebabb1f35d015f71198"
)
def test_hkdf__any_input__output_is_32_bytes(self):
okm = hkdf_sha256(ikm=b"\x00" * 32, salt=b"\x00" * 16, info=b"test")
assert len(okm) == 32
def test_hkdf__different_info__different_output(self):
a = hkdf_sha256(ikm=b"\x00" * 32, salt=b"\x00" * 16, info=b"a")
b = hkdf_sha256(ikm=b"\x00" * 32, salt=b"\x00" * 16, info=b"b")
assert a != b
class TestUUIDBytesOrder:
def test_uuid_bytes__known_uuid__matches_java_big_endian(self):
u = uuid.UUID("550e8400-e29b-41d4-a716-446655440000")
assert u.bytes.hex() == "550e8400e29b41d4a716446655440000"
class TestBuildPairingLink:
def test_build_pairing_link__valid_inputs__correct_payload_layout(self):
session_id = "550e8400-e29b-41d4-a716-446655440000"
project_id = "660e8400-e29b-41d4-a716-446655440000"
activation_key = b"\xaa" * 32
runner_name = "my-runner"
link = _link(
base_url="https://www.comet.com/opik/api/",
session_id=session_id,
activation_key=activation_key,
project_id=project_id,
runner_name=runner_name,
)
assert link.startswith("https://www.comet.com/opik/pair/v1?")
fragment = link.split("#", 1)[1]
padding_needed = (4 - len(fragment) % 4) % 4
payload = base64.urlsafe_b64decode(fragment + "=" * padding_needed)
assert payload[0:16] == uuid.UUID(session_id).bytes
assert payload[16:48] == activation_key
assert payload[48:64] == uuid.UUID(project_id).bytes
assert payload[64] == len(runner_name.encode("utf-8"))
name_end = 65 + payload[64]
assert payload[65:name_end] == runner_name.encode("utf-8")
# Default runner type is CONNECT (0x00)
assert payload[name_end] == 0x00
def test_build_pairing_link__endpoint_type__encodes_0x01(self):
link = _link(
base_url="http://localhost:5173/api/",
session_id="550e8400-e29b-41d4-a716-446655440000",
activation_key=b"\x00" * 32,
project_id="660e8400-e29b-41d4-a716-446655440000",
runner_name="r",
runner_type=RunnerType.ENDPOINT,
)
fragment = link.split("#", 1)[1]
padding_needed = (4 - len(fragment) % 4) % 4
payload = base64.urlsafe_b64decode(fragment + "=" * padding_needed)
# name_len=1, name="r", then type byte
assert payload[66] == 0x01
def test_build_pairing_link__cloud_url__no_double_opik_path(self):
link = _link(
base_url="https://www.comet.com/opik/api/",
session_id="550e8400-e29b-41d4-a716-446655440000",
activation_key=b"\x00" * 32,
project_id="660e8400-e29b-41d4-a716-446655440000",
runner_name="r",
)
assert "/opik/opik/" not in link
def test_build_pairing_link__localhost_url__correct_prefix(self):
link = _link(
base_url="http://localhost:5173/api/",
session_id="550e8400-e29b-41d4-a716-446655440000",
activation_key=b"\x00" * 32,
project_id="660e8400-e29b-41d4-a716-446655440000",
runner_name="r",
)
assert link.startswith("http://localhost:5173/opik/pair/v1?")
def test_build_pairing_link__always_includes_url_param(self):
link = _link(
base_url="http://localhost:5173/api/",
workspace=None,
)
# `/api` is stripped — we send the URL the user would visit in a
# browser, not the internal API endpoint.
assert "url=http%3A%2F%2Flocalhost%3A5173%2F" in link
assert "%2Fapi%2F" not in link
def test_build_pairing_link__cloud_api_url__strips_api_suffix(self):
link = _link(
base_url="https://www.comet.com/opik/api/",
workspace=None,
)
assert "url=https%3A%2F%2Fwww.comet.com%2Fopik%2F" in link
def test_build_pairing_link__url_param_precedes_workspace(self):
link = _link(
base_url="http://localhost:5173/api/",
workspace="team-a",
)
# All query params present; order is stable for FE parsing.
assert "url=" in link
assert "project=my-proj" in link
assert "workspace=team-a" in link
def test_build_pairing_link__project_name_with_special_chars__url_encoded(
self,
):
link = _link(project_name="My Project / Demo")
# Spaces and `/` must be percent-encoded so the query stays parseable.
assert "project=My%20Project%20%2F%20Demo" in link
def test_build_pairing_link__workspace_with_special_chars__url_encoded(
self,
):
# `&` and `=` in a workspace name would otherwise split the query and
# the FE's URLSearchParams would read a truncated value.
link = _link(workspace="weird&name=oops")
assert "workspace=weird%26name%3Doops" in link
class TestResolveProjectId:
def test_resolve_project_id__project_exists__returns_id(self):
api = MagicMock()
project = MagicMock()
project.id = "proj-uuid-123"
api.projects.retrieve_project.return_value = project
result = _resolve(api, "my-project")
assert result == "proj-uuid-123"
api.projects.retrieve_project.assert_called_once_with(name="my-project")
def test_resolve_project_id__project_missing__uses_404_hint(self):
from opik.rest_api.core.api_error import ApiError
# Project retrieve returns Opik's custom ErrorMessage shape.
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["Project not found"]}
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(api, "nonexistent")
message = exc_info.value.message
assert "Could not retrieve project 'nonexistent'" in message
assert "Project not found" in message
assert "check the project name and try again" in message
def test_resolve_project_id__unauthorized__surfaces_server_message_and_docs(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=401,
body={"code": 401, "message": "API key should be provided"},
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(api, "my-project")
message = exc_info.value.message
assert "Could not retrieve project 'my-project'" in message
assert "API key should be provided" in message
assert "Run: opik configure" in message
assert (
"https://www.comet.com/docs/opik/tracing/advanced/sdk_configuration"
in message
)
def test_resolve_project_id__forbidden__shares_auth_hint(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=403,
body={"code": 403, "message": "User is not allowed to access workspace"},
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(api, "my-project")
message = exc_info.value.message
assert "User is not allowed to access workspace" in message
assert "Run: opik configure" in message
def test_resolve_project_id__server_error__falls_back_to_generic_hint(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=500,
body="Internal Server Error",
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(api, "my-project")
message = exc_info.value.message
assert "Internal Server Error" in message
assert "verify your Opik configuration and connectivity" in message
assert (
"https://www.comet.com/docs/opik/tracing/advanced/sdk_configuration"
in message
)
def test_resolve_project_id__empty_body__falls_back_to_status_text(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(status_code=502, body=None)
with pytest.raises(click.ClickException) as exc_info:
_resolve(api, "my-project")
assert "server returned HTTP 502" in exc_info.value.message
def test_resolve_project_id__missing_and_create_flag__creates_and_resolves(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
created = MagicMock()
created.id = "proj-uuid-new"
api.projects.retrieve_project.side_effect = [
ApiError(status_code=404, body={"errors": ["Project not found"]}),
created,
]
result = _resolve(api, "new-project", create_if_missing=True)
assert result == "proj-uuid-new"
api.projects.create_project.assert_called_once_with(name="new-project")
assert api.projects.retrieve_project.call_count == 2
def test_resolve_project_id__known_missing__skips_initial_lookup(self):
api = MagicMock()
created = MagicMock()
created.id = "proj-uuid-new"
api.projects.retrieve_project.return_value = created
result = _resolve(
api,
"new-project",
create_if_missing=True,
project_known_missing=True,
)
assert result == "proj-uuid-new"
api.projects.create_project.assert_called_once_with(name="new-project")
# Interactive preflight already saw the 404 — only the post-create
# lookup should run.
assert api.projects.retrieve_project.call_count == 1
def test_resolve_project_id__missing_no_flag__does_not_create(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["Project not found"]}
)
with pytest.raises(click.ClickException):
_resolve(api, "missing", create_if_missing=False)
api.projects.create_project.assert_not_called()
def test_resolve_project_id__non_404_with_flag__does_not_create(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=401, body={"message": "unauthorized"}
)
with pytest.raises(click.ClickException):
_resolve(api, "x", create_if_missing=True)
api.projects.create_project.assert_not_called()
def test_resolve_project_id__create_fails__raises_clean_error(self):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["Project not found"]}
)
api.projects.create_project.side_effect = ApiError(
status_code=403, body={"message": "User cannot create projects"}
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(api, "blocked", create_if_missing=True)
message = exc_info.value.message
assert "Could not create project 'blocked'" in message
assert "User cannot create projects" in message
def test_resolve_project_id__retrieval_error_with_config_context__appends_workspace_and_url(
self,
):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["Project not found"]}
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(
api,
"missing-proj",
workspace="team-a",
base_url="https://opik.example.com/api/",
)
message = exc_info.value.message
assert "team-a" in message
assert "https://opik.example.com/api/" in message
def test_resolve_project_id__retrieval_error_without_config_file__appends_opik_configure_hint(
self,
):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=401, body={"message": "API key should be provided"}
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(
api,
"my-project",
workspace="default",
base_url="https://www.comet.com/opik/api/",
config_file_exists=False,
)
message = exc_info.value.message
assert "~/.opik.config" in message
assert "Run: opik configure" in message
def test_resolve_project_id__retrieval_error_with_config_file__omits_no_config_hint(
self,
):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=500, body="Internal Server Error"
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(
api,
"p",
workspace="default",
base_url="https://www.comet.com/opik/api/",
config_file_exists=True,
)
assert "~/.opik.config" not in exc_info.value.message
def test_resolve_project_id__create_fails_with_config_context__includes_workspace_and_url(
self,
):
from opik.rest_api.core.api_error import ApiError
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["Project not found"]}
)
api.projects.create_project.side_effect = ApiError(
status_code=403, body={"message": "User cannot create projects"}
)
with pytest.raises(click.ClickException) as exc_info:
_resolve(
api,
"blocked",
create_if_missing=True,
workspace="team-a",
base_url="https://opik.example.com/api/",
config_file_exists=False,
)
message = exc_info.value.message
assert "Could not create project 'blocked'" in message
assert "team-a" in message
assert "https://opik.example.com/api/" in message
assert "Run: opik configure" in message
class TestValidateRunnerName:
def test_validate_runner_name__valid_name__passes(self):
validate_runner_name("my-runner")
def test_validate_runner_name__empty__raises(self):
with pytest.raises(Exception, match="empty"):
validate_runner_name("")
def test_validate_runner_name__whitespace_only__raises(self):
with pytest.raises(Exception, match="empty"):
validate_runner_name(" ")
def test_validate_runner_name__over_128_chars__raises(self):
with pytest.raises(Exception, match="128 characters"):
validate_runner_name("x" * 129)
def test_validate_runner_name__over_255_utf8_bytes__raises(self):
name = "\U0001f600" * 64
with pytest.raises(Exception, match="255 UTF-8 bytes"):
validate_runner_name(name)
class TestGenerateRunnerName:
def test_generate_runner_name__explicit_name__returns_it(self):
assert generate_runner_name("my-name") == "my-name"
def test_generate_runner_name__none__generates_random_hex(self):
name = generate_runner_name(None)
assert "-" in name
hex_part = name.rsplit("-", 1)[1]
assert len(hex_part) == 6
int(hex_part, 16)
class TestRunPairing:
SESSION_ID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
RUNNER_ID = "11111111-2222-3333-4444-555555555555"
PROJECT_ID = "66666666-7777-8888-9999-aaaaaaaaaaaa"
def _make_api(self, project_id=None):
if project_id is None:
project_id = self.PROJECT_ID
api = MagicMock()
project = MagicMock()
project.id = project_id
api.projects.retrieve_project.return_value = project
session_resp = MagicMock()
session_resp.session_id = self.SESSION_ID
session_resp.runner_id = self.RUNNER_ID
api.pairing.create_pairing_session.return_value = session_resp
runner = MagicMock()
runner.status = "connected"
api.runners.get_runner.return_value = runner
return api
@patch("opik.cli.local_runner.pairing.time.sleep")
def test_run_pairing__happyflow__returns_result(self, mock_sleep):
api = self._make_api()
result = _pair(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.CONNECT,
base_url="http://localhost:5173/api/",
ttl_seconds=1,
)
assert isinstance(result, PairingResult)
assert result.runner_id == self.RUNNER_ID
assert result.project_name == "my-proj"
assert result.project_id == self.PROJECT_ID
assert len(result.bridge_key) == 32
@patch("opik.cli.local_runner.pairing.time.sleep")
def test_run_pairing__with_tui__calls_started_and_completed(self, mock_sleep):
api = self._make_api()
tui = MagicMock()
_pair(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.CONNECT,
base_url="http://localhost:5173/api/",
tui=tui,
ttl_seconds=1,
)
tui.pairing_started.assert_called_once()
tui.pairing_completed.assert_called_once()
@patch("opik.cli.local_runner.pairing.time.sleep")
def test_run_pairing__404_during_poll__retries_until_connected(self, mock_sleep):
api = self._make_api()
err = NotFoundError(body=None)
runner = MagicMock()
runner.status = "connected"
api.runners.get_runner.side_effect = [err, err, runner]
result = _pair(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.ENDPOINT,
base_url="http://localhost:5173/api/",
ttl_seconds=1,
)
assert result.runner_id == self.RUNNER_ID
assert api.runners.get_runner.call_count == 3
@patch("opik.cli.local_runner.pairing.time.monotonic")
@patch("opik.cli.local_runner.pairing.time.sleep")
def test_run_pairing__timeout__calls_tui_pairing_failed(
self, mock_sleep, mock_monotonic
):
api = self._make_api()
runner = MagicMock()
runner.status = "pairing"
api.runners.get_runner.return_value = runner
# Strictly-increasing clock: each call returns previous + 1000s. This
# guarantees the while-loop exits on the call immediately after
# `deadline = monotonic() + ttl_seconds` is computed, regardless of
# how many other calls (from logging, coverage, plugins, etc.) land
# on the patched monotonic before or after the test's own calls.
clock = {"t": 0.0}
def advancing_monotonic():
clock["t"] += 1000.0
return clock["t"]
mock_monotonic.side_effect = advancing_monotonic
tui = MagicMock()
with pytest.raises(click.ClickException) as exc_info:
_pair(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.CONNECT,
base_url="http://localhost:5173/api/",
tui=tui,
ttl_seconds=300,
)
assert "timed out" in exc_info.value.message
tui.pairing_failed.assert_called_once_with("timed out")
@patch("opik.cli.local_runner.pairing.time.sleep")
def test_run_pairing__keyboard_interrupt__calls_tui_pairing_failed(
self, mock_sleep
):
api = self._make_api()
api.runners.get_runner.side_effect = KeyboardInterrupt
tui = MagicMock()
with pytest.raises(KeyboardInterrupt):
_pair(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.CONNECT,
base_url="http://localhost:5173/api/",
tui=tui,
ttl_seconds=1,
)
tui.pairing_failed.assert_called_once_with("interrupted")
@patch("opik.cli.local_runner.pairing.time.sleep")
def test_run_pairing__non_404_api_error__propagates(self, mock_sleep):
from opik.rest_api.core.api_error import ApiError
api = self._make_api()
api.runners.get_runner.side_effect = ApiError(
status_code=429, body="rate limited"
)
with pytest.raises(ApiError) as exc_info:
_pair(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.CONNECT,
base_url="http://localhost:5173/api/",
ttl_seconds=1,
)
assert exc_info.value.status_code == 429
class TestRunHeadless:
SESSION_ID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
RUNNER_ID = "11111111-2222-3333-4444-555555555555"
PROJECT_ID = "66666666-7777-8888-9999-aaaaaaaaaaaa"
def _make_api(self):
api = MagicMock()
project = MagicMock()
project.id = self.PROJECT_ID
api.projects.retrieve_project.return_value = project
session_resp = MagicMock()
session_resp.session_id = self.SESSION_ID
session_resp.runner_id = self.RUNNER_ID
api.pairing.create_pairing_session.return_value = session_resp
api.pairing.activate_pairing_session.return_value = None
return api
def test_run_headless__creates_and_self_activates(self):
api = self._make_api()
result = _headless(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.ENDPOINT,
)
assert result.runner_id == self.RUNNER_ID
assert result.project_id == self.PROJECT_ID
assert result.bridge_key == b""
api.pairing.create_pairing_session.assert_called_once()
api.pairing.activate_pairing_session.assert_called_once()
def test_run_headless__no_polling(self):
api = self._make_api()
_headless(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.ENDPOINT,
)
# No get_runner polling — headless activates immediately
api.runners.get_runner.assert_not_called()
def test_run_headless__connect_type__raises(self):
api = self._make_api()
with pytest.raises(click.ClickException, match="not supported"):
_headless(
api=api,
project_name="my-proj",
runner_name="test-runner",
runner_type=RunnerType.CONNECT,
)
+175
View File
@@ -0,0 +1,175 @@
from unittest.mock import MagicMock, patch
import pytest
from opik.cli.local_runner.preflight import (
maybe_auto_configure,
should_create_project,
)
from opik.rest_api.core.api_error import ApiError
class TestShouldCreateProject:
def test__headless__returns_create_without_lookup(self):
api = MagicMock()
result = should_create_project(api, "any", workspace=None, headless=True)
# Headless skips preflight, so we want to create but don't yet know
# whether the project is actually missing — the resolver must look up.
assert result == (True, False)
api.projects.retrieve_project.assert_not_called()
def test__project_exists__returns_no_create(self):
api = MagicMock()
api.projects.retrieve_project.return_value = MagicMock(id="proj-1")
result = should_create_project(api, "exists", workspace="ws", headless=False)
assert result == (False, False)
def test__non_404_error__returns_no_create_to_let_downstream_format(self):
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=401, body={"message": "unauthorized"}
)
result = should_create_project(api, "x", workspace="ws", headless=False)
assert result == (False, False)
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__missing_and_not_tty__returns_no_create(self, mock_stdin):
mock_stdin.isatty.return_value = False
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["not found"]}
)
result = should_create_project(api, "missing", workspace="ws", headless=False)
assert result == (False, False)
@patch("opik.cli.local_runner.preflight.click.confirm", return_value=True)
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__missing_and_tty_user_confirms__returns_create_and_known_missing(
self, mock_stdin, mock_confirm
):
mock_stdin.isatty.return_value = True
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["not found"]}
)
result = should_create_project(api, "missing", workspace="ws", headless=False)
# Interactive preflight saw 404 and user confirmed — downstream can
# skip the redundant lookup.
assert result == (True, True)
prompt_text = mock_confirm.call_args[0][0]
assert "missing" in prompt_text
assert "ws" in prompt_text
@patch("opik.cli.local_runner.preflight.click.confirm", return_value=False)
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__missing_and_tty_user_declines__returns_no_create(
self, mock_stdin, mock_confirm
):
mock_stdin.isatty.return_value = True
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(
status_code=404, body={"errors": ["not found"]}
)
result = should_create_project(api, "missing", workspace="ws", headless=False)
assert result == (False, False)
@patch("opik.cli.local_runner.preflight.click.confirm", return_value=True)
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__no_workspace__prompt_omits_workspace_label(
self, mock_stdin, mock_confirm
):
mock_stdin.isatty.return_value = True
api = MagicMock()
api.projects.retrieve_project.side_effect = ApiError(status_code=404, body={})
should_create_project(api, "missing", workspace=None, headless=False)
prompt_text = mock_confirm.call_args[0][0]
assert "in workspace" not in prompt_text
class TestMaybeAutoConfigure:
def _patch_env(self, monkeypatch, opik_api_key=None):
if opik_api_key is None:
monkeypatch.delenv("OPIK_API_KEY", raising=False)
else:
monkeypatch.setenv("OPIK_API_KEY", opik_api_key)
def _config_probe(self, *, config_file_exists=False, api_key=None):
probe = MagicMock()
probe.config_file_exists = config_file_exists
probe.api_key = api_key
return probe
@patch("opik.cli.configure.run_interactive_configure")
def test__non_interactive_flag__skips(self, mock_configure, monkeypatch):
self._patch_env(monkeypatch)
maybe_auto_configure(api_key_arg=None, non_interactive=True, headless=False)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
def test__headless__skips(self, mock_configure, monkeypatch):
self._patch_env(monkeypatch)
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=True)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
def test__api_key_arg__skips(self, mock_configure, monkeypatch):
self._patch_env(monkeypatch)
maybe_auto_configure(api_key_arg="abc", non_interactive=False, headless=False)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
def test__env_api_key__skips(self, mock_configure, monkeypatch):
self._patch_env(monkeypatch, opik_api_key="from-env")
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__stdin_not_tty__skips(self, mock_stdin, mock_configure, monkeypatch):
self._patch_env(monkeypatch)
mock_stdin.isatty.return_value = False
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
@patch("opik.cli.local_runner.preflight.OpikConfig")
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__config_file_already_exists__skips(
self, mock_stdin, mock_config_cls, mock_configure, monkeypatch
):
self._patch_env(monkeypatch)
mock_stdin.isatty.return_value = True
mock_config_cls.return_value = self._config_probe(config_file_exists=True)
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
@patch("opik.cli.local_runner.preflight.OpikConfig")
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__probe_resolves_api_key_from_env_chain__skips(
self, mock_stdin, mock_config_cls, mock_configure, monkeypatch
):
# The OpikConfig probe layer can pick up an API key from elsewhere
# (e.g. .env file) even when OPIK_API_KEY isn't in os.environ. We
# treat that as "already configured enough" and skip the prompt.
self._patch_env(monkeypatch)
mock_stdin.isatty.return_value = True
mock_config_cls.return_value = self._config_probe(api_key="from-dotenv")
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
mock_configure.assert_not_called()
@patch("opik.cli.configure.run_interactive_configure")
@patch("opik.cli.local_runner.preflight.OpikConfig")
@patch("opik.cli.local_runner.preflight.sys.stdin")
def test__no_config_and_interactive__runs_configure(
self, mock_stdin, mock_config_cls, mock_configure, monkeypatch
):
self._patch_env(monkeypatch)
mock_stdin.isatty.return_value = True
mock_config_cls.return_value = self._config_probe()
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
mock_configure.assert_called_once()
if __name__ == "__main__":
pytest.main([__file__, "-v"])
@@ -0,0 +1,440 @@
"""Unit tests verifying that item tags round-trip through CLI export/import.
Covers the regression where ``opik export`` / ``opik import`` silently dropped
tags for prompts, datasets, and experiments because the exporters used
hand-written field allowlists that omitted ``tags`` (and the importers never
forwarded them). Traces and spans already round-tripped correctly.
"""
import json
import sys
from pathlib import Path
from unittest.mock import Mock, MagicMock, patch
# The experiment/prompt import modules pull in a prompt module that is awkward
# to import in isolation; mirror the shim used by test_import_experiment.py.
sys.modules.setdefault("opik.api_objects.prompt.prompt", MagicMock())
from opik.cli.imports.prompt import import_prompts_from_directory # noqa: E402
from opik.cli.imports.dataset import import_datasets_from_directory # noqa: E402
from opik.cli.exports.prompt import export_single_prompt # noqa: E402
from opik.cli.imports.experiment import ( # noqa: E402
ExperimentData,
recreate_experiment,
)
from opik.cli.exports.utils import create_experiment_data_structure # noqa: E402
class TestPromptTagsImport:
def test_import_text_prompt__with_tags__forwards_tags(self, tmp_path: Path) -> None:
prompt_file = tmp_path / "prompt_p1.json"
prompt_file.write_text(
json.dumps(
{
"name": "greeting",
"current_version": {
"prompt": "Hello {{name}}",
"type": "mustache",
"template_structure": "text",
"tags": ["prod", "greeting"],
},
"history": [],
}
)
)
client = Mock()
client.create_prompt = Mock()
import_prompts_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.create_prompt.assert_called_once()
assert client.create_prompt.call_args.kwargs["tags"] == ["prod", "greeting"]
def test_import_chat_prompt__with_tags__forwards_tags(self, tmp_path: Path) -> None:
prompt_file = tmp_path / "prompt_c1.json"
prompt_file.write_text(
json.dumps(
{
"name": "chat",
"current_version": {
"prompt": [{"role": "user", "content": "hi"}],
"type": "mustache",
"template_structure": "chat",
"tags": ["chatty"],
},
"history": [],
}
)
)
client = Mock()
client.create_chat_prompt = Mock()
import_prompts_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.create_chat_prompt.assert_called_once()
assert client.create_chat_prompt.call_args.kwargs["tags"] == ["chatty"]
class TestDatasetTagsImport:
def _write_dataset(self, tmp_path: Path, payload: dict) -> None:
(tmp_path / "dataset_d1.json").write_text(json.dumps(payload))
def _make_client(self) -> Mock:
client = Mock()
# Force the create path (get_dataset raises -> create_dataset used).
client.get_dataset = Mock(side_effect=Exception("not found"))
created = Mock()
created.id = "new-ds-id"
client.create_dataset = Mock(return_value=created)
client.rest_client = Mock()
client.rest_client.datasets = Mock()
client.rest_client.datasets.update_dataset = Mock()
return client
def test_import_dataset__flat_format_with_tags__applies_tags(
self, tmp_path: Path
) -> None:
self._write_dataset(
tmp_path,
{"name": "ds", "tags": ["a", "b"], "items": []},
)
client = self._make_client()
import_datasets_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.rest_client.datasets.update_dataset.assert_called_once()
call = client.rest_client.datasets.update_dataset.call_args
assert call.args[0] == "new-ds-id"
assert call.kwargs["tags"] == ["a", "b"]
def test_import_dataset__nested_format_with_tags__applies_tags(
self, tmp_path: Path
) -> None:
self._write_dataset(
tmp_path,
{"dataset": {"name": "ds", "id": "x", "tags": ["c"]}, "items": []},
)
client = self._make_client()
import_datasets_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.rest_client.datasets.update_dataset.assert_called_once()
assert client.rest_client.datasets.update_dataset.call_args.kwargs["tags"] == [
"c"
]
def test_import_dataset__no_tags_field__skips_update(self, tmp_path: Path) -> None:
self._write_dataset(tmp_path, {"name": "ds", "items": []})
client = self._make_client()
import_datasets_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.rest_client.datasets.update_dataset.assert_not_called()
def test_import_dataset__empty_tags_list__clears_tags(self, tmp_path: Path) -> None:
# An explicit empty list must still call the REST update so a
# re-import can clear pre-existing tags on the destination.
self._write_dataset(tmp_path, {"name": "ds", "tags": [], "items": []})
client = self._make_client()
import_datasets_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.rest_client.datasets.update_dataset.assert_called_once()
assert client.rest_client.datasets.update_dataset.call_args.kwargs["tags"] == []
def test_import_dataset__existing_dataset_with_description__preserves_description(
self, tmp_path: Path
) -> None:
# Tags-only import must not silently null out a description the
# destination dataset already had (the backend's update PUT doesn't
# COALESCE description against the existing row).
self._write_dataset(
tmp_path,
{"name": "ds", "tags": ["a"], "items": []},
)
client = self._make_client()
# Re-import onto a dataset that already exists (get_dataset succeeds).
existing_dataset = Mock()
existing_dataset.id = "existing-ds-id"
client.get_dataset = Mock(return_value=existing_dataset)
existing_remote = Mock()
existing_remote.description = "original description"
existing_remote.visibility = "private"
client.rest_client.datasets.get_dataset_by_id = Mock(
return_value=existing_remote
)
import_datasets_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
client.rest_client.datasets.update_dataset.assert_called_once()
call = client.rest_client.datasets.update_dataset.call_args
assert call.kwargs["description"] == "original description"
assert call.kwargs["visibility"] == "private"
assert call.kwargs["tags"] == ["a"]
def test_import_dataset__update_dataset_raises__continues_and_inserts_items(
self, tmp_path: Path
) -> None:
# A tag-update failure must be swallowed with a warning so the rest of
# the import (item insertion, count, manifest) still proceeds.
self._write_dataset(
tmp_path,
{"name": "ds", "tags": ["a"], "items": [{"id": "1", "input": "x"}]},
)
client = self._make_client()
client.rest_client.datasets.update_dataset = Mock(side_effect=Exception("boom"))
result = import_datasets_from_directory(
client=client,
source_dir=tmp_path,
project_name="proj",
dry_run=False,
name_pattern=None,
debug=False,
)
# The import did not abort: items were still inserted and the dataset
# counted as imported.
client.create_dataset.return_value.insert.assert_called_once()
assert result["datasets"] == 1
assert result["datasets_errors"] == 0
class TestPromptTagsExport:
"""Prompt tags are container-level and are dropped by the direct
get_prompt() lookup; the exporter must recover them via search_prompts.
"""
def test_export_single_prompt__object_has_tags__skips_search(
self, tmp_path: Path
) -> None:
prompt = Mock()
prompt.id = "p1"
prompt.name = "greeting"
prompt.tags = ["prod"] # direct lookup already carries the tags
client = Mock()
# No history available; _safe_prompt_history swallows the error.
client.get_prompt_history = Mock(side_effect=Exception("no history"))
count = export_single_prompt(
client=client,
prompt=prompt,
output_dir=tmp_path,
project_name="proj",
max_results=None,
force=True,
debug=False,
format="json",
)
assert count == 1
data = json.loads((tmp_path / "prompt_p1.json").read_text())
assert data["current_version"]["tags"] == ["prod"]
# Tags were already present, so no recovery search is needed.
client.search_prompts.assert_not_called()
def test_export_single_prompt__container_only_tags__recovered_from_search(
self, tmp_path: Path
) -> None:
prompt = Mock()
prompt.id = "p1"
prompt.name = "greeting"
prompt.tags = [] # direct lookup dropped the container-level tags
candidate = Mock()
candidate.name = "greeting"
candidate.tags = ["prod", "greeting"]
client = Mock()
client.search_prompts = Mock(return_value=[candidate])
# No history available; _safe_prompt_history swallows the error.
client.get_prompt_history = Mock(side_effect=Exception("no history"))
count = export_single_prompt(
client=client,
prompt=prompt,
output_dir=tmp_path,
project_name="proj",
max_results=None,
force=True,
debug=False,
format="json",
)
assert count == 1
data = json.loads((tmp_path / "prompt_p1.json").read_text())
assert data["current_version"]["tags"] == ["prod", "greeting"]
def test_export_single_prompt__search_fails__still_exports_with_fallback(
self, tmp_path: Path
) -> None:
# Tag resolution is best-effort: a search_prompts failure is logged but
# must not abort the export — the prompt still exports with whatever tags
# the object carried.
prompt = Mock()
prompt.id = "p1"
prompt.name = "greeting"
prompt.tags = [] # nothing on the object; recovery will be attempted
client = Mock()
client.search_prompts = Mock(side_effect=Exception("api down"))
client.get_prompt_history = Mock(side_effect=Exception("no history"))
count = export_single_prompt(
client=client,
prompt=prompt,
output_dir=tmp_path,
project_name="proj",
max_results=None,
force=True,
debug=False,
format="json",
)
assert count == 1
data = json.loads((tmp_path / "prompt_p1.json").read_text())
assert data["current_version"]["tags"] == []
class TestExperimentTags:
def test_export_experiment_structure__with_tags__includes_tags(self) -> None:
experiment = Mock()
experiment.id = "exp-1"
experiment.name = "e"
experiment.dataset_name = "ds"
data_obj = Mock()
data_obj.tags = ["t1", "t2"]
experiment.get_experiment_data = Mock(return_value=data_obj)
structure = create_experiment_data_structure(experiment, [])
assert structure["experiment"]["tags"] == ["t1", "t2"]
def test_import_experiment_from_disk__with_tags__forwards_tags(self) -> None:
client = Mock()
client.flush = Mock(return_value=True)
client.get_or_create_dataset = Mock(return_value=Mock(name="ds"))
created_experiment = Mock()
created_experiment.id = "exp-123"
client.create_experiment = Mock(return_value=created_experiment)
client._rest_client = Mock()
experiment_data = ExperimentData(
experiment={
"id": "exp-123",
"name": "test-experiment",
"dataset_name": "test-dataset",
"type": "regular",
"tags": ["golden", "regression"],
},
items=[],
)
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
mock_id_helpers.generate_id = Mock(return_value="generated-id")
recreate_experiment(
client,
experiment_data,
"test-project",
{}, # trace_id_map
{}, # dataset_item_id_map
dry_run=False,
debug=False,
# target_project_name defaults to None -> import-from-disk path
)
client.create_experiment.assert_called_once()
assert client.create_experiment.call_args.kwargs["tags"] == [
"golden",
"regression",
]
def test_import_experiment_from_disk__empty_tags_list__forwards_empty_tags(
self,
) -> None:
# An explicit empty list must round-trip as tags=[], not collapse to
# tags=None (which would drop a deliberately cleared tag set).
client = Mock()
client.flush = Mock(return_value=True)
client.get_or_create_dataset = Mock(return_value=Mock(name="ds"))
created_experiment = Mock()
created_experiment.id = "exp-123"
client.create_experiment = Mock(return_value=created_experiment)
client._rest_client = Mock()
experiment_data = ExperimentData(
experiment={
"id": "exp-123",
"name": "test-experiment",
"dataset_name": "test-dataset",
"type": "regular",
"tags": [],
},
items=[],
)
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
mock_id_helpers.generate_id = Mock(return_value="generated-id")
recreate_experiment(
client,
experiment_data,
"test-project",
{}, # trace_id_map
{}, # dataset_item_id_map
dry_run=False,
debug=False,
)
client.create_experiment.assert_called_once()
assert client.create_experiment.call_args.kwargs["tags"] == []
@@ -0,0 +1,557 @@
"""Unit tests for matches_trace_filter in exports/trace_filter.py."""
import sys
from types import ModuleType
from unittest.mock import MagicMock, patch
from opik.cli.exports.trace_filter import matches_trace_filter
def _make_oql_module(mock_oql: MagicMock) -> ModuleType:
"""Return a fake opik.api_objects.opik_query_language module whose
OpikQueryLanguage.for_traces() returns *mock_oql*."""
mod = ModuleType("opik.api_objects.opik_query_language")
cls = MagicMock()
cls.for_traces.return_value = mock_oql
mod.OpikQueryLanguage = cls # type: ignore[attr-defined]
return mod
class TestMatchesTraceFilter:
def test_matches_trace_filter__date_time_gte__matches_trace_after_cutoff(self):
"""A trace whose created_at is after the cutoff must pass the filter."""
trace = {
"created_at": "2024-06-01T12:00:00+00:00",
}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "created_at",
"operator": ">=",
"value": "2024-01-01T00:00:00+00:00",
"type": "date_time",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
assert result is True
def test_matches_trace_filter__date_time_gte__excludes_trace_before_cutoff(self):
"""A trace whose created_at is before the cutoff must be excluded."""
trace = {
"created_at": "2023-06-01T12:00:00+00:00",
}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "created_at",
"operator": ">=",
"value": "2024-01-01T00:00:00+00:00",
"type": "date_time",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
assert result is False
def test_matches_trace_filter__invalid_filter__returns_true_with_warning(self):
"""An unparseable filter string must return True and log a warning."""
import opik.cli.exports.trace_filter as trace_filter_module
trace = {"name": "my-trace"}
fake_mod = ModuleType("opik.api_objects.opik_query_language")
cls = MagicMock()
cls.for_traces.side_effect = ValueError("bad filter syntax")
fake_mod.OpikQueryLanguage = cls # type: ignore[attr-defined]
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
with patch.object(trace_filter_module.logger, "warning") as mock_warn:
result = matches_trace_filter(trace, "THIS IS NOT VALID OQL")
assert result is True
mock_warn.assert_called_once()
# The warning message should contain both the filter string and the exception
warning_args = mock_warn.call_args
assert "bad filter syntax" in str(warning_args) or any(
"bad filter syntax" in str(a) for a in warning_args.args
)
def test_matches_trace_filter__string_contains__happyflow(self):
"""A string 'contains' filter must match when the substring is present."""
trace = {"name": "evaluation-run-42"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "contains",
"value": "evaluation",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name contains "evaluation"')
assert result is True
def test_matches_trace_filter__no_expressions__returns_true(self):
"""When the filter parses to zero expressions every trace must pass."""
trace = {"name": "any-trace"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = []
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, "")
assert result is True
# ------------------------------------------------------------------
# number type
# ------------------------------------------------------------------
def test_matches_trace_filter__number_gte__matches(self):
"""A number >= filter must match when the field value satisfies it."""
trace = {"usage": {"total_tokens": 500}}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "usage.total_tokens",
"operator": ">=",
"value": "100",
"type": "number",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'usage.total_tokens >= "100"')
assert result is True
def test_matches_trace_filter__number_lt__excludes(self):
"""A number < filter must exclude a trace that doesn't satisfy it."""
trace = {"usage": {"total_tokens": 50}}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "usage.total_tokens",
"operator": "<",
"value": "10",
"type": "number",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'usage.total_tokens < "10"')
assert result is False
# ------------------------------------------------------------------
# is_empty / is_not_empty
# ------------------------------------------------------------------
def test_matches_trace_filter__is_empty__none_field__matches(self):
"""is_empty must match when the field is absent / None."""
trace = {} # 'error' key is absent
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "error",
"operator": "is_empty",
"value": "",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, "error is_empty")
assert result is True
def test_matches_trace_filter__is_empty__non_empty_field__excludes(self):
"""is_empty must exclude when the field has a value."""
trace = {"error": "something went wrong"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "error",
"operator": "is_empty",
"value": "",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, "error is_empty")
assert result is False
def test_matches_trace_filter__is_not_empty__non_empty_field__matches(self):
"""is_not_empty must match when the field is present and non-empty."""
trace = {"name": "my-trace"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "is_not_empty",
"value": "",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, "name is_not_empty")
assert result is True
def test_matches_trace_filter__is_not_empty__none_field__excludes(self):
"""is_not_empty must exclude when the field is absent."""
trace = {}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "is_not_empty",
"value": "",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, "name is_not_empty")
assert result is False
# ------------------------------------------------------------------
# Additional string operators
# ------------------------------------------------------------------
def test_matches_trace_filter__string_not_contains__excludes(self):
"""not_contains must exclude when the substring is present."""
trace = {"name": "evaluation-run-42"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "not_contains",
"value": "evaluation",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name not_contains "evaluation"')
assert result is False
def test_matches_trace_filter__string_starts_with__matches(self):
"""starts_with must match when the field begins with the prefix."""
trace = {"name": "evaluation-run-42"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "starts_with",
"value": "evaluation",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name starts_with "evaluation"')
assert result is True
def test_matches_trace_filter__string_ends_with__matches(self):
"""ends_with must match when the field ends with the suffix."""
trace = {"name": "evaluation-run-42"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "ends_with",
"value": "42",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name ends_with "42"')
assert result is True
# ------------------------------------------------------------------
# Nested / dotted field access
# ------------------------------------------------------------------
def test_matches_trace_filter__dotted_field__matches(self):
"""Dotted fields like 'usage.total_tokens' should be resolved correctly."""
trace = {"usage": {"total_tokens": 200}}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "usage.total_tokens",
"operator": "=",
"value": "200",
"type": "number",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'usage.total_tokens = "200"')
assert result is True
def test_matches_trace_filter__key_field_access__matches(self):
"""Expressions with a non-empty 'key' resolve via trace_dict[field][key]."""
trace = {"metadata": {"env": "production"}}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "metadata",
"operator": "=",
"value": "production",
"type": "string",
"key": "env",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'metadata.env = "production"')
assert result is True
# ------------------------------------------------------------------
# None field value → False
# ------------------------------------------------------------------
def test_matches_trace_filter__missing_field__returns_false(self):
"""When a non-empty-check expression references a missing field the trace must be excluded."""
trace = {} # 'name' is absent
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "=",
"value": "something",
"type": "string",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name = "something"')
assert result is False
# ------------------------------------------------------------------
# Multiple expressions (AND semantics)
# ------------------------------------------------------------------
def test_matches_trace_filter__multiple_expressions__all_must_pass(self):
"""When two expressions are given both must be satisfied (AND logic)."""
trace = {
"name": "evaluation-run-42",
"created_at": "2024-06-01T12:00:00+00:00",
}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "contains",
"value": "evaluation",
"type": "string",
"key": "",
},
{
"field": "created_at",
"operator": ">=",
"value": "2024-01-01T00:00:00+00:00",
"type": "date_time",
"key": "",
},
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name contains "evaluation"')
assert result is True
def test_matches_trace_filter__multiple_expressions__first_fails__returns_false(
self,
):
"""When the first of two expressions fails the trace must be excluded."""
trace = {
"name": "other-run",
"created_at": "2024-06-01T12:00:00+00:00",
}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "name",
"operator": "contains",
"value": "evaluation",
"type": "string",
"key": "",
},
{
"field": "created_at",
"operator": ">=",
"value": "2024-01-01T00:00:00+00:00",
"type": "date_time",
"key": "",
},
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'name contains "evaluation"')
assert result is False
# ------------------------------------------------------------------
# date_time: naive datetime field value (no tzinfo)
# ------------------------------------------------------------------
def test_matches_trace_filter__date_time_naive__treated_as_utc(self):
"""A naive datetime field value must be compared as UTC."""
from datetime import datetime
trace = {"created_at": datetime(2024, 6, 1, 12, 0, 0)} # no tzinfo
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "created_at",
"operator": ">=",
"value": "2024-01-01T00:00:00+00:00",
"type": "date_time",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
assert result is True
# ------------------------------------------------------------------
# date_time: unparseable field value → result=False (not exception)
# ------------------------------------------------------------------
def test_matches_trace_filter__date_time_unparseable_value__returns_false(self):
"""An unparseable date_time field value must exclude the trace, not raise."""
trace = {"created_at": "not-a-date"}
mock_oql = MagicMock()
mock_oql.get_filter_expressions.return_value = [
{
"field": "created_at",
"operator": ">=",
"value": "2024-01-01T00:00:00+00:00",
"type": "date_time",
"key": "",
}
]
fake_mod = _make_oql_module(mock_oql)
with patch.dict(
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
):
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
assert result is False