chore: import upstream snapshot with attribution
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1 @@
|
||||
"""Unit tests for CLI functionality."""
|
||||
@@ -0,0 +1,196 @@
|
||||
"""Shared mock helpers for ``opik migrate`` test modules.
|
||||
|
||||
Used by both ``test_migrate_dataset_exclude_versions.py`` (Slice 1 paths) and
|
||||
``test_migrate_dataset_version_replay.py`` (Slice 2 paths). Lives as a plain
|
||||
module (not conftest.py) because these are helper classes, not pytest
|
||||
fixtures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Dict, List, Optional
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
|
||||
class _DatasetRow:
|
||||
def __init__(
|
||||
self,
|
||||
id: str,
|
||||
name: str,
|
||||
description: Optional[str] = None,
|
||||
items: int = 0,
|
||||
type: Optional[str] = "dataset",
|
||||
visibility: Optional[str] = "private",
|
||||
tags: Optional[List[str]] = None,
|
||||
# ``project_id=None`` represents a workspace-scoped dataset (V1
|
||||
# entity, or anything left at workspace scope after auto-migration).
|
||||
# Tests that want a project-scoped source pass an explicit id.
|
||||
project_id: Optional[str] = None,
|
||||
) -> None:
|
||||
self.id = id
|
||||
self.name = name
|
||||
self.description = description
|
||||
self.dataset_items_count = items
|
||||
self.type = type
|
||||
self.visibility = visibility
|
||||
self.tags = tags
|
||||
self.project_id = project_id
|
||||
|
||||
|
||||
class _Page:
|
||||
def __init__(self, content: List[_DatasetRow]) -> None:
|
||||
self.content = content
|
||||
|
||||
|
||||
def _named(name: str) -> MagicMock:
|
||||
obj = MagicMock()
|
||||
obj.name = name
|
||||
return obj
|
||||
|
||||
|
||||
def _planner_rest_client(
|
||||
find_side_effects: List[_Page],
|
||||
*,
|
||||
target_project_exists: bool = True,
|
||||
workspace_project_names: Optional[List[str]] = None,
|
||||
) -> MagicMock:
|
||||
"""Build a rest_client mock for direct planner unit tests."""
|
||||
rest_client = MagicMock()
|
||||
if target_project_exists:
|
||||
target_project = MagicMock()
|
||||
target_project.id = "target-project-id"
|
||||
rest_client.projects.retrieve_project.return_value = target_project
|
||||
else:
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
rest_client.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={}
|
||||
)
|
||||
# Suggestion lookup queries find_projects on the 404 path.
|
||||
candidates = [_named(name) for name in (workspace_project_names or [])]
|
||||
rest_client.projects.find_projects.return_value = _Page(candidates)
|
||||
rest_client.datasets.find_datasets.side_effect = find_side_effects
|
||||
return rest_client
|
||||
|
||||
|
||||
def _planner_client(rest_client: MagicMock) -> MagicMock:
|
||||
"""Wrap a planner rest_client mock as an ``opik.Opik``-shaped client.
|
||||
|
||||
The planner now takes the high-level client (so the resolver can route
|
||||
``get_project_by_id`` through ``client.get_project`` instead of the
|
||||
Fern surface). Tests still build the low-level rest_client mock and
|
||||
drive its side_effects, then wrap with this helper to satisfy the new
|
||||
planner signature.
|
||||
|
||||
``client.get_project(id=...)`` delegates to
|
||||
``rest_client.projects.get_project_by_id(id=...)`` so any per-test
|
||||
stub on the rest_client side still flows through unchanged.
|
||||
"""
|
||||
client = MagicMock()
|
||||
client.rest_client = rest_client
|
||||
client.get_project = MagicMock(
|
||||
side_effect=lambda id: rest_client.projects.get_project_by_id(id=id)
|
||||
)
|
||||
return client
|
||||
|
||||
|
||||
def _build_fake_client(
|
||||
*,
|
||||
source_rows: List[_DatasetRow],
|
||||
destination_rows: List[_DatasetRow],
|
||||
items: List[Dict[str, object]],
|
||||
target_project_exists: bool = True,
|
||||
stale_temp_rows: Optional[List[_DatasetRow]] = None,
|
||||
) -> MagicMock:
|
||||
"""Construct an opik.Opik mock matching the executor's call surface.
|
||||
|
||||
The planner makes three workspace ``find_datasets`` lookups in a fixed
|
||||
order: (1) source resolution, (2) the ``<name>_v1`` rename-target
|
||||
collision pre-flight, and (3) the ``<name>__migrating`` stale-temp
|
||||
lookup (OPIK-7162). A side-effect list drives all three deterministically.
|
||||
``destination_rows`` feeds the ``_v1`` check; the stale-temp lookup
|
||||
returns empty by default (no leftover temp) unless ``stale_temp_rows``
|
||||
is supplied.
|
||||
|
||||
``items`` is a list of dicts; we materialize them as DatasetItem
|
||||
dataclasses for the streaming mock (matches the real `__internal_api__
|
||||
stream_items_as_dataclasses__` shape) so per-item fidelity assertions
|
||||
have somewhere to land.
|
||||
"""
|
||||
from opik.api_objects.dataset import dataset_item
|
||||
|
||||
rest_client = MagicMock()
|
||||
if target_project_exists:
|
||||
target_project = MagicMock()
|
||||
target_project.id = "target-project-id"
|
||||
rest_client.projects.retrieve_project.return_value = target_project
|
||||
else:
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
rest_client.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={}
|
||||
)
|
||||
rest_client.datasets.find_datasets.side_effect = [
|
||||
_Page(source_rows),
|
||||
_Page(destination_rows),
|
||||
_Page(stale_temp_rows or []),
|
||||
]
|
||||
rest_client.datasets.update_dataset = MagicMock()
|
||||
rest_client.datasets.create_dataset = MagicMock()
|
||||
rest_client.datasets.delete_dataset = MagicMock()
|
||||
|
||||
client = MagicMock()
|
||||
client.rest_client = rest_client
|
||||
client._workspace = "default"
|
||||
|
||||
# Build DatasetItem dataclasses from the provided dicts so the executor's
|
||||
# dataclass-form stream returns a realistic shape. Top-level fields like
|
||||
# `description` / `source` / `trace_id` / `span_id` can be passed via the
|
||||
# dict (other keys get stuffed into `data`/extra).
|
||||
top_level = {
|
||||
"id",
|
||||
"trace_id",
|
||||
"span_id",
|
||||
"source",
|
||||
"description",
|
||||
"evaluators",
|
||||
"execution_policy",
|
||||
}
|
||||
source_items: List[dataset_item.DatasetItem] = []
|
||||
for raw in items:
|
||||
kwargs = {k: v for k, v in raw.items() if k in top_level and k != "id"}
|
||||
data = {k: v for k, v in raw.items() if k not in top_level}
|
||||
ds_item = dataset_item.DatasetItem(**kwargs, **data)
|
||||
if "id" in raw:
|
||||
ds_item.id = raw["id"] # type: ignore[assignment]
|
||||
source_items.append(ds_item)
|
||||
|
||||
# MagicMock treats dunder-prefixed names as magic and blocks them by
|
||||
# default; pre-attach plain MagicMocks so attribute access works.
|
||||
source_dataset = MagicMock()
|
||||
stream_mock = MagicMock(return_value=iter(source_items))
|
||||
source_dataset.__internal_api__stream_items_as_dataclasses__ = stream_mock
|
||||
|
||||
dest_dataset = MagicMock()
|
||||
insert_mock = MagicMock()
|
||||
dest_dataset.__internal_api__insert_items_as_dataclasses__ = insert_mock
|
||||
|
||||
# Under the OPIK-7162 ordering the destination is written under the temp
|
||||
# name ``<orig>__migrating`` for the whole copy, then promoted to the
|
||||
# original name at the end. The executor resolves the destination by name
|
||||
# in ``_replay_versions`` (temp name) and again in ``PromoteDestination``
|
||||
# (temp name -> id). The source is never resolved via ``get_dataset`` any
|
||||
# more (item reads stream by name on the rest_client), so route the temp
|
||||
# name (and anything else) to the destination dataset.
|
||||
source_orig_name = source_rows[0].name if source_rows else ""
|
||||
temp_name = f"{source_orig_name}__migrating"
|
||||
|
||||
def _get_dataset(name: str, project_name: Optional[str] = None) -> MagicMock:
|
||||
if name == temp_name:
|
||||
return dest_dataset
|
||||
return source_dataset
|
||||
|
||||
client.get_dataset.side_effect = _get_dataset
|
||||
client.create_dataset = MagicMock()
|
||||
client.delete_dataset = MagicMock()
|
||||
return client, source_dataset, dest_dataset
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Shared mock helpers for ``opik migrate prompt`` test modules.
|
||||
|
||||
Used by ``test_migrate_prompt_planner.py``,
|
||||
``test_migrate_prompt_executor.py``, and
|
||||
``test_migrate_prompt_version_replay.py``. Lives as a plain module (not
|
||||
a conftest.py) because these are helper classes, not pytest fixtures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, List, Optional
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
|
||||
class _PromptRow:
|
||||
"""Minimal stand-in for ``PromptPublic`` rows returned by ``get_prompts``."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
id: str,
|
||||
name: str,
|
||||
*,
|
||||
description: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
# None = workspace-scoped (legacy v1 prompts that never had a
|
||||
# project). Tests that want a project-scoped source pass an id.
|
||||
project_id: Optional[str] = None,
|
||||
template_structure: Optional[str] = "text",
|
||||
) -> None:
|
||||
self.id = id
|
||||
self.name = name
|
||||
self.description = description
|
||||
self.tags = tags
|
||||
self.project_id = project_id
|
||||
self.template_structure = template_structure
|
||||
|
||||
|
||||
class _PromptVersionRow:
|
||||
"""Stand-in for ``PromptVersionPublic`` rows from ``get_prompt_versions``.
|
||||
|
||||
Field set mirrors the Fern model: ``id``, ``prompt_id``, ``commit``,
|
||||
``template`` (required), plus optional metadata / type / change_description
|
||||
/ tags / environments / template_structure.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
id: str,
|
||||
prompt_id: str,
|
||||
commit: str,
|
||||
template: str,
|
||||
*,
|
||||
metadata: Optional[Any] = None,
|
||||
type: Optional[str] = None,
|
||||
change_description: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
environments: Optional[List[str]] = None,
|
||||
template_structure: Optional[str] = "text",
|
||||
) -> None:
|
||||
self.id = id
|
||||
self.prompt_id = prompt_id
|
||||
self.commit = commit
|
||||
self.template = template
|
||||
self.metadata = metadata
|
||||
self.type = type
|
||||
self.change_description = change_description
|
||||
self.tags = tags
|
||||
self.environments = environments
|
||||
self.template_structure = template_structure
|
||||
|
||||
|
||||
class _Page:
|
||||
def __init__(self, content: List[Any]) -> None:
|
||||
self.content = content
|
||||
|
||||
|
||||
def _named(name: str) -> MagicMock:
|
||||
obj = MagicMock()
|
||||
obj.name = name
|
||||
return obj
|
||||
|
||||
|
||||
def _planner_rest_client(
|
||||
find_prompts_side_effects: List[_Page],
|
||||
*,
|
||||
target_project_exists: bool = True,
|
||||
workspace_project_names: Optional[List[str]] = None,
|
||||
) -> MagicMock:
|
||||
"""Build a rest_client mock for direct prompt-planner unit tests.
|
||||
|
||||
``find_prompts_side_effects`` is the side-effect list for
|
||||
``rest_client.prompts.get_prompts``; the planner calls it twice (once
|
||||
for source resolution, once for the rename-collision preflight) so
|
||||
callers pass two pages.
|
||||
"""
|
||||
rest_client = MagicMock()
|
||||
if target_project_exists:
|
||||
target_project = MagicMock()
|
||||
target_project.id = "target-project-id"
|
||||
rest_client.projects.retrieve_project.return_value = target_project
|
||||
else:
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
rest_client.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={}
|
||||
)
|
||||
candidates = [_named(n) for n in (workspace_project_names or [])]
|
||||
rest_client.projects.find_projects.return_value = _Page(candidates)
|
||||
rest_client.prompts.get_prompts.side_effect = find_prompts_side_effects
|
||||
return rest_client
|
||||
|
||||
|
||||
def _planner_client(rest_client: MagicMock) -> MagicMock:
|
||||
"""Wrap a rest_client mock as an ``opik.Opik``-shaped client.
|
||||
|
||||
The resolver routes project-id lookups through ``client.get_project``
|
||||
(high-level surface) so the project-name resolution still benefits
|
||||
from any test-side stubs on ``rest_client.projects.get_project_by_id``.
|
||||
"""
|
||||
client = MagicMock()
|
||||
client.rest_client = rest_client
|
||||
client.get_project = MagicMock(
|
||||
side_effect=lambda id: rest_client.projects.get_project_by_id(id=id)
|
||||
)
|
||||
return client
|
||||
@@ -0,0 +1,313 @@
|
||||
"""Tests for opik.cli.local_runner.stop and the `opik <type> stop` CLI surface."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli.local_runner import stop as stop_module
|
||||
from opik.cli.local_runner.pairing import RunnerType
|
||||
from opik.cli.local_runner.stop import do_stop
|
||||
from opik.cli.main import cli
|
||||
from opik.runner import pid_file
|
||||
from opik.runner.pid_file import RunnerInfo
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_runners_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
runners = tmp_path / "runners"
|
||||
monkeypatch.setattr(pid_file, "_RUNNERS_DIR", runners)
|
||||
return runners
|
||||
|
||||
|
||||
def _write_runner_file(
|
||||
dir: Path,
|
||||
runner_id: str,
|
||||
runner_type: str,
|
||||
project_name: str,
|
||||
pid: int,
|
||||
workspace: Optional[str] = None,
|
||||
) -> Path:
|
||||
dir.mkdir(parents=True, exist_ok=True)
|
||||
path = dir / f"{runner_type}-{runner_id}.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"pid": pid,
|
||||
"runner_id": runner_id,
|
||||
"runner_type": runner_type,
|
||||
"project_name": project_name,
|
||||
"workspace": workspace,
|
||||
"started_at": 1.0,
|
||||
}
|
||||
)
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
class TestDoStopFilters:
|
||||
def test_do_stop__no_filter__usage_error(self, tmp_runners_dir: Path) -> None:
|
||||
# No --project / --runner / --all → usage error before any signaling.
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
do_stop(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name=None,
|
||||
all_flag=False,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
# click.UsageError prints "Error: ..." and is a Click exception.
|
||||
assert "specify" in str(exc_info.value).lower()
|
||||
|
||||
def test_do_stop__no_match__prints_no_match_message(
|
||||
self, tmp_runners_dir: Path, capsys: pytest.CaptureFixture
|
||||
) -> None:
|
||||
do_stop(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name="missing",
|
||||
all_flag=False,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
out = capsys.readouterr().out
|
||||
assert "No local 'connect' runners found" in out
|
||||
|
||||
def test_do_stop__project_filter__signals_matching_only(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
|
||||
_write_runner_file(tmp_runners_dir, "r-2", "connect", "beta", os.getpid())
|
||||
|
||||
seen = []
|
||||
|
||||
def fake_signal(info: RunnerInfo):
|
||||
seen.append(info.runner_id)
|
||||
return True, ""
|
||||
|
||||
with patch.object(stop_module, "_signal_until_gone", side_effect=fake_signal):
|
||||
do_stop(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name="alpha",
|
||||
all_flag=False,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
|
||||
assert seen == ["r-1"]
|
||||
# The matched runner's pid file is cleaned up.
|
||||
assert not (tmp_runners_dir / "connect-r-1.json").exists()
|
||||
# Non-matched is left alone.
|
||||
assert (tmp_runners_dir / "connect-r-2.json").exists()
|
||||
|
||||
def test_do_stop__all_flag__signals_only_target_runner_type(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
|
||||
_write_runner_file(tmp_runners_dir, "r-2", "connect", "beta", os.getpid())
|
||||
# An endpoint runner must NOT be touched by `connect stop --all`.
|
||||
_write_runner_file(tmp_runners_dir, "r-3", "endpoint", "alpha", os.getpid())
|
||||
|
||||
seen = []
|
||||
with patch.object(
|
||||
stop_module,
|
||||
"_signal_until_gone",
|
||||
lambda info: (seen.append(info.runner_id) or (True, "")),
|
||||
):
|
||||
do_stop(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name=None,
|
||||
all_flag=True,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
assert sorted(seen) == ["r-1", "r-2"]
|
||||
assert (tmp_runners_dir / "endpoint-r-3.json").exists()
|
||||
|
||||
def test_do_stop__runner_id_filter__signals_matching_only(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
|
||||
_write_runner_file(tmp_runners_dir, "r-2", "connect", "alpha", os.getpid())
|
||||
|
||||
seen = []
|
||||
with patch.object(
|
||||
stop_module,
|
||||
"_signal_until_gone",
|
||||
lambda info: (seen.append(info.runner_id) or (True, "")),
|
||||
):
|
||||
do_stop(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name=None,
|
||||
all_flag=False,
|
||||
runner_id_filter="r-2",
|
||||
)
|
||||
assert seen == ["r-2"]
|
||||
|
||||
|
||||
class TestSignalUntilGone:
|
||||
"""Logic tests for the SIGTERM → wait → SIGKILL flow.
|
||||
|
||||
Real-subprocess kill is exercised by `tests/unit/runner/test_pid_file.py::TestSignalRoundtrip`;
|
||||
those cases call `proc.wait()` to reap zombies, which pytest's parent-child
|
||||
relationship requires. Here we mock the alive-check + signal so we can assert
|
||||
the escalation order without fighting zombie state.
|
||||
"""
|
||||
|
||||
def _make_info(self, tmp_runners_dir: Path, pid: int = 12345) -> RunnerInfo:
|
||||
return RunnerInfo(
|
||||
pid=pid,
|
||||
runner_id="r-1",
|
||||
runner_type="connect",
|
||||
project_name="p",
|
||||
workspace=None,
|
||||
started_at=0.0,
|
||||
path=tmp_runners_dir / "connect-r-1.json",
|
||||
)
|
||||
|
||||
def test_signal_until_gone__target_exits_after_sigterm__returns_ok(
|
||||
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.setattr(stop_module, "_POLL_INTERVAL_SECONDS", 0.01)
|
||||
sent_signals = []
|
||||
monkeypatch.setattr(
|
||||
stop_module.os, "kill", lambda pid, sig: sent_signals.append((pid, sig))
|
||||
)
|
||||
# First poll says alive, next says gone.
|
||||
alive_seq = iter([True, False])
|
||||
monkeypatch.setattr(
|
||||
stop_module.pid_file, "is_pid_alive", lambda _pid: next(alive_seq)
|
||||
)
|
||||
|
||||
ok, _ = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
|
||||
assert ok
|
||||
assert sent_signals == [(12345, signal.SIGTERM)]
|
||||
|
||||
def test_signal_until_gone__sigterm_ignored__escalates_to_sigkill(
|
||||
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
monkeypatch.setattr(stop_module, "_SIGTERM_GRACE_SECONDS", 0.05)
|
||||
monkeypatch.setattr(stop_module, "_POLL_INTERVAL_SECONDS", 0.01)
|
||||
sent_signals = []
|
||||
state = {"alive": True}
|
||||
|
||||
def fake_kill(pid: int, sig: int) -> None:
|
||||
sent_signals.append((pid, sig))
|
||||
if sig == signal.SIGKILL:
|
||||
state["alive"] = False
|
||||
|
||||
monkeypatch.setattr(stop_module.os, "kill", fake_kill)
|
||||
monkeypatch.setattr(
|
||||
stop_module.pid_file, "is_pid_alive", lambda _pid: state["alive"]
|
||||
)
|
||||
|
||||
ok, _ = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
|
||||
assert ok
|
||||
assert [sig for _, sig in sent_signals] == [signal.SIGTERM, signal.SIGKILL]
|
||||
|
||||
def test_signal_until_gone__already_dead_pid__returns_ok(
|
||||
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
def fake_kill(_pid: int, _sig: int) -> None:
|
||||
raise ProcessLookupError()
|
||||
|
||||
monkeypatch.setattr(stop_module.os, "kill", fake_kill)
|
||||
ok, reason = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
|
||||
assert ok
|
||||
assert "exited" in reason
|
||||
|
||||
def test_signal_until_gone__permission_error__returns_failure(
|
||||
self, tmp_runners_dir: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
def fake_kill(_pid: int, _sig: int) -> None:
|
||||
raise PermissionError("not yours")
|
||||
|
||||
monkeypatch.setattr(stop_module.os, "kill", fake_kill)
|
||||
ok, reason = stop_module._signal_until_gone(self._make_info(tmp_runners_dir))
|
||||
assert not ok
|
||||
assert "permission" in reason.lower()
|
||||
|
||||
|
||||
class TestStopExitCode:
|
||||
def test_do_stop__signal_failed__raises_systemexit_1(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
_write_runner_file(tmp_runners_dir, "r-1", "connect", "alpha", os.getpid())
|
||||
with patch.object(
|
||||
stop_module, "_signal_until_gone", return_value=(False, "denied")
|
||||
):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
do_stop(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name="alpha",
|
||||
all_flag=False,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
|
||||
class TestConnectStopCli:
|
||||
def test_cli_connect_stop__no_filter__exits_with_usage_error(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["connect", "stop"])
|
||||
assert result.exit_code == 2
|
||||
assert "Specify" in result.output or "specify" in result.output
|
||||
|
||||
def test_cli_connect_stop__with_project__routes_to_do_stop(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
|
||||
result = runner.invoke(cli, ["connect", "stop", "--project", "p"])
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_stop.assert_called_once()
|
||||
assert mock_stop.call_args.kwargs == dict(
|
||||
runner_type=RunnerType.CONNECT,
|
||||
project_name="p",
|
||||
all_flag=False,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
|
||||
def test_cli_connect_stop__all_flag__sets_all_flag_kwarg(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
|
||||
result = runner.invoke(cli, ["connect", "stop", "--all"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert mock_stop.call_args.kwargs["all_flag"] is True
|
||||
|
||||
def test_cli_connect_stop__runner_flag__sets_runner_id_filter(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
|
||||
result = runner.invoke(cli, ["connect", "stop", "--runner", "r-1"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert mock_stop.call_args.kwargs["runner_id_filter"] == "r-1"
|
||||
|
||||
|
||||
class TestEndpointStopCli:
|
||||
def test_cli_endpoint_stop__with_project__routes_to_do_stop(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
|
||||
result = runner.invoke(cli, ["endpoint", "stop", "--project", "p"])
|
||||
assert result.exit_code == 0, result.output
|
||||
mock_stop.assert_called_once()
|
||||
assert mock_stop.call_args.kwargs == dict(
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
project_name="p",
|
||||
all_flag=False,
|
||||
runner_id_filter=None,
|
||||
)
|
||||
|
||||
def test_cli_endpoint_stop__all_flag__sets_all_flag_kwarg(
|
||||
self, tmp_runners_dir: Path
|
||||
) -> None:
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.local_runner.stop.do_stop") as mock_stop:
|
||||
result = runner.invoke(cli, ["endpoint", "stop", "--all"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert mock_stop.call_args.kwargs["all_flag"] is True
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,445 @@
|
||||
"""Tests for CLI commands."""
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
from contextlib import contextmanager
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli import cli
|
||||
|
||||
|
||||
class TestDownloadCommand:
|
||||
"""Test the download CLI command."""
|
||||
|
||||
def test_export_group_help(self):
|
||||
"""Test that the export group shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["export", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Export data from an Opik project" in result.output
|
||||
assert "dataset" in result.output
|
||||
assert "traces" in result.output
|
||||
assert "experiment" in result.output
|
||||
|
||||
def test_export_dataset_help(self):
|
||||
"""Test that the export dataset command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["export", "default", "proj", "dataset", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Export a dataset by exact name" in result.output
|
||||
assert "--force" in result.output
|
||||
|
||||
def test_export_traces_help(self):
|
||||
"""Test that the export traces command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["export", "default", "proj", "traces", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Export the project's traces" in result.output
|
||||
|
||||
def test_export_experiment_help(self):
|
||||
"""Test that the export experiment command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli, ["export", "default", "proj", "experiment", "--help"]
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "Export an experiment by exact name" in result.output
|
||||
|
||||
|
||||
class TestUploadCommand:
|
||||
"""Test the upload CLI command."""
|
||||
|
||||
def test_import_group_help(self):
|
||||
"""Test that the import group shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["import", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Import data into an Opik project" in result.output
|
||||
assert "dataset" in result.output
|
||||
assert "traces" in result.output
|
||||
assert "experiment" in result.output
|
||||
|
||||
def test_import_dataset_help(self):
|
||||
"""Test that the import dataset command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["import", "default", "proj", "dataset", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Import datasets from" in result.output
|
||||
assert "--dry-run" in result.output
|
||||
|
||||
def test_import_traces_help(self):
|
||||
"""Test that the import traces command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["import", "default", "proj", "traces", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Import the project's traces" in result.output
|
||||
assert "--dry-run" in result.output
|
||||
|
||||
def test_import_experiment_help(self):
|
||||
"""Test that the import experiment command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli, ["import", "default", "proj", "experiment", "--help"]
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "Import experiments from" in result.output
|
||||
assert "--dry-run" in result.output
|
||||
|
||||
|
||||
class TestSmokeTestCommand:
|
||||
"""Test the smoke-test functionality via healthcheck command."""
|
||||
|
||||
def test_smoke_test_help(self):
|
||||
"""Test that the healthcheck --smoke-test command shows help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["healthcheck", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "--smoke-test" in result.output
|
||||
assert "--project-name" in result.output
|
||||
assert "Project name for the smoke test" in result.output
|
||||
assert "WORKSPACE" in result.output
|
||||
assert "Run a smoke test to verify Opik integration" in result.output
|
||||
|
||||
def test_smoke_test_minimal_args_parsing(self):
|
||||
"""Test that healthcheck --smoke-test command requires workspace value."""
|
||||
runner = CliRunner()
|
||||
# Test that help shows workspace is required
|
||||
result = runner.invoke(cli, ["healthcheck", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "WORKSPACE" in result.output
|
||||
# Test that missing workspace value causes error
|
||||
result = runner.invoke(cli, ["healthcheck", "--smoke-test"])
|
||||
assert result.exit_code != 0
|
||||
assert "Error" in result.output or "Missing" in result.output
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik.Opik")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik.start_as_current_trace")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_trace")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_span")
|
||||
@patch("opik.cli.healthcheck.smoke_test.create_opik_logo_image")
|
||||
@patch("opik.cli.healthcheck.smoke_test.track")
|
||||
def test_smoke_test_with_workspace_and_project_name(
|
||||
self,
|
||||
mock_track,
|
||||
mock_create_logo,
|
||||
mock_update_span,
|
||||
mock_update_trace,
|
||||
mock_start_trace,
|
||||
mock_opik_class,
|
||||
mock_healthcheck_run,
|
||||
):
|
||||
"""Test healthcheck --smoke-test command with workspace and --project-name arguments."""
|
||||
# Setup mocks
|
||||
mock_client = MagicMock()
|
||||
# Mock search_traces to raise an exception immediately to skip verification polling
|
||||
# This prevents the verification from running and triggering real client creation
|
||||
mock_client.search_traces = MagicMock(
|
||||
side_effect=AttributeError("Mock client - search_traces not available")
|
||||
)
|
||||
mock_opik_class.return_value = mock_client
|
||||
|
||||
# Mock the context manager for start_as_current_trace
|
||||
@contextmanager
|
||||
def mock_trace_context(*args, **kwargs):
|
||||
yield MagicMock()
|
||||
|
||||
mock_start_trace.return_value = mock_trace_context()
|
||||
|
||||
# Mock create_opik_logo_image to return a fake path
|
||||
mock_logo_path = Path("/tmp/fake_logo.png")
|
||||
mock_create_logo.return_value = mock_logo_path
|
||||
|
||||
# Mock the track decorator to return the function unchanged
|
||||
def track_decorator(*args, **kwargs):
|
||||
def decorator(func):
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
mock_track.side_effect = track_decorator
|
||||
|
||||
# Run the command
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"healthcheck",
|
||||
"--smoke-test",
|
||||
"test-workspace",
|
||||
"--project-name",
|
||||
"test-project",
|
||||
],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
# Assertions
|
||||
assert result.exit_code == 0
|
||||
# Verify client was created with correct arguments
|
||||
mock_opik_class.assert_called_once()
|
||||
call_kwargs = mock_opik_class.call_args[1]
|
||||
assert call_kwargs["workspace"] == "test-workspace"
|
||||
assert call_kwargs["project_name"] == "test-project"
|
||||
# Verify trace was started
|
||||
mock_start_trace.assert_called_once()
|
||||
# Verify explicit client was flushed and ended
|
||||
mock_client.flush.assert_called_once()
|
||||
mock_client.end.assert_called_once()
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik.Opik")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik.start_as_current_trace")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_trace")
|
||||
@patch("opik.cli.healthcheck.smoke_test.opik_context.update_current_span")
|
||||
@patch("opik.cli.healthcheck.smoke_test.create_opik_logo_image")
|
||||
@patch("opik.cli.healthcheck.smoke_test.track")
|
||||
def test_smoke_test_with_default_project_name(
|
||||
self,
|
||||
mock_track,
|
||||
mock_create_logo,
|
||||
mock_update_span,
|
||||
mock_update_trace,
|
||||
mock_start_trace,
|
||||
mock_opik_class,
|
||||
mock_healthcheck_run,
|
||||
):
|
||||
"""Test healthcheck --smoke-test command with workspace only (uses default project name)."""
|
||||
# Setup mocks
|
||||
mock_client = MagicMock()
|
||||
# Mock search_traces to raise an exception immediately to skip verification polling
|
||||
# This prevents the verification from running and triggering real client creation
|
||||
mock_client.search_traces = MagicMock(
|
||||
side_effect=AttributeError("Mock client - search_traces not available")
|
||||
)
|
||||
mock_opik_class.return_value = mock_client
|
||||
|
||||
# Mock the context manager for start_as_current_trace
|
||||
@contextmanager
|
||||
def mock_trace_context(*args, **kwargs):
|
||||
yield MagicMock()
|
||||
|
||||
mock_start_trace.return_value = mock_trace_context()
|
||||
|
||||
# Mock create_opik_logo_image to return a fake path
|
||||
mock_logo_path = Path("/tmp/fake_logo.png")
|
||||
mock_create_logo.return_value = mock_logo_path
|
||||
|
||||
# Mock the track decorator to return the function unchanged
|
||||
def track_decorator(*args, **kwargs):
|
||||
def decorator(func):
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
mock_track.side_effect = track_decorator
|
||||
|
||||
# Run the command without --project-name (should use default)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["healthcheck", "--smoke-test", "test-workspace"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
# Assertions
|
||||
assert result.exit_code == 0
|
||||
# Verify client was created with default project name
|
||||
mock_opik_class.assert_called_once()
|
||||
call_kwargs = mock_opik_class.call_args[1]
|
||||
assert call_kwargs["workspace"] == "test-workspace"
|
||||
assert call_kwargs["project_name"] == "smoke-test-project" # Default value
|
||||
# Verify trace was started with default project name
|
||||
mock_start_trace.assert_called_once()
|
||||
trace_call_kwargs = mock_start_trace.call_args[1]
|
||||
assert trace_call_kwargs["project_name"] == "smoke-test-project"
|
||||
|
||||
|
||||
class TestCheckPermissionsCommand:
|
||||
"""Test the --check-permissions functionality via healthcheck command."""
|
||||
|
||||
def test_check_permissions_option__in_healthcheck_help__shown(self):
|
||||
"""Test that --check-permissions option is shown in healthcheck help."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["healthcheck", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "--check-permissions" in result.output
|
||||
assert "WORKSPACE" in result.output
|
||||
|
||||
def test_check_permissions__missing_workspace__raises_error(self):
|
||||
"""Test that --check-permissions without a workspace value raises an error."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["healthcheck", "--check-permissions"])
|
||||
assert result.exit_code != 0
|
||||
assert "Error" in result.output or "Missing" in result.output
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
|
||||
def test_check_permissions__workspace_provided__calls_run_with_workspace(
|
||||
self,
|
||||
mock_permissions_run,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that --check-permissions pass the workspace to check_user_permissions.run."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["healthcheck", "--check-permissions", "my-workspace"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
assert result.exit_code == 0
|
||||
mock_permissions_run.assert_called_once_with(
|
||||
api_key=None, workspace="my-workspace"
|
||||
)
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
|
||||
def test_check_permissions__api_key_in_cli_context__passes_api_key_to_run(
|
||||
self,
|
||||
mock_permissions_run,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that --check-permissions forward the api_key from CLI context."""
|
||||
from opik.cli import cli as opik_cli
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
opik_cli,
|
||||
[
|
||||
"--api-key",
|
||||
"test-api-key",
|
||||
"healthcheck",
|
||||
"--check-permissions",
|
||||
"my-workspace",
|
||||
],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
assert result.exit_code == 0
|
||||
mock_permissions_run.assert_called_once_with(
|
||||
api_key="test-api-key", workspace="my-workspace"
|
||||
)
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
|
||||
def test_check_permissions__empty_workspace__raises_bad_parameter(
|
||||
self,
|
||||
mock_permissions_run,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that --check-permissions with an empty workspace string raises BadParameter."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["healthcheck", "--check-permissions", ""],
|
||||
)
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert (
|
||||
"Error" in result.output
|
||||
or "requires a non-empty workspace name" in result.output
|
||||
)
|
||||
mock_permissions_run.assert_not_called()
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.cli.check_user_permissions.run")
|
||||
def test_check_permissions__no_flag__not_called(
|
||||
self,
|
||||
mock_permissions_run,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that check_user_permissions.run is not called when --check-permissions is absent."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["healthcheck"], catch_exceptions=False)
|
||||
|
||||
assert result.exit_code == 0
|
||||
mock_permissions_run.assert_not_called()
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.check_user_permissions.get_user_permissions")
|
||||
@patch("opik.cli.healthcheck.check_user_permissions.config.OpikConfig")
|
||||
def test_check_permissions__api_returns_user_and_workspace__displays_info(
|
||||
self,
|
||||
mock_opik_config,
|
||||
mock_get_permissions,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that user and workspace info from API response are printed."""
|
||||
mock_config = MagicMock()
|
||||
mock_config.api_key = "test-api-key"
|
||||
mock_config.workspace = "my-workspace"
|
||||
mock_config.url_override = "https://opik.example.com"
|
||||
mock_opik_config.return_value = mock_config
|
||||
|
||||
mock_get_permissions.return_value = {
|
||||
"user_name": "alice",
|
||||
"workspace_name": "my-workspace",
|
||||
"permissions": [
|
||||
{"permission_name": "read", "permission_value": True},
|
||||
{"permission_name": "write", "permission_value": False},
|
||||
],
|
||||
}
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["healthcheck", "--check-permissions", "my-workspace"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "alice" in result.output
|
||||
assert "my-workspace" in result.output
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.check_user_permissions.get_user_permissions")
|
||||
@patch("opik.cli.healthcheck.check_user_permissions.config.OpikConfig")
|
||||
def test_check_permissions__connection_error__prints_error(
|
||||
self,
|
||||
mock_opik_config,
|
||||
mock_get_permissions,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that a ConnectionError from the API prints an error message."""
|
||||
mock_config = MagicMock()
|
||||
mock_config.api_key = "test-api-key"
|
||||
mock_config.workspace = "my-workspace"
|
||||
mock_config.url_override = "https://opik.example.com"
|
||||
mock_opik_config.return_value = mock_config
|
||||
|
||||
mock_get_permissions.side_effect = ConnectionError("Network error: timeout")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["healthcheck", "--check-permissions", "my-workspace"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "Failed to fetch user permissions" in result.output
|
||||
|
||||
@patch("opik.cli.healthcheck.cli.standard_check.run")
|
||||
@patch("opik.cli.healthcheck.check_user_permissions.config.OpikConfig")
|
||||
def test_check_permissions__missing_api_key__raises_value_error(
|
||||
self,
|
||||
mock_opik_config,
|
||||
mock_standard_run,
|
||||
):
|
||||
"""Test that missing api_key (not in CLI context and not in config) raises ValueError."""
|
||||
mock_config = MagicMock()
|
||||
mock_config.api_key = None
|
||||
mock_config.workspace = None
|
||||
mock_config.url_override = "https://opik.example.com"
|
||||
mock_opik_config.return_value = mock_config
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["healthcheck", "--check-permissions", "my-workspace"],
|
||||
)
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert isinstance(result.exception, ValueError)
|
||||
assert "API key is required" in str(result.exception)
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Tests for the ``opik configure`` command group."""
|
||||
|
||||
import pathlib
|
||||
from unittest import mock
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli import cli
|
||||
from opik.cli import configure as configure_cli
|
||||
|
||||
|
||||
def test_configure_status__prints_config_summary():
|
||||
runner = CliRunner()
|
||||
config = mock.Mock(
|
||||
config_file_exists=True,
|
||||
config_file_fullpath=pathlib.Path("/home/u/.opik.config"),
|
||||
url_override="https://dev.comet.com/opik/api/",
|
||||
workspace="my-ws",
|
||||
)
|
||||
with mock.patch.object(
|
||||
configure_cli.opik_config, "OpikConfig", return_value=config
|
||||
):
|
||||
result = runner.invoke(cli, ["configure", "status"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "Your Opik configuration" in result.output
|
||||
assert "https://dev.comet.com/opik/api/" in result.output
|
||||
assert "my-ws" in result.output
|
||||
|
||||
|
||||
def test_configure_status__not_configured__points_to_configure():
|
||||
runner = CliRunner()
|
||||
config = mock.Mock(
|
||||
config_file_exists=False,
|
||||
config_file_fullpath=pathlib.Path("/home/u/.opik.config"),
|
||||
)
|
||||
with mock.patch.object(
|
||||
configure_cli.opik_config, "OpikConfig", return_value=config
|
||||
):
|
||||
result = runner.invoke(cli, ["configure", "status"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "not found" in result.output
|
||||
assert "opik configure" in result.output
|
||||
|
||||
|
||||
def test_configure_no_subcommand__runs_configurator():
|
||||
runner = CliRunner()
|
||||
with mock.patch.object(configure_cli, "run_interactive_configure") as spy:
|
||||
result = runner.invoke(cli, ["configure", "--use-local"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
spy.assert_called_once()
|
||||
assert spy.call_args.kwargs["use_local"] is True
|
||||
@@ -0,0 +1,230 @@
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import httpx
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli.main import cli
|
||||
from opik.cli.local_runner.pairing import PairingResult, RunnerType
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
|
||||
class TestConnect:
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__with_project__calls_pairing_and_supervisor(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-abc",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["connect", "--project", "my-proj"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_run_pairing.assert_called_once()
|
||||
call_kwargs = mock_run_pairing.call_args[1]
|
||||
assert call_kwargs["project_name"] == "my-proj"
|
||||
assert call_kwargs["runner_type"] == RunnerType.CONNECT
|
||||
|
||||
mock_launch.assert_called_once()
|
||||
launch_kwargs = mock_launch.call_args[1]
|
||||
assert launch_kwargs["command"] is None
|
||||
|
||||
def test_connect__no_project__shows_error(self):
|
||||
runner = CliRunner()
|
||||
# Any arg flips the `_EndpointGroup`-style fallback into the hidden _run
|
||||
# subcommand, where `--project` is enforced as required by Click. Bare
|
||||
# `opik connect` prints group help instead; covered by the next test.
|
||||
result = runner.invoke(cli, ["connect", "--name", "test-runner"])
|
||||
assert result.exit_code == 2
|
||||
assert "--project" in result.output
|
||||
|
||||
def test_connect__no_args__shows_help(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["connect"])
|
||||
assert result.exit_code == 0
|
||||
# Help body lists `stop` and the docstring mentions `--project`.
|
||||
assert "stop" in result.output
|
||||
assert "--project" in result.output
|
||||
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__network_failure__shows_clean_error(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.side_effect = httpx.ConnectError("Connection refused")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["connect", "--project", "my-proj"])
|
||||
assert result.exit_code != 0
|
||||
assert "Could not connect to Opik backend" in result.output
|
||||
assert "https://api.test/" in result.output
|
||||
assert "Run: opik configure" in result.output
|
||||
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__api_error__shows_error_body(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.side_effect = ApiError(status_code=409, body="conflict")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["connect", "--project", "my-proj"])
|
||||
assert result.exit_code != 0
|
||||
assert "conflict" in result.output
|
||||
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__tui_stopped_on_pairing_failure(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.side_effect = ApiError(status_code=500, body="boom")
|
||||
|
||||
runner = CliRunner()
|
||||
runner.invoke(cli, ["connect", "--project", "my-proj"])
|
||||
|
||||
tui_instance = mock_tui_cls.return_value
|
||||
tui_instance.stop.assert_called_once()
|
||||
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__workspace_and_api_key_passed__forwarded_to_opik_constructor(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
client.config.config_file_exists = True
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-abc",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"connect",
|
||||
"--project",
|
||||
"my-proj",
|
||||
"--workspace",
|
||||
"my-ws",
|
||||
"--api-key",
|
||||
"my-key",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_opik_cls.assert_called_once_with(
|
||||
project_name="my-proj",
|
||||
api_key="my-key",
|
||||
workspace="my-ws",
|
||||
_show_misconfiguration_message=False,
|
||||
)
|
||||
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__local_api_key_overrides_global(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
client.config.config_file_exists = True
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-abc",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--api-key",
|
||||
"global-key",
|
||||
"connect",
|
||||
"--project",
|
||||
"my-proj",
|
||||
"--api-key",
|
||||
"local-key",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_opik_cls.assert_called_once_with(
|
||||
project_name="my-proj",
|
||||
api_key="local-key",
|
||||
workspace=None,
|
||||
_show_misconfiguration_message=False,
|
||||
)
|
||||
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_connect__no_local_api_key__falls_back_to_global(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
client.config.config_file_exists = True
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-abc",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--api-key", "global-key", "connect", "--project", "my-proj"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_opik_cls.assert_called_once_with(
|
||||
project_name="my-proj",
|
||||
api_key="global-key",
|
||||
workspace=None,
|
||||
_show_misconfiguration_message=False,
|
||||
)
|
||||
@@ -0,0 +1,240 @@
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import httpx
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli.main import cli
|
||||
from opik.cli.local_runner.pairing import PairingResult, RunnerType
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
|
||||
class TestEndpoint:
|
||||
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_endpoint__with_command__calls_pairing_and_supervisor(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-xyz",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli, ["endpoint", "--project", "my-proj", "--", "echo", "hello"]
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_run_pairing.assert_called_once()
|
||||
call_kwargs = mock_run_pairing.call_args[1]
|
||||
assert call_kwargs["project_name"] == "my-proj"
|
||||
assert call_kwargs["runner_type"] == RunnerType.ENDPOINT
|
||||
|
||||
mock_launch.assert_called_once()
|
||||
launch_kwargs = mock_launch.call_args[1]
|
||||
assert launch_kwargs["command"] == ["echo", "hello"]
|
||||
|
||||
def test_endpoint__no_project__shows_error(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["endpoint", "--", "echo", "hello"])
|
||||
assert result.exit_code == 2
|
||||
assert "--project" in result.output
|
||||
|
||||
def test_endpoint__no_command__shows_error(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["endpoint", "--project", "my-proj"])
|
||||
assert result.exit_code == 2
|
||||
|
||||
def test_endpoint__nonexistent_binary__shows_error(self):
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["endpoint", "--project", "my-proj", "--", "nonexistent-binary-xyz-12345"],
|
||||
)
|
||||
assert result.exit_code == 2
|
||||
assert "not found" in result.output.lower()
|
||||
|
||||
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_endpoint__network_failure__shows_clean_error(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.side_effect = httpx.ConnectError("Connection refused")
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli, ["endpoint", "--project", "my-proj", "--", "echo", "hello"]
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "Could not connect to Opik backend" in result.output
|
||||
assert "https://api.test/" in result.output
|
||||
assert "Run: opik configure" in result.output
|
||||
|
||||
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_endpoint__tui_stopped_on_pairing_failure(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.side_effect = ApiError(status_code=500, body="boom")
|
||||
|
||||
runner = CliRunner()
|
||||
runner.invoke(cli, ["endpoint", "--project", "my-proj", "--", "echo", "hello"])
|
||||
|
||||
tui_instance = mock_tui_cls.return_value
|
||||
tui_instance.stop.assert_called_once()
|
||||
|
||||
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_endpoint__workspace_and_api_key_passed__forwarded_to_opik_constructor(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
client.config.config_file_exists = True
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-xyz",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"endpoint",
|
||||
"--project",
|
||||
"my-proj",
|
||||
"--workspace",
|
||||
"my-ws",
|
||||
"--api-key",
|
||||
"my-key",
|
||||
"--",
|
||||
"echo",
|
||||
"hello",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_opik_cls.assert_called_once_with(
|
||||
project_name="my-proj",
|
||||
api_key="my-key",
|
||||
workspace="my-ws",
|
||||
_show_misconfiguration_message=False,
|
||||
)
|
||||
|
||||
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_endpoint__local_api_key_overrides_global(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
client.config.config_file_exists = True
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-xyz",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--api-key",
|
||||
"global-key",
|
||||
"endpoint",
|
||||
"--project",
|
||||
"my-proj",
|
||||
"--api-key",
|
||||
"local-key",
|
||||
"--",
|
||||
"echo",
|
||||
"hello",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_opik_cls.assert_called_once_with(
|
||||
project_name="my-proj",
|
||||
api_key="local-key",
|
||||
workspace=None,
|
||||
_show_misconfiguration_message=False,
|
||||
)
|
||||
|
||||
@patch("opik.runner.snapshot.has_entrypoint", return_value=True)
|
||||
@patch("opik.cli.local_runner._run.RunnerTUI")
|
||||
@patch("opik.cli.local_runner.pairing.launch_supervisor")
|
||||
@patch("opik.cli.local_runner.pairing.run_pairing")
|
||||
@patch("opik.cli.local_runner._run.Opik")
|
||||
def test_endpoint__no_local_api_key__falls_back_to_global(
|
||||
self, mock_opik_cls, mock_run_pairing, mock_launch, mock_tui_cls, _mock_ep
|
||||
):
|
||||
client = MagicMock()
|
||||
client.config.url_override = "https://api.test/"
|
||||
client.config.config_file_exists = True
|
||||
mock_opik_cls.return_value = client
|
||||
|
||||
mock_run_pairing.return_value = PairingResult(
|
||||
runner_id="r-xyz",
|
||||
project_name="my-proj",
|
||||
project_id="p-123",
|
||||
bridge_key=b"\x00" * 32,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--api-key",
|
||||
"global-key",
|
||||
"endpoint",
|
||||
"--project",
|
||||
"my-proj",
|
||||
"--",
|
||||
"echo",
|
||||
"hello",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
|
||||
mock_opik_cls.assert_called_once_with(
|
||||
project_name="my-proj",
|
||||
api_key="global-key",
|
||||
workspace=None,
|
||||
_show_misconfiguration_message=False,
|
||||
)
|
||||
@@ -0,0 +1,282 @@
|
||||
"""Unit tests for export error-handling fixes.
|
||||
|
||||
Covers three HIGH-priority fixes to prevent silent data loss during export:
|
||||
|
||||
HIGH-1 experiment.export_traces_by_ids: when get_trace_content raises 429,
|
||||
the internal _fetch_trace_data retry decorator must kick in so the
|
||||
trace is eventually exported, not silently dropped. Only genuine 404s
|
||||
should be skipped without retrying.
|
||||
|
||||
HIGH-2 experiment.export_experiment_by_id: must re-raise on error so that
|
||||
the all.py caller can set had_errors=True (previously it swallowed
|
||||
all exceptions and returned a zero-stats tuple).
|
||||
|
||||
HIGH-3 all._fetch_experiments_page_raw: 429 must trigger a retry via the
|
||||
_export_rest_retry decorator instead of silently returning ([], 0)
|
||||
and losing an entire page of experiments.
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import httpx
|
||||
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
from opik import exceptions as opik_exceptions
|
||||
|
||||
_EXPERIMENT_MODULE = "opik.cli.exports.experiment"
|
||||
_ALL_MODULE = "opik.cli.exports.all"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HIGH-1: export_traces_by_ids retries 429 rather than dropping the trace
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportTracesByIdsRetry:
|
||||
def _make_mock_trace(self, trace_id: str) -> MagicMock:
|
||||
t = MagicMock()
|
||||
t.model_dump.return_value = {"id": trace_id}
|
||||
return t
|
||||
|
||||
def test_export_traces_by_ids__429_is_retried_not_dropped(self, tmp_path):
|
||||
"""A 429 ApiError must trigger the retry decorator so the trace is
|
||||
exported, not silently dropped."""
|
||||
from opik.cli.exports.experiment import export_traces_by_ids
|
||||
|
||||
call_count = 0
|
||||
|
||||
def get_trace_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count == 1:
|
||||
raise ApiError(status_code=429, headers={})
|
||||
return self._make_mock_trace("t1")
|
||||
|
||||
client = MagicMock()
|
||||
client.get_trace_content.side_effect = get_trace_side_effect
|
||||
client.search_spans.return_value = []
|
||||
|
||||
with patch(f"{_EXPERIMENT_MODULE}._fetch_trace_data.retry.sleep"):
|
||||
exported, skipped = export_traces_by_ids(
|
||||
client, ["t1"], tmp_path, "proj", None, "json", False, False
|
||||
)
|
||||
|
||||
assert exported == 1, "429 must not silently drop the trace"
|
||||
assert call_count == 2, "429 must trigger a retry"
|
||||
|
||||
def test_export_traces_by_ids__rate_limited_exception_is_retried(self, tmp_path):
|
||||
"""OpikCloudRequestsRateLimited must also be retried."""
|
||||
from opik.cli.exports.experiment import export_traces_by_ids
|
||||
|
||||
call_count = 0
|
||||
|
||||
def get_trace_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count == 1:
|
||||
raise opik_exceptions.OpikCloudRequestsRateLimited(
|
||||
headers={}, retry_after=1.0
|
||||
)
|
||||
return self._make_mock_trace("t2")
|
||||
|
||||
client = MagicMock()
|
||||
client.get_trace_content.side_effect = get_trace_side_effect
|
||||
client.search_spans.return_value = []
|
||||
|
||||
with patch(f"{_EXPERIMENT_MODULE}._fetch_trace_data.retry.sleep"):
|
||||
exported, skipped = export_traces_by_ids(
|
||||
client, ["t2"], tmp_path, "proj", None, "json", False, False
|
||||
)
|
||||
|
||||
assert exported == 1, "rate-limit exception must not silently drop the trace"
|
||||
assert call_count == 2
|
||||
|
||||
def test_export_traces_by_ids__404_is_skipped_not_retried(self, tmp_path):
|
||||
"""A genuine 404 (trace not found) should be skipped cleanly without retrying."""
|
||||
from opik.cli.exports.experiment import export_traces_by_ids
|
||||
|
||||
call_count = 0
|
||||
|
||||
def get_trace_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
raise ApiError(status_code=404, headers={})
|
||||
|
||||
client = MagicMock()
|
||||
client.get_trace_content.side_effect = get_trace_side_effect
|
||||
|
||||
exported, skipped = export_traces_by_ids(
|
||||
client, ["missing-id"], tmp_path, "proj", None, "json", False, False
|
||||
)
|
||||
|
||||
assert exported == 0, "404 trace must not be exported"
|
||||
assert call_count == 1, "404 must not trigger a retry"
|
||||
|
||||
def test_export_traces_by_ids__success__exports_trace(self, tmp_path):
|
||||
"""Happy path: trace is written to disk and counted as exported."""
|
||||
from opik.cli.exports.experiment import export_traces_by_ids
|
||||
|
||||
client = MagicMock()
|
||||
client.get_trace_content.return_value = self._make_mock_trace("t3")
|
||||
client.search_spans.return_value = []
|
||||
|
||||
exported, skipped = export_traces_by_ids(
|
||||
client, ["t3"], tmp_path, "proj", None, "json", False, False
|
||||
)
|
||||
|
||||
assert exported == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HIGH-2: export_experiment_by_id propagates errors instead of swallowing them
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportExperimentByIdPropagatesErrors:
|
||||
def test_export_experiment_by_id__api_error_propagates(self, tmp_path):
|
||||
"""An unexpected error must propagate so the caller can set had_errors=True."""
|
||||
from opik.cli.exports.experiment import export_experiment_by_id
|
||||
|
||||
client = MagicMock()
|
||||
# Simulate a non-retriable API error (e.g. 403 Forbidden)
|
||||
client.get_experiment_by_id.side_effect = ApiError(status_code=403, headers={})
|
||||
|
||||
with pytest.raises(ApiError):
|
||||
export_experiment_by_id(
|
||||
client=client,
|
||||
output_dir=tmp_path,
|
||||
project_name="proj",
|
||||
experiment_id="exp-1",
|
||||
max_traces=None,
|
||||
force=True,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
def test_export_all_experiments__experiment_failure_sets_had_errors(self, tmp_path):
|
||||
"""When export_experiment_by_id raises, _export_all_experiments must set
|
||||
had_errors=True rather than silently counting it as skipped."""
|
||||
from opik.cli.exports.all import _export_all_experiments
|
||||
|
||||
client = MagicMock()
|
||||
experiments_dir = tmp_path / "experiments"
|
||||
experiments_dir.mkdir()
|
||||
|
||||
# Two experiments: first succeeds, second raises.
|
||||
exp1 = MagicMock()
|
||||
exp1.id = "e1"
|
||||
exp1.name = "exp-one"
|
||||
exp2 = MagicMock()
|
||||
exp2.id = "e2"
|
||||
exp2.name = "exp-two"
|
||||
|
||||
def export_side_effect(
|
||||
client, output_dir, project_name, exp_id, *args, **kwargs
|
||||
):
|
||||
if exp_id == "e2":
|
||||
raise RuntimeError("transient failure")
|
||||
m = MagicMock()
|
||||
m.get_all_trace_ids.return_value = []
|
||||
return {}, 1, m
|
||||
|
||||
with (
|
||||
patch(
|
||||
f"{_ALL_MODULE}._paginate_experiments",
|
||||
return_value=[exp1, exp2],
|
||||
),
|
||||
patch(
|
||||
f"{_ALL_MODULE}.export_experiment_by_id",
|
||||
side_effect=export_side_effect,
|
||||
),
|
||||
patch(f"{_ALL_MODULE}.export_collected_trace_ids", return_value=(0, 0)),
|
||||
):
|
||||
_, _, _, _, had_errors = _export_all_experiments(
|
||||
client=client,
|
||||
project_dir=tmp_path,
|
||||
project_name="proj",
|
||||
project_id=None,
|
||||
experiments_dir=experiments_dir,
|
||||
max_results=None,
|
||||
force=False,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert had_errors is True, "a failed experiment must set had_errors=True"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HIGH-3: _fetch_experiments_page_raw retries 429 instead of returning [], 0
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFetchExperimentsPageRawRetry:
|
||||
def _make_httpx_response(self, status_code: int, headers=None, json_body=None):
|
||||
"""Build a minimal httpx.Response for raise_for_status() tests."""
|
||||
headers = headers or {}
|
||||
body = json_body or {"content": [], "total": 0}
|
||||
import json as _json
|
||||
|
||||
# httpx.Response.raise_for_status() requires a request to be attached.
|
||||
request = httpx.Request("GET", "http://test/v1/private/experiments")
|
||||
response = httpx.Response(
|
||||
status_code=status_code,
|
||||
headers=headers,
|
||||
content=_json.dumps(body).encode(),
|
||||
request=request,
|
||||
)
|
||||
return response
|
||||
|
||||
def test_fetch_experiments_page_raw__429_raises_rate_limited_not_empty_list(
|
||||
self,
|
||||
):
|
||||
"""A 429 response must raise OpikCloudRequestsRateLimited (for the retry
|
||||
decorator), NOT silently return ([], 0)."""
|
||||
from opik.cli.exports.all import _fetch_experiments_page_raw
|
||||
|
||||
client = MagicMock()
|
||||
httpx_mock = MagicMock()
|
||||
client.rest_client.experiments._raw_client._client_wrapper.httpx_client = (
|
||||
httpx_mock
|
||||
)
|
||||
httpx_mock.request.return_value = self._make_httpx_response(
|
||||
429, headers={"Retry-After": "5"}
|
||||
)
|
||||
|
||||
with pytest.raises(opik_exceptions.OpikCloudRequestsRateLimited):
|
||||
_fetch_experiments_page_raw.__wrapped__(client, 1, None)
|
||||
|
||||
def test_fetch_experiments_page_raw__429_is_retried_by_decorator(self):
|
||||
"""When the underlying call raises 429, the decorator retries rather
|
||||
than letting a silent ([], 0) escape to the caller."""
|
||||
from opik.cli.exports.all import _fetch_experiments_page_raw
|
||||
|
||||
call_count = 0
|
||||
|
||||
def request_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count == 1:
|
||||
return self._make_httpx_response(429, headers={"Retry-After": "1"})
|
||||
return self._make_httpx_response(
|
||||
200,
|
||||
json_body={
|
||||
"content": [{"id": "exp-1", "name": "my-exp"}],
|
||||
"total": 1,
|
||||
},
|
||||
)
|
||||
|
||||
client = MagicMock()
|
||||
httpx_mock = MagicMock()
|
||||
client.rest_client.experiments._raw_client._client_wrapper.httpx_client = (
|
||||
httpx_mock
|
||||
)
|
||||
httpx_mock.request.side_effect = request_side_effect
|
||||
|
||||
with patch(f"{_ALL_MODULE}._fetch_experiments_page_raw.retry.sleep"):
|
||||
items, total = _fetch_experiments_page_raw(client, 1, None)
|
||||
|
||||
assert call_count == 2, "429 must trigger a retry"
|
||||
assert len(items) == 1, "successful retry must return the page contents"
|
||||
assert total == 1
|
||||
@@ -0,0 +1,788 @@
|
||||
"""Tests for rate-limiting resilience in project export.
|
||||
|
||||
Covers the changes introduced to handle server-side 429 throttling:
|
||||
|
||||
Retry wait behaviour (via export_traces public API)
|
||||
- 429 with Retry-After header → sleep honours header value
|
||||
- 429 with Retry-After exceeding cap → sleep falls back to 30 s
|
||||
- 429 with no Retry-After → sleep is 30 s
|
||||
- non-429 transient error → sleep is in exponential backoff range [2, 60] s
|
||||
|
||||
export_traces (page-fetch failures)
|
||||
- 429 on page N → page is skipped, remaining pages still fetched, had_errors=True
|
||||
- all pages succeed → had_errors=False, all traces exported
|
||||
- middle page failure → subsequent pages still exported
|
||||
|
||||
export_traces (span-fetch failures)
|
||||
- bulk span fetch raises repeatedly → traces exported with empty spans, had_errors=True
|
||||
|
||||
export_traces (inter-page delay)
|
||||
- time.sleep NOT called between successful full pages (inter-page sleep removed)
|
||||
- time.sleep not called after partial (terminal) page
|
||||
|
||||
export_single_project (had_errors propagation)
|
||||
- returns 4-tuple; had_errors=False on clean run, True when errors occurred
|
||||
- first element is 0 when no traces exported or skipped, 1 otherwise
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
_MODULE = "opik.cli.exports.project"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_mock_trace(trace_id: str) -> MagicMock:
|
||||
t = MagicMock()
|
||||
t.id = trace_id
|
||||
t.name = f"trace-{trace_id}"
|
||||
t.model_dump.return_value = {
|
||||
"id": trace_id,
|
||||
"name": f"trace-{trace_id}",
|
||||
"start_time": None,
|
||||
"end_time": None,
|
||||
"input": {},
|
||||
"output": {},
|
||||
"metadata": {},
|
||||
"tags": [],
|
||||
"feedback_scores": [],
|
||||
"error_info": None,
|
||||
"thread_id": None,
|
||||
"created_at": None,
|
||||
"created_by": None,
|
||||
"last_updated_at": None,
|
||||
"last_updated_by": None,
|
||||
"visibility_mode": None,
|
||||
"ttft": None,
|
||||
"project_name": "test-project",
|
||||
}
|
||||
return t
|
||||
|
||||
|
||||
def _make_page(traces, total=None):
|
||||
page = MagicMock()
|
||||
page.content = traces
|
||||
page.total = total if total is not None else len(traces)
|
||||
return page
|
||||
|
||||
|
||||
def _make_full_page(n=500, offset=0):
|
||||
"""Return a page with exactly *n* mock traces (matches the internal page_size=500).
|
||||
|
||||
Use *offset* to generate distinct trace IDs across pages.
|
||||
"""
|
||||
return _make_page([_make_mock_trace(f"bulk-{offset + i}") for i in range(n)])
|
||||
|
||||
|
||||
def _uuid7(n: int) -> str:
|
||||
"""Deterministic, valid UUIDv7 string for trace index *n*.
|
||||
|
||||
Version nibble is 7 and the variant nibble is RFC-4122 (8), so
|
||||
``uuid.UUID(...).version == 7`` — this is what makes the exporter build a
|
||||
bounded ``trace_id`` range filter instead of falling back to a full scan.
|
||||
"""
|
||||
high = f"{0x0190A1B2C3D4 + n:012x}"
|
||||
return f"{high[:8]}-{high[8:12]}-7000-8000-{n:012x}"
|
||||
|
||||
|
||||
def _make_full_page_uuid7(n=500, offset=0):
|
||||
"""Like ``_make_full_page`` but with UUIDv7 trace ids (bounded-scan path)."""
|
||||
return _make_page([_make_mock_trace(_uuid7(offset + i)) for i in range(n)])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Retry wait behaviour — tested via export_traces (public API)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportTracesRetryWaitBehaviour:
|
||||
"""Verify that the export-specific retry decorator applies the correct wait
|
||||
durations when the underlying page-fetch raises errors. Tests go through
|
||||
the public export_traces entrypoint so the actual retry decorator fires;
|
||||
time.sleep is patched to capture what tenacity passes to it.
|
||||
"""
|
||||
|
||||
def _run_with_first_call_raising(self, exc, tmp_path):
|
||||
"""Run export_traces where the first page fetch raises *exc* then succeeds."""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
trace = _make_mock_trace("t1")
|
||||
call_count = 0
|
||||
|
||||
def get_traces_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count == 1:
|
||||
raise exc
|
||||
return _make_page([trace])
|
||||
|
||||
mock_client = MagicMock()
|
||||
mock_client.rest_client.traces.get_traces_by_project.side_effect = (
|
||||
get_traces_side_effect
|
||||
)
|
||||
|
||||
with patch(f"{_MODULE}._fetch_traces_page.retry.sleep") as mock_sleep:
|
||||
export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=tmp_path,
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
return [c.args[0] for c in mock_sleep.call_args_list]
|
||||
|
||||
def test_export_traces__page_fetch_429_with_retry_after_header__sleep_honours_header(
|
||||
self, tmp_path
|
||||
):
|
||||
exc = ApiError(status_code=429, headers={"retry-after": "45"})
|
||||
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
|
||||
# Jitter adds 0–5 s on top of the header value.
|
||||
assert any(45.0 <= s <= 50.0 for s in sleep_calls)
|
||||
|
||||
def test_export_traces__page_fetch_429_retry_after_exceeds_cap__sleep_clamped_to_max(
|
||||
self, tmp_path
|
||||
):
|
||||
# 200 s > _EXPORT_MAX_RETRY_AFTER_SECONDS (120 s) → clamped to 120 s
|
||||
exc = ApiError(status_code=429, headers={"retry-after": "200"})
|
||||
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
|
||||
# Jitter adds 0–5 s on top of the 120 s cap.
|
||||
assert any(120.0 <= s <= 125.0 for s in sleep_calls)
|
||||
|
||||
def test_export_traces__page_fetch_429_retry_after_http_date__sleep_honours_header(
|
||||
self, tmp_path
|
||||
):
|
||||
import email.utils
|
||||
import time
|
||||
|
||||
# Build an HTTP-date Retry-After 60 seconds in the future.
|
||||
future = email.utils.formatdate(timeval=time.time() + 60, usegmt=True)
|
||||
exc = ApiError(status_code=429, headers={"retry-after": future})
|
||||
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
|
||||
# Should sleep approximately 60 s (allow ±2 s for execution) plus 0–5 s jitter.
|
||||
assert any(58.0 <= s <= 67.0 for s in sleep_calls)
|
||||
|
||||
def test_export_traces__page_fetch_429_no_retry_after_header__sleep_is_30s(
|
||||
self, tmp_path
|
||||
):
|
||||
exc = ApiError(status_code=429, headers={})
|
||||
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
|
||||
# Jitter adds 0–5 s on top of the 30 s fallback.
|
||||
assert any(30.0 <= s <= 35.0 for s in sleep_calls)
|
||||
|
||||
def test_export_traces__page_fetch_non_429_transient_error__sleep_is_exponential_backoff(
|
||||
self, tmp_path
|
||||
):
|
||||
exc = ApiError(status_code=503)
|
||||
sleep_calls = self._run_with_first_call_raising(exc, tmp_path)
|
||||
# Backoff is exponential starting at 2 s, capped at 60 s.
|
||||
assert any(2.0 <= s <= 60.0 for s in sleep_calls)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# export_traces — page-fetch failures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportTracesPageFetchFailures:
|
||||
def test_export_traces__first_page_429__skips_page_and_continues(self):
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
trace_p2 = _make_mock_trace("t-p2")
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
mock_fetch.side_effect = [
|
||||
ApiError(status_code=429, body="rate limited"),
|
||||
_make_page([trace_p2]),
|
||||
_make_page([]),
|
||||
]
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
assert had_errors is True
|
||||
assert exported == 1
|
||||
assert skipped == 0
|
||||
|
||||
def test_export_traces__all_pages_succeed__happyflow(self):
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
traces = [_make_mock_trace("t1"), _make_mock_trace("t2")]
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
mock_fetch.side_effect = [
|
||||
_make_page(traces),
|
||||
_make_page([]),
|
||||
]
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
assert had_errors is False
|
||||
assert exported == 2
|
||||
|
||||
def test_export_traces__middle_page_fails__subsequent_pages_still_exported(self):
|
||||
"""A failed page in the middle must not prevent later pages from being fetched."""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
# Use page_size=2 so pages with 2 traces are "full" (2 == page_size).
|
||||
# Keeps file-write count tiny (4 files) while still exercising the
|
||||
# pagination loop — the loop only breaks early on a *short* page.
|
||||
page1 = _make_full_page(2, offset=0)
|
||||
page3 = _make_full_page(2, offset=2)
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
mock_fetch.side_effect = [
|
||||
page1, # page 1: OK, 2 traces
|
||||
ApiError(status_code=429), # page 2: rate limited
|
||||
page3, # page 3: OK, 2 traces
|
||||
_make_page([]), # page 4: end
|
||||
]
|
||||
with patch(f"{_MODULE}._fetch_spans_page", return_value=_make_page([])):
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
page_size=2,
|
||||
)
|
||||
|
||||
assert had_errors is True
|
||||
assert exported == 4 # page 1 and page 3 traces both exported
|
||||
|
||||
def test_export_traces__consecutive_failures_reach_cap__export_aborted(self):
|
||||
"""After MAX_CONSECUTIVE_PAGE_FAILURES pages in a row fail, the loop aborts."""
|
||||
from opik.cli.exports.project import (
|
||||
export_traces,
|
||||
MAX_CONSECUTIVE_PAGE_FAILURES,
|
||||
)
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
# Every page raises a transient error
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
mock_fetch.side_effect = ApiError(status_code=503)
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
assert had_errors is True
|
||||
assert exported == 0
|
||||
# Should have stopped after MAX_CONSECUTIVE_PAGE_FAILURES attempts
|
||||
assert mock_fetch.call_count == MAX_CONSECUTIVE_PAGE_FAILURES
|
||||
|
||||
def test_export_traces__permanent_api_error__raises_instead_of_skipping(self):
|
||||
"""A 400 Bad Request must not be silently skipped — it should propagate."""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
mock_fetch.side_effect = ApiError(status_code=400, body="bad request")
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
with pytest.raises(ApiError) as exc_info:
|
||||
export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
assert exc_info.value.status_code == 400
|
||||
# Should have raised immediately, not retried
|
||||
assert mock_fetch.call_count == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# export_traces — span-fetch failures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportTracesSpanFetchFailures:
|
||||
def test_export_traces__bulk_span_fetch_fails__traces_exported_with_empty_spans(
|
||||
self,
|
||||
):
|
||||
"""When the bulk span page fetch keeps failing, traces are still written
|
||||
(with empty span lists) and had_errors is set to True."""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
t1 = _make_mock_trace("t1")
|
||||
t2 = _make_mock_trace("t2")
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_trace_fetch:
|
||||
mock_trace_fetch.side_effect = [
|
||||
_make_page([t1, t2]),
|
||||
_make_page([]),
|
||||
]
|
||||
with patch(f"{_MODULE}._fetch_spans_page") as mock_span_fetch:
|
||||
mock_span_fetch.side_effect = ApiError(status_code=503)
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
assert had_errors is True
|
||||
assert exported == 2 # both trace files written, just without spans
|
||||
assert skipped == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# export_traces — bounded-chunk (memory) behaviour
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportTracesChunking:
|
||||
def test_export_traces__large_project_flushes_in_bounded_chunks(self):
|
||||
"""A multi-page export writes traces incrementally, one bounded chunk at
|
||||
a time, instead of buffering every trace and span before the first write.
|
||||
Incremental flushing is what keeps peak memory from scaling with the
|
||||
whole project (the previous all-at-once design OOM'd on large projects).
|
||||
|
||||
The export is driven purely through the public ``export_traces`` API,
|
||||
mocking only the public REST client boundary
|
||||
(``traces.get_traces_by_project`` / ``spans.get_spans_by_project``) — no
|
||||
private helpers are patched. Correctness is asserted on the returned
|
||||
tuple and the final ``trace_*.json`` artifacts. The bounded-chunk
|
||||
property is asserted through robust invariants observed at the public
|
||||
span endpoint (files already on disk when each span fetch fires), not a
|
||||
brittle exact-call-order snapshot: the assertions hold regardless of the
|
||||
exact chunk boundary the implementation picks.
|
||||
"""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
# UUIDv7 ids so each chunk builds a bounded trace_id range filter and
|
||||
# scans spans per chunk (the common, memory-bounded path).
|
||||
page1 = _make_full_page_uuid7(2, offset=0)
|
||||
page2 = _make_full_page_uuid7(2, offset=2)
|
||||
page3 = _make_full_page_uuid7(2, offset=4)
|
||||
mock_client = MagicMock()
|
||||
expected_ids = [_uuid7(i) for i in range(6)]
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
project_dir = Path(tmp)
|
||||
files_on_disk_at_each_span_fetch = []
|
||||
|
||||
def span_fetch_side_effect(*args, **kwargs):
|
||||
# How many trace files already exist when a span fetch begins.
|
||||
# A growing, sub-total count proves earlier chunks were flushed
|
||||
# to disk before later chunks are processed (incremental), rather
|
||||
# than one all-at-once sweep after buffering everything.
|
||||
files_on_disk_at_each_span_fetch.append(
|
||||
len(list(project_dir.glob("trace_*.json")))
|
||||
)
|
||||
return _make_page([])
|
||||
|
||||
mock_client.rest_client.traces.get_traces_by_project.side_effect = [
|
||||
page1,
|
||||
page2,
|
||||
page3,
|
||||
_make_page([]),
|
||||
]
|
||||
mock_client.rest_client.spans.get_spans_by_project.side_effect = (
|
||||
span_fetch_side_effect
|
||||
)
|
||||
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=project_dir,
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
page_size=2,
|
||||
)
|
||||
|
||||
# Public result contract.
|
||||
assert (exported, skipped, had_errors) == (6, 0, False)
|
||||
|
||||
# Final artifacts: exactly the six expected trace files were written.
|
||||
written_ids = sorted(
|
||||
p.name[len("trace_") : -len(".json")]
|
||||
for p in project_dir.glob("trace_*.json")
|
||||
)
|
||||
assert written_ids == sorted(expected_ids)
|
||||
|
||||
# Bounded-chunk invariants (robust to the exact chunk boundary):
|
||||
# - more than one span-fetch session → work is chunked, not one sweep;
|
||||
# - counts never decrease → files accumulate as chunks flush;
|
||||
# - a later fetch sees files already written → earlier chunks were
|
||||
# flushed to disk before the last chunk was processed, so only ~one
|
||||
# chunk's worth of traces is held in memory at a time.
|
||||
assert len(files_on_disk_at_each_span_fetch) > 1
|
||||
assert files_on_disk_at_each_span_fetch == sorted(
|
||||
files_on_disk_at_each_span_fetch
|
||||
)
|
||||
assert files_on_disk_at_each_span_fetch[0] == 0
|
||||
assert files_on_disk_at_each_span_fetch[-1] > 0
|
||||
|
||||
def test_export_traces__non_uuid7_ids__span_scan_bounded_per_chunk(self):
|
||||
"""When trace ids aren't UUIDv7 the span scan can't be bounded by a
|
||||
trace_id range, so the backend streams back every span in the project.
|
||||
|
||||
Rather than cache that whole-project span set — which would reintroduce
|
||||
the unbounded memory peak this chunked export exists to prevent — each
|
||||
chunk runs its own scan and keeps only its own traces' spans. Peak
|
||||
memory therefore stays bounded to one chunk regardless of project size;
|
||||
the trade-off is one unfiltered scan per chunk on this rare fallback.
|
||||
"""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
# Non-UUIDv7 ids ("bulk-N") force the unfiltered-scan fallback.
|
||||
page1 = _make_full_page(2, offset=0)
|
||||
page2 = _make_full_page(2, offset=2)
|
||||
page3 = _make_full_page(2, offset=4)
|
||||
mock_client = MagicMock()
|
||||
expected_ids = [f"bulk-{i}" for i in range(6)]
|
||||
|
||||
# Each per-chunk scan pages GET /spans starting at page 1; a fresh page-1
|
||||
# request marks a new scan session.
|
||||
scan_sessions = []
|
||||
|
||||
def span_fetch_side_effect(*args, **kwargs):
|
||||
page = kwargs.get("page")
|
||||
if page == 1:
|
||||
scan_sessions.append([])
|
||||
scan_sessions[-1].append(page)
|
||||
return _make_page([])
|
||||
|
||||
mock_client.rest_client.traces.get_traces_by_project.side_effect = [
|
||||
page1,
|
||||
page2,
|
||||
page3,
|
||||
_make_page([]),
|
||||
]
|
||||
mock_client.rest_client.spans.get_spans_by_project.side_effect = (
|
||||
span_fetch_side_effect
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
project_dir = Path(tmp)
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=project_dir,
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
page_size=2,
|
||||
)
|
||||
|
||||
assert (exported, skipped, had_errors) == (6, 0, False)
|
||||
written_ids = sorted(
|
||||
p.name[len("trace_") : -len(".json")]
|
||||
for p in project_dir.glob("trace_*.json")
|
||||
)
|
||||
assert written_ids == sorted(expected_ids)
|
||||
|
||||
# One unfiltered scan per flushed chunk (3) — not a single cached
|
||||
# whole-project scan — which is what keeps peak memory bounded to a chunk.
|
||||
assert len(scan_sessions) == 3
|
||||
|
||||
def test_export_traces__span_scan_error_isolated_to_its_chunk(self):
|
||||
"""A transient span-scan failure is confined to the chunk that hit it.
|
||||
|
||||
With per-chunk scanning there is no shared span cache to poison: the
|
||||
failed chunk is exported with partial spans and had_errors is set, while
|
||||
later chunks scan independently and export normally. The failure is
|
||||
injected at the public REST boundary (``spans.get_spans_by_project``),
|
||||
per the SDK testing guidelines, not at a private transport helper.
|
||||
"""
|
||||
from opik.cli.exports.project import export_traces
|
||||
|
||||
# Non-UUIDv7 ids force the unfiltered per-chunk scan fallback.
|
||||
page1 = _make_full_page(2, offset=0)
|
||||
page2 = _make_full_page(2, offset=2)
|
||||
page3 = _make_full_page(2, offset=4)
|
||||
mock_client = MagicMock()
|
||||
expected_ids = [f"bulk-{i}" for i in range(6)]
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
project_dir = Path(tmp)
|
||||
|
||||
def get_spans_side_effect(*args, **kwargs):
|
||||
# Fail every span page while the first chunk is being scanned
|
||||
# (nothing flushed yet); once earlier chunks have written their
|
||||
# files, succeed — proving the error didn't leak across chunks.
|
||||
if not any(project_dir.glob("trace_*.json")):
|
||||
raise ApiError(status_code=503)
|
||||
return _make_page([])
|
||||
|
||||
mock_client.rest_client.traces.get_traces_by_project.side_effect = [
|
||||
page1,
|
||||
page2,
|
||||
page3,
|
||||
_make_page([]),
|
||||
]
|
||||
mock_client.rest_client.spans.get_spans_by_project.side_effect = (
|
||||
get_spans_side_effect
|
||||
)
|
||||
|
||||
# Neutralise the retry decorator's sleep (public boundary triggers
|
||||
# the real 8-attempt retry) and the inter-page delay.
|
||||
with patch(f"{_MODULE}._fetch_spans_page.retry.sleep"):
|
||||
with patch(f"{_MODULE}.time.sleep"):
|
||||
exported, skipped, had_errors = export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=project_dir,
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
page_size=2,
|
||||
)
|
||||
|
||||
# The failed chunk still wrote its traces (with partial spans) and the
|
||||
# failure is flagged; later chunks were unaffected, so all 6 landed.
|
||||
assert (exported, skipped) == (6, 0)
|
||||
assert had_errors is True
|
||||
written_ids = sorted(
|
||||
p.name[len("trace_") : -len(".json")]
|
||||
for p in project_dir.glob("trace_*.json")
|
||||
)
|
||||
assert written_ids == sorted(expected_ids)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# export_traces — inter-page delay
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportTracesInterPageDelay:
|
||||
def test_export_traces__multiple_full_pages__sleep_not_called_on_success(self):
|
||||
"""sleep is NOT called between successful pages — the inter-page delay was
|
||||
removed because server-side 429s are already handled by the retry decorator."""
|
||||
from opik.cli.exports.project import export_traces, _PAGE_FETCH_DELAY_SECONDS
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
mock_fetch.side_effect = [
|
||||
_make_full_page(2, offset=0), # page 1: full, IDs 0–1
|
||||
_make_full_page(2, offset=2), # page 2: full, IDs 2–3
|
||||
_make_page([]), # page 3: empty — end
|
||||
]
|
||||
with patch(f"{_MODULE}._fetch_spans_page", return_value=_make_page([])):
|
||||
with patch(f"{_MODULE}.time.sleep") as mock_sleep:
|
||||
export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
page_size=2,
|
||||
)
|
||||
|
||||
from unittest.mock import call as mock_call
|
||||
|
||||
delay_calls = mock_sleep.call_args_list.count(
|
||||
mock_call(_PAGE_FETCH_DELAY_SECONDS)
|
||||
)
|
||||
assert delay_calls == 0
|
||||
|
||||
def test_export_traces__single_partial_page__sleep_not_called(self):
|
||||
"""A partial (< 100 trace) page means we're at the end — no sleep needed."""
|
||||
from opik.cli.exports.project import export_traces, _PAGE_FETCH_DELAY_SECONDS
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
with patch(f"{_MODULE}._fetch_traces_page") as mock_fetch:
|
||||
# Single partial page — loop breaks immediately on short-page check
|
||||
mock_fetch.side_effect = [
|
||||
_make_page([_make_mock_trace("t1")]),
|
||||
]
|
||||
with patch(f"{_MODULE}.time.sleep") as mock_sleep:
|
||||
export_traces(
|
||||
client=mock_client,
|
||||
project_name="proj",
|
||||
project_dir=Path(tmp),
|
||||
max_results=None,
|
||||
filter_string=None,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
from unittest.mock import call as mock_call
|
||||
|
||||
delay_calls = mock_sleep.call_args_list.count(
|
||||
mock_call(_PAGE_FETCH_DELAY_SECONDS)
|
||||
)
|
||||
assert delay_calls == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# export_single_project — had_errors and project_exported propagation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportSingleProjectReturnValues:
|
||||
def _make_project(self, name="test-proj"):
|
||||
p = MagicMock()
|
||||
p.name = name
|
||||
p.id = "proj-id-1"
|
||||
return p
|
||||
|
||||
def test_export_single_project__clean_run__had_errors_false(self):
|
||||
from opik.cli.exports.project import export_single_project
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
projects_dir = Path(tmp) / "projects"
|
||||
projects_dir.mkdir()
|
||||
|
||||
with (
|
||||
patch(f"{_MODULE}.export_traces") as mock_export,
|
||||
patch(f"{_MODULE}.time.sleep"),
|
||||
):
|
||||
mock_export.return_value = (3, 0, False)
|
||||
result = export_single_project(
|
||||
client=mock_client,
|
||||
project_name="test-proj",
|
||||
project_dir=projects_dir,
|
||||
filter_string=None,
|
||||
max_results=None,
|
||||
force=False,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert len(result) == 4
|
||||
proj_count, _exported, _skipped, had_errors = result
|
||||
assert had_errors is False
|
||||
assert proj_count == 1 # traces were exported
|
||||
|
||||
def test_export_single_project__run_with_errors__had_errors_true(self):
|
||||
from opik.cli.exports.project import export_single_project
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
projects_dir = Path(tmp) / "projects"
|
||||
projects_dir.mkdir()
|
||||
|
||||
with (
|
||||
patch(f"{_MODULE}.export_traces") as mock_export,
|
||||
patch(f"{_MODULE}.time.sleep"),
|
||||
):
|
||||
mock_export.return_value = (2, 1, True)
|
||||
result = export_single_project(
|
||||
client=mock_client,
|
||||
project_name="test-proj",
|
||||
project_dir=projects_dir,
|
||||
filter_string=None,
|
||||
max_results=None,
|
||||
force=False,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert len(result) == 4
|
||||
_proj_count, _exported, _skipped, had_errors = result
|
||||
assert had_errors is True
|
||||
|
||||
def test_export_single_project__no_traces_found__project_count_is_zero(self):
|
||||
"""When there are no traces at all, the project-exported flag must be 0."""
|
||||
from opik.cli.exports.project import export_single_project
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
projects_dir = Path(tmp) / "projects"
|
||||
projects_dir.mkdir()
|
||||
|
||||
with (
|
||||
patch(f"{_MODULE}.export_traces") as mock_export,
|
||||
patch(f"{_MODULE}.time.sleep"),
|
||||
):
|
||||
mock_export.return_value = (0, 0, False) # nothing exported or skipped
|
||||
result = export_single_project(
|
||||
client=mock_client,
|
||||
project_name="test-proj",
|
||||
project_dir=projects_dir,
|
||||
filter_string=None,
|
||||
max_results=None,
|
||||
force=False,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
proj_count, _exported, _skipped, _had_errors = result
|
||||
assert proj_count == 0
|
||||
|
||||
def test_export_single_project__exception_during_export__returns_had_errors_true(
|
||||
self,
|
||||
):
|
||||
from opik.cli.exports.project import export_single_project
|
||||
|
||||
mock_client = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
projects_dir = Path(tmp) / "projects"
|
||||
projects_dir.mkdir()
|
||||
|
||||
with patch(f"{_MODULE}.export_traces") as mock_export:
|
||||
mock_export.side_effect = RuntimeError("unexpected failure")
|
||||
result = export_single_project(
|
||||
client=mock_client,
|
||||
project_name="test-proj",
|
||||
project_dir=projects_dir,
|
||||
filter_string=None,
|
||||
max_results=None,
|
||||
force=False,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert result == (0, 0, 0, True)
|
||||
@@ -0,0 +1,900 @@
|
||||
"""Unit tests for experiment import functionality."""
|
||||
|
||||
import json
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
import pytest
|
||||
|
||||
# Mock the problematic imports before importing
|
||||
# Mock the prompt import that's causing issues
|
||||
sys.modules["opik.api_objects.prompt.prompt"] = MagicMock()
|
||||
|
||||
# Now we can import normally
|
||||
from opik.cli.imports.experiment import ( # noqa: E402
|
||||
ExperimentData,
|
||||
load_experiment_data,
|
||||
recreate_experiment,
|
||||
_import_traces_for_project,
|
||||
)
|
||||
from opik.cli.imports.utils import ( # noqa: E402
|
||||
translate_trace_id as utils_translate_trace_id,
|
||||
sort_spans_topologically,
|
||||
)
|
||||
|
||||
|
||||
class TestExperimentData:
|
||||
"""Test ExperimentData dataclass."""
|
||||
|
||||
def test_experiment_data_from_dict(self) -> None:
|
||||
"""Test creating ExperimentData from dictionary."""
|
||||
data = {
|
||||
"experiment": {
|
||||
"id": "exp-123",
|
||||
"name": "test-experiment",
|
||||
"dataset_name": "test-dataset",
|
||||
},
|
||||
"items": [
|
||||
{"id": "item-1", "trace_id": "trace-1"},
|
||||
{"id": "item-2", "trace_id": "trace-2"},
|
||||
],
|
||||
"downloaded_at": "2024-01-01T00:00:00",
|
||||
}
|
||||
|
||||
exp_data = ExperimentData.from_dict(data)
|
||||
|
||||
assert exp_data.experiment["id"] == "exp-123"
|
||||
assert exp_data.experiment["name"] == "test-experiment"
|
||||
assert len(exp_data.items) == 2
|
||||
assert exp_data.downloaded_at == "2024-01-01T00:00:00"
|
||||
|
||||
def test_experiment_data_from_dict_minimal(self) -> None:
|
||||
"""Test creating ExperimentData with minimal data."""
|
||||
data = {
|
||||
"experiment": {"id": "exp-123", "dataset_name": "test-dataset"},
|
||||
"items": [],
|
||||
}
|
||||
|
||||
exp_data = ExperimentData.from_dict(data)
|
||||
|
||||
assert exp_data.experiment["id"] == "exp-123"
|
||||
assert exp_data.items == []
|
||||
assert exp_data.downloaded_at is None
|
||||
|
||||
def test_load_experiment_data_from_file(self, tmp_path: Path) -> None:
|
||||
"""Test loading experiment data from JSON file."""
|
||||
experiment_file = tmp_path / "experiment_test.json"
|
||||
data = {
|
||||
"experiment": {
|
||||
"id": "exp-123",
|
||||
"name": "test-experiment",
|
||||
"dataset_name": "test-dataset",
|
||||
},
|
||||
"items": [{"id": "item-1"}],
|
||||
}
|
||||
|
||||
with open(experiment_file, "w") as f:
|
||||
json.dump(data, f)
|
||||
|
||||
exp_data = load_experiment_data(experiment_file)
|
||||
|
||||
assert isinstance(exp_data, ExperimentData)
|
||||
assert exp_data.experiment["id"] == "exp-123"
|
||||
assert len(exp_data.items) == 1
|
||||
|
||||
|
||||
class TestTranslateTraceId:
|
||||
"""Test translate_trace_id function."""
|
||||
|
||||
def test_translate_trace_id_found(self) -> None:
|
||||
"""Test translating trace ID when mapping exists."""
|
||||
trace_id_map = {"old-trace-1": "new-trace-1", "old-trace-2": "new-trace-2"}
|
||||
|
||||
result = utils_translate_trace_id("old-trace-1", trace_id_map)
|
||||
|
||||
assert result == "new-trace-1"
|
||||
|
||||
def test_translate_trace_id_not_found(self) -> None:
|
||||
"""Test translating trace ID when mapping doesn't exist."""
|
||||
trace_id_map = {"old-trace-1": "new-trace-1"}
|
||||
|
||||
result = utils_translate_trace_id("old-trace-2", trace_id_map)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_translate_trace_id_empty_map(self) -> None:
|
||||
"""Test translating trace ID with empty map."""
|
||||
trace_id_map: Dict[str, str] = {}
|
||||
|
||||
result = utils_translate_trace_id("old-trace-1", trace_id_map)
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_translate_trace_id_requires_dict(self) -> None:
|
||||
"""Test that translate_trace_id requires Dict, not Optional."""
|
||||
# This test verifies the type signature is correct
|
||||
# If someone tries to pass None, type checker should catch it
|
||||
trace_id_map: Dict[str, str] = {} # Required, not Optional
|
||||
|
||||
result = utils_translate_trace_id("trace-1", trace_id_map)
|
||||
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestRecreateExperiment:
|
||||
"""Test recreate_experiment function."""
|
||||
|
||||
@staticmethod
|
||||
def _extract_items_arg_from_call_args(call_args: Any) -> Any:
|
||||
"""Helper to extract the items argument from call_args.
|
||||
|
||||
Handles both positional and keyword arguments.
|
||||
"""
|
||||
if hasattr(call_args, "args") and call_args.args:
|
||||
return call_args.args[0]
|
||||
if hasattr(call_args, "kwargs"):
|
||||
if "items" in call_args.kwargs:
|
||||
return call_args.kwargs["items"]
|
||||
for value in call_args.kwargs.values():
|
||||
if isinstance(value, list) and len(value) > 0:
|
||||
return value
|
||||
return None
|
||||
|
||||
@pytest.fixture
|
||||
def mock_client(self) -> Mock:
|
||||
"""Create a mock Opik client."""
|
||||
client = Mock()
|
||||
# Ensure flush returns True to indicate success
|
||||
client.flush = Mock(return_value=True)
|
||||
|
||||
# Mock dataset
|
||||
mock_dataset = Mock()
|
||||
mock_dataset.name = "test-dataset"
|
||||
mock_dataset.__internal_api__insert_items_as_dataclasses__ = Mock()
|
||||
|
||||
# Mock experiment
|
||||
mock_experiment = Mock()
|
||||
mock_experiment.insert = Mock()
|
||||
mock_experiment.id = "exp-123"
|
||||
|
||||
client.get_or_create_dataset = Mock(return_value=mock_dataset)
|
||||
client.create_experiment = Mock(return_value=mock_experiment)
|
||||
# Mock REST client for experiment items creation
|
||||
client._rest_client = Mock()
|
||||
client._rest_client.experiments = Mock()
|
||||
client._rest_client.experiments.create_experiment_items = Mock()
|
||||
|
||||
return client
|
||||
|
||||
@pytest.fixture
|
||||
def experiment_data(self) -> ExperimentData:
|
||||
"""Create sample experiment data."""
|
||||
return ExperimentData(
|
||||
experiment={
|
||||
"id": "exp-123",
|
||||
"name": "test-experiment",
|
||||
"dataset_name": "test-dataset",
|
||||
"type": "regular",
|
||||
},
|
||||
items=[
|
||||
{
|
||||
"trace_id": "trace-1",
|
||||
"dataset_item_id": "ds-item-1",
|
||||
"dataset_item_data": {
|
||||
"input": "test input",
|
||||
"expected_output": "test output",
|
||||
},
|
||||
},
|
||||
{
|
||||
"trace_id": "trace-2",
|
||||
"dataset_item_id": "ds-item-2",
|
||||
"dataset_item_data": {"input": "test input 2"},
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
def test_recreate_experiment_requires_trace_id_map(
|
||||
self, mock_client: Mock, experiment_data: ExperimentData
|
||||
) -> None:
|
||||
"""Test that recreate_experiment requires trace_id_map (not Optional)."""
|
||||
# This test verifies the type signature
|
||||
trace_id_map: Dict[str, str] = {
|
||||
"trace-1": "new-trace-1",
|
||||
"trace-2": "new-trace-2",
|
||||
}
|
||||
dataset_item_id_map: Dict[str, str] = {
|
||||
"ds-item-1": "new-ds-item-1",
|
||||
"ds-item-2": "new-ds-item-2",
|
||||
}
|
||||
|
||||
# Should not accept None - type checker would catch this
|
||||
# We test that it works with a dict
|
||||
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
|
||||
mock_id_helpers.generate_id = Mock(return_value="generated-id")
|
||||
|
||||
recreate_experiment(
|
||||
mock_client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
trace_id_map, # Required, not Optional
|
||||
dataset_item_id_map, # Required for mapping dataset items
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
# Verify experiment items were created via REST API
|
||||
assert mock_client._rest_client.experiments.create_experiment_items.called
|
||||
|
||||
def test_recreate_experiment_batches_dataset_items(
|
||||
self, mock_client: Mock, experiment_data: ExperimentData
|
||||
) -> None:
|
||||
"""Test that experiment items are created in batch, not one at a time."""
|
||||
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
|
||||
mock_id_helpers.generate_id = Mock(side_effect=["exp-item-1", "exp-item-2"])
|
||||
|
||||
trace_id_map = {"trace-1": "new-trace-1", "trace-2": "new-trace-2"}
|
||||
dataset_item_id_map = {
|
||||
"ds-item-1": "new-ds-item-1",
|
||||
"ds-item-2": "new-ds-item-2",
|
||||
}
|
||||
|
||||
recreate_experiment(
|
||||
mock_client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
trace_id_map,
|
||||
dataset_item_id_map,
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
# Verify batch insert was called ONCE with all items
|
||||
assert (
|
||||
mock_client._rest_client.experiments.create_experiment_items.call_count
|
||||
== 1
|
||||
)
|
||||
|
||||
# Verify it was called with a list of items (batch)
|
||||
call_args = (
|
||||
mock_client._rest_client.experiments.create_experiment_items.call_args
|
||||
)
|
||||
assert call_args is not None
|
||||
|
||||
# Extract experiment_items argument from call_args
|
||||
experiment_items_arg = None
|
||||
if hasattr(call_args, "kwargs") and "experiment_items" in call_args.kwargs:
|
||||
experiment_items_arg = call_args.kwargs["experiment_items"]
|
||||
elif hasattr(call_args, "args") and call_args.args:
|
||||
experiment_items_arg = call_args.args[0]
|
||||
|
||||
assert experiment_items_arg is not None, (
|
||||
f"Could not find experiment_items in call_args: {call_args}"
|
||||
)
|
||||
assert len(experiment_items_arg) == 2, (
|
||||
f"Expected 2 items in batch, got {len(experiment_items_arg)}"
|
||||
)
|
||||
|
||||
def test_recreate_experiment_uses_module_names_correctly(
|
||||
self, mock_client: Mock, experiment_data: ExperimentData
|
||||
) -> None:
|
||||
"""Test that module names (id_helpers_module) are used correctly."""
|
||||
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
|
||||
mock_id_helpers.generate_id = Mock(return_value="generated-id")
|
||||
|
||||
trace_id_map = {"trace-1": "new-trace-1"}
|
||||
dataset_item_id_map = {"ds-item-1": "new-ds-item-1"}
|
||||
|
||||
recreate_experiment(
|
||||
mock_client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
trace_id_map,
|
||||
dataset_item_id_map,
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
# Verify id_helpers module is used (not checked for None)
|
||||
assert mock_id_helpers.generate_id.called
|
||||
|
||||
def test_recreate_experiment_handles_empty_trace_id_map(
|
||||
self, mock_client: Mock, experiment_data: ExperimentData
|
||||
) -> None:
|
||||
"""Test that empty trace_id_map is handled correctly."""
|
||||
trace_id_map: Dict[str, str] = {} # Empty but valid
|
||||
dataset_item_id_map: Dict[str, str] = {} # Empty but valid
|
||||
|
||||
recreate_experiment(
|
||||
mock_client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
trace_id_map,
|
||||
dataset_item_id_map,
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
# Should still create experiment and dataset, but skip items
|
||||
assert mock_client.get_or_create_dataset.called
|
||||
assert mock_client.create_experiment.called
|
||||
|
||||
def test_recreate_experiment_dry_run(
|
||||
self, mock_client: Mock, experiment_data: ExperimentData
|
||||
) -> None:
|
||||
"""Test dry run mode."""
|
||||
trace_id_map = {"trace-1": "new-trace-1"}
|
||||
dataset_item_id_map = {"ds-item-1": "new-ds-item-1"}
|
||||
|
||||
result = recreate_experiment(
|
||||
mock_client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
trace_id_map,
|
||||
dataset_item_id_map,
|
||||
dry_run=True,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
assert result is True
|
||||
# Should not create anything in dry run
|
||||
assert not mock_client.get_or_create_dataset.called
|
||||
assert not mock_client.create_experiment.called
|
||||
|
||||
def test_recreate_experiment_chunks_items_within_be_cap(
|
||||
self, mock_client: Mock
|
||||
) -> None:
|
||||
# BE rejects a single ``create_experiment_items`` POST whose item
|
||||
# count exceeds ``ExperimentItemsBatch``'s ``@Size(max=…)``. The
|
||||
# actual cap value is BE-configured and may change over time, so
|
||||
# we read it from the module-level constant the chunker uses and
|
||||
# assert behavior against that, not against a hardcoded literal.
|
||||
# Contract: every batch ``<= cap``, total across batches ==
|
||||
# full item count, multiple batches when items > cap.
|
||||
from opik.cli.imports.experiment import _EXPERIMENT_ITEMS_INSERT_BATCH_SIZE
|
||||
|
||||
cap = _EXPERIMENT_ITEMS_INSERT_BATCH_SIZE
|
||||
# 2.5 × cap exercises the multi-batch path with a partial-last-
|
||||
# batch remainder; any value above ``cap`` works for the contract.
|
||||
n_items = cap * 5 // 2
|
||||
experiment_data = ExperimentData(
|
||||
experiment={
|
||||
"id": "exp-big",
|
||||
"name": "big-experiment",
|
||||
"dataset_name": "test-dataset",
|
||||
"type": "regular",
|
||||
},
|
||||
items=[
|
||||
{
|
||||
"trace_id": f"trace-{i}",
|
||||
"dataset_item_id": f"ds-item-{i}",
|
||||
"dataset_item_data": {"input": f"test input {i}"},
|
||||
}
|
||||
for i in range(n_items)
|
||||
],
|
||||
)
|
||||
trace_id_map = {f"trace-{i}": f"new-trace-{i}" for i in range(n_items)}
|
||||
dataset_item_id_map = {
|
||||
f"ds-item-{i}": f"new-ds-item-{i}" for i in range(n_items)
|
||||
}
|
||||
|
||||
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
|
||||
mock_id_helpers.generate_id = Mock(
|
||||
side_effect=[f"exp-item-{i}" for i in range(n_items)]
|
||||
)
|
||||
|
||||
recreate_experiment(
|
||||
mock_client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
trace_id_map,
|
||||
dataset_item_id_map,
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
create_calls = mock_client._rest_client.experiments.create_experiment_items.call_args_list
|
||||
# ceil(n_items / cap) batched POSTs.
|
||||
expected_batches = (n_items + cap - 1) // cap
|
||||
assert len(create_calls) == expected_batches, (
|
||||
f"expected {expected_batches} chunked create_experiment_items "
|
||||
f"calls for {n_items} items at cap={cap}, got {len(create_calls)}"
|
||||
)
|
||||
# Every batch must respect the BE cap; total must equal the
|
||||
# full item count (nothing dropped, nothing duplicated).
|
||||
batch_sizes = [
|
||||
len(call.kwargs.get("experiment_items", []))
|
||||
if call.kwargs.get("experiment_items") is not None
|
||||
else len(call.args[0])
|
||||
for call in create_calls
|
||||
]
|
||||
assert all(size <= cap for size in batch_sizes), (
|
||||
f"every batch must respect the BE cap (={cap}), got {batch_sizes}"
|
||||
)
|
||||
assert sum(batch_sizes) == n_items, (
|
||||
f"all {n_items} items must end up across the chunked batches, "
|
||||
f"got sum={sum(batch_sizes)}"
|
||||
)
|
||||
|
||||
|
||||
class TestImportTracesWithSpans:
|
||||
"""Test trace import with span parent_span_id preservation."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_client(self) -> Mock:
|
||||
"""Create a mock Opik client."""
|
||||
client = Mock()
|
||||
client.flush = Mock()
|
||||
|
||||
# Mock trace creation
|
||||
mock_trace = Mock()
|
||||
mock_trace.id = "new-trace-1"
|
||||
client.trace = Mock(return_value=mock_trace)
|
||||
|
||||
# Mock span creation
|
||||
mock_spans = []
|
||||
for i in range(3):
|
||||
mock_span = Mock()
|
||||
mock_span.id = f"new-span-{i + 1}"
|
||||
mock_spans.append(mock_span)
|
||||
|
||||
client.span = Mock(side_effect=mock_spans)
|
||||
|
||||
return client
|
||||
|
||||
def test_import_traces_preserves_span_hierarchy(
|
||||
self, mock_client: Mock, tmp_path: Path
|
||||
) -> None:
|
||||
"""Test that span parent_span_id relationships are preserved."""
|
||||
# Create test trace file with spans
|
||||
projects_dir = tmp_path / "projects" / "test-project"
|
||||
projects_dir.mkdir(parents=True)
|
||||
|
||||
trace_data = {
|
||||
"trace": {
|
||||
"id": "original-trace-1",
|
||||
"name": "test-trace",
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"id": "span-1",
|
||||
"name": "root-span",
|
||||
"parent_span_id": None, # Root span
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
{
|
||||
"id": "span-2",
|
||||
"name": "child-span",
|
||||
"parent_span_id": "span-1", # Child of span-1
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
{
|
||||
"id": "span-3",
|
||||
"name": "grandchild-span",
|
||||
"parent_span_id": "span-2", # Child of span-2
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
trace_file = projects_dir / "trace_original-trace-1.json"
|
||||
with open(trace_file, "w") as f:
|
||||
json.dump(trace_data, f)
|
||||
|
||||
# Import traces
|
||||
trace_id_map, _ = _import_traces_for_project(
|
||||
mock_client, projects_dir, "test-project", dry_run=False, debug=False
|
||||
)
|
||||
|
||||
# Verify spans were created
|
||||
assert mock_client.span.call_count == 3
|
||||
|
||||
# Verify spans were created in correct order (root first, then children)
|
||||
span_calls = mock_client.span.call_args_list
|
||||
|
||||
# First span should be root (no parent_span_id)
|
||||
first_call = span_calls[0]
|
||||
assert first_call.kwargs.get("parent_span_id") is None
|
||||
|
||||
# Second span should have parent_span_id set to first span's new ID
|
||||
# Note: We can't easily verify the exact ID mapping without more complex mocking,
|
||||
# but we can verify that parent_span_id is being passed
|
||||
second_call = span_calls[1]
|
||||
# The parent_span_id should be set (not None) since span-1 was created first
|
||||
# and its new ID should be in span_id_map
|
||||
assert "parent_span_id" in second_call.kwargs
|
||||
|
||||
# Third span (grandchild) should have parent_span_id set to second span's new ID
|
||||
third_call = span_calls[2]
|
||||
assert "parent_span_id" in third_call.kwargs
|
||||
assert third_call.kwargs.get("parent_span_id") is not None
|
||||
|
||||
# Verify trace was created
|
||||
assert mock_client.trace.called
|
||||
assert "original-trace-1" in trace_id_map
|
||||
|
||||
def test_import_traces_preserves_deep_hierarchy(
|
||||
self, mock_client: Mock, tmp_path: Path
|
||||
) -> None:
|
||||
"""Test that deep hierarchies (4+ levels) are preserved correctly."""
|
||||
projects_dir = tmp_path / "projects" / "test-project"
|
||||
projects_dir.mkdir(parents=True)
|
||||
|
||||
# Create a 4-level hierarchy: root -> child -> grandchild -> great-grandchild
|
||||
# Spans are intentionally in wrong order to test sorting
|
||||
trace_data = {
|
||||
"trace": {
|
||||
"id": "original-trace-1",
|
||||
"name": "test-trace",
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"id": "span-4",
|
||||
"name": "great-grandchild",
|
||||
"parent_span_id": "span-3",
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
{
|
||||
"id": "span-2",
|
||||
"name": "child",
|
||||
"parent_span_id": "span-1",
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
{
|
||||
"id": "span-1",
|
||||
"name": "root",
|
||||
"parent_span_id": None,
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
{
|
||||
"id": "span-3",
|
||||
"name": "grandchild",
|
||||
"parent_span_id": "span-2",
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
trace_file = projects_dir / "trace_original-trace-1.json"
|
||||
with open(trace_file, "w") as f:
|
||||
json.dump(trace_data, f)
|
||||
|
||||
# Import traces
|
||||
trace_id_map, _ = _import_traces_for_project(
|
||||
mock_client, projects_dir, "test-project", dry_run=False, debug=False
|
||||
)
|
||||
|
||||
# Verify all spans were created
|
||||
assert mock_client.span.call_count == 4
|
||||
|
||||
span_calls = mock_client.span.call_args_list
|
||||
|
||||
# Verify order: root -> child -> grandchild -> great-grandchild
|
||||
# First span should be root (no parent)
|
||||
assert span_calls[0].kwargs.get("parent_span_id") is None
|
||||
|
||||
# Second span should be child (has parent)
|
||||
assert span_calls[1].kwargs.get("parent_span_id") is not None
|
||||
|
||||
# Third span should be grandchild (has parent)
|
||||
assert span_calls[2].kwargs.get("parent_span_id") is not None
|
||||
|
||||
# Fourth span should be great-grandchild (has parent)
|
||||
assert span_calls[3].kwargs.get("parent_span_id") is not None
|
||||
|
||||
# Verify trace was created
|
||||
assert mock_client.trace.called
|
||||
assert "original-trace-1" in trace_id_map
|
||||
|
||||
def test_import_traces_sorts_spans_correctly(
|
||||
self, mock_client: Mock, tmp_path: Path
|
||||
) -> None:
|
||||
"""Test that spans are sorted (root spans first, then children)."""
|
||||
projects_dir = tmp_path / "projects" / "test-project"
|
||||
projects_dir.mkdir(parents=True)
|
||||
|
||||
trace_data = {
|
||||
"trace": {
|
||||
"id": "original-trace-1",
|
||||
"name": "test-trace",
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"id": "span-2",
|
||||
"name": "child-span",
|
||||
"parent_span_id": "span-1", # Child - should come after root
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
{
|
||||
"id": "span-1",
|
||||
"name": "root-span",
|
||||
"parent_span_id": None, # Root - should come first
|
||||
"input": {},
|
||||
"output": {},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
trace_file = projects_dir / "trace_original-trace-1.json"
|
||||
with open(trace_file, "w") as f:
|
||||
json.dump(trace_data, f)
|
||||
|
||||
# Import traces
|
||||
_, _ = _import_traces_for_project(
|
||||
mock_client, projects_dir, "test-project", dry_run=False, debug=False
|
||||
) # Returns (trace_id_map, stats), but we don't need them for this test
|
||||
|
||||
# Verify spans were created in correct order
|
||||
span_calls = mock_client.span.call_args_list
|
||||
|
||||
# First span should be root (no parent)
|
||||
assert span_calls[0].kwargs.get("parent_span_id") is None
|
||||
|
||||
# Second span should have parent_span_id
|
||||
assert span_calls[1].kwargs.get("parent_span_id") is not None
|
||||
|
||||
|
||||
class TestTopologicalSort:
|
||||
"""Test the topological sort function for spans."""
|
||||
|
||||
def test_sort_spans_simple_hierarchy(self) -> None:
|
||||
"""Test sorting with a simple 2-level hierarchy."""
|
||||
spans = [
|
||||
{"id": "span-2", "name": "child", "parent_span_id": "span-1"},
|
||||
{"id": "span-1", "name": "root", "parent_span_id": None},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Root should come first
|
||||
assert sorted_spans[0]["id"] == "span-1"
|
||||
assert sorted_spans[0]["parent_span_id"] is None
|
||||
# Child should come second
|
||||
assert sorted_spans[1]["id"] == "span-2"
|
||||
assert sorted_spans[1]["parent_span_id"] == "span-1"
|
||||
|
||||
def test_sort_spans_multi_level_hierarchy(self) -> None:
|
||||
"""Test sorting with a 4-level hierarchy (root -> child -> grandchild -> great-grandchild)."""
|
||||
spans = [
|
||||
{"id": "span-4", "name": "great-grandchild", "parent_span_id": "span-3"},
|
||||
{"id": "span-2", "name": "child", "parent_span_id": "span-1"},
|
||||
{"id": "span-1", "name": "root", "parent_span_id": None},
|
||||
{"id": "span-3", "name": "grandchild", "parent_span_id": "span-2"},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Verify order: root -> child -> grandchild -> great-grandchild
|
||||
assert sorted_spans[0]["id"] == "span-1"
|
||||
assert sorted_spans[0]["parent_span_id"] is None
|
||||
|
||||
assert sorted_spans[1]["id"] == "span-2"
|
||||
assert sorted_spans[1]["parent_span_id"] == "span-1"
|
||||
|
||||
assert sorted_spans[2]["id"] == "span-3"
|
||||
assert sorted_spans[2]["parent_span_id"] == "span-2"
|
||||
|
||||
assert sorted_spans[3]["id"] == "span-4"
|
||||
assert sorted_spans[3]["parent_span_id"] == "span-3"
|
||||
|
||||
def test_sort_spans_multiple_roots(self) -> None:
|
||||
"""Test sorting with multiple root spans."""
|
||||
spans = [
|
||||
{"id": "span-3", "name": "child-of-2", "parent_span_id": "span-2"},
|
||||
{"id": "span-1", "name": "root-1", "parent_span_id": None},
|
||||
{"id": "span-2", "name": "root-2", "parent_span_id": None},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Both roots should come before the child
|
||||
root_ids = {sorted_spans[0]["id"], sorted_spans[1]["id"]}
|
||||
assert root_ids == {"span-1", "span-2"}
|
||||
assert sorted_spans[0]["parent_span_id"] is None
|
||||
assert sorted_spans[1]["parent_span_id"] is None
|
||||
|
||||
# Child should come last
|
||||
assert sorted_spans[2]["id"] == "span-3"
|
||||
assert sorted_spans[2]["parent_span_id"] == "span-2"
|
||||
|
||||
def test_sort_spans_multiple_children(self) -> None:
|
||||
"""Test sorting with a root that has multiple children."""
|
||||
spans = [
|
||||
{"id": "span-3", "name": "child-2", "parent_span_id": "span-1"},
|
||||
{"id": "span-1", "name": "root", "parent_span_id": None},
|
||||
{"id": "span-2", "name": "child-1", "parent_span_id": "span-1"},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Root should come first
|
||||
assert sorted_spans[0]["id"] == "span-1"
|
||||
assert sorted_spans[0]["parent_span_id"] is None
|
||||
|
||||
# Both children should come after root (order doesn't matter for siblings)
|
||||
child_ids = {sorted_spans[1]["id"], sorted_spans[2]["id"]}
|
||||
assert child_ids == {"span-2", "span-3"}
|
||||
assert sorted_spans[1]["parent_span_id"] == "span-1"
|
||||
assert sorted_spans[2]["parent_span_id"] == "span-1"
|
||||
|
||||
def test_sort_spans_missing_parent(self) -> None:
|
||||
"""Test sorting when a span references a non-existent parent."""
|
||||
spans = [
|
||||
{"id": "span-1", "name": "root", "parent_span_id": None},
|
||||
{"id": "span-2", "name": "orphan", "parent_span_id": "nonexistent"},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Both should be treated as roots (orphan becomes root)
|
||||
assert len(sorted_spans) == 2
|
||||
# Both should have no parent or invalid parent
|
||||
for span in sorted_spans:
|
||||
assert (
|
||||
span["parent_span_id"] is None
|
||||
or span["parent_span_id"] == "nonexistent"
|
||||
)
|
||||
|
||||
def test_sort_spans_empty_list(self) -> None:
|
||||
"""Test sorting with empty list."""
|
||||
spans: list = []
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
assert sorted_spans == []
|
||||
|
||||
def test_sort_spans_single_root(self) -> None:
|
||||
"""Test sorting with single root span."""
|
||||
spans = [{"id": "span-1", "name": "root", "parent_span_id": None}]
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
assert len(sorted_spans) == 1
|
||||
assert sorted_spans[0]["id"] == "span-1"
|
||||
assert sorted_spans[0]["parent_span_id"] is None
|
||||
|
||||
def test_sort_spans_all_have_parents(self) -> None:
|
||||
"""Test sorting when all spans have parents (no explicit root).
|
||||
|
||||
This tests the fix for the bug where empty root_spans would cause
|
||||
the function to return an empty list, silently dropping all spans.
|
||||
"""
|
||||
spans = [
|
||||
{"id": "span-1", "name": "child-1", "parent_span_id": "span-2"},
|
||||
{"id": "span-2", "name": "child-2", "parent_span_id": "span-1"},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Should not return empty list - all spans should be included
|
||||
assert len(sorted_spans) == 2
|
||||
# Verify all spans are present
|
||||
span_ids = {span["id"] for span in sorted_spans}
|
||||
assert span_ids == {"span-1", "span-2"}
|
||||
|
||||
def test_sort_spans_cycle(self) -> None:
|
||||
"""Test sorting with a cycle in the span graph.
|
||||
|
||||
This tests that cycles don't cause infinite loops and all spans
|
||||
are still included in the result.
|
||||
"""
|
||||
spans = [
|
||||
{"id": "span-1", "name": "span-1", "parent_span_id": "span-2"},
|
||||
{"id": "span-2", "name": "span-2", "parent_span_id": "span-3"},
|
||||
{"id": "span-3", "name": "span-3", "parent_span_id": "span-1"},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# Should not return empty list - all spans should be included
|
||||
assert len(sorted_spans) == 3
|
||||
# Verify all spans are present
|
||||
span_ids = {span["id"] for span in sorted_spans}
|
||||
assert span_ids == {"span-1", "span-2", "span-3"}
|
||||
|
||||
def test_sort_spans_disconnected_components(self) -> None:
|
||||
"""Test sorting with disconnected components (multiple separate graphs).
|
||||
|
||||
This tests that spans not reachable from root spans are still included.
|
||||
"""
|
||||
spans = [
|
||||
{"id": "span-1", "name": "root-1", "parent_span_id": None},
|
||||
{"id": "span-2", "name": "child-1", "parent_span_id": "span-1"},
|
||||
{"id": "span-3", "name": "disconnected-1", "parent_span_id": "span-4"},
|
||||
{"id": "span-4", "name": "disconnected-2", "parent_span_id": "span-3"},
|
||||
]
|
||||
|
||||
sorted_spans = sort_spans_topologically(spans)
|
||||
|
||||
# All spans should be included
|
||||
assert len(sorted_spans) == 4
|
||||
# Verify all spans are present
|
||||
span_ids = {span["id"] for span in sorted_spans}
|
||||
assert span_ids == {"span-1", "span-2", "span-3", "span-4"}
|
||||
# Root span should come first
|
||||
assert sorted_spans[0]["id"] == "span-1"
|
||||
assert sorted_spans[0]["parent_span_id"] is None
|
||||
|
||||
|
||||
class TestModuleNameUsage:
|
||||
"""Test that module names are used correctly (not checked for None)."""
|
||||
|
||||
def test_module_names_are_modules_not_variables(self) -> None:
|
||||
"""Test that dataset_item_module and id_helpers_module are modules."""
|
||||
from opik.cli.imports.experiment import dataset_item_module, id_helpers_module
|
||||
|
||||
# Modules should exist and be importable
|
||||
assert dataset_item_module is not None
|
||||
assert id_helpers_module is not None
|
||||
|
||||
# They should be modules, not None
|
||||
assert isinstance(dataset_item_module, types.ModuleType)
|
||||
assert isinstance(id_helpers_module, types.ModuleType)
|
||||
|
||||
|
||||
class TestProjectTraceImport:
|
||||
"""Test importing a project's traces from its project directory."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_client(self) -> Mock:
|
||||
"""Create a minimal mock Opik client."""
|
||||
client = Mock()
|
||||
client.flush = Mock()
|
||||
mock_trace = Mock()
|
||||
mock_trace.id = "new-trace-id"
|
||||
client.trace = Mock(return_value=mock_trace)
|
||||
client.span = Mock()
|
||||
return client
|
||||
|
||||
def test_trace_files_in_project_dir_are_imported(
|
||||
self, mock_client: Mock, tmp_path: Path
|
||||
) -> None:
|
||||
"""trace_{id}.json files directly under the project dir are imported and
|
||||
appear in the returned trace_id_map, and the trace is created in the
|
||||
named project."""
|
||||
project_dir = tmp_path / "projects" / "my-project"
|
||||
project_dir.mkdir(parents=True)
|
||||
|
||||
trace_id = "abc123"
|
||||
trace_data: Dict[str, Any] = {
|
||||
"trace": {"id": trace_id, "name": "t", "input": {}, "output": {}},
|
||||
"spans": [],
|
||||
}
|
||||
trace_file = project_dir / f"trace_{trace_id}.json"
|
||||
with open(trace_file, "w") as f:
|
||||
json.dump(trace_data, f)
|
||||
|
||||
trace_id_map, _ = _import_traces_for_project(
|
||||
mock_client, project_dir, "my-project", dry_run=False, debug=False
|
||||
)
|
||||
|
||||
# The original trace ID must appear in the returned map
|
||||
assert trace_id in trace_id_map
|
||||
# The trace must be created in the named project (no "default" fallback)
|
||||
assert mock_client.trace.call_args.kwargs.get("project_name") == "my-project"
|
||||
|
||||
def test_empty_project_dir_returns_empty_map(
|
||||
self, mock_client: Mock, tmp_path: Path
|
||||
) -> None:
|
||||
"""A project dir with no trace files yields an empty map without error."""
|
||||
project_dir = tmp_path / "projects" / "empty-project"
|
||||
project_dir.mkdir(parents=True)
|
||||
|
||||
trace_id_map, stats = _import_traces_for_project(
|
||||
mock_client, project_dir, "empty-project", dry_run=False, debug=False
|
||||
)
|
||||
|
||||
assert trace_id_map == {}
|
||||
assert stats["traces"] == 0
|
||||
assert not mock_client.trace.called
|
||||
@@ -0,0 +1,208 @@
|
||||
"""Tests for the ``opik mcp configure`` command."""
|
||||
|
||||
import pathlib
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli import cli
|
||||
from opik.cli import mcp as mcp_cli
|
||||
from opik.config import OpikConfig
|
||||
|
||||
|
||||
def _config(**overrides) -> OpikConfig:
|
||||
values = dict(url_override="https://www.comet.com/opik/api/", workspace="acme-ai")
|
||||
values.update(overrides)
|
||||
return OpikConfig(**values)
|
||||
|
||||
|
||||
class TestResolveSetupParams:
|
||||
def test_cloud__no_url_flags(self):
|
||||
params = mcp_cli._resolve_setup_params(
|
||||
_config(api_key="key", url_override="https://www.comet.com/opik/api/")
|
||||
)
|
||||
assert params["use_local"] is False
|
||||
assert params["self_hosted_comet"] is False
|
||||
assert params["api_url"] == "https://www.comet.com/opik/api/"
|
||||
|
||||
def test_self_hosted_comet__detected_from_opik_api_path(self):
|
||||
params = mcp_cli._resolve_setup_params(
|
||||
_config(api_key="key", url_override="https://opik.acme.com/opik/api/")
|
||||
)
|
||||
assert params["self_hosted_comet"] is True
|
||||
assert params["use_local"] is False
|
||||
assert params["base_url"] == "https://opik.acme.com/"
|
||||
|
||||
def test_localhost__is_use_local(self):
|
||||
params = mcp_cli._resolve_setup_params(
|
||||
_config(api_key=None, url_override="http://localhost:5173/api/")
|
||||
)
|
||||
assert params["use_local"] is True
|
||||
assert params["self_hosted_comet"] is False
|
||||
|
||||
def test_self_hosted_oss__non_opik_path_is_use_local(self):
|
||||
params = mcp_cli._resolve_setup_params(
|
||||
_config(api_key=None, url_override="https://opik.acme.com/api/")
|
||||
)
|
||||
assert params["use_local"] is True
|
||||
assert params["self_hosted_comet"] is False
|
||||
|
||||
|
||||
class TestInstallCommand:
|
||||
def test_install__reads_config_and_calls_setup(self):
|
||||
runner = CliRunner()
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.interactive_helpers, "is_interactive", return_value=True
|
||||
),
|
||||
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "configure"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
setup_spy.assert_called_once()
|
||||
assert setup_spy.call_args.kwargs["api_key"] == "key"
|
||||
assert setup_spy.call_args.kwargs["workspace"] == "acme-ai"
|
||||
assert setup_spy.call_args.kwargs["force_local_server"] is False
|
||||
|
||||
def test_install__local_server_flag__forces_local(self):
|
||||
runner = CliRunner()
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.interactive_helpers, "is_interactive", return_value=True
|
||||
),
|
||||
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "configure", "--local-server"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
setup_spy.assert_called_once()
|
||||
assert setup_spy.call_args.kwargs["force_local_server"] is True
|
||||
|
||||
def test_install__non_interactive__errors(self):
|
||||
runner = CliRunner()
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.interactive_helpers, "is_interactive", return_value=False
|
||||
),
|
||||
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "configure"])
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "interactive terminal" in result.output
|
||||
setup_spy.assert_not_called()
|
||||
|
||||
def test_install__no_config_user_declines__errors(self):
|
||||
runner = CliRunner()
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key=None)
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.interactive_helpers, "is_interactive", return_value=True
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.configure_cli, "run_interactive_configure"
|
||||
) as configure_spy,
|
||||
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "configure"], input="n\n")
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "opik configure" in result.output
|
||||
configure_spy.assert_not_called()
|
||||
setup_spy.assert_not_called()
|
||||
|
||||
def test_install__no_config_user_accepts__runs_configure_then_installs(self):
|
||||
runner = CliRunner()
|
||||
configs = iter([_config(api_key=None), _config(api_key="new-key")])
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config, "OpikConfig", side_effect=lambda: next(configs)
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.interactive_helpers, "is_interactive", return_value=True
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.configure_cli, "run_interactive_configure"
|
||||
) as configure_spy,
|
||||
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "configure"], input="y\n")
|
||||
|
||||
assert result.exit_code == 0
|
||||
configure_spy.assert_called_once_with(install_mcp=False)
|
||||
setup_spy.assert_called_once()
|
||||
assert setup_spy.call_args.kwargs["api_key"] == "new-key"
|
||||
|
||||
def test_status__lists_sdk_env_and_host_drift(self):
|
||||
runner = CliRunner()
|
||||
host = mcp_cli.mcp_status.HostStatus(
|
||||
display_name="Claude Code",
|
||||
config_path=pathlib.Path("/home/u/.claude.json"),
|
||||
detected=True,
|
||||
registered=True,
|
||||
transport=mcp_cli.mcp_status.TRANSPORT_LOCAL,
|
||||
points_to="http://localhost:5173/api/",
|
||||
workspace="default",
|
||||
in_sync=False,
|
||||
)
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.mcp_status, "collect_host_statuses", return_value=[host]
|
||||
),
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "status"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "Your Opik configuration" in result.output
|
||||
assert "configured for 1 AI assistant" in result.output
|
||||
assert "Claude Code" in result.output
|
||||
assert "OUT OF SYNC with your Opik configuration" in result.output
|
||||
assert "http://localhost:5173/api/" in result.output
|
||||
assert "default" in result.output
|
||||
|
||||
def test_status__none_configured__suggests_configure(self):
|
||||
runner = CliRunner()
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config, "OpikConfig", return_value=_config(api_key="key")
|
||||
),
|
||||
patch.object(mcp_cli.mcp_status, "collect_host_statuses", return_value=[]),
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "status"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "not configured for any AI assistant" in result.output
|
||||
assert "opik mcp configure" in result.output
|
||||
|
||||
def test_install__local_without_api_key__proceeds(self):
|
||||
runner = CliRunner()
|
||||
with (
|
||||
patch.object(
|
||||
mcp_cli.opik_config,
|
||||
"OpikConfig",
|
||||
return_value=_config(
|
||||
api_key=None, url_override="http://localhost:5173/api/"
|
||||
),
|
||||
),
|
||||
patch.object(
|
||||
mcp_cli.interactive_helpers, "is_interactive", return_value=True
|
||||
),
|
||||
patch.object(mcp_cli.mcp_installer, "setup_mcp_server") as setup_spy,
|
||||
):
|
||||
result = runner.invoke(cli, ["mcp", "configure"])
|
||||
|
||||
assert result.exit_code == 0
|
||||
setup_spy.assert_called_once()
|
||||
assert setup_spy.call_args.kwargs["use_local"] is True
|
||||
@@ -0,0 +1,553 @@
|
||||
"""Tests for ``opik migrate dataset`` checkpoint/resume (OPIK-7168).
|
||||
|
||||
Two layers:
|
||||
|
||||
* ``TestMigrationCheckpoint`` -- the checkpoint store in isolation
|
||||
(``opik.cli.migrate.checkpoint``): key derivation, atomic round-trip,
|
||||
in-flight tracking, corrupt/foreign-schema tolerance, delete lifecycle.
|
||||
|
||||
* ``TestCascadeResume`` -- the cascade's resume behaviour
|
||||
(``cascade_experiments`` with a ``checkpoint``): skip already-completed
|
||||
experiments, re-migrate an interrupted experiment after deleting its partial
|
||||
destination data, and seed the progress callback so a resumed run reports the
|
||||
right completed count instead of starting at 0.
|
||||
|
||||
The cascade tests reuse the elaborate REST/client fakes from
|
||||
``test_migrate_dataset_experiments_cascade`` so the resume behaviour is
|
||||
exercised against the same call surface the real cascade drives.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, List, Tuple
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from opik.cli.migrate import checkpoint as checkpoint_module
|
||||
from opik.cli.migrate.checkpoint import (
|
||||
SCHEMA_VERSION,
|
||||
MigrationCheckpoint,
|
||||
checkpoint_key,
|
||||
checkpoint_path,
|
||||
load_or_create,
|
||||
)
|
||||
from opik.cli.migrate.datasets.experiments import cascade_experiments
|
||||
|
||||
from .test_migrate_dataset_experiments_cascade import (
|
||||
_Experiment,
|
||||
_ExperimentItem,
|
||||
_Trace,
|
||||
_audit,
|
||||
_cascade_rest_client,
|
||||
_client_with_recreate_capture,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Checkpoint store (unit)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolate_checkpoint_dir(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
"""Point the checkpoint store at a tmp dir instead of the real ~/.opik.
|
||||
|
||||
The checkpoint now lives at a fixed per-user path (``checkpoint_dir``),
|
||||
independent of cwd/--audit-log. Redirecting it here keeps the unit tests
|
||||
hermetic and off the developer's home directory.
|
||||
"""
|
||||
cp_dir = tmp_path / "migrate-checkpoints"
|
||||
cp_dir.mkdir(parents=True, exist_ok=True)
|
||||
monkeypatch.setattr(checkpoint_module, "checkpoint_dir", lambda: cp_dir)
|
||||
|
||||
|
||||
class TestMigrationCheckpoint:
|
||||
def test_checkpoint_key__stable_and_distinct_per_tuple(self) -> None:
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
# Deterministic: same tuple -> same key across calls.
|
||||
assert key == checkpoint_key("ws", "proj", "ds")
|
||||
# Distinct tuples -> distinct keys, including the ambiguous shift of a
|
||||
# separator across the three components (the \x00 join prevents
|
||||
# "a"+"bc" colliding with "ab"+"c").
|
||||
assert checkpoint_key("a", "bc", "d") != checkpoint_key("ab", "c", "d")
|
||||
|
||||
def test_checkpoint_path__lives_in_checkpoint_dir_keyed_by_hash(self) -> None:
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
path = checkpoint_path(key)
|
||||
assert path.parent == checkpoint_module.checkpoint_dir()
|
||||
assert path.name == f"opik-migrate-checkpoint-{key}.json"
|
||||
|
||||
def test_load_or_create__no_file__starts_fresh(self, tmp_path: Path) -> None:
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="proj",
|
||||
dataset="ds",
|
||||
)
|
||||
assert cp.completed_count == 0
|
||||
assert cp.in_flight is None
|
||||
assert not cp.path.exists()
|
||||
|
||||
def test_flush_then_load__round_trips_completed_and_in_flight(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
cp.total_experiments = 3
|
||||
cp.mark_in_flight(
|
||||
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-ds"
|
||||
)
|
||||
cp.record_dest_trace_ids(["dest-trace-a", "dest-trace-b"])
|
||||
cp.mark_completed("src-exp-1")
|
||||
cp.flush()
|
||||
|
||||
reloaded = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert reloaded.total_experiments == 3
|
||||
assert reloaded.completed_experiment_ids == {"src-exp-1"}
|
||||
# ``mark_completed`` clears in_flight, so only the completed set survives.
|
||||
assert reloaded.in_flight is None
|
||||
|
||||
def test_flush_preserves_in_flight_when_not_completed(self, tmp_path: Path) -> None:
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
cp.mark_in_flight(
|
||||
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-ds"
|
||||
)
|
||||
cp.record_dest_trace_ids(["dest-trace-a"])
|
||||
cp.flush()
|
||||
|
||||
reloaded = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert reloaded.in_flight is not None
|
||||
assert reloaded.in_flight.source_experiment_id == "src-exp-2"
|
||||
assert reloaded.in_flight.experiment_name == "exp-2"
|
||||
assert reloaded.in_flight.dest_dataset_id == "dest-ds"
|
||||
assert reloaded.in_flight.dest_trace_ids == ["dest-trace-a"]
|
||||
|
||||
def test_load_or_create__corrupt_file__starts_fresh(self, tmp_path: Path) -> None:
|
||||
# A truncated / unparseable checkpoint (e.g. an interrupted write on a
|
||||
# filesystem without atomic replace) must be treated as "no progress",
|
||||
# not crash the migration.
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
checkpoint_path(key).write_text('{"schema_version": 1, "comp')
|
||||
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert cp.completed_count == 0
|
||||
assert cp.in_flight is None
|
||||
|
||||
def test_load_or_create__non_bool_dataset_phase_done__starts_fresh(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# ``dataset_phase_done`` gates resume-vs-full-plan, so a truthy non-bool
|
||||
# (the string "false", a non-empty list) must NOT be coerced to True and
|
||||
# force a resume that skips rename/create/replay. A non-bool is
|
||||
# malformed -> fresh.
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
for bad in ('"false"', "[1]", "1"):
|
||||
# Use the CURRENT schema so the load reaches the non-bool guard
|
||||
# rather than short-circuiting on a schema-version mismatch.
|
||||
checkpoint_path(key).write_text(
|
||||
f'{{"schema_version": {SCHEMA_VERSION}, "dataset_phase_done": {bad}}}'
|
||||
)
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert cp.dataset_phase_done is False
|
||||
|
||||
def test_load_or_create__malformed_in_flight__starts_fresh(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# A checkpoint with the right schema_version but a wrong-shaped
|
||||
# ``in_flight`` (here a bare string, or a dict missing the required
|
||||
# ``source_experiment_id``) must fall back to fresh rather than crash
|
||||
# the CLI with a TypeError/KeyError.
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
for bad_in_flight in ('"not-a-dict"', '{"experiment_name": "x"}'):
|
||||
checkpoint_path(key).write_text(
|
||||
'{"schema_version": 1, "completed_experiment_ids": ["e1"], '
|
||||
f'"in_flight": {bad_in_flight}}}'
|
||||
)
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert cp.completed_count == 0
|
||||
assert cp.in_flight is None
|
||||
|
||||
def test_load_or_create__corrupt_completed_ids__starts_fresh(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# ``set("abc")`` / ``set([1, 2])`` don't raise, so a corrupt
|
||||
# ``completed_experiment_ids`` (a bare string, or a list with non-string
|
||||
# entries) would silently seed the wrong completed set and make the
|
||||
# cascade skip/re-run the wrong experiments. It must fall back to fresh.
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
for bad in ('"abc"', "[1, 2, 3]", "{}"):
|
||||
checkpoint_path(key).write_text(
|
||||
f'{{"schema_version": 1, "completed_experiment_ids": {bad}}}'
|
||||
)
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert cp.completed_experiment_ids == set()
|
||||
|
||||
def test_load_or_create__corrupt_dest_trace_ids__starts_fresh(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# Same silent-corruption guard for the in-flight trace-id list.
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
checkpoint_path(key).write_text(
|
||||
'{"schema_version": 1, "in_flight": '
|
||||
'{"source_experiment_id": "e2", "dest_trace_ids": "trace-1"}}'
|
||||
)
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert cp.in_flight is None
|
||||
assert cp.completed_count == 0
|
||||
|
||||
def test_load_or_create__non_string_resume_names__starts_fresh(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# OPIK-7162: source_dataset_id / source_name / temp_dest_name are the
|
||||
# resume handles that flow unvalidated into client.get_dataset(name=...)
|
||||
# and stream_dataset_items(dataset_name=...). A hand-edited non-string
|
||||
# would otherwise reach the API; it must fall back to fresh, matching
|
||||
# the same corrupt-recovery contract as the id collections. Uses the
|
||||
# CURRENT schema so the load reaches the field guard (not the schema
|
||||
# short-circuit).
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
for field in ("source_dataset_id", "source_name", "temp_dest_name"):
|
||||
for bad in ("[1]", "{}", "5"):
|
||||
checkpoint_path(key).write_text(
|
||||
f'{{"schema_version": {SCHEMA_VERSION}, '
|
||||
f'"dataset_phase_done": true, "{field}": {bad}}}'
|
||||
)
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
# Fell back to fresh: no phase-done carried over from the bad file.
|
||||
assert cp.dataset_phase_done is False
|
||||
assert getattr(cp, field) is None
|
||||
|
||||
def test_load_or_create__unresolvable_home__returns_none(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
# A homeless environment (Path.home() raising, as in some CI/containers)
|
||||
# must NOT crash the migration before it starts -- load_or_create
|
||||
# returns None and the caller runs without resume support.
|
||||
def _boom() -> Path:
|
||||
raise RuntimeError("Could not determine home directory")
|
||||
|
||||
monkeypatch.setattr(checkpoint_module, "checkpoint_dir", _boom, raising=True)
|
||||
assert load_or_create(workspace="ws", project="proj", dataset="ds") is None
|
||||
|
||||
def test_flush__write_failure__swallowed_not_raised(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
# A checkpoint is only a resume aid: a read-only / full disk on flush
|
||||
# must be logged and swallowed, never abort an otherwise-healthy
|
||||
# migration.
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert cp is not None
|
||||
|
||||
def _readonly(*_a: Any, **_k: Any) -> None:
|
||||
raise OSError("Read-only file system")
|
||||
|
||||
monkeypatch.setattr(Path, "mkdir", _readonly, raising=True)
|
||||
cp.flush() # must not raise
|
||||
|
||||
def test_flush_then_load__round_trips_dest_experiment_id(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
cp.mark_in_flight(
|
||||
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-ds"
|
||||
)
|
||||
cp.record_dest_experiment_id("dest-exp-99")
|
||||
cp.flush()
|
||||
|
||||
reloaded = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
assert reloaded.in_flight is not None
|
||||
assert reloaded.in_flight.dest_experiment_id == "dest-exp-99"
|
||||
|
||||
def test_load_or_create__foreign_schema_version__starts_fresh(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
key = checkpoint_key("ws", "proj", "ds")
|
||||
checkpoint_path(key).write_text(
|
||||
'{"schema_version": 999, "completed_experiment_ids": ["src-exp-1"]}'
|
||||
)
|
||||
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
# A newer schema we can't interpret is ignored -> fresh start.
|
||||
assert cp.completed_count == 0
|
||||
|
||||
def test_delete__removes_file_and_is_idempotent(self, tmp_path: Path) -> None:
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
cp.flush()
|
||||
assert cp.path.exists()
|
||||
cp.delete()
|
||||
assert not cp.path.exists()
|
||||
# Idempotent: deleting again is a no-op, not an error.
|
||||
cp.delete()
|
||||
|
||||
def test_flush__atomic__no_stray_tmp_file_left(self, tmp_path: Path) -> None:
|
||||
cp = load_or_create(workspace="ws", project="proj", dataset="ds")
|
||||
cp.flush()
|
||||
# The temp file used for the atomic write must have been renamed away.
|
||||
tmp_files = list(tmp_path.glob("*.tmp"))
|
||||
assert tmp_files == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cascade resume behaviour
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _two_experiment_rig() -> Tuple[Any, Any]:
|
||||
"""Two source experiments, each with one item/trace, wired end-to-end.
|
||||
|
||||
Returns ``(rest_client, client)`` ready to pass to ``cascade_experiments``.
|
||||
"""
|
||||
exp1 = _Experiment(id="src-exp-1", name="exp-1", dataset_version_id="src-v-1")
|
||||
exp2 = _Experiment(id="src-exp-2", name="exp-2", dataset_version_id="src-v-1")
|
||||
item1 = _ExperimentItem(
|
||||
id="i1",
|
||||
experiment_id="src-exp-1",
|
||||
trace_id="src-trace-1",
|
||||
dataset_item_id="src-ds-item-1",
|
||||
)
|
||||
item2 = _ExperimentItem(
|
||||
id="i2",
|
||||
experiment_id="src-exp-2",
|
||||
trace_id="src-trace-2",
|
||||
dataset_item_id="src-ds-item-1",
|
||||
)
|
||||
rest_client = _cascade_rest_client(
|
||||
experiments_by_dataset={"src-dataset-1": [exp1, exp2]},
|
||||
items_by_experiment={"exp-1": [item1], "exp-2": [item2]},
|
||||
traces_by_id={
|
||||
"src-trace-1": _Trace(id="src-trace-1"),
|
||||
"src-trace-2": _Trace(id="src-trace-2"),
|
||||
},
|
||||
spans_by_trace={"src-trace-1": [], "src-trace-2": []},
|
||||
)
|
||||
client = _client_with_recreate_capture(rest_client)
|
||||
return rest_client, client
|
||||
|
||||
|
||||
def _run_cascade(
|
||||
rest_client: Any, client: Any, checkpoint: MigrationCheckpoint, **overrides: Any
|
||||
) -> Tuple[Any, List[Tuple[int, int, str]]]:
|
||||
"""Run ``cascade_experiments`` with a progress-capturing callback."""
|
||||
progress_calls: List[Tuple[int, int, str]] = []
|
||||
|
||||
kwargs: dict = dict(
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
target_dataset_id="dest-dataset-1",
|
||||
version_remap={"src-v-1": "dest-v-1"},
|
||||
item_id_remap={"src-ds-item-1": "dest-ds-item-1"},
|
||||
audit=_audit(),
|
||||
checkpoint=checkpoint,
|
||||
progress_callback=lambda c, t, label: progress_calls.append((c, t, label)),
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
result = cascade_experiments(client, rest_client, **kwargs)
|
||||
return result, progress_calls
|
||||
|
||||
|
||||
class TestCascadeResume:
|
||||
def test_skip_completed__does_not_recreate_already_done_experiment(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
# Pretend exp-1 already migrated on a prior run.
|
||||
cp.mark_completed("src-exp-1")
|
||||
|
||||
result, _ = _run_cascade(rest_client, client, cp)
|
||||
|
||||
# Only the not-yet-done experiment is recreated.
|
||||
assert result.experiments_migrated == 1
|
||||
assert client.create_experiment.call_count == 1
|
||||
created_names = [
|
||||
call.kwargs["name"] for call in client.create_experiment.call_args_list
|
||||
]
|
||||
assert created_names == ["exp-2"]
|
||||
|
||||
def test_skip_completed__all_done__no_recreation(self, tmp_path: Path) -> None:
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
cp.mark_completed("src-exp-1")
|
||||
cp.mark_completed("src-exp-2")
|
||||
|
||||
result, _ = _run_cascade(rest_client, client, cp)
|
||||
|
||||
assert result.experiments_migrated == 0
|
||||
client.create_experiment.assert_not_called()
|
||||
|
||||
def test_happyflow__marks_each_experiment_completed_and_flushes(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
|
||||
result, _ = _run_cascade(rest_client, client, cp)
|
||||
|
||||
assert result.experiments_migrated == 2
|
||||
assert cp.completed_experiment_ids == {"src-exp-1", "src-exp-2"}
|
||||
# in_flight is cleared once the last experiment completes.
|
||||
assert cp.in_flight is None
|
||||
# Progress was flushed to disk (a re-run would see both done).
|
||||
reloaded = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
assert reloaded.completed_experiment_ids == {"src-exp-1", "src-exp-2"}
|
||||
|
||||
def test_re_migrate_incomplete__deletes_partial_dest_data_before_rerun(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
# exp-1 finished on the prior run; exp-2 was interrupted mid-flight with
|
||||
# two destination traces and its destination experiment row already
|
||||
# created (id recorded before creation).
|
||||
cp.mark_completed("src-exp-1")
|
||||
cp.mark_in_flight(
|
||||
"src-exp-2", experiment_name="exp-2", dest_dataset_id="dest-dataset-1"
|
||||
)
|
||||
cp.record_dest_trace_ids(["stale-dest-trace-1", "stale-dest-trace-2"])
|
||||
cp.record_dest_experiment_id("stale-dest-exp")
|
||||
|
||||
_run_cascade(rest_client, client, cp)
|
||||
|
||||
# Partial traces were deleted (spans cascade on the BE), so they don't
|
||||
# duplicate on the re-run.
|
||||
rest_client.traces.delete_traces.assert_called_once()
|
||||
deleted_trace_ids = rest_client.traces.delete_traces.call_args.kwargs["ids"]
|
||||
assert set(deleted_trace_ids) == {"stale-dest-trace-1", "stale-dest-trace-2"}
|
||||
# The exact recorded destination experiment row was deleted BY ID --
|
||||
# cleanup never does a name lookup, so no same-named peer can be hit.
|
||||
rest_client.experiments.delete_experiments_by_id.assert_called_once()
|
||||
deleted_exp_ids = (
|
||||
rest_client.experiments.delete_experiments_by_id.call_args.kwargs["ids"]
|
||||
)
|
||||
assert deleted_exp_ids == ["stale-dest-exp"]
|
||||
# in_flight is cleared after cleanup so a further crash won't re-delete.
|
||||
assert cp.in_flight is None
|
||||
|
||||
def test_re_migrate_incomplete__no_experiment_row_yet__only_traces_deleted(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# Interruption landed after traces were flushed but before the
|
||||
# experiment row was created: dest_experiment_id is None, so cleanup
|
||||
# deletes the traces and skips the experiment delete entirely (no
|
||||
# name lookup that could hit a peer).
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
cp.mark_in_flight(
|
||||
"src-exp-1", experiment_name="exp-1", dest_dataset_id="dest-dataset-1"
|
||||
)
|
||||
cp.record_dest_trace_ids(["stale-dest-trace-1"])
|
||||
# no record_dest_experiment_id -> dest_experiment_id stays None
|
||||
|
||||
_run_cascade(rest_client, client, cp)
|
||||
|
||||
rest_client.traces.delete_traces.assert_called_once()
|
||||
rest_client.experiments.delete_experiments_by_id.assert_not_called()
|
||||
|
||||
def test_re_migrate_incomplete__records_dest_ids_before_backend_write(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# OPIK-7168 (#533/#536): the destination trace ids and experiment id
|
||||
# must be flushed to the checkpoint BEFORE the backend write that
|
||||
# persists them, so a crash in that window still leaves them recorded
|
||||
# for the next run's cleanup. We assert the checkpoint file on disk
|
||||
# already carries this experiment's dest trace ids by the time the
|
||||
# trace-flush happens.
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
|
||||
flushed_state: dict = {}
|
||||
original_flush = client.flush
|
||||
|
||||
def _capture_on_first_flush() -> None:
|
||||
# On the first client.flush() (the trace flush), read back the
|
||||
# checkpoint from disk and snapshot its in-flight trace ids.
|
||||
if "dest_trace_ids" not in flushed_state:
|
||||
reloaded = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
flushed_state["dest_trace_ids"] = (
|
||||
list(reloaded.in_flight.dest_trace_ids)
|
||||
if reloaded.in_flight
|
||||
else []
|
||||
)
|
||||
return original_flush()
|
||||
|
||||
client.flush = MagicMock(side_effect=_capture_on_first_flush)
|
||||
|
||||
_run_cascade(rest_client, client, cp)
|
||||
|
||||
# By the time the backend trace flush ran, the checkpoint on disk had
|
||||
# already recorded this experiment's destination trace id.
|
||||
assert len(flushed_state["dest_trace_ids"]) == 1
|
||||
|
||||
def test_progress_seeded__resumed_run_starts_at_completed_count(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
rest_client, client = _two_experiment_rig()
|
||||
cp = load_or_create(
|
||||
workspace="ws",
|
||||
project="DestProject",
|
||||
dataset="MyDataset",
|
||||
)
|
||||
cp.mark_completed("src-exp-1")
|
||||
|
||||
_, progress_calls = _run_cascade(rest_client, client, cp)
|
||||
|
||||
# First progress tick reports 1 already-completed of 2 total (not 0),
|
||||
# so the bar opens at 50% rather than restarting.
|
||||
assert progress_calls[0] == (1, 2, "exp-2")
|
||||
# Final tick snaps to total/total "done".
|
||||
assert progress_calls[-1] == (2, 2, "done")
|
||||
|
||||
def test_no_checkpoint__cascade_unchanged(self) -> None:
|
||||
# Without a checkpoint the cascade behaves exactly as before: both
|
||||
# experiments migrate, no delete calls, progress counts from 0.
|
||||
rest_client, client = _two_experiment_rig()
|
||||
progress_calls: List[Tuple[int, int, str]] = []
|
||||
|
||||
result = cascade_experiments(
|
||||
client,
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
version_remap={"src-v-1": "dest-v-1"},
|
||||
item_id_remap={"src-ds-item-1": "dest-ds-item-1"},
|
||||
audit=_audit(),
|
||||
progress_callback=lambda c, t, label: progress_calls.append((c, t, label)),
|
||||
)
|
||||
|
||||
assert result.experiments_migrated == 2
|
||||
rest_client.traces.delete_traces.assert_not_called()
|
||||
assert progress_calls[0] == (0, 2, "exp-1")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,496 @@
|
||||
"""Tests for ``opik migrate dataset`` Slice 5 — optimization cascade.
|
||||
|
||||
Slice 5 sits between Slice 2 (``ReplayVersions``) and Slice 3
|
||||
(``CascadeExperiments``). It recreates every optimization referencing
|
||||
the source dataset under the destination project and populates
|
||||
``plan.optimization_id_remap`` so the experiment cascade can re-point
|
||||
each experiment's ``optimization_id`` FK.
|
||||
|
||||
Scope this module covers (ticket AC for OPIK-6532):
|
||||
|
||||
* Optimization fidelity round-trip (name / objective_name / status /
|
||||
metadata / studio_config carried verbatim)
|
||||
* READ-ONLY aggregates (num_trials / feedback_scores / baseline_* /
|
||||
best_* / total_optimization_cost) not forwarded
|
||||
* Fresh client-side UUID minted per source optimization; ``id_remap``
|
||||
populated with source -> destination
|
||||
* Empty optimization (zero trials) round-trips and produces a remap entry
|
||||
* Zero-optimization datasets are a no-op (returns empty result, no
|
||||
REST call)
|
||||
* Per-optimization ``migrate_optimization`` audit record emitted
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from opik.cli.migrate.audit import AuditLog
|
||||
from opik.cli.migrate.datasets import optimizations as optimizations_module
|
||||
from opik.cli.migrate.datasets.optimizations import cascade_optimizations
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stand-in wire shapes for the cascade's reads.
|
||||
#
|
||||
# The cascade reads ``OptimizationPublic`` from ``find_optimizations`` and
|
||||
# writes via ``create_optimization``. Constructing the real pydantic
|
||||
# models would impose ~20 fields per row; the cascade only inspects a
|
||||
# handful, so we use plain objects with the fields it actually reads.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _Optimization:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
id: str,
|
||||
name: Optional[str] = "opt",
|
||||
objective_name: str = "accuracy",
|
||||
status: str = "completed",
|
||||
dataset_name: str = "MyDataset",
|
||||
dataset_id: Optional[str] = "src-dataset-1",
|
||||
metadata: Optional[Any] = None,
|
||||
studio_config: Optional[Any] = None,
|
||||
last_updated_at: Optional[Any] = None,
|
||||
# READ-ONLY aggregates -- attached so we can assert the cascade
|
||||
# does NOT forward them on the create call.
|
||||
num_trials: Optional[int] = None,
|
||||
feedback_scores: Optional[List[Any]] = None,
|
||||
experiment_scores: Optional[List[Any]] = None,
|
||||
baseline_objective_score: Optional[float] = None,
|
||||
best_objective_score: Optional[float] = None,
|
||||
baseline_duration: Optional[float] = None,
|
||||
best_duration: Optional[float] = None,
|
||||
baseline_cost: Optional[float] = None,
|
||||
best_cost: Optional[float] = None,
|
||||
total_optimization_cost: Optional[float] = None,
|
||||
) -> None:
|
||||
self.id = id
|
||||
self.name = name
|
||||
self.objective_name = objective_name
|
||||
self.status = status
|
||||
self.dataset_name = dataset_name
|
||||
self.dataset_id = dataset_id
|
||||
self.metadata = metadata
|
||||
self.studio_config = studio_config
|
||||
self.last_updated_at = last_updated_at
|
||||
self.num_trials = num_trials
|
||||
self.feedback_scores = feedback_scores
|
||||
self.experiment_scores = experiment_scores
|
||||
self.baseline_objective_score = baseline_objective_score
|
||||
self.best_objective_score = best_objective_score
|
||||
self.baseline_duration = baseline_duration
|
||||
self.best_duration = best_duration
|
||||
self.baseline_cost = baseline_cost
|
||||
self.best_cost = best_cost
|
||||
self.total_optimization_cost = total_optimization_cost
|
||||
|
||||
|
||||
class _Page:
|
||||
def __init__(self, content: List[_Optimization]) -> None:
|
||||
self.content = content
|
||||
|
||||
|
||||
def _audit() -> AuditLog:
|
||||
"""Real AuditLog so the cascade's per-optimization ``record`` calls
|
||||
land in a list we can assert on. The audit log carries no side
|
||||
effects until ``write`` is called, so using the real type in tests
|
||||
is cheaper than building a coupled mock."""
|
||||
return AuditLog(command="migrate dataset", args={})
|
||||
|
||||
|
||||
def _cascade_rest_client(
|
||||
pages: List[_Page],
|
||||
*,
|
||||
create_side_effect: Optional[Any] = None,
|
||||
) -> MagicMock:
|
||||
"""Build a rest_client mock for the optimization cascade.
|
||||
|
||||
``find_optimizations`` is called once per page until a short page
|
||||
breaks the loop. ``create_optimization`` is captured so individual
|
||||
tests can assert on its kwargs.
|
||||
"""
|
||||
rest_client = MagicMock()
|
||||
rest_client.optimizations.find_optimizations.side_effect = pages
|
||||
if create_side_effect is not None:
|
||||
rest_client.optimizations.create_optimization.side_effect = create_side_effect
|
||||
else:
|
||||
rest_client.optimizations.create_optimization.return_value = None
|
||||
return rest_client
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cascade tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCascadeOptimizations:
|
||||
def test_no_source_optimizations__noop_no_remap_no_create_calls(self) -> None:
|
||||
# Empty result page -> no create_optimization calls, empty
|
||||
# remap, counters at zero. Slice 5 always emits the action; the
|
||||
# cascade handles the zero case quietly.
|
||||
rest_client = _cascade_rest_client([_Page([])])
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
assert result.id_remap == {}
|
||||
assert result.optimizations_total == 0
|
||||
assert result.optimizations_migrated == 0
|
||||
assert result.optimizations_skipped == 0
|
||||
rest_client.optimizations.create_optimization.assert_not_called()
|
||||
|
||||
def test_one_optimization__fidelity_fields_forwarded_and_remap_populated(
|
||||
self,
|
||||
) -> None:
|
||||
# Full fidelity copy of the fields the BE round-trips:
|
||||
# name, objective_name, status, metadata, studio_config,
|
||||
# last_updated_at. Dest dataset_name + project_name are taken
|
||||
# from the action, NOT from the source row (the source name was
|
||||
# renamed to <name>_v1 by the time the cascade runs).
|
||||
opt = _Optimization(
|
||||
id="src-opt-1",
|
||||
name="overnight-tune",
|
||||
objective_name="f1",
|
||||
status="completed",
|
||||
metadata={"trial_count": 12, "user_note": "kept"},
|
||||
studio_config=None,
|
||||
num_trials=12,
|
||||
baseline_objective_score=0.5,
|
||||
best_objective_score=0.81,
|
||||
total_optimization_cost=4.20,
|
||||
)
|
||||
rest_client = _cascade_rest_client([_Page([opt])])
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
rest_client.optimizations.create_optimization.assert_called_once()
|
||||
kwargs = rest_client.optimizations.create_optimization.call_args.kwargs
|
||||
# Fresh destination id -- not the source id.
|
||||
assert kwargs["id"] != "src-opt-1"
|
||||
assert kwargs["id"] == result.id_remap["src-opt-1"]
|
||||
# Fidelity fields forwarded verbatim.
|
||||
assert kwargs["name"] == "overnight-tune"
|
||||
assert kwargs["objective_name"] == "f1"
|
||||
assert kwargs["status"] == "completed"
|
||||
assert kwargs["metadata"] == {"trial_count": 12, "user_note": "kept"}
|
||||
# Destination scoping comes from the action, not the source row.
|
||||
assert kwargs["dataset_name"] == "MyDataset"
|
||||
assert kwargs["project_name"] == "DestProject"
|
||||
|
||||
def test_readonly_aggregates__not_forwarded_on_create(self) -> None:
|
||||
# ``num_trials`` / ``feedback_scores`` / ``baseline_*`` / ``best_*``
|
||||
# / ``total_optimization_cost`` are recomputed by the BE at read
|
||||
# time from constituent experiments. The cascade must NOT forward
|
||||
# them -- if it did, they'd be silently dropped on write today
|
||||
# but might persist (with stale numbers) under a future BE
|
||||
# schema change.
|
||||
opt = _Optimization(
|
||||
id="src-opt-1",
|
||||
num_trials=99,
|
||||
feedback_scores=[MagicMock()],
|
||||
experiment_scores=[MagicMock()],
|
||||
baseline_objective_score=0.1,
|
||||
best_objective_score=0.9,
|
||||
baseline_duration=10.0,
|
||||
best_duration=2.0,
|
||||
baseline_cost=1.0,
|
||||
best_cost=0.5,
|
||||
total_optimization_cost=42.0,
|
||||
)
|
||||
rest_client = _cascade_rest_client([_Page([opt])])
|
||||
|
||||
cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
kwargs = rest_client.optimizations.create_optimization.call_args.kwargs
|
||||
for forbidden in (
|
||||
"num_trials",
|
||||
"feedback_scores",
|
||||
"experiment_scores",
|
||||
"baseline_objective_score",
|
||||
"best_objective_score",
|
||||
"baseline_duration",
|
||||
"best_duration",
|
||||
"baseline_cost",
|
||||
"best_cost",
|
||||
"total_optimization_cost",
|
||||
):
|
||||
assert forbidden not in kwargs, (
|
||||
f"{forbidden} is BE-computed; cascade must not forward it"
|
||||
)
|
||||
|
||||
def test_empty_optimization__zero_trials__still_migrated(self) -> None:
|
||||
# Optimization with zero constituent experiments is still a
|
||||
# user-visible row in the Optimization Studio UI; migrating it
|
||||
# preserves the user's setup even if no trials ran yet.
|
||||
opt = _Optimization(id="src-opt-1", num_trials=0, name="never-ran")
|
||||
rest_client = _cascade_rest_client([_Page([opt])])
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
assert result.optimizations_migrated == 1
|
||||
assert "src-opt-1" in result.id_remap
|
||||
rest_client.optimizations.create_optimization.assert_called_once()
|
||||
|
||||
def test_multiple_optimizations__each_gets_distinct_destination_id(
|
||||
self,
|
||||
) -> None:
|
||||
opt1 = _Optimization(id="src-opt-1", name="run-a")
|
||||
opt2 = _Optimization(id="src-opt-2", name="run-b")
|
||||
opt3 = _Optimization(id="src-opt-3", name="run-c")
|
||||
rest_client = _cascade_rest_client([_Page([opt1, opt2, opt3])])
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
assert result.optimizations_migrated == 3
|
||||
assert set(result.id_remap.keys()) == {
|
||||
"src-opt-1",
|
||||
"src-opt-2",
|
||||
"src-opt-3",
|
||||
}
|
||||
# Destination ids must be unique (fresh UUIDs).
|
||||
assert len(set(result.id_remap.values())) == 3
|
||||
|
||||
def test_audit__per_optimization_record_emitted(self) -> None:
|
||||
# Each successful migration emits a ``migrate_optimization`` audit
|
||||
# record carrying source_id -> destination_id mapping plus
|
||||
# name / objective_name / status so users can trace what moved.
|
||||
opt = _Optimization(
|
||||
id="src-opt-1",
|
||||
name="overnight",
|
||||
objective_name="accuracy",
|
||||
status="completed",
|
||||
)
|
||||
rest_client = _cascade_rest_client([_Page([opt])])
|
||||
audit = _audit()
|
||||
|
||||
cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=audit,
|
||||
)
|
||||
|
||||
migrate_records = [
|
||||
r for r in audit.actions if r["type"] == "migrate_optimization"
|
||||
]
|
||||
assert len(migrate_records) == 1
|
||||
record = migrate_records[0]
|
||||
# Audit-level status conveys action success/failure -- must not
|
||||
# be shadowed by the optimization's own status.
|
||||
assert record["status"] == "ok"
|
||||
assert record["source_id"] == "src-opt-1"
|
||||
assert record["destination_id"] != "src-opt-1"
|
||||
assert record["name"] == "overnight"
|
||||
assert record["objective_name"] == "accuracy"
|
||||
# Optimization's own status lives under a non-shadowing key.
|
||||
assert record["optimization_status"] == "completed"
|
||||
|
||||
def test_pagination__exhausts_pages_until_short(self) -> None:
|
||||
# find_optimizations is paginated; the cascade walks until a
|
||||
# short page (< page_size). With page_size=100 in production,
|
||||
# two pages of 100 and a final short page of 50 should fire
|
||||
# three reads in order.
|
||||
page_size = optimizations_module._OPTIMIZATION_PAGE_SIZE
|
||||
page_a = _Page([_Optimization(id=f"src-{i}") for i in range(page_size)])
|
||||
page_b = _Page(
|
||||
[_Optimization(id=f"src-{page_size + i}") for i in range(page_size)]
|
||||
)
|
||||
page_c = _Page([_Optimization(id=f"src-{2 * page_size + i}") for i in range(3)])
|
||||
rest_client = _cascade_rest_client([page_a, page_b, page_c])
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
assert result.optimizations_total == 2 * page_size + 3
|
||||
# 3 reads + N writes.
|
||||
assert rest_client.optimizations.find_optimizations.call_count == 3
|
||||
|
||||
def test_create_optimization_raises__exception_propagates(self) -> None:
|
||||
# The BE returns 409 on id conflict (we mint client-side UUIDs
|
||||
# so within-run collisions are infinitesimal but possible across
|
||||
# retries). Surface as a cascade error rather than silently
|
||||
# continuing; the audit log's surrounding ``failed`` entry will
|
||||
# capture partial progress via the umbrella action.
|
||||
opt = _Optimization(id="src-opt-1")
|
||||
rest_client = _cascade_rest_client(
|
||||
[_Page([opt])],
|
||||
create_side_effect=RuntimeError("409 conflict"),
|
||||
)
|
||||
|
||||
with pytest.raises(RuntimeError, match="409 conflict"):
|
||||
cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
def test_source_missing_id__skipped_not_remapped(self) -> None:
|
||||
# Defensive path: BE returns an optimization without an id.
|
||||
# Skip rather than fail -- other optimizations are still
|
||||
# recoverable, and a non-zero ``optimizations_skipped`` counter
|
||||
# in the audit surfaces the malformed row for investigation.
|
||||
ok = _Optimization(id="src-opt-1")
|
||||
bad = _Optimization(id="")
|
||||
rest_client = _cascade_rest_client([_Page([ok, bad])])
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-dataset-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
|
||||
assert result.optimizations_migrated == 1
|
||||
assert result.optimizations_skipped == 1
|
||||
assert list(result.id_remap.keys()) == ["src-opt-1"]
|
||||
|
||||
|
||||
class TestCascadeOptimizationsWithExperiments:
|
||||
"""End-to-end planner-shape assertion: the optimization remap built
|
||||
by Slice 5 is observable to Slice 3's experiment cascade via the
|
||||
shared ``MigrationPlan``. The actual experiment-cascade FK forwarding
|
||||
is asserted in ``test_migrate_dataset_experiments_cascade``; this
|
||||
test only pins the wiring (action ordering + remap propagation).
|
||||
"""
|
||||
|
||||
def test_plan_remap__populated_by_cascade__readable_by_subsequent_actions(
|
||||
self,
|
||||
) -> None:
|
||||
from opik.cli.migrate.datasets.planner import MigrationPlan
|
||||
from opik.cli.migrate.datasets.resolver import ResolvedDataset
|
||||
|
||||
opt = _Optimization(id="src-opt-1", name="trained")
|
||||
rest_client = _cascade_rest_client([_Page([opt])])
|
||||
plan = MigrationPlan(
|
||||
source=ResolvedDataset(
|
||||
id="src-1",
|
||||
name="MyDataset",
|
||||
project_name=None,
|
||||
description=None,
|
||||
visibility=None,
|
||||
tags=None,
|
||||
type="dataset",
|
||||
),
|
||||
target_name="MyDataset",
|
||||
to_project="DestProject",
|
||||
)
|
||||
|
||||
result = cascade_optimizations(
|
||||
rest_client,
|
||||
source_dataset_id="src-1",
|
||||
target_dataset_name="MyDataset",
|
||||
target_project_name="DestProject",
|
||||
audit=_audit(),
|
||||
)
|
||||
plan.optimization_id_remap.update(result.id_remap)
|
||||
|
||||
# Subsequent CascadeExperiments would read plan.optimization_id_remap;
|
||||
# the wiring works as long as result.id_remap propagates here.
|
||||
assert plan.optimization_id_remap == result.id_remap
|
||||
assert "src-opt-1" in plan.optimization_id_remap
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Studio config Read -> Write reconstruction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStudioConfigReconstruction:
|
||||
"""The Fern-generated Read and Write variants of
|
||||
``OptimizationStudioConfig`` are structurally identical but
|
||||
nominally distinct; the cascade has to reconstruct the Write
|
||||
variant from the Read variant so the typed ``create_optimization``
|
||||
parameter is satisfied. Pin the round-trip so any future Fern
|
||||
regeneration that changes the shape on one side gets caught.
|
||||
"""
|
||||
|
||||
def test_none_in_none_out(self) -> None:
|
||||
from opik.cli.migrate.datasets.optimizations import (
|
||||
_studio_config_public_to_write,
|
||||
)
|
||||
|
||||
assert _studio_config_public_to_write(None) is None
|
||||
|
||||
def test_round_trip__fields_preserved(self) -> None:
|
||||
from opik.cli.migrate.datasets.optimizations import (
|
||||
_studio_config_public_to_write,
|
||||
)
|
||||
from opik.rest_api.types.optimization_studio_config_public import (
|
||||
OptimizationStudioConfigPublic,
|
||||
)
|
||||
from opik.rest_api.types.studio_evaluation_public import (
|
||||
StudioEvaluationPublic,
|
||||
)
|
||||
from opik.rest_api.types.studio_llm_model_public import (
|
||||
StudioLlmModelPublic,
|
||||
)
|
||||
from opik.rest_api.types.studio_message_public import StudioMessagePublic
|
||||
from opik.rest_api.types.studio_metric_public import StudioMetricPublic
|
||||
from opik.rest_api.types.studio_optimizer_public import (
|
||||
StudioOptimizerPublic,
|
||||
)
|
||||
from opik.rest_api.types.studio_prompt_public import StudioPromptPublic
|
||||
|
||||
public = OptimizationStudioConfigPublic(
|
||||
dataset_name="MyDataset",
|
||||
prompt=StudioPromptPublic(
|
||||
messages=[StudioMessagePublic(role="system", content="you are helpful")]
|
||||
),
|
||||
llm_model=StudioLlmModelPublic(
|
||||
model="gpt-4o", parameters={"temperature": 0.2}
|
||||
),
|
||||
evaluation=StudioEvaluationPublic(
|
||||
metrics=[StudioMetricPublic(type="accuracy")]
|
||||
),
|
||||
optimizer=StudioOptimizerPublic(type="grid", parameters={"steps": 5}),
|
||||
)
|
||||
write = _studio_config_public_to_write(public)
|
||||
assert write is not None
|
||||
# Round-trip via model_dump -- the typed Write variant accepts
|
||||
# extra fields so this also covers any forward-compatible BE
|
||||
# additions on the Public variant.
|
||||
write_dump: Dict[str, Any] = write.model_dump()
|
||||
public_dump: Dict[str, Any] = public.model_dump()
|
||||
assert write_dump == public_dump
|
||||
@@ -0,0 +1,950 @@
|
||||
"""Planner + CLI-help tests for ``opik migrate dataset``.
|
||||
|
||||
The planner cases (conflict, project-not-found, default flow ordering)
|
||||
live here; the meaty version-replay tests live in
|
||||
``test_migrate_dataset_version_replay.py`` and the cascade tests live in
|
||||
``test_migrate_dataset_experiments_cascade.py``.
|
||||
|
||||
Shared helpers (``_DatasetRow``, ``_Page``, ``_planner_rest_client``)
|
||||
come from ``_migrate_helpers``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli import cli
|
||||
from opik.cli.migrate.audit import AuditLog
|
||||
from opik.cli.migrate.checkpoint import MigrationCheckpoint
|
||||
from opik.cli.migrate.datasets import planner as planner_module
|
||||
from opik.cli.migrate.datasets.planner import TEMP_MIGRATION_MARKER_TAG
|
||||
from opik.cli.migrate.datasets.resume import ReconstructedRemaps
|
||||
from opik.cli.migrate.errors import (
|
||||
ConflictError,
|
||||
DatasetNotFoundError,
|
||||
MigrationError,
|
||||
ProjectNotFoundError,
|
||||
)
|
||||
|
||||
from ._migrate_helpers import (
|
||||
_build_fake_client,
|
||||
_DatasetRow,
|
||||
_Page,
|
||||
_planner_client,
|
||||
_planner_rest_client,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Elapsed-time formatter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFormatElapsed:
|
||||
"""Pin the wall-clock duration renderer used in the migrate success /
|
||||
failure lines. Sub-minute → one decimal of seconds; past a minute →
|
||||
integer ``Mm Ss`` or ``Hh Mm Ss`` (no fractional seconds).
|
||||
"""
|
||||
|
||||
def test_sub_minute__one_decimal_seconds(self) -> None:
|
||||
from opik.cli.migrate.main import _format_elapsed
|
||||
|
||||
assert _format_elapsed(0.0) == "0.0s"
|
||||
assert _format_elapsed(12.34) == "12.3s"
|
||||
assert _format_elapsed(59.99) == "60.0s"
|
||||
|
||||
def test_minute_range__integer_m_s(self) -> None:
|
||||
from opik.cli.migrate.main import _format_elapsed
|
||||
|
||||
assert _format_elapsed(60.0) == "1m 0s"
|
||||
assert _format_elapsed(125.7) == "2m 5s"
|
||||
|
||||
def test_hour_range__integer_h_m_s(self) -> None:
|
||||
from opik.cli.migrate.main import _format_elapsed
|
||||
|
||||
assert _format_elapsed(3600.0) == "1h 0m 0s"
|
||||
assert _format_elapsed(3725.0) == "1h 2m 5s"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OPIK-6599: loud-fail on skipped items
|
||||
#
|
||||
# When the cascade emits any ``skip`` audit record, the migrate must:
|
||||
# 1. Finalize the audit log to ``failed`` (not ``ok``)
|
||||
# 2. Print a SKIP_SUMMARY line to stderr (not stdout) so CI gates can
|
||||
# grep without parsing the JSON
|
||||
# 3. Exit non-zero
|
||||
#
|
||||
# Tests below cover ``_finalize_with_skips_or_ok`` directly so they don't
|
||||
# need a fully-wired Opik client + REST server stub. The CLI is only the
|
||||
# wrapper around this helper.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFinalizeWithSkipsOrOk:
|
||||
def _make_audit_with_skips(self) -> AuditLog:
|
||||
audit = AuditLog(command="opik migrate dataset", args={})
|
||||
audit.record(
|
||||
type="skip",
|
||||
status="skipped",
|
||||
details={
|
||||
"reason": "items_missing_dataset_item_remap",
|
||||
"experiment_id": "src-exp-1",
|
||||
"experiment_name": "exp-1",
|
||||
"count": 2500,
|
||||
"sample_source_ids": ["src-ds-item-1", "src-ds-item-2"],
|
||||
},
|
||||
)
|
||||
return audit
|
||||
|
||||
def test_skips_present__finalizes_failed_exits_1_stderr_summary(
|
||||
self, tmp_path, capsys
|
||||
) -> None:
|
||||
from opik.cli.migrate.main import _finalize_with_skips_or_ok
|
||||
|
||||
audit = self._make_audit_with_skips()
|
||||
audit_path = tmp_path / "audit.json"
|
||||
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
_finalize_with_skips_or_ok(
|
||||
audit,
|
||||
audit_path,
|
||||
name="MyDataset",
|
||||
target_label="MyDataset",
|
||||
target_project="DestProject",
|
||||
elapsed_seconds=12.3,
|
||||
)
|
||||
|
||||
# AC 1: non-zero exit code
|
||||
assert exc.value.code == 1
|
||||
|
||||
captured = capsys.readouterr()
|
||||
# AC 3: skip message on stderr (not stdout), with the
|
||||
# machine-parseable SKIP_SUMMARY suffix
|
||||
assert "SKIP_SUMMARY:" in captured.err
|
||||
assert "items_skipped_missing_item=2500" in captured.err
|
||||
assert "experiments_skipped=0" in captured.err
|
||||
assert "items_skipped_missing_trace=0" in captured.err
|
||||
assert "NOT rolled back" in captured.err
|
||||
# Rollback hint names the entities the operator must remove,
|
||||
# the destination project, and the rename-back step on the source.
|
||||
assert "roll back manually" in captured.err
|
||||
assert "DestProject" in captured.err
|
||||
assert "MyDataset_v1" in captured.err
|
||||
|
||||
# AC 2: audit finalized to failed with skip record intact
|
||||
on_disk = json.loads(audit_path.read_text())
|
||||
assert on_disk["status"] == "failed"
|
||||
assert any(a.get("status") == "skipped" for a in on_disk["actions"])
|
||||
|
||||
def test_no_skips__finalizes_ok_no_exit_no_stderr(self, tmp_path, capsys) -> None:
|
||||
from opik.cli.migrate.main import _finalize_with_skips_or_ok
|
||||
|
||||
audit = AuditLog(command="opik migrate dataset", args={})
|
||||
audit.record(type="rename_source", status="ok", details={})
|
||||
audit_path = tmp_path / "audit.json"
|
||||
|
||||
# Happy path returns without raising; happy-path message goes to
|
||||
# stdout, stderr stays clean.
|
||||
_finalize_with_skips_or_ok(
|
||||
audit,
|
||||
audit_path,
|
||||
name="MyDataset",
|
||||
target_label="MyDataset",
|
||||
target_project="DestProject",
|
||||
elapsed_seconds=5.0,
|
||||
)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "SKIP_SUMMARY:" not in captured.err
|
||||
on_disk = json.loads(audit_path.read_text())
|
||||
assert on_disk["status"] == "ok"
|
||||
|
||||
def test_multiple_skip_records__totals_aggregated_by_reason(
|
||||
self, tmp_path, capsys
|
||||
) -> None:
|
||||
from opik.cli.migrate.main import _finalize_with_skips_or_ok
|
||||
|
||||
# Two experiments, each contributing skips for both reasons.
|
||||
audit = AuditLog(command="opik migrate dataset", args={})
|
||||
for exp_id in ("src-exp-1", "src-exp-2"):
|
||||
audit.record(
|
||||
type="skip",
|
||||
status="skipped",
|
||||
details={
|
||||
"reason": "items_missing_trace_remap",
|
||||
"experiment_id": exp_id,
|
||||
"experiment_name": exp_id,
|
||||
"count": 7,
|
||||
"sample_source_ids": [],
|
||||
},
|
||||
)
|
||||
audit.record(
|
||||
type="skip",
|
||||
status="skipped",
|
||||
details={
|
||||
"reason": "items_missing_dataset_item_remap",
|
||||
"experiment_id": exp_id,
|
||||
"experiment_name": exp_id,
|
||||
"count": 3,
|
||||
"sample_source_ids": [],
|
||||
},
|
||||
)
|
||||
|
||||
with pytest.raises(SystemExit):
|
||||
_finalize_with_skips_or_ok(
|
||||
audit,
|
||||
tmp_path / "audit.json",
|
||||
name="MyDataset",
|
||||
target_label="MyDataset",
|
||||
target_project="DestProject",
|
||||
elapsed_seconds=1.0,
|
||||
)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
# 7 + 7 = 14 trace skips, 3 + 3 = 6 dataset-item skips
|
||||
assert "items_skipped_missing_trace=14" in captured.err
|
||||
assert "items_skipped_missing_item=6" in captured.err
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Help text
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMigrateHelp:
|
||||
def test_migrate_group__help_invoked__lists_subcommands(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["migrate", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "Migrate Opik entities" in result.output
|
||||
assert "dataset" in result.output
|
||||
|
||||
def test_migrate_dataset__help_invoked__lists_required_flags(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["migrate", "dataset", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "--to-project" in result.output
|
||||
assert "--from-project" in result.output
|
||||
assert "--dry-run" in result.output
|
||||
|
||||
def test_migrate_dataset__help_invoked__lists_exclude_experiments(self) -> None:
|
||||
# OPIK-7161 AC: the opt-out flag must be discoverable in --help.
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["migrate", "dataset", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "--exclude-experiments" in result.output
|
||||
|
||||
def test_migrate_dataset__exclude_experiments__cli_run_skips_and_reports(
|
||||
self, tmp_path
|
||||
) -> None:
|
||||
# OPIK-7161: exercise the flag through the public Click entrypoint,
|
||||
# not just the finalize helper, so the option -> build_dataset_plan
|
||||
# -> finalize plumbing in migrate_dataset_command is covered end to
|
||||
# end (per .agents/skills/python-sdk/testing.md: test the public API).
|
||||
# The fake client mocks the whole rename/create/replay surface; with
|
||||
# --exclude-experiments the plan carries no cascade actions, so the
|
||||
# command reaches the success finalize with zero experiment work.
|
||||
client, _, _ = _build_fake_client(
|
||||
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
|
||||
destination_rows=[],
|
||||
items=[{"id": "item-a", "input": "hello"}],
|
||||
)
|
||||
audit_path = tmp_path / "audit.json"
|
||||
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.migrate.main._build_client", return_value=client):
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"migrate",
|
||||
"dataset",
|
||||
"MyDataset",
|
||||
"--to-project",
|
||||
"B",
|
||||
"--exclude-experiments",
|
||||
"--audit-log",
|
||||
str(audit_path),
|
||||
],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
# User-facing output makes the intentional skip clear.
|
||||
assert "--exclude-experiments" in result.output
|
||||
assert "skipped" in result.output
|
||||
# No experiment cascade ran: the source dataset was never queried for
|
||||
# experiments (find_experiments belongs only to the cascade path).
|
||||
assert client.rest_client.experiments.find_experiments.call_count == 0
|
||||
# Audit log finalized ok and recorded the flag in its args.
|
||||
on_disk = json.loads(audit_path.read_text())
|
||||
assert on_disk["status"] == "ok"
|
||||
assert on_disk["args"]["exclude_experiments"] is True
|
||||
cascade_types = {a.get("type") for a in on_disk["actions"]}
|
||||
assert "cascade_experiments" not in cascade_types
|
||||
assert "cascade_optimizations" not in cascade_types
|
||||
|
||||
|
||||
class TestTempDestRenameOnSuccess:
|
||||
"""OPIK-7162 acceptance criteria, exercised through the public Click
|
||||
entrypoint: the source keeps its name until the copy succeeds, then the
|
||||
handoff runs (source -> _v1, temp -> original). A mid-run failure leaves
|
||||
the source name untouched, and a re-run after failure is safe/idempotent.
|
||||
|
||||
All cases use ``--exclude-experiments`` so the plan is the minimal
|
||||
Create -> Replay -> Rename -> Promote shape and the assertions stay
|
||||
focused on the handoff, not the cascade.
|
||||
"""
|
||||
|
||||
def _run(self, client, tmp_path, extra_args=()):
|
||||
audit_path = tmp_path / "audit.json"
|
||||
runner = CliRunner()
|
||||
with patch("opik.cli.migrate.main._build_client", return_value=client):
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"migrate",
|
||||
"dataset",
|
||||
"MyDataset",
|
||||
"--to-project",
|
||||
"B",
|
||||
"--exclude-experiments",
|
||||
"--audit-log",
|
||||
str(audit_path),
|
||||
*extra_args,
|
||||
],
|
||||
)
|
||||
return result, audit_path
|
||||
|
||||
def test_success__source_renamed_to_v1_and_dest_promoted_to_original(
|
||||
self, tmp_path
|
||||
) -> None:
|
||||
# AC: a successful migration leaves the destination under the
|
||||
# original name and the source under the _v1 suffix. The two renames
|
||||
# happen via update_dataset PUTs; the destination is created under
|
||||
# the temp name first.
|
||||
client, _, _ = _build_fake_client(
|
||||
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
|
||||
destination_rows=[],
|
||||
items=[{"id": "item-a", "input": "hello"}],
|
||||
)
|
||||
result, audit_path = self._run(client, tmp_path)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
rest = client.rest_client
|
||||
# Destination created under the temp name (not the final name), and
|
||||
# stamped with the migration marker tag so a future re-run can prove
|
||||
# it's ours before discarding it.
|
||||
create_kwargs = rest.datasets.create_dataset.call_args.kwargs
|
||||
assert create_kwargs["name"] == "MyDataset__migrating"
|
||||
assert TEMP_MIGRATION_MARKER_TAG in (create_kwargs["tags"] or [])
|
||||
# Two rename PUTs: source -> _v1, then temp -> original. The promote PUT
|
||||
# re-passes the source's ORIGINAL tags (marker stripped).
|
||||
rename_calls = [c.kwargs for c in rest.datasets.update_dataset.call_args_list]
|
||||
source_rename = next(c for c in rename_calls if c["id"] == "src-1")
|
||||
assert source_rename["name"] == "MyDataset_v1"
|
||||
promote = next(c for c in rename_calls if c.get("name") == "MyDataset")
|
||||
assert promote["name"] == "MyDataset"
|
||||
# The promote PUT must pass tags as an EXPLICIT list (never None), so the
|
||||
# BE actually overwrites and drops the marker. A live backend treats
|
||||
# ``tags=None`` as "leave unchanged", which would strand the marker on a
|
||||
# source with no tags — so assert the concrete list, not just marker
|
||||
# absence. Source here has no tags -> promote clears with [].
|
||||
assert promote["tags"] == []
|
||||
assert TEMP_MIGRATION_MARKER_TAG not in promote["tags"]
|
||||
# Audit ends ok and records the handoff actions in order.
|
||||
on_disk = json.loads(audit_path.read_text())
|
||||
assert on_disk["status"] == "ok"
|
||||
ok_types = [a["type"] for a in on_disk["actions"] if a.get("status") == "ok"]
|
||||
assert ok_types.index("rename_source") < ok_types.index("promote_destination")
|
||||
assert ok_types.index("create_destination") < ok_types.index("rename_source")
|
||||
|
||||
def test_success__source_tags_preserved_marker_stripped(self, tmp_path) -> None:
|
||||
# When the source has real tags, the temp create adds the marker
|
||||
# alongside them, and the promote re-passes exactly the source's
|
||||
# originals (marker dropped, real tags kept).
|
||||
client, _, _ = _build_fake_client(
|
||||
source_rows=[
|
||||
_DatasetRow(id="src-1", name="MyDataset", tags=["team-a", "prod"])
|
||||
],
|
||||
destination_rows=[],
|
||||
items=[{"id": "item-a", "input": "hello"}],
|
||||
)
|
||||
result, _ = self._run(client, tmp_path)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
rest = client.rest_client
|
||||
create_tags = rest.datasets.create_dataset.call_args.kwargs["tags"]
|
||||
assert set(create_tags) == {"team-a", "prod", TEMP_MIGRATION_MARKER_TAG}
|
||||
promote = next(
|
||||
c.kwargs
|
||||
for c in rest.datasets.update_dataset.call_args_list
|
||||
if c.kwargs.get("name") == "MyDataset"
|
||||
)
|
||||
assert promote["tags"] == ["team-a", "prod"]
|
||||
assert TEMP_MIGRATION_MARKER_TAG not in promote["tags"]
|
||||
|
||||
def test_midrun_failure__source_name_untouched(self, tmp_path) -> None:
|
||||
# AC: a migration interrupted mid-run leaves the source name
|
||||
# untouched. Blow up the destination create (the first copy action);
|
||||
# the source-rename PUT must never fire.
|
||||
client, _, _ = _build_fake_client(
|
||||
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
|
||||
destination_rows=[],
|
||||
items=[{"id": "item-a", "input": "hello"}],
|
||||
)
|
||||
client.rest_client.datasets.create_dataset.side_effect = RuntimeError(
|
||||
"boom mid-copy"
|
||||
)
|
||||
result, audit_path = self._run(client, tmp_path)
|
||||
|
||||
assert result.exit_code == 1
|
||||
# No update_dataset PUT touched the source id -> its name is intact.
|
||||
source_touched = [
|
||||
c
|
||||
for c in client.rest_client.datasets.update_dataset.call_args_list
|
||||
if c.kwargs.get("id") == "src-1"
|
||||
]
|
||||
assert source_touched == []
|
||||
on_disk = json.loads(audit_path.read_text())
|
||||
assert on_disk["status"] == "failed"
|
||||
# The handoff actions never reached ``ok``.
|
||||
ok_types = {a["type"] for a in on_disk["actions"] if a.get("status") == "ok"}
|
||||
assert "rename_source" not in ok_types
|
||||
assert "promote_destination" not in ok_types
|
||||
|
||||
def test_rerun_after_failure__discards_stale_temp_then_completes(
|
||||
self, tmp_path
|
||||
) -> None:
|
||||
# AC: re-running after an interrupted run completes with no manual
|
||||
# cleanup. A stale ``MyDataset__migrating`` from the prior failed run —
|
||||
# carrying the migration marker tag that proves it's ours — is
|
||||
# discovered and deleted before the destination is recreated.
|
||||
client, _, _ = _build_fake_client(
|
||||
source_rows=[_DatasetRow(id="src-1", name="MyDataset")],
|
||||
destination_rows=[],
|
||||
items=[{"id": "item-a", "input": "hello"}],
|
||||
stale_temp_rows=[
|
||||
_DatasetRow(
|
||||
id="stale-1",
|
||||
name="MyDataset__migrating",
|
||||
tags=[TEMP_MIGRATION_MARKER_TAG],
|
||||
)
|
||||
],
|
||||
)
|
||||
result, audit_path = self._run(client, tmp_path)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
# The stale temp was deleted by id before recreate.
|
||||
client.rest_client.datasets.delete_dataset.assert_called_once_with(id="stale-1")
|
||||
on_disk = json.loads(audit_path.read_text())
|
||||
assert on_disk["status"] == "ok"
|
||||
action_types = [
|
||||
a["type"] for a in on_disk["actions"] if a.get("status") == "ok"
|
||||
]
|
||||
assert action_types.index("discard_stale_temp") < action_types.index(
|
||||
"create_destination"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Planner unit tests (no Click invocation)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPlanBuilding:
|
||||
def test_build_dataset_plan__default_flow__orders_create_replay_cascades_then_handoff(
|
||||
self,
|
||||
) -> None:
|
||||
# OPIK-7162: the plan builds the destination under a temp name FIRST
|
||||
# (source keeps its name), runs the copy + cascades, then does the
|
||||
# name handoff LAST: rename source -> <name>_v1, promote temp ->
|
||||
# <name>. The order is load-bearing on two axes:
|
||||
# * CascadeOptimizations before CascadeExperiments (opt-id remap).
|
||||
# * RenameSource before PromoteDestination (source-away then
|
||||
# destination-in, so <name> is never held by two rows at once).
|
||||
# Three find_datasets pages: source resolve, _v1 collision check,
|
||||
# __migrating stale-temp lookup.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset", description="d")]),
|
||||
_Page([]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == [
|
||||
"CreateDestination",
|
||||
"ReplayVersions",
|
||||
"CascadeOptimizations",
|
||||
"CascadeExperiments",
|
||||
"RenameSource",
|
||||
"PromoteDestination",
|
||||
]
|
||||
# Destination is created under the temp name, not the final name.
|
||||
create = plan.actions[0]
|
||||
assert create.name == "MyDataset__migrating"
|
||||
replay = plan.actions[1]
|
||||
assert replay.source_name == "MyDataset"
|
||||
assert replay.dest_name == "MyDataset__migrating"
|
||||
# Handoff: source away first, destination in second.
|
||||
rename = plan.actions[4]
|
||||
assert rename.from_name == "MyDataset"
|
||||
assert rename.to_name == "MyDataset_v1"
|
||||
promote = plan.actions[5]
|
||||
assert promote.from_name == "MyDataset__migrating"
|
||||
assert promote.to_name == "MyDataset"
|
||||
assert plan.target_name == "MyDataset"
|
||||
# New remap dict starts empty; _cascade_optimizations populates it.
|
||||
assert plan.optimization_id_remap == {}
|
||||
|
||||
def test_build_dataset_plan__exclude_experiments__omits_both_cascades(
|
||||
self,
|
||||
) -> None:
|
||||
# OPIK-7161: --exclude-experiments drops the experiment stage AND
|
||||
# the optimization stage (optimizations are containers for the
|
||||
# skipped experiments). No cascade actions, but the name handoff
|
||||
# (rename + promote) still runs after the dataset + versions copy.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset", description="d")]),
|
||||
_Page([]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
exclude_experiments=True,
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == [
|
||||
"CreateDestination",
|
||||
"ReplayVersions",
|
||||
"RenameSource",
|
||||
"PromoteDestination",
|
||||
]
|
||||
assert not any(
|
||||
isinstance(a, planner_module.CascadeExperiments) for a in plan.actions
|
||||
)
|
||||
assert not any(
|
||||
isinstance(a, planner_module.CascadeOptimizations) for a in plan.actions
|
||||
)
|
||||
|
||||
def test_build_dataset_plan__exclude_experiments_default_false__keeps_cascades(
|
||||
self,
|
||||
) -> None:
|
||||
# Default (flag off) is unchanged: both cascades still emitted.
|
||||
# Guards the opt-out default so a plain migrate never silently
|
||||
# starts skipping experiments.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
|
||||
_Page([]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == [
|
||||
"CreateDestination",
|
||||
"ReplayVersions",
|
||||
"CascadeOptimizations",
|
||||
"CascadeExperiments",
|
||||
"RenameSource",
|
||||
"PromoteDestination",
|
||||
]
|
||||
|
||||
def test_build_dataset_plan__test_suite__type_forwarded_to_destination(
|
||||
self,
|
||||
) -> None:
|
||||
# Test suites flow through the same plan shape as plain datasets;
|
||||
# the only difference is ``CreateDestination.type`` being forwarded
|
||||
# so the target accepts suite-level evaluators + execution_policy
|
||||
# via ``ReplayVersions``.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page(
|
||||
[_DatasetRow(id="src-1", name="MySuite", type="evaluation_suite")]
|
||||
),
|
||||
_Page([]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MySuite",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == [
|
||||
"CreateDestination",
|
||||
"ReplayVersions",
|
||||
"CascadeOptimizations",
|
||||
"CascadeExperiments",
|
||||
"RenameSource",
|
||||
"PromoteDestination",
|
||||
]
|
||||
replay = plan.actions[1]
|
||||
assert replay.is_test_suite is True
|
||||
|
||||
def test_build_dataset_plan__rename_target_collides_workspace_wide__raises_conflict(
|
||||
self,
|
||||
) -> None:
|
||||
# The eventual source-rename target "<source>_v1" collides with
|
||||
# another dataset in the workspace — caught up-front so a doomed run
|
||||
# never does any copy work.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
|
||||
_Page([_DatasetRow(id="other-1", name="MyDataset_v1")]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
with pytest.raises(ConflictError) as exc_info:
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
assert "MyDataset_v1" in str(exc_info.value)
|
||||
|
||||
def test_build_dataset_plan__rename_target_match_is_source_itself__no_conflict(
|
||||
self,
|
||||
) -> None:
|
||||
# When find_datasets returns the source itself for the _v1 check, we
|
||||
# must not treat that as a collision — it's about to be renamed.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset_v1")]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
# Should NOT raise: the only "match" is the source dataset itself.
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
assert plan.target_name == "MyDataset"
|
||||
|
||||
def test_build_dataset_plan__marked_stale_temp__prepends_discard_action(
|
||||
self,
|
||||
) -> None:
|
||||
# OPIK-7162 safe re-run: a leftover "<name>__migrating" carrying the
|
||||
# migration marker tag (proof it's ours) is detected and a
|
||||
# DiscardStaleTemp action is prepended so the re-run starts clean.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
|
||||
_Page([]),
|
||||
_Page(
|
||||
[
|
||||
_DatasetRow(
|
||||
id="stale-1",
|
||||
name="MyDataset__migrating",
|
||||
tags=[TEMP_MIGRATION_MARKER_TAG],
|
||||
)
|
||||
]
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == [
|
||||
"DiscardStaleTemp",
|
||||
"CreateDestination",
|
||||
"ReplayVersions",
|
||||
"CascadeOptimizations",
|
||||
"CascadeExperiments",
|
||||
"RenameSource",
|
||||
"PromoteDestination",
|
||||
]
|
||||
discard = plan.actions[0]
|
||||
assert discard.temp_id == "stale-1"
|
||||
assert discard.temp_name == "MyDataset__migrating"
|
||||
|
||||
def test_build_dataset_plan__unmarked_name_collision__raises_conflict(
|
||||
self,
|
||||
) -> None:
|
||||
# A dataset named "<name>__migrating" WITHOUT the migration marker is a
|
||||
# real user dataset that merely shares the name — it must NOT be
|
||||
# deleted. The planner aborts with ConflictError instead.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
|
||||
_Page([]),
|
||||
_Page([_DatasetRow(id="user-1", name="MyDataset__migrating")]),
|
||||
]
|
||||
)
|
||||
|
||||
with pytest.raises(ConflictError) as exc_info:
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
assert "MyDataset__migrating" in str(exc_info.value)
|
||||
assert TEMP_MIGRATION_MARKER_TAG in str(exc_info.value)
|
||||
|
||||
def test_build_dataset_plan__same_from_and_to_project_flag__raises_conflict(
|
||||
self,
|
||||
) -> None:
|
||||
# Cheap early-out: user literally passed --from-project A --to-project A.
|
||||
# Rejected before any lookup.
|
||||
rest_client = _planner_rest_client(
|
||||
[_Page([_DatasetRow(id="src-1", name="MyDataset")])]
|
||||
)
|
||||
|
||||
with pytest.raises(ConflictError, match="same project"):
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="A",
|
||||
from_project="A",
|
||||
)
|
||||
|
||||
def test_build_dataset_plan__omitted_flag_source_in_dest_project__raises_conflict(
|
||||
self,
|
||||
) -> None:
|
||||
# The gap the flag-only check missed: --from-project is OMITTED, but the
|
||||
# source actually lives in the destination project. resolve_source
|
||||
# populates source.project_name from the row's project_id, so the
|
||||
# authoritative post-resolve guard still catches it.
|
||||
source_row = _DatasetRow(id="src-1", name="MyDataset", project_id="proj-A")
|
||||
rest_client = _planner_rest_client([_Page([source_row])])
|
||||
# project_name_for_row -> client.get_project(id="proj-A").name == "A".
|
||||
proj = MagicMock()
|
||||
proj.name = "A"
|
||||
rest_client.projects.get_project_by_id.return_value = proj
|
||||
|
||||
with pytest.raises(ConflictError, match="same project"):
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="A",
|
||||
)
|
||||
|
||||
def test_build_dataset_plan__workspace_scoped_source__no_same_project_abort(
|
||||
self,
|
||||
) -> None:
|
||||
# A workspace-scoped source (no project_id -> project_name is None) has
|
||||
# no single project to collide with --to-project, so a workspace-scoped
|
||||
# -> project migrate is legitimate and must NOT be blocked.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset", project_id=None)]),
|
||||
_Page([]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="A",
|
||||
)
|
||||
assert plan.target_name == "MyDataset"
|
||||
|
||||
def test_build_dataset_plan__no_stale_temp__no_discard_action(self) -> None:
|
||||
# The common case: no leftover temp, so no DiscardStaleTemp emitted.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_DatasetRow(id="src-1", name="MyDataset")]),
|
||||
_Page([]),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
assert not any(
|
||||
isinstance(a, planner_module.DiscardStaleTemp) for a in plan.actions
|
||||
)
|
||||
|
||||
def test_build_dataset_plan__source_name_not_found__raises_dataset_not_found(
|
||||
self,
|
||||
) -> None:
|
||||
rest_client = _planner_rest_client([_Page([])])
|
||||
|
||||
with pytest.raises(DatasetNotFoundError) as exc_info:
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="Missing",
|
||||
to_project="B",
|
||||
)
|
||||
assert "Missing" in str(exc_info.value)
|
||||
|
||||
def test_build_dataset_plan__source_name_resolves_to_many__raises_conflict(
|
||||
self,
|
||||
) -> None:
|
||||
# Workspace uniqueness is enforced by the BE (UNIQUE
|
||||
# (workspace_id, name)); if the BE invariant is somehow
|
||||
# violated, surface it as ConflictError rather than silently
|
||||
# picking a row.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page(
|
||||
[
|
||||
_DatasetRow(id="a", name="MyDataset"),
|
||||
_DatasetRow(id="b", name="MyDataset"),
|
||||
]
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
with pytest.raises(ConflictError):
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
def test_build_dataset_plan__destination_project_missing__raises_project_not_found(
|
||||
self,
|
||||
) -> None:
|
||||
rest_client = _planner_rest_client(
|
||||
find_side_effects=[],
|
||||
target_project_exists=False,
|
||||
)
|
||||
|
||||
with pytest.raises(ProjectNotFoundError) as exc_info:
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="DoesNotExist",
|
||||
)
|
||||
assert "DoesNotExist" in str(exc_info.value)
|
||||
|
||||
def test_build_dataset_plan__destination_project_missing__suggests_similar_names(
|
||||
self,
|
||||
) -> None:
|
||||
rest_client = _planner_rest_client(
|
||||
find_side_effects=[],
|
||||
target_project_exists=False,
|
||||
workspace_project_names=["production", "staging", "Beat", "Best"],
|
||||
)
|
||||
|
||||
with pytest.raises(ProjectNotFoundError) as exc_info:
|
||||
planner_module.build_dataset_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyDataset",
|
||||
to_project="Beta",
|
||||
)
|
||||
message = str(exc_info.value)
|
||||
assert "Beta" in message
|
||||
# difflib should surface the close one-letter neighbours.
|
||||
assert "Did you mean" in message
|
||||
assert "Beat" in message or "Best" in message
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OPIK-7162 + OPIK-7168 integration: resume reuses the temp destination and
|
||||
# finishes the pending handoff (rename source -> _v1, promote temp -> original).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestBuildResumePlanTempDest:
|
||||
def _checkpoint(self) -> MigrationCheckpoint:
|
||||
return MigrationCheckpoint(
|
||||
key="k",
|
||||
workspace="ws",
|
||||
project="B",
|
||||
dataset="MyDataset",
|
||||
path=Path("/tmp/does-not-matter.json"),
|
||||
dataset_phase_done=True,
|
||||
source_dataset_id="src-1",
|
||||
source_name="MyDataset",
|
||||
temp_dest_name="MyDataset__migrating",
|
||||
)
|
||||
|
||||
def _resume_client(self, source_id: str = "src-1") -> MagicMock:
|
||||
# resolve_source(MyDataset) -> the still-unrenamed source;
|
||||
# get_dataset(MyDataset__migrating) -> the temp destination.
|
||||
rest_client = _planner_rest_client(
|
||||
[_Page([_DatasetRow(id=source_id, name="MyDataset")])]
|
||||
)
|
||||
client = _planner_client(rest_client)
|
||||
dest = MagicMock()
|
||||
dest.id = "temp-dest-1"
|
||||
client.get_dataset = MagicMock(return_value=dest)
|
||||
return client
|
||||
|
||||
def test_resume__reuses_temp_and_appends_cascade_then_handoff(self) -> None:
|
||||
# A dataset_phase_done checkpoint means create-temp/replay/optimizations
|
||||
# already ran into MyDataset__migrating and the source still holds its
|
||||
# original name. The resume plan must NOT re-create or re-replay; it
|
||||
# resolves the temp destination, then emits the pending tail:
|
||||
# CascadeExperiments -> RenameSource -> PromoteDestination.
|
||||
with patch.object(
|
||||
planner_module, "reconstruct_remaps", return_value=ReconstructedRemaps()
|
||||
):
|
||||
plan = planner_module.build_dataset_plan(
|
||||
client=self._resume_client(),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
resume_checkpoint=self._checkpoint(),
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == ["CascadeExperiments", "RenameSource", "PromoteDestination"]
|
||||
assert plan.is_resume is True
|
||||
# The cascade + promote target the TEMP destination (promote hasn't run
|
||||
# yet); the source rename moves the original name to _v1.
|
||||
cascade = plan.actions[0]
|
||||
assert cascade.dest_name == "MyDataset__migrating"
|
||||
rename = plan.actions[1]
|
||||
assert rename.from_name == "MyDataset"
|
||||
assert rename.to_name == "MyDataset_v1"
|
||||
promote = plan.actions[2]
|
||||
assert promote.from_name == "MyDataset__migrating"
|
||||
assert promote.to_name == "MyDataset"
|
||||
|
||||
def test_resume__source_id_mismatch__raises(self) -> None:
|
||||
# If the user-supplied name now resolves to a DIFFERENT dataset than the
|
||||
# interrupted run's source, resume must refuse rather than migrate the
|
||||
# wrong dataset.
|
||||
with patch.object(
|
||||
planner_module, "reconstruct_remaps", return_value=ReconstructedRemaps()
|
||||
):
|
||||
with pytest.raises(MigrationError, match="different dataset"):
|
||||
planner_module.build_dataset_plan(
|
||||
client=self._resume_client(source_id="DIFFERENT-id"),
|
||||
name="MyDataset",
|
||||
to_project="B",
|
||||
resume_checkpoint=self._checkpoint(),
|
||||
)
|
||||
@@ -0,0 +1,241 @@
|
||||
"""Unit tests for read-only remap reconstruction on resume (OPIK-7168).
|
||||
|
||||
``reconstruct_remaps`` rebuilds version_remap / item_id_remap /
|
||||
optimization_id_remap from an already-migrated destination without writing
|
||||
anything. These tests mock the REST reads (versions, per-version items,
|
||||
optimizations) and assert the pairing/identity/fallback logic and the
|
||||
fail-loud guard on a version-count mismatch.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from opik.cli.migrate.datasets.resume import (
|
||||
ResumeReconstructionError,
|
||||
reconstruct_remaps,
|
||||
)
|
||||
|
||||
|
||||
class _Version:
|
||||
def __init__(self, id: str, version_hash: str) -> None:
|
||||
self.id = id
|
||||
self.version_hash = version_hash
|
||||
|
||||
|
||||
class _Item:
|
||||
def __init__(
|
||||
self,
|
||||
id: str,
|
||||
data: Optional[Dict[str, Any]] = None,
|
||||
description: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
evaluators: Optional[List[Any]] = None,
|
||||
execution_policy: Optional[Any] = None,
|
||||
trace_id: Optional[str] = None,
|
||||
span_id: Optional[str] = None,
|
||||
source: Optional[str] = None,
|
||||
) -> None:
|
||||
self.id = id
|
||||
self.data = data or {"q": id}
|
||||
self.description = description
|
||||
self.tags = tags
|
||||
self.evaluators = evaluators
|
||||
self.execution_policy = execution_policy
|
||||
self.trace_id = trace_id
|
||||
self.span_id = span_id
|
||||
self.source = source
|
||||
|
||||
|
||||
class _Optimization:
|
||||
def __init__(self, id: str, name: str) -> None:
|
||||
self.id = id
|
||||
self.name = name
|
||||
|
||||
|
||||
def _page(content: List[Any]) -> MagicMock:
|
||||
return MagicMock(content=content)
|
||||
|
||||
|
||||
def _rest_client(
|
||||
*,
|
||||
source_versions: List[_Version],
|
||||
dest_versions: List[_Version],
|
||||
items_by_version_hash: Dict[str, List[_Item]],
|
||||
source_optimizations: Optional[List[_Optimization]] = None,
|
||||
dest_optimizations: Optional[List[_Optimization]] = None,
|
||||
) -> MagicMock:
|
||||
rest = MagicMock()
|
||||
|
||||
def _list_versions(id: str, page: int, size: int) -> MagicMock:
|
||||
# reconstruct calls _iter_source_versions_chronological which lists
|
||||
# NEWEST-first then reverses; return newest-first here.
|
||||
if page != 1:
|
||||
return _page([])
|
||||
if id == "src-ds":
|
||||
return _page(list(reversed(source_versions)))
|
||||
return _page(list(reversed(dest_versions)))
|
||||
|
||||
rest.datasets.list_dataset_versions.side_effect = _list_versions
|
||||
|
||||
def _stream(dataset_name: str, steam_limit: int, dataset_version: str, **kw: Any):
|
||||
# _stream_version_items_raw parses a byte stream via rest_stream_parser;
|
||||
# tests patch that helper directly (see the patch_stream fixture), so
|
||||
# this raw endpoint should never be hit.
|
||||
raise AssertionError("stream_dataset_items should be patched in tests")
|
||||
|
||||
rest.datasets.stream_dataset_items.side_effect = _stream
|
||||
|
||||
def _find_opts(dataset_id: str, page: int, size: int) -> MagicMock:
|
||||
if page != 1:
|
||||
return _page([])
|
||||
if dataset_id == "src-ds":
|
||||
return _page(source_optimizations or [])
|
||||
return _page(dest_optimizations or [])
|
||||
|
||||
rest.optimizations.find_optimizations.side_effect = _find_opts
|
||||
return rest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def patch_stream(monkeypatch: pytest.MonkeyPatch):
|
||||
"""Patch ``_stream_version_items_raw`` (byte-stream parser) with a direct
|
||||
lookup keyed by version_hash, so tests supply plain ``_Item`` lists."""
|
||||
registry: Dict[str, List[_Item]] = {}
|
||||
|
||||
def _fake_stream(
|
||||
rest_client: Any,
|
||||
*,
|
||||
dataset_name: str,
|
||||
project_name: Optional[str],
|
||||
version_hash: Optional[str],
|
||||
) -> List[_Item]:
|
||||
return registry.get(version_hash or "", [])
|
||||
|
||||
monkeypatch.setattr(
|
||||
"opik.cli.migrate.datasets.resume._stream_version_items_raw", _fake_stream
|
||||
)
|
||||
return registry
|
||||
|
||||
|
||||
def test_reconstruct__paired_versions_and_identity_items__remaps_built(
|
||||
patch_stream: Dict[str, List[_Item]],
|
||||
) -> None:
|
||||
source_versions = [_Version("sv1", "h1"), _Version("sv2", "h2")]
|
||||
# Destination hashes DIFFER from source (BE recomputes them) — pairing must
|
||||
# be positional, not by hash.
|
||||
dest_versions = [_Version("dv1", "dh1"), _Version("dv2", "dh2")]
|
||||
# Identity: dest item ids equal source item ids (copy_from preserves them).
|
||||
patch_stream["h1"] = [_Item("i1"), _Item("i2")]
|
||||
patch_stream["dh1"] = [_Item("i1"), _Item("i2")]
|
||||
patch_stream["h2"] = [_Item("i3")]
|
||||
patch_stream["dh2"] = [_Item("i3")]
|
||||
|
||||
rest = _rest_client(
|
||||
source_versions=source_versions,
|
||||
dest_versions=dest_versions,
|
||||
items_by_version_hash={},
|
||||
)
|
||||
|
||||
remaps = reconstruct_remaps(
|
||||
rest,
|
||||
source_dataset_id="src-ds",
|
||||
source_name="ds",
|
||||
source_project_name=None,
|
||||
dest_dataset_id="dst-ds",
|
||||
dest_name="ds",
|
||||
dest_project_name="proj",
|
||||
)
|
||||
|
||||
assert remaps.version_remap == {"sv1": "dv1", "sv2": "dv2"}
|
||||
assert remaps.item_id_remap == {"i1": "i1", "i2": "i2", "i3": "i3"}
|
||||
|
||||
|
||||
def test_reconstruct__dest_ids_differ__content_hash_fallback_maps_them(
|
||||
patch_stream: Dict[str, List[_Item]],
|
||||
) -> None:
|
||||
# First-version read-back case: dest item id was minted fresh, so identity
|
||||
# match fails and we fall back to content-hash matching.
|
||||
source_versions = [_Version("sv1", "h1")]
|
||||
dest_versions = [_Version("dv1", "dh1")]
|
||||
patch_stream["h1"] = [_Item("src-i1", data={"q": "same"})]
|
||||
patch_stream["dh1"] = [_Item("dst-i1", data={"q": "same"})]
|
||||
|
||||
rest = _rest_client(
|
||||
source_versions=source_versions,
|
||||
dest_versions=dest_versions,
|
||||
items_by_version_hash={},
|
||||
)
|
||||
|
||||
remaps = reconstruct_remaps(
|
||||
rest,
|
||||
source_dataset_id="src-ds",
|
||||
source_name="ds",
|
||||
source_project_name=None,
|
||||
dest_dataset_id="dst-ds",
|
||||
dest_name="ds",
|
||||
dest_project_name="proj",
|
||||
)
|
||||
|
||||
assert remaps.item_id_remap == {"src-i1": "dst-i1"}
|
||||
|
||||
|
||||
def test_reconstruct__version_count_mismatch__raises_reconstruction_error(
|
||||
patch_stream: Dict[str, List[_Item]],
|
||||
) -> None:
|
||||
# Destination has fewer versions than source (e.g. crash mid-replay) — must
|
||||
# fail loud rather than mis-pair.
|
||||
rest = _rest_client(
|
||||
source_versions=[_Version("sv1", "h1"), _Version("sv2", "h2")],
|
||||
dest_versions=[_Version("dv1", "dh1")],
|
||||
items_by_version_hash={},
|
||||
)
|
||||
|
||||
with pytest.raises(ResumeReconstructionError, match="Delete the destination"):
|
||||
reconstruct_remaps(
|
||||
rest,
|
||||
source_dataset_id="src-ds",
|
||||
source_name="ds",
|
||||
source_project_name=None,
|
||||
dest_dataset_id="dst-ds",
|
||||
dest_name="ds",
|
||||
dest_project_name="proj",
|
||||
)
|
||||
|
||||
|
||||
def test_reconstruct__optimizations__matched_by_name_into_remap(
|
||||
patch_stream: Dict[str, List[_Item]],
|
||||
) -> None:
|
||||
source_versions = [_Version("sv1", "h1")]
|
||||
dest_versions = [_Version("dv1", "dh1")]
|
||||
patch_stream["h1"] = []
|
||||
patch_stream["dh1"] = []
|
||||
|
||||
rest = _rest_client(
|
||||
source_versions=source_versions,
|
||||
dest_versions=dest_versions,
|
||||
items_by_version_hash={},
|
||||
source_optimizations=[
|
||||
_Optimization("so1", "opt-a"),
|
||||
_Optimization("so2", "opt-b"),
|
||||
],
|
||||
dest_optimizations=[
|
||||
_Optimization("do1", "opt-a"),
|
||||
_Optimization("do2", "opt-b"),
|
||||
],
|
||||
)
|
||||
|
||||
remaps = reconstruct_remaps(
|
||||
rest,
|
||||
source_dataset_id="src-ds",
|
||||
source_name="ds",
|
||||
source_project_name=None,
|
||||
dest_dataset_id="dst-ds",
|
||||
dest_name="ds",
|
||||
dest_project_name="proj",
|
||||
)
|
||||
|
||||
assert remaps.optimization_id_remap == {"so1": "do1", "so2": "do2"}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,305 @@
|
||||
"""Executor tests for ``opik migrate prompt``.
|
||||
|
||||
Drives ``execute_plan`` against fully-mocked rest_client surfaces and
|
||||
asserts that each action dispatches to the right Fern call with the
|
||||
right kwargs. The verbatim-commit replay path and per-version audit
|
||||
shape are exercised here (the dataset-side analogue lives in
|
||||
``test_migrate_dataset_version_replay.py``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, List
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from opik.cli.migrate.audit import AuditLog
|
||||
from opik.cli.migrate.prompts import executor as executor_module
|
||||
from opik.cli.migrate.prompts.planner import (
|
||||
CreateDestination,
|
||||
MigrationPlan,
|
||||
RenameSource,
|
||||
ReplayVersions,
|
||||
)
|
||||
from opik.cli.migrate.prompts.resolver import ResolvedPrompt
|
||||
from opik.rest_api.types.prompt_version_detail import PromptVersionDetail
|
||||
|
||||
from ._migrate_prompt_helpers import _Page, _PromptVersionRow
|
||||
|
||||
|
||||
def _client_with_rest_mock(version_pages: List[_Page]) -> Any:
|
||||
rest_client = MagicMock()
|
||||
rest_client.prompts.update_prompt = MagicMock()
|
||||
rest_client.prompts.create_prompt = MagicMock()
|
||||
rest_client.prompts.create_prompt_version = MagicMock(
|
||||
side_effect=lambda **kw: MagicMock(
|
||||
id=f"dest-version-{kw['version'].commit}",
|
||||
commit=kw["version"].commit,
|
||||
environments=kw["version"].environments,
|
||||
)
|
||||
)
|
||||
rest_client.prompts.get_prompt_versions.side_effect = version_pages
|
||||
|
||||
client = MagicMock()
|
||||
client.rest_client = rest_client
|
||||
return client, rest_client
|
||||
|
||||
|
||||
def _make_plan() -> MigrationPlan:
|
||||
source = ResolvedPrompt(
|
||||
id="src-1",
|
||||
name="MyPrompt",
|
||||
project_id=None,
|
||||
project_name=None,
|
||||
description="orig",
|
||||
tags=["a"],
|
||||
template_structure="text",
|
||||
)
|
||||
plan = MigrationPlan(source=source, target_name="MyPrompt", to_project="B")
|
||||
plan.actions = [
|
||||
RenameSource(
|
||||
source_id="src-1",
|
||||
from_name="MyPrompt",
|
||||
to_name="MyPrompt_v1",
|
||||
description="orig",
|
||||
tags=["a"],
|
||||
),
|
||||
CreateDestination(
|
||||
name="MyPrompt",
|
||||
project_name="B",
|
||||
description="orig",
|
||||
tags=["a"],
|
||||
template_structure="text",
|
||||
),
|
||||
ReplayVersions(
|
||||
source_prompt_id="src-1",
|
||||
source_name_after_rename="MyPrompt_v1",
|
||||
source_project_name=None,
|
||||
dest_name="MyPrompt",
|
||||
dest_project_name="B",
|
||||
template_structure="text",
|
||||
),
|
||||
]
|
||||
return plan
|
||||
|
||||
|
||||
class TestExecuteActions:
|
||||
def test_rename_source__re_passes_description_and_tags(self) -> None:
|
||||
# The BE rename PUT wipes description (no COALESCE in the UPDATE);
|
||||
# the executor must re-pass it so the source description survives.
|
||||
client, rest_client = _client_with_rest_mock(
|
||||
[_Page([])], # no versions
|
||||
)
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
rest_client.prompts.update_prompt.assert_called_once_with(
|
||||
id="src-1",
|
||||
name="MyPrompt_v1",
|
||||
description="orig",
|
||||
tags=["a"],
|
||||
)
|
||||
|
||||
def test_create_destination__omits_template_so_be_does_not_auto_mint_v1(
|
||||
self,
|
||||
) -> None:
|
||||
# If we passed a template here, the BE would auto-create a v1
|
||||
# with its own commit hash, defeating the verbatim-replay
|
||||
# guarantee. The executor must NOT include template in the
|
||||
# create_prompt call.
|
||||
client, rest_client = _client_with_rest_mock([_Page([])])
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
rest_client.prompts.create_prompt.assert_called_once()
|
||||
kwargs = rest_client.prompts.create_prompt.call_args.kwargs
|
||||
assert kwargs["name"] == "MyPrompt"
|
||||
assert kwargs["project_name"] == "B"
|
||||
assert kwargs["description"] == "orig"
|
||||
assert kwargs["tags"] == ["a"]
|
||||
assert kwargs["template_structure"] == "text"
|
||||
# Critical: no template, no type, no metadata at the container
|
||||
# level — those are version-level fields handled by the replay.
|
||||
assert "template" not in kwargs
|
||||
assert "type" not in kwargs
|
||||
assert "metadata" not in kwargs
|
||||
|
||||
def test_replay_versions__preserves_commit_hashes_verbatim(self) -> None:
|
||||
# The verbatim-commit invariant is the architectural reason this
|
||||
# slice exists: source versions are minted on the destination
|
||||
# with the source's exact commit hash, made possible by the
|
||||
# destination prompt's fresh id (so (workspace_id, prompt_id,
|
||||
# commit) never collides).
|
||||
v1 = _PromptVersionRow(
|
||||
id="src-v-1",
|
||||
prompt_id="src-1",
|
||||
commit="aaaaaaaa",
|
||||
template="hello {{name}}",
|
||||
type="mustache",
|
||||
)
|
||||
v2 = _PromptVersionRow(
|
||||
id="src-v-2",
|
||||
prompt_id="src-1",
|
||||
commit="bbbbbbbb",
|
||||
template="hello {{name}} v2",
|
||||
type="mustache",
|
||||
)
|
||||
# BE returns newest-first (pv.id DESC); the replay loop reverses
|
||||
# to oldest-first. Pass them in newest-first order to mirror the
|
||||
# BE response shape.
|
||||
client, rest_client = _client_with_rest_mock([_Page([v2, v1])])
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
# Two POST /v1/private/prompts/versions calls, oldest-first.
|
||||
assert rest_client.prompts.create_prompt_version.call_count == 2
|
||||
first_call = rest_client.prompts.create_prompt_version.call_args_list[0]
|
||||
second_call = rest_client.prompts.create_prompt_version.call_args_list[1]
|
||||
|
||||
first_version = first_call.kwargs["version"]
|
||||
second_version = second_call.kwargs["version"]
|
||||
assert isinstance(first_version, PromptVersionDetail)
|
||||
assert isinstance(second_version, PromptVersionDetail)
|
||||
assert first_version.commit == "aaaaaaaa"
|
||||
assert second_version.commit == "bbbbbbbb"
|
||||
assert first_version.template == "hello {{name}}"
|
||||
assert second_version.template == "hello {{name}} v2"
|
||||
|
||||
def test_replay_versions__carries_environments_verbatim(self) -> None:
|
||||
# Environment ownership is per-version data the BE accepts inline on
|
||||
# create_prompt_version. The history mixes an env-less version, a
|
||||
# non-latest version owning a single env, and a latest version owning
|
||||
# multiple envs — proving single + multi env work and that a
|
||||
# non-latest version's env is preserved (not just the newest one's).
|
||||
v1 = _PromptVersionRow(
|
||||
id="src-v-1",
|
||||
prompt_id="src-1",
|
||||
commit="aaaaaaaa",
|
||||
template="hello",
|
||||
)
|
||||
v2 = _PromptVersionRow(
|
||||
id="src-v-2",
|
||||
prompt_id="src-1",
|
||||
commit="bbbbbbbb",
|
||||
template="hi",
|
||||
environments=["development"],
|
||||
)
|
||||
v3 = _PromptVersionRow(
|
||||
id="src-v-3",
|
||||
prompt_id="src-1",
|
||||
commit="cccccccc",
|
||||
template="hey",
|
||||
environments=["production", "staging"],
|
||||
)
|
||||
# BE returns newest-first; the loop reverses to oldest-first.
|
||||
client, rest_client = _client_with_rest_mock([_Page([v3, v2, v1])])
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
calls = rest_client.prompts.create_prompt_version.call_args_list
|
||||
assert calls[0].kwargs["version"].environments is None
|
||||
assert calls[1].kwargs["version"].environments == ["development"]
|
||||
assert calls[2].kwargs["version"].environments == ["production", "staging"]
|
||||
|
||||
def test_replay_versions__records_environments_in_audit(self) -> None:
|
||||
v1 = _PromptVersionRow(
|
||||
id="src-v-1",
|
||||
prompt_id="src-1",
|
||||
commit="aaaaaaaa",
|
||||
template="hello",
|
||||
environments=["staging", "production"],
|
||||
)
|
||||
client, _ = _client_with_rest_mock([_Page([v1])])
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
record = next(a for a in audit.actions if a["type"] == "replay_prompt_version")
|
||||
# Environments are recorded verbatim (order is irrelevant), so compare
|
||||
# order-insensitively.
|
||||
assert sorted(record["source_environments"]) == ["production", "staging"]
|
||||
assert sorted(record["target_environments"]) == ["production", "staging"]
|
||||
|
||||
def test_replay_versions__populates_prompt_version_id_remap(self) -> None:
|
||||
# Slice 7 (OPIK-6575) reads ``plan.prompt_version_id_remap`` to
|
||||
# remap experiment FK references. The executor must populate it.
|
||||
v1 = _PromptVersionRow(
|
||||
id="src-v-1",
|
||||
prompt_id="src-1",
|
||||
commit="aaaaaaaa",
|
||||
template="hello",
|
||||
)
|
||||
v2 = _PromptVersionRow(
|
||||
id="src-v-2",
|
||||
prompt_id="src-1",
|
||||
commit="bbbbbbbb",
|
||||
template="hi",
|
||||
)
|
||||
client, rest_client = _client_with_rest_mock([_Page([v2, v1])])
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
assert plan.prompt_version_id_remap == {
|
||||
"src-v-1": "dest-version-aaaaaaaa",
|
||||
"src-v-2": "dest-version-bbbbbbbb",
|
||||
}
|
||||
|
||||
def test_replay_versions__records_one_audit_entry_per_version(self) -> None:
|
||||
v1 = _PromptVersionRow(
|
||||
id="src-v-1",
|
||||
prompt_id="src-1",
|
||||
commit="aaaaaaaa",
|
||||
template="hello",
|
||||
)
|
||||
v2 = _PromptVersionRow(
|
||||
id="src-v-2",
|
||||
prompt_id="src-1",
|
||||
commit="bbbbbbbb",
|
||||
template="hi",
|
||||
)
|
||||
client, _ = _client_with_rest_mock([_Page([v2, v1])])
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.execute_plan(client, plan, audit)
|
||||
|
||||
per_version_records = [
|
||||
a for a in audit.actions if a["type"] == "replay_prompt_version"
|
||||
]
|
||||
assert len(per_version_records) == 2
|
||||
# All per-version records are recorded as ``ok`` (the per-version
|
||||
# audit happens after a successful create_prompt_version).
|
||||
assert all(a["status"] == "ok" for a in per_version_records)
|
||||
assert {a["source_commit"] for a in per_version_records} == {
|
||||
"aaaaaaaa",
|
||||
"bbbbbbbb",
|
||||
}
|
||||
|
||||
|
||||
class TestRecordPlanned:
|
||||
def test_record_planned__emits_one_planned_entry_per_action(self) -> None:
|
||||
# Dry-run path: every action gets a ``planned`` record, no REST
|
||||
# calls fire.
|
||||
plan = _make_plan()
|
||||
audit = AuditLog(command="opik migrate prompt", args={})
|
||||
|
||||
executor_module.record_planned(plan, audit)
|
||||
|
||||
statuses = [a["status"] for a in audit.actions]
|
||||
types = [a["type"] for a in audit.actions]
|
||||
assert statuses == ["planned", "planned", "planned"]
|
||||
assert types == [
|
||||
"rename_source",
|
||||
"create_destination",
|
||||
"replay_versions",
|
||||
]
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Planner tests for ``opik migrate prompt``.
|
||||
|
||||
The dataset planner tests are next door; this file tracks the same
|
||||
patterns (happy-path action ordering, conflict detection, project-not-
|
||||
found, source-not-found) for the prompt slice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from opik.cli import cli
|
||||
from opik.cli.migrate.errors import (
|
||||
ConflictError,
|
||||
ProjectNotFoundError,
|
||||
PromptNotFoundError,
|
||||
)
|
||||
from opik.cli.migrate.prompts import planner as prompt_planner
|
||||
|
||||
from ._migrate_prompt_helpers import (
|
||||
_Page,
|
||||
_PromptRow,
|
||||
_planner_client,
|
||||
_planner_rest_client,
|
||||
)
|
||||
|
||||
|
||||
class TestMigratePromptHelp:
|
||||
def test_migrate_group_help__lists_prompt_subcommand(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["migrate", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "prompt" in result.output
|
||||
|
||||
def test_migrate_prompt_help__lists_required_flags(self) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["migrate", "prompt", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "--to-project" in result.output
|
||||
assert "--from-project" in result.output
|
||||
assert "--dry-run" in result.output
|
||||
|
||||
|
||||
class TestPlanBuilding:
|
||||
def test_build_prompt_plan__default_flow__orders_rename_create_replay(
|
||||
self,
|
||||
) -> None:
|
||||
# Action ordering invariant: the rename frees the workspace-unique
|
||||
# name BEFORE the destination claims it, and the replay loop runs
|
||||
# AFTER the destination container exists.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_PromptRow(id="src-1", name="MyPrompt", description="d")]),
|
||||
_Page([]), # rename-collision preflight finds nothing
|
||||
]
|
||||
)
|
||||
|
||||
plan = prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
types = [type(a).__name__ for a in plan.actions]
|
||||
assert types == ["RenameSource", "CreateDestination", "ReplayVersions"]
|
||||
|
||||
def test_build_prompt_plan__rename_re_passes_description_and_tags(
|
||||
self,
|
||||
) -> None:
|
||||
# The BE rename PUT wipes description (UPDATE has no COALESCE on
|
||||
# description); the planner forwards the source's description and
|
||||
# tags into the RenameSource record so the executor re-passes them.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page(
|
||||
[
|
||||
_PromptRow(
|
||||
id="src-1",
|
||||
name="MyPrompt",
|
||||
description="orig description",
|
||||
tags=["a", "b"],
|
||||
)
|
||||
]
|
||||
),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
rename = plan.actions[0]
|
||||
assert isinstance(rename, prompt_planner.RenameSource)
|
||||
assert rename.from_name == "MyPrompt"
|
||||
assert rename.to_name == "MyPrompt_v1"
|
||||
assert rename.description == "orig description"
|
||||
assert rename.tags == ["a", "b"]
|
||||
|
||||
def test_build_prompt_plan__create_destination_carries_metadata(
|
||||
self,
|
||||
) -> None:
|
||||
# CreateDestination forwards container-level metadata so the
|
||||
# destination inherits description / tags / template_structure
|
||||
# from the source.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page(
|
||||
[
|
||||
_PromptRow(
|
||||
id="src-1",
|
||||
name="MyPrompt",
|
||||
description="orig",
|
||||
tags=["a"],
|
||||
template_structure="chat",
|
||||
)
|
||||
]
|
||||
),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
create = plan.actions[1]
|
||||
assert isinstance(create, prompt_planner.CreateDestination)
|
||||
assert create.name == "MyPrompt"
|
||||
assert create.project_name == "B"
|
||||
assert create.description == "orig"
|
||||
assert create.tags == ["a"]
|
||||
assert create.template_structure == "chat"
|
||||
|
||||
def test_build_prompt_plan__replay_versions_carries_source_id_and_template_structure(
|
||||
self,
|
||||
) -> None:
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page(
|
||||
[
|
||||
_PromptRow(
|
||||
id="src-1",
|
||||
name="MyPrompt",
|
||||
template_structure="chat",
|
||||
)
|
||||
]
|
||||
),
|
||||
_Page([]),
|
||||
]
|
||||
)
|
||||
|
||||
plan = prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
replay = plan.actions[2]
|
||||
assert isinstance(replay, prompt_planner.ReplayVersions)
|
||||
assert replay.source_prompt_id == "src-1"
|
||||
assert replay.source_name_after_rename == "MyPrompt_v1"
|
||||
assert replay.dest_name == "MyPrompt"
|
||||
assert replay.dest_project_name == "B"
|
||||
assert replay.template_structure == "chat"
|
||||
|
||||
|
||||
class TestPreflightErrors:
|
||||
def test_build_prompt_plan__destination_project_missing__raises(
|
||||
self,
|
||||
) -> None:
|
||||
rest_client = _planner_rest_client(
|
||||
[], # no prompt lookups happen if project preflight fails
|
||||
target_project_exists=False,
|
||||
)
|
||||
|
||||
with pytest.raises(ProjectNotFoundError) as excinfo:
|
||||
prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="DoesNotExist",
|
||||
)
|
||||
|
||||
assert "DoesNotExist" in str(excinfo.value)
|
||||
|
||||
def test_build_prompt_plan__source_prompt_missing__raises(self) -> None:
|
||||
rest_client = _planner_rest_client(
|
||||
[_Page([])] # source lookup returns no matches
|
||||
)
|
||||
|
||||
with pytest.raises(PromptNotFoundError) as excinfo:
|
||||
prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="Missing",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
assert "Missing" in str(excinfo.value)
|
||||
assert "workspace" in str(excinfo.value).lower()
|
||||
|
||||
def test_build_prompt_plan__from_project_set_and_missing__raises_with_project_scope(
|
||||
self,
|
||||
) -> None:
|
||||
# When --from-project is passed and the source prompt is not in that
|
||||
# project, the not-found error should name the project (not "the
|
||||
# workspace") so the user knows where we looked.
|
||||
rest_client = _planner_rest_client(
|
||||
[_Page([])] # source lookup returns no matches in the project
|
||||
)
|
||||
|
||||
with pytest.raises(PromptNotFoundError) as excinfo:
|
||||
prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="B",
|
||||
from_project="A",
|
||||
)
|
||||
|
||||
assert "MyPrompt" in str(excinfo.value)
|
||||
assert "project 'A'" in str(excinfo.value)
|
||||
|
||||
def test_build_prompt_plan__rename_collision__raises(self) -> None:
|
||||
# Source resolves, but the rename target ``MyPrompt_v1`` is already
|
||||
# taken by another prompt -> ConflictError.
|
||||
rest_client = _planner_rest_client(
|
||||
[
|
||||
_Page([_PromptRow(id="src-1", name="MyPrompt")]),
|
||||
_Page([_PromptRow(id="other-1", name="MyPrompt_v1")]),
|
||||
]
|
||||
)
|
||||
|
||||
with pytest.raises(ConflictError) as excinfo:
|
||||
prompt_planner.build_prompt_plan(
|
||||
client=_planner_client(rest_client),
|
||||
name="MyPrompt",
|
||||
to_project="B",
|
||||
)
|
||||
|
||||
assert "MyPrompt_v1" in str(excinfo.value)
|
||||
@@ -0,0 +1,358 @@
|
||||
"""Tests for the migration manifest used by opik import for resumable migrations."""
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from opik.cli.migration_manifest import MigrationManifest, MANIFEST_FILENAME
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_base(tmp_path: Path) -> Path:
|
||||
return tmp_path
|
||||
|
||||
|
||||
class TestMigrationManifestLifecycle:
|
||||
def test_manifest__fresh_instance__status_is_not_started(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
assert manifest.status == "not_started"
|
||||
assert not manifest.is_in_progress
|
||||
assert not manifest.is_completed
|
||||
|
||||
def test_start__fresh_manifest__status_becomes_in_progress_and_file_written(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.start()
|
||||
assert manifest.is_in_progress
|
||||
assert (tmp_base / MANIFEST_FILENAME).exists()
|
||||
|
||||
def test_complete__after_start__status_becomes_completed(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.start()
|
||||
manifest.complete()
|
||||
assert manifest.is_completed
|
||||
assert not manifest.is_in_progress
|
||||
|
||||
def test_reset__after_start_with_data__all_state_cleared(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.start()
|
||||
trace_file = tmp_base / "projects" / "p1" / "trace_abc.json"
|
||||
trace_file.parent.mkdir(parents=True)
|
||||
trace_file.touch()
|
||||
manifest.mark_file_completed(trace_file)
|
||||
manifest.add_trace_mapping("src-1", "dest-1")
|
||||
|
||||
manifest.reset()
|
||||
|
||||
assert manifest.status == "not_started"
|
||||
assert manifest.completed_count() == 0
|
||||
assert manifest.get_trace_id_map() == {}
|
||||
|
||||
def test_exists__before_any_save__returns_false(self, tmp_base: Path) -> None:
|
||||
assert not MigrationManifest.exists(tmp_base)
|
||||
|
||||
def test_exists__after_start__returns_true(self, tmp_base: Path) -> None:
|
||||
MigrationManifest(tmp_base).start()
|
||||
assert MigrationManifest.exists(tmp_base)
|
||||
|
||||
def test_start__called_twice__started_at_preserved(self, tmp_base: Path) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.start()
|
||||
|
||||
conn = sqlite3.connect(str(tmp_base / MANIFEST_FILENAME))
|
||||
first_started_at = conn.execute(
|
||||
"SELECT value FROM status WHERE key = 'started_at'"
|
||||
).fetchone()[0]
|
||||
conn.close()
|
||||
|
||||
manifest.start() # second call must not overwrite started_at
|
||||
|
||||
conn = sqlite3.connect(str(tmp_base / MANIFEST_FILENAME))
|
||||
second_started_at = conn.execute(
|
||||
"SELECT value FROM status WHERE key = 'started_at'"
|
||||
).fetchone()[0]
|
||||
conn.close()
|
||||
|
||||
assert second_started_at == first_started_at
|
||||
|
||||
|
||||
class TestFileTracking:
|
||||
def _make_file(self, base: Path, relative: str) -> Path:
|
||||
p = base / relative
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.touch()
|
||||
return p
|
||||
|
||||
def test_is_file_completed__fresh_manifest__returns_false(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
f = self._make_file(tmp_base, "datasets/dataset_foo.json")
|
||||
assert not manifest.is_file_completed(f)
|
||||
|
||||
def test_mark_file_completed__new_file__recorded_and_count_incremented(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
f = self._make_file(tmp_base, "datasets/dataset_foo.json")
|
||||
manifest.mark_file_completed(f)
|
||||
assert manifest.is_file_completed(f)
|
||||
assert manifest.completed_count() == 1
|
||||
|
||||
def test_mark_file_completed__same_file_twice__count_remains_one(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
f = self._make_file(tmp_base, "datasets/dataset_foo.json")
|
||||
manifest.mark_file_completed(f)
|
||||
manifest.mark_file_completed(f)
|
||||
assert manifest.completed_count() == 1
|
||||
|
||||
def test_mark_file_failed__new_file__recorded_with_error(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
f = self._make_file(tmp_base, "projects/p1/trace_abc.json")
|
||||
manifest.mark_file_failed(f, "timeout")
|
||||
assert manifest.failed_count() == 1
|
||||
assert manifest.get_failed_files()[manifest.relative_path(f)] == "timeout"
|
||||
|
||||
def test_mark_file_completed__previously_failed_file__removed_from_failed(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
f = self._make_file(tmp_base, "projects/p1/trace_abc.json")
|
||||
manifest.mark_file_failed(f, "timeout")
|
||||
manifest.mark_file_completed(f)
|
||||
assert manifest.failed_count() == 0
|
||||
assert manifest.is_file_completed(f)
|
||||
|
||||
def test_relative_path__nested_file__returns_path_relative_to_base(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
f = self._make_file(tmp_base, "projects/my-project/trace_xyz.json")
|
||||
assert manifest.relative_path(f) == str(
|
||||
Path("projects") / "my-project" / "trace_xyz.json"
|
||||
)
|
||||
|
||||
|
||||
class TestTraceIdMapping:
|
||||
def test_add_trace_mapping__single_entry__retrievable_via_get_map(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.add_trace_mapping("src-111", "dest-222")
|
||||
manifest.save()
|
||||
assert manifest.get_trace_id_map() == {"src-111": "dest-222"}
|
||||
|
||||
def test_trace_id_map__save_then_new_instance__mapping_survives(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
m1 = MigrationManifest(tmp_base)
|
||||
m1.start()
|
||||
m1.add_trace_mapping("src-aaa", "dest-bbb")
|
||||
m1.save()
|
||||
|
||||
m2 = MigrationManifest(tmp_base)
|
||||
assert m2.get_trace_id_map() == {"src-aaa": "dest-bbb"}
|
||||
|
||||
def test_get_trace_id_map__mutate_returned_dict__original_unaffected(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.add_trace_mapping("src-1", "dest-1")
|
||||
manifest.save()
|
||||
copy = manifest.get_trace_id_map()
|
||||
copy["src-2"] = "dest-2" # mutate the copy
|
||||
assert "src-2" not in manifest.get_trace_id_map()
|
||||
|
||||
def test_add_trace_mapping__same_src_id_twice__last_dest_wins(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.add_trace_mapping("src-1", "dest-old")
|
||||
manifest.add_trace_mapping("src-1", "dest-new")
|
||||
manifest.save()
|
||||
assert manifest.get_trace_id_map()["src-1"] == "dest-new"
|
||||
|
||||
|
||||
class TestDatabaseIntegrity:
|
||||
def test_manifest_file__after_start_and_save__is_valid_sqlite_with_expected_tables(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.start()
|
||||
manifest.add_trace_mapping("a", "b")
|
||||
manifest.save()
|
||||
|
||||
conn = sqlite3.connect(str(tmp_base / MANIFEST_FILENAME))
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'"
|
||||
).fetchall()
|
||||
}
|
||||
conn.close()
|
||||
|
||||
assert {"status", "completed_files", "failed_files", "trace_id_map"} <= tables
|
||||
|
||||
def test_save__after_start__no_tmp_file_created(self, tmp_base: Path) -> None:
|
||||
manifest = MigrationManifest(tmp_base)
|
||||
manifest.start()
|
||||
assert not (tmp_base / "migration_manifest.tmp").exists()
|
||||
|
||||
def test_mark_file_completed__duplicate_write__count_remains_one(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
"""INSERT OR IGNORE means re-flushing the same path never duplicates rows."""
|
||||
manifest = MigrationManifest(tmp_base, batch_size=1)
|
||||
f = tmp_base / "projects" / "p" / "trace_x.json"
|
||||
f.parent.mkdir(parents=True)
|
||||
f.touch()
|
||||
manifest.mark_file_completed(f)
|
||||
manifest.mark_file_completed(f)
|
||||
assert manifest.completed_count() == 1
|
||||
|
||||
|
||||
class TestBatchingBehavior:
|
||||
"""Document and verify the crash trade-off introduced by batched writes."""
|
||||
|
||||
def _make_trace(self, base: Path, name: str) -> Path:
|
||||
p = base / "projects" / "proj" / name
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.touch()
|
||||
return p
|
||||
|
||||
def test_mark_file_completed__within_batch__visible_via_api_before_flush(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
"""Buffered completions are visible through the public API (which flushes
|
||||
before querying) even before the batch threshold is reached."""
|
||||
manifest = MigrationManifest(tmp_base, batch_size=50)
|
||||
f = self._make_trace(tmp_base, "trace_001.json")
|
||||
manifest.mark_file_completed(f)
|
||||
# completed_count() flushes first, so the buffered write IS visible.
|
||||
assert manifest.completed_count() == 1
|
||||
assert manifest.is_file_completed(f)
|
||||
|
||||
def test_mark_file_completed__crash_before_flush__unflushed_data_lost(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
"""Simulates a crash (new instance, no save()) with batch_size > pending count.
|
||||
|
||||
This is the documented trade-off: up to batch_size-1 completions may be
|
||||
absent from the manifest after a crash. Those files will simply be
|
||||
re-imported on resume.
|
||||
"""
|
||||
f = self._make_trace(tmp_base, "trace_001.json")
|
||||
|
||||
m1 = MigrationManifest(tmp_base, batch_size=50)
|
||||
m1.start() # always flushes immediately
|
||||
m1.mark_file_completed(f) # buffered — NOT yet on disk
|
||||
# Simulate crash: m1 is abandoned without save() or complete().
|
||||
# __del__ is NOT called here because m1 is still in scope when m2 is created.
|
||||
|
||||
m2 = MigrationManifest(tmp_base, batch_size=50)
|
||||
# The completion is not on disk — resume will re-process this file.
|
||||
assert not m2.is_file_completed(f)
|
||||
assert m2.completed_count() == 0
|
||||
|
||||
def test_mark_file_completed__batch_size_one__each_write_immediately_durable(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
"""With batch_size=1 every completion is immediately committed to disk."""
|
||||
f = self._make_trace(tmp_base, "trace_001.json")
|
||||
|
||||
m1 = MigrationManifest(tmp_base, batch_size=1)
|
||||
m1.start()
|
||||
m1.mark_file_completed(f) # auto-flushes (batch_size=1)
|
||||
|
||||
# New instance reads from disk — must see the completion.
|
||||
m2 = MigrationManifest(tmp_base, batch_size=1)
|
||||
assert m2.is_file_completed(f)
|
||||
assert m2.completed_count() == 1
|
||||
|
||||
def test_mark_file_completed__batch_threshold_reached__auto_flushed_to_disk(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
"""When pending count hits batch_size the buffer is auto-flushed."""
|
||||
batch_size = 3
|
||||
files = [self._make_trace(tmp_base, f"trace_{i:03d}.json") for i in range(3)]
|
||||
|
||||
m1 = MigrationManifest(tmp_base, batch_size=batch_size)
|
||||
m1.start()
|
||||
for f in files:
|
||||
m1.mark_file_completed(f) # third call triggers auto-flush
|
||||
|
||||
# New instance must see all three completions.
|
||||
m2 = MigrationManifest(tmp_base, batch_size=batch_size)
|
||||
assert m2.completed_count() == 3
|
||||
for f in files:
|
||||
assert m2.is_file_completed(f)
|
||||
|
||||
|
||||
class TestResumeScenario:
|
||||
"""Simulate a real interrupted + resumed migration."""
|
||||
|
||||
def test_resume__interrupted_import__completed_files_skipped_and_id_map_available(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
"""After an interrupted run the second run skips already-flushed files.
|
||||
|
||||
batch_size=1 so every mark_file_completed is immediately durable —
|
||||
this models the boundary between two process invocations where only
|
||||
flushed data survives.
|
||||
"""
|
||||
trace_files = []
|
||||
for i in range(3):
|
||||
p = tmp_base / "projects" / "proj" / f"trace_{i:03d}.json"
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.touch()
|
||||
trace_files.append(p)
|
||||
|
||||
m1 = MigrationManifest(tmp_base, batch_size=1)
|
||||
m1.start()
|
||||
m1.add_trace_mapping("src-0", "dest-0")
|
||||
m1.mark_file_completed(trace_files[0])
|
||||
m1.add_trace_mapping("src-1", "dest-1")
|
||||
m1.mark_file_completed(trace_files[1])
|
||||
# Process crashes before trace_files[2] — no complete() call.
|
||||
|
||||
m2 = MigrationManifest(tmp_base)
|
||||
assert m2.is_in_progress
|
||||
assert m2.completed_count() == 2
|
||||
assert m2.is_file_completed(trace_files[0])
|
||||
assert m2.is_file_completed(trace_files[1])
|
||||
assert not m2.is_file_completed(trace_files[2])
|
||||
|
||||
id_map = m2.get_trace_id_map()
|
||||
assert id_map["src-0"] == "dest-0"
|
||||
assert id_map["src-1"] == "dest-1"
|
||||
assert "src-2" not in id_map
|
||||
|
||||
def test_reset__after_completed_files__all_state_cleared(
|
||||
self, tmp_base: Path
|
||||
) -> None:
|
||||
trace_file = tmp_base / "projects" / "p" / "trace_abc.json"
|
||||
trace_file.parent.mkdir(parents=True)
|
||||
trace_file.touch()
|
||||
|
||||
m1 = MigrationManifest(tmp_base)
|
||||
m1.start()
|
||||
m1.mark_file_completed(trace_file)
|
||||
|
||||
m2 = MigrationManifest(tmp_base)
|
||||
m2.reset()
|
||||
|
||||
assert m2.completed_count() == 0
|
||||
assert not m2.is_file_completed(trace_file)
|
||||
assert m2.status == "not_started"
|
||||
@@ -0,0 +1,723 @@
|
||||
import base64
|
||||
import uuid
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import click
|
||||
import pytest
|
||||
|
||||
from opik.cli.local_runner.pairing import (
|
||||
PairingResult,
|
||||
RunnerType,
|
||||
build_pairing_link,
|
||||
generate_runner_name,
|
||||
hkdf_sha256,
|
||||
resolve_project_id,
|
||||
run_headless,
|
||||
run_pairing,
|
||||
validate_runner_name,
|
||||
)
|
||||
from opik.rest_api.errors.not_found_error import NotFoundError
|
||||
|
||||
|
||||
# Helpers — fill in the keyword-only args that production callers always supply
|
||||
# so tests don't have to repeat them. Tests override what they actually care
|
||||
# about via **overrides.
|
||||
def _resolve(api, project_name, **overrides):
|
||||
kwargs = dict(
|
||||
create_if_missing=False,
|
||||
workspace=None,
|
||||
base_url=None,
|
||||
config_file_exists=True,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return resolve_project_id(api, project_name, **kwargs)
|
||||
|
||||
|
||||
def _headless(api, **overrides):
|
||||
kwargs = dict(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="r-1",
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
create_if_missing=False,
|
||||
workspace=None,
|
||||
base_url=None,
|
||||
config_file_exists=True,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return run_headless(**kwargs)
|
||||
|
||||
|
||||
def _link(**overrides):
|
||||
kwargs = dict(
|
||||
base_url="https://www.comet.com/opik/api/",
|
||||
session_id="550e8400-e29b-41d4-a716-446655440000",
|
||||
activation_key=b"\x00" * 32,
|
||||
project_id="660e8400-e29b-41d4-a716-446655440000",
|
||||
project_name="my-proj",
|
||||
runner_name="r",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
workspace=None,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return build_pairing_link(**kwargs)
|
||||
|
||||
|
||||
def _pair(api, **overrides):
|
||||
kwargs = dict(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="r-1",
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
workspace=None,
|
||||
tui=None,
|
||||
create_if_missing=False,
|
||||
config_file_exists=True,
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return run_pairing(**kwargs)
|
||||
|
||||
|
||||
class TestHKDF:
|
||||
def test_hkdf__pinned_vector__matches_expected(self):
|
||||
ikm = bytes(range(32))
|
||||
salt = uuid.UUID("00000000-0000-0000-0000-000000000001").bytes
|
||||
info = b"opik-bridge-v1"
|
||||
|
||||
okm = hkdf_sha256(ikm=ikm, salt=salt, info=info)
|
||||
|
||||
assert (
|
||||
okm.hex()
|
||||
== "9647b7959765ecad68dfef02f31cfca7f7901a9076c15ebabb1f35d015f71198"
|
||||
)
|
||||
|
||||
def test_hkdf__any_input__output_is_32_bytes(self):
|
||||
okm = hkdf_sha256(ikm=b"\x00" * 32, salt=b"\x00" * 16, info=b"test")
|
||||
assert len(okm) == 32
|
||||
|
||||
def test_hkdf__different_info__different_output(self):
|
||||
a = hkdf_sha256(ikm=b"\x00" * 32, salt=b"\x00" * 16, info=b"a")
|
||||
b = hkdf_sha256(ikm=b"\x00" * 32, salt=b"\x00" * 16, info=b"b")
|
||||
assert a != b
|
||||
|
||||
|
||||
class TestUUIDBytesOrder:
|
||||
def test_uuid_bytes__known_uuid__matches_java_big_endian(self):
|
||||
u = uuid.UUID("550e8400-e29b-41d4-a716-446655440000")
|
||||
assert u.bytes.hex() == "550e8400e29b41d4a716446655440000"
|
||||
|
||||
|
||||
class TestBuildPairingLink:
|
||||
def test_build_pairing_link__valid_inputs__correct_payload_layout(self):
|
||||
session_id = "550e8400-e29b-41d4-a716-446655440000"
|
||||
project_id = "660e8400-e29b-41d4-a716-446655440000"
|
||||
activation_key = b"\xaa" * 32
|
||||
runner_name = "my-runner"
|
||||
|
||||
link = _link(
|
||||
base_url="https://www.comet.com/opik/api/",
|
||||
session_id=session_id,
|
||||
activation_key=activation_key,
|
||||
project_id=project_id,
|
||||
runner_name=runner_name,
|
||||
)
|
||||
|
||||
assert link.startswith("https://www.comet.com/opik/pair/v1?")
|
||||
|
||||
fragment = link.split("#", 1)[1]
|
||||
padding_needed = (4 - len(fragment) % 4) % 4
|
||||
payload = base64.urlsafe_b64decode(fragment + "=" * padding_needed)
|
||||
|
||||
assert payload[0:16] == uuid.UUID(session_id).bytes
|
||||
assert payload[16:48] == activation_key
|
||||
assert payload[48:64] == uuid.UUID(project_id).bytes
|
||||
assert payload[64] == len(runner_name.encode("utf-8"))
|
||||
name_end = 65 + payload[64]
|
||||
assert payload[65:name_end] == runner_name.encode("utf-8")
|
||||
# Default runner type is CONNECT (0x00)
|
||||
assert payload[name_end] == 0x00
|
||||
|
||||
def test_build_pairing_link__endpoint_type__encodes_0x01(self):
|
||||
link = _link(
|
||||
base_url="http://localhost:5173/api/",
|
||||
session_id="550e8400-e29b-41d4-a716-446655440000",
|
||||
activation_key=b"\x00" * 32,
|
||||
project_id="660e8400-e29b-41d4-a716-446655440000",
|
||||
runner_name="r",
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
)
|
||||
fragment = link.split("#", 1)[1]
|
||||
padding_needed = (4 - len(fragment) % 4) % 4
|
||||
payload = base64.urlsafe_b64decode(fragment + "=" * padding_needed)
|
||||
# name_len=1, name="r", then type byte
|
||||
assert payload[66] == 0x01
|
||||
|
||||
def test_build_pairing_link__cloud_url__no_double_opik_path(self):
|
||||
link = _link(
|
||||
base_url="https://www.comet.com/opik/api/",
|
||||
session_id="550e8400-e29b-41d4-a716-446655440000",
|
||||
activation_key=b"\x00" * 32,
|
||||
project_id="660e8400-e29b-41d4-a716-446655440000",
|
||||
runner_name="r",
|
||||
)
|
||||
assert "/opik/opik/" not in link
|
||||
|
||||
def test_build_pairing_link__localhost_url__correct_prefix(self):
|
||||
link = _link(
|
||||
base_url="http://localhost:5173/api/",
|
||||
session_id="550e8400-e29b-41d4-a716-446655440000",
|
||||
activation_key=b"\x00" * 32,
|
||||
project_id="660e8400-e29b-41d4-a716-446655440000",
|
||||
runner_name="r",
|
||||
)
|
||||
assert link.startswith("http://localhost:5173/opik/pair/v1?")
|
||||
|
||||
def test_build_pairing_link__always_includes_url_param(self):
|
||||
link = _link(
|
||||
base_url="http://localhost:5173/api/",
|
||||
workspace=None,
|
||||
)
|
||||
# `/api` is stripped — we send the URL the user would visit in a
|
||||
# browser, not the internal API endpoint.
|
||||
assert "url=http%3A%2F%2Flocalhost%3A5173%2F" in link
|
||||
assert "%2Fapi%2F" not in link
|
||||
|
||||
def test_build_pairing_link__cloud_api_url__strips_api_suffix(self):
|
||||
link = _link(
|
||||
base_url="https://www.comet.com/opik/api/",
|
||||
workspace=None,
|
||||
)
|
||||
assert "url=https%3A%2F%2Fwww.comet.com%2Fopik%2F" in link
|
||||
|
||||
def test_build_pairing_link__url_param_precedes_workspace(self):
|
||||
link = _link(
|
||||
base_url="http://localhost:5173/api/",
|
||||
workspace="team-a",
|
||||
)
|
||||
# All query params present; order is stable for FE parsing.
|
||||
assert "url=" in link
|
||||
assert "project=my-proj" in link
|
||||
assert "workspace=team-a" in link
|
||||
|
||||
def test_build_pairing_link__project_name_with_special_chars__url_encoded(
|
||||
self,
|
||||
):
|
||||
link = _link(project_name="My Project / Demo")
|
||||
# Spaces and `/` must be percent-encoded so the query stays parseable.
|
||||
assert "project=My%20Project%20%2F%20Demo" in link
|
||||
|
||||
def test_build_pairing_link__workspace_with_special_chars__url_encoded(
|
||||
self,
|
||||
):
|
||||
# `&` and `=` in a workspace name would otherwise split the query and
|
||||
# the FE's URLSearchParams would read a truncated value.
|
||||
link = _link(workspace="weird&name=oops")
|
||||
assert "workspace=weird%26name%3Doops" in link
|
||||
|
||||
|
||||
class TestResolveProjectId:
|
||||
def test_resolve_project_id__project_exists__returns_id(self):
|
||||
api = MagicMock()
|
||||
project = MagicMock()
|
||||
project.id = "proj-uuid-123"
|
||||
api.projects.retrieve_project.return_value = project
|
||||
|
||||
result = _resolve(api, "my-project")
|
||||
assert result == "proj-uuid-123"
|
||||
api.projects.retrieve_project.assert_called_once_with(name="my-project")
|
||||
|
||||
def test_resolve_project_id__project_missing__uses_404_hint(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
# Project retrieve returns Opik's custom ErrorMessage shape.
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["Project not found"]}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(api, "nonexistent")
|
||||
message = exc_info.value.message
|
||||
assert "Could not retrieve project 'nonexistent'" in message
|
||||
assert "Project not found" in message
|
||||
assert "check the project name and try again" in message
|
||||
|
||||
def test_resolve_project_id__unauthorized__surfaces_server_message_and_docs(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=401,
|
||||
body={"code": 401, "message": "API key should be provided"},
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(api, "my-project")
|
||||
message = exc_info.value.message
|
||||
assert "Could not retrieve project 'my-project'" in message
|
||||
assert "API key should be provided" in message
|
||||
assert "Run: opik configure" in message
|
||||
assert (
|
||||
"https://www.comet.com/docs/opik/tracing/advanced/sdk_configuration"
|
||||
in message
|
||||
)
|
||||
|
||||
def test_resolve_project_id__forbidden__shares_auth_hint(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=403,
|
||||
body={"code": 403, "message": "User is not allowed to access workspace"},
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(api, "my-project")
|
||||
message = exc_info.value.message
|
||||
assert "User is not allowed to access workspace" in message
|
||||
assert "Run: opik configure" in message
|
||||
|
||||
def test_resolve_project_id__server_error__falls_back_to_generic_hint(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=500,
|
||||
body="Internal Server Error",
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(api, "my-project")
|
||||
message = exc_info.value.message
|
||||
assert "Internal Server Error" in message
|
||||
assert "verify your Opik configuration and connectivity" in message
|
||||
assert (
|
||||
"https://www.comet.com/docs/opik/tracing/advanced/sdk_configuration"
|
||||
in message
|
||||
)
|
||||
|
||||
def test_resolve_project_id__empty_body__falls_back_to_status_text(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(status_code=502, body=None)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(api, "my-project")
|
||||
assert "server returned HTTP 502" in exc_info.value.message
|
||||
|
||||
def test_resolve_project_id__missing_and_create_flag__creates_and_resolves(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
created = MagicMock()
|
||||
created.id = "proj-uuid-new"
|
||||
api.projects.retrieve_project.side_effect = [
|
||||
ApiError(status_code=404, body={"errors": ["Project not found"]}),
|
||||
created,
|
||||
]
|
||||
|
||||
result = _resolve(api, "new-project", create_if_missing=True)
|
||||
assert result == "proj-uuid-new"
|
||||
api.projects.create_project.assert_called_once_with(name="new-project")
|
||||
assert api.projects.retrieve_project.call_count == 2
|
||||
|
||||
def test_resolve_project_id__known_missing__skips_initial_lookup(self):
|
||||
api = MagicMock()
|
||||
created = MagicMock()
|
||||
created.id = "proj-uuid-new"
|
||||
api.projects.retrieve_project.return_value = created
|
||||
|
||||
result = _resolve(
|
||||
api,
|
||||
"new-project",
|
||||
create_if_missing=True,
|
||||
project_known_missing=True,
|
||||
)
|
||||
assert result == "proj-uuid-new"
|
||||
api.projects.create_project.assert_called_once_with(name="new-project")
|
||||
# Interactive preflight already saw the 404 — only the post-create
|
||||
# lookup should run.
|
||||
assert api.projects.retrieve_project.call_count == 1
|
||||
|
||||
def test_resolve_project_id__missing_no_flag__does_not_create(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["Project not found"]}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException):
|
||||
_resolve(api, "missing", create_if_missing=False)
|
||||
api.projects.create_project.assert_not_called()
|
||||
|
||||
def test_resolve_project_id__non_404_with_flag__does_not_create(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=401, body={"message": "unauthorized"}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException):
|
||||
_resolve(api, "x", create_if_missing=True)
|
||||
api.projects.create_project.assert_not_called()
|
||||
|
||||
def test_resolve_project_id__create_fails__raises_clean_error(self):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["Project not found"]}
|
||||
)
|
||||
api.projects.create_project.side_effect = ApiError(
|
||||
status_code=403, body={"message": "User cannot create projects"}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(api, "blocked", create_if_missing=True)
|
||||
message = exc_info.value.message
|
||||
assert "Could not create project 'blocked'" in message
|
||||
assert "User cannot create projects" in message
|
||||
|
||||
def test_resolve_project_id__retrieval_error_with_config_context__appends_workspace_and_url(
|
||||
self,
|
||||
):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["Project not found"]}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(
|
||||
api,
|
||||
"missing-proj",
|
||||
workspace="team-a",
|
||||
base_url="https://opik.example.com/api/",
|
||||
)
|
||||
message = exc_info.value.message
|
||||
assert "team-a" in message
|
||||
assert "https://opik.example.com/api/" in message
|
||||
|
||||
def test_resolve_project_id__retrieval_error_without_config_file__appends_opik_configure_hint(
|
||||
self,
|
||||
):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=401, body={"message": "API key should be provided"}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(
|
||||
api,
|
||||
"my-project",
|
||||
workspace="default",
|
||||
base_url="https://www.comet.com/opik/api/",
|
||||
config_file_exists=False,
|
||||
)
|
||||
message = exc_info.value.message
|
||||
assert "~/.opik.config" in message
|
||||
assert "Run: opik configure" in message
|
||||
|
||||
def test_resolve_project_id__retrieval_error_with_config_file__omits_no_config_hint(
|
||||
self,
|
||||
):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=500, body="Internal Server Error"
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(
|
||||
api,
|
||||
"p",
|
||||
workspace="default",
|
||||
base_url="https://www.comet.com/opik/api/",
|
||||
config_file_exists=True,
|
||||
)
|
||||
assert "~/.opik.config" not in exc_info.value.message
|
||||
|
||||
def test_resolve_project_id__create_fails_with_config_context__includes_workspace_and_url(
|
||||
self,
|
||||
):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["Project not found"]}
|
||||
)
|
||||
api.projects.create_project.side_effect = ApiError(
|
||||
status_code=403, body={"message": "User cannot create projects"}
|
||||
)
|
||||
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_resolve(
|
||||
api,
|
||||
"blocked",
|
||||
create_if_missing=True,
|
||||
workspace="team-a",
|
||||
base_url="https://opik.example.com/api/",
|
||||
config_file_exists=False,
|
||||
)
|
||||
message = exc_info.value.message
|
||||
assert "Could not create project 'blocked'" in message
|
||||
assert "team-a" in message
|
||||
assert "https://opik.example.com/api/" in message
|
||||
assert "Run: opik configure" in message
|
||||
|
||||
|
||||
class TestValidateRunnerName:
|
||||
def test_validate_runner_name__valid_name__passes(self):
|
||||
validate_runner_name("my-runner")
|
||||
|
||||
def test_validate_runner_name__empty__raises(self):
|
||||
with pytest.raises(Exception, match="empty"):
|
||||
validate_runner_name("")
|
||||
|
||||
def test_validate_runner_name__whitespace_only__raises(self):
|
||||
with pytest.raises(Exception, match="empty"):
|
||||
validate_runner_name(" ")
|
||||
|
||||
def test_validate_runner_name__over_128_chars__raises(self):
|
||||
with pytest.raises(Exception, match="128 characters"):
|
||||
validate_runner_name("x" * 129)
|
||||
|
||||
def test_validate_runner_name__over_255_utf8_bytes__raises(self):
|
||||
name = "\U0001f600" * 64
|
||||
with pytest.raises(Exception, match="255 UTF-8 bytes"):
|
||||
validate_runner_name(name)
|
||||
|
||||
|
||||
class TestGenerateRunnerName:
|
||||
def test_generate_runner_name__explicit_name__returns_it(self):
|
||||
assert generate_runner_name("my-name") == "my-name"
|
||||
|
||||
def test_generate_runner_name__none__generates_random_hex(self):
|
||||
name = generate_runner_name(None)
|
||||
assert "-" in name
|
||||
hex_part = name.rsplit("-", 1)[1]
|
||||
assert len(hex_part) == 6
|
||||
int(hex_part, 16)
|
||||
|
||||
|
||||
class TestRunPairing:
|
||||
SESSION_ID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
|
||||
RUNNER_ID = "11111111-2222-3333-4444-555555555555"
|
||||
PROJECT_ID = "66666666-7777-8888-9999-aaaaaaaaaaaa"
|
||||
|
||||
def _make_api(self, project_id=None):
|
||||
if project_id is None:
|
||||
project_id = self.PROJECT_ID
|
||||
api = MagicMock()
|
||||
project = MagicMock()
|
||||
project.id = project_id
|
||||
api.projects.retrieve_project.return_value = project
|
||||
|
||||
session_resp = MagicMock()
|
||||
session_resp.session_id = self.SESSION_ID
|
||||
session_resp.runner_id = self.RUNNER_ID
|
||||
api.pairing.create_pairing_session.return_value = session_resp
|
||||
|
||||
runner = MagicMock()
|
||||
runner.status = "connected"
|
||||
api.runners.get_runner.return_value = runner
|
||||
|
||||
return api
|
||||
|
||||
@patch("opik.cli.local_runner.pairing.time.sleep")
|
||||
def test_run_pairing__happyflow__returns_result(self, mock_sleep):
|
||||
api = self._make_api()
|
||||
result = _pair(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
ttl_seconds=1,
|
||||
)
|
||||
|
||||
assert isinstance(result, PairingResult)
|
||||
assert result.runner_id == self.RUNNER_ID
|
||||
assert result.project_name == "my-proj"
|
||||
assert result.project_id == self.PROJECT_ID
|
||||
assert len(result.bridge_key) == 32
|
||||
|
||||
@patch("opik.cli.local_runner.pairing.time.sleep")
|
||||
def test_run_pairing__with_tui__calls_started_and_completed(self, mock_sleep):
|
||||
api = self._make_api()
|
||||
tui = MagicMock()
|
||||
|
||||
_pair(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
tui=tui,
|
||||
ttl_seconds=1,
|
||||
)
|
||||
|
||||
tui.pairing_started.assert_called_once()
|
||||
tui.pairing_completed.assert_called_once()
|
||||
|
||||
@patch("opik.cli.local_runner.pairing.time.sleep")
|
||||
def test_run_pairing__404_during_poll__retries_until_connected(self, mock_sleep):
|
||||
api = self._make_api()
|
||||
err = NotFoundError(body=None)
|
||||
runner = MagicMock()
|
||||
runner.status = "connected"
|
||||
api.runners.get_runner.side_effect = [err, err, runner]
|
||||
|
||||
result = _pair(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
ttl_seconds=1,
|
||||
)
|
||||
assert result.runner_id == self.RUNNER_ID
|
||||
assert api.runners.get_runner.call_count == 3
|
||||
|
||||
@patch("opik.cli.local_runner.pairing.time.monotonic")
|
||||
@patch("opik.cli.local_runner.pairing.time.sleep")
|
||||
def test_run_pairing__timeout__calls_tui_pairing_failed(
|
||||
self, mock_sleep, mock_monotonic
|
||||
):
|
||||
api = self._make_api()
|
||||
runner = MagicMock()
|
||||
runner.status = "pairing"
|
||||
api.runners.get_runner.return_value = runner
|
||||
|
||||
# Strictly-increasing clock: each call returns previous + 1000s. This
|
||||
# guarantees the while-loop exits on the call immediately after
|
||||
# `deadline = monotonic() + ttl_seconds` is computed, regardless of
|
||||
# how many other calls (from logging, coverage, plugins, etc.) land
|
||||
# on the patched monotonic before or after the test's own calls.
|
||||
clock = {"t": 0.0}
|
||||
|
||||
def advancing_monotonic():
|
||||
clock["t"] += 1000.0
|
||||
return clock["t"]
|
||||
|
||||
mock_monotonic.side_effect = advancing_monotonic
|
||||
|
||||
tui = MagicMock()
|
||||
with pytest.raises(click.ClickException) as exc_info:
|
||||
_pair(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
tui=tui,
|
||||
ttl_seconds=300,
|
||||
)
|
||||
assert "timed out" in exc_info.value.message
|
||||
|
||||
tui.pairing_failed.assert_called_once_with("timed out")
|
||||
|
||||
@patch("opik.cli.local_runner.pairing.time.sleep")
|
||||
def test_run_pairing__keyboard_interrupt__calls_tui_pairing_failed(
|
||||
self, mock_sleep
|
||||
):
|
||||
api = self._make_api()
|
||||
api.runners.get_runner.side_effect = KeyboardInterrupt
|
||||
|
||||
tui = MagicMock()
|
||||
with pytest.raises(KeyboardInterrupt):
|
||||
_pair(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
tui=tui,
|
||||
ttl_seconds=1,
|
||||
)
|
||||
|
||||
tui.pairing_failed.assert_called_once_with("interrupted")
|
||||
|
||||
@patch("opik.cli.local_runner.pairing.time.sleep")
|
||||
def test_run_pairing__non_404_api_error__propagates(self, mock_sleep):
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
api = self._make_api()
|
||||
api.runners.get_runner.side_effect = ApiError(
|
||||
status_code=429, body="rate limited"
|
||||
)
|
||||
|
||||
with pytest.raises(ApiError) as exc_info:
|
||||
_pair(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
base_url="http://localhost:5173/api/",
|
||||
ttl_seconds=1,
|
||||
)
|
||||
assert exc_info.value.status_code == 429
|
||||
|
||||
|
||||
class TestRunHeadless:
|
||||
SESSION_ID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
|
||||
RUNNER_ID = "11111111-2222-3333-4444-555555555555"
|
||||
PROJECT_ID = "66666666-7777-8888-9999-aaaaaaaaaaaa"
|
||||
|
||||
def _make_api(self):
|
||||
api = MagicMock()
|
||||
project = MagicMock()
|
||||
project.id = self.PROJECT_ID
|
||||
api.projects.retrieve_project.return_value = project
|
||||
|
||||
session_resp = MagicMock()
|
||||
session_resp.session_id = self.SESSION_ID
|
||||
session_resp.runner_id = self.RUNNER_ID
|
||||
api.pairing.create_pairing_session.return_value = session_resp
|
||||
api.pairing.activate_pairing_session.return_value = None
|
||||
|
||||
return api
|
||||
|
||||
def test_run_headless__creates_and_self_activates(self):
|
||||
api = self._make_api()
|
||||
result = _headless(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
)
|
||||
|
||||
assert result.runner_id == self.RUNNER_ID
|
||||
assert result.project_id == self.PROJECT_ID
|
||||
assert result.bridge_key == b""
|
||||
api.pairing.create_pairing_session.assert_called_once()
|
||||
api.pairing.activate_pairing_session.assert_called_once()
|
||||
|
||||
def test_run_headless__no_polling(self):
|
||||
api = self._make_api()
|
||||
_headless(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.ENDPOINT,
|
||||
)
|
||||
# No get_runner polling — headless activates immediately
|
||||
api.runners.get_runner.assert_not_called()
|
||||
|
||||
def test_run_headless__connect_type__raises(self):
|
||||
api = self._make_api()
|
||||
with pytest.raises(click.ClickException, match="not supported"):
|
||||
_headless(
|
||||
api=api,
|
||||
project_name="my-proj",
|
||||
runner_name="test-runner",
|
||||
runner_type=RunnerType.CONNECT,
|
||||
)
|
||||
@@ -0,0 +1,175 @@
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from opik.cli.local_runner.preflight import (
|
||||
maybe_auto_configure,
|
||||
should_create_project,
|
||||
)
|
||||
from opik.rest_api.core.api_error import ApiError
|
||||
|
||||
|
||||
class TestShouldCreateProject:
|
||||
def test__headless__returns_create_without_lookup(self):
|
||||
api = MagicMock()
|
||||
result = should_create_project(api, "any", workspace=None, headless=True)
|
||||
# Headless skips preflight, so we want to create but don't yet know
|
||||
# whether the project is actually missing — the resolver must look up.
|
||||
assert result == (True, False)
|
||||
api.projects.retrieve_project.assert_not_called()
|
||||
|
||||
def test__project_exists__returns_no_create(self):
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.return_value = MagicMock(id="proj-1")
|
||||
result = should_create_project(api, "exists", workspace="ws", headless=False)
|
||||
assert result == (False, False)
|
||||
|
||||
def test__non_404_error__returns_no_create_to_let_downstream_format(self):
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=401, body={"message": "unauthorized"}
|
||||
)
|
||||
result = should_create_project(api, "x", workspace="ws", headless=False)
|
||||
assert result == (False, False)
|
||||
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__missing_and_not_tty__returns_no_create(self, mock_stdin):
|
||||
mock_stdin.isatty.return_value = False
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["not found"]}
|
||||
)
|
||||
result = should_create_project(api, "missing", workspace="ws", headless=False)
|
||||
assert result == (False, False)
|
||||
|
||||
@patch("opik.cli.local_runner.preflight.click.confirm", return_value=True)
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__missing_and_tty_user_confirms__returns_create_and_known_missing(
|
||||
self, mock_stdin, mock_confirm
|
||||
):
|
||||
mock_stdin.isatty.return_value = True
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["not found"]}
|
||||
)
|
||||
result = should_create_project(api, "missing", workspace="ws", headless=False)
|
||||
# Interactive preflight saw 404 and user confirmed — downstream can
|
||||
# skip the redundant lookup.
|
||||
assert result == (True, True)
|
||||
prompt_text = mock_confirm.call_args[0][0]
|
||||
assert "missing" in prompt_text
|
||||
assert "ws" in prompt_text
|
||||
|
||||
@patch("opik.cli.local_runner.preflight.click.confirm", return_value=False)
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__missing_and_tty_user_declines__returns_no_create(
|
||||
self, mock_stdin, mock_confirm
|
||||
):
|
||||
mock_stdin.isatty.return_value = True
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(
|
||||
status_code=404, body={"errors": ["not found"]}
|
||||
)
|
||||
result = should_create_project(api, "missing", workspace="ws", headless=False)
|
||||
assert result == (False, False)
|
||||
|
||||
@patch("opik.cli.local_runner.preflight.click.confirm", return_value=True)
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__no_workspace__prompt_omits_workspace_label(
|
||||
self, mock_stdin, mock_confirm
|
||||
):
|
||||
mock_stdin.isatty.return_value = True
|
||||
api = MagicMock()
|
||||
api.projects.retrieve_project.side_effect = ApiError(status_code=404, body={})
|
||||
should_create_project(api, "missing", workspace=None, headless=False)
|
||||
prompt_text = mock_confirm.call_args[0][0]
|
||||
assert "in workspace" not in prompt_text
|
||||
|
||||
|
||||
class TestMaybeAutoConfigure:
|
||||
def _patch_env(self, monkeypatch, opik_api_key=None):
|
||||
if opik_api_key is None:
|
||||
monkeypatch.delenv("OPIK_API_KEY", raising=False)
|
||||
else:
|
||||
monkeypatch.setenv("OPIK_API_KEY", opik_api_key)
|
||||
|
||||
def _config_probe(self, *, config_file_exists=False, api_key=None):
|
||||
probe = MagicMock()
|
||||
probe.config_file_exists = config_file_exists
|
||||
probe.api_key = api_key
|
||||
return probe
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
def test__non_interactive_flag__skips(self, mock_configure, monkeypatch):
|
||||
self._patch_env(monkeypatch)
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=True, headless=False)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
def test__headless__skips(self, mock_configure, monkeypatch):
|
||||
self._patch_env(monkeypatch)
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=True)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
def test__api_key_arg__skips(self, mock_configure, monkeypatch):
|
||||
self._patch_env(monkeypatch)
|
||||
maybe_auto_configure(api_key_arg="abc", non_interactive=False, headless=False)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
def test__env_api_key__skips(self, mock_configure, monkeypatch):
|
||||
self._patch_env(monkeypatch, opik_api_key="from-env")
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__stdin_not_tty__skips(self, mock_stdin, mock_configure, monkeypatch):
|
||||
self._patch_env(monkeypatch)
|
||||
mock_stdin.isatty.return_value = False
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
@patch("opik.cli.local_runner.preflight.OpikConfig")
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__config_file_already_exists__skips(
|
||||
self, mock_stdin, mock_config_cls, mock_configure, monkeypatch
|
||||
):
|
||||
self._patch_env(monkeypatch)
|
||||
mock_stdin.isatty.return_value = True
|
||||
mock_config_cls.return_value = self._config_probe(config_file_exists=True)
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
@patch("opik.cli.local_runner.preflight.OpikConfig")
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__probe_resolves_api_key_from_env_chain__skips(
|
||||
self, mock_stdin, mock_config_cls, mock_configure, monkeypatch
|
||||
):
|
||||
# The OpikConfig probe layer can pick up an API key from elsewhere
|
||||
# (e.g. .env file) even when OPIK_API_KEY isn't in os.environ. We
|
||||
# treat that as "already configured enough" and skip the prompt.
|
||||
self._patch_env(monkeypatch)
|
||||
mock_stdin.isatty.return_value = True
|
||||
mock_config_cls.return_value = self._config_probe(api_key="from-dotenv")
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
|
||||
mock_configure.assert_not_called()
|
||||
|
||||
@patch("opik.cli.configure.run_interactive_configure")
|
||||
@patch("opik.cli.local_runner.preflight.OpikConfig")
|
||||
@patch("opik.cli.local_runner.preflight.sys.stdin")
|
||||
def test__no_config_and_interactive__runs_configure(
|
||||
self, mock_stdin, mock_config_cls, mock_configure, monkeypatch
|
||||
):
|
||||
self._patch_env(monkeypatch)
|
||||
mock_stdin.isatty.return_value = True
|
||||
mock_config_cls.return_value = self._config_probe()
|
||||
maybe_auto_configure(api_key_arg=None, non_interactive=False, headless=False)
|
||||
mock_configure.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,440 @@
|
||||
"""Unit tests verifying that item tags round-trip through CLI export/import.
|
||||
|
||||
Covers the regression where ``opik export`` / ``opik import`` silently dropped
|
||||
tags for prompts, datasets, and experiments because the exporters used
|
||||
hand-written field allowlists that omitted ``tags`` (and the importers never
|
||||
forwarded them). Traces and spans already round-tripped correctly.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
# The experiment/prompt import modules pull in a prompt module that is awkward
|
||||
# to import in isolation; mirror the shim used by test_import_experiment.py.
|
||||
sys.modules.setdefault("opik.api_objects.prompt.prompt", MagicMock())
|
||||
|
||||
from opik.cli.imports.prompt import import_prompts_from_directory # noqa: E402
|
||||
from opik.cli.imports.dataset import import_datasets_from_directory # noqa: E402
|
||||
from opik.cli.exports.prompt import export_single_prompt # noqa: E402
|
||||
from opik.cli.imports.experiment import ( # noqa: E402
|
||||
ExperimentData,
|
||||
recreate_experiment,
|
||||
)
|
||||
from opik.cli.exports.utils import create_experiment_data_structure # noqa: E402
|
||||
|
||||
|
||||
class TestPromptTagsImport:
|
||||
def test_import_text_prompt__with_tags__forwards_tags(self, tmp_path: Path) -> None:
|
||||
prompt_file = tmp_path / "prompt_p1.json"
|
||||
prompt_file.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"name": "greeting",
|
||||
"current_version": {
|
||||
"prompt": "Hello {{name}}",
|
||||
"type": "mustache",
|
||||
"template_structure": "text",
|
||||
"tags": ["prod", "greeting"],
|
||||
},
|
||||
"history": [],
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
client = Mock()
|
||||
client.create_prompt = Mock()
|
||||
|
||||
import_prompts_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.create_prompt.assert_called_once()
|
||||
assert client.create_prompt.call_args.kwargs["tags"] == ["prod", "greeting"]
|
||||
|
||||
def test_import_chat_prompt__with_tags__forwards_tags(self, tmp_path: Path) -> None:
|
||||
prompt_file = tmp_path / "prompt_c1.json"
|
||||
prompt_file.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"name": "chat",
|
||||
"current_version": {
|
||||
"prompt": [{"role": "user", "content": "hi"}],
|
||||
"type": "mustache",
|
||||
"template_structure": "chat",
|
||||
"tags": ["chatty"],
|
||||
},
|
||||
"history": [],
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
client = Mock()
|
||||
client.create_chat_prompt = Mock()
|
||||
|
||||
import_prompts_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.create_chat_prompt.assert_called_once()
|
||||
assert client.create_chat_prompt.call_args.kwargs["tags"] == ["chatty"]
|
||||
|
||||
|
||||
class TestDatasetTagsImport:
|
||||
def _write_dataset(self, tmp_path: Path, payload: dict) -> None:
|
||||
(tmp_path / "dataset_d1.json").write_text(json.dumps(payload))
|
||||
|
||||
def _make_client(self) -> Mock:
|
||||
client = Mock()
|
||||
# Force the create path (get_dataset raises -> create_dataset used).
|
||||
client.get_dataset = Mock(side_effect=Exception("not found"))
|
||||
created = Mock()
|
||||
created.id = "new-ds-id"
|
||||
client.create_dataset = Mock(return_value=created)
|
||||
client.rest_client = Mock()
|
||||
client.rest_client.datasets = Mock()
|
||||
client.rest_client.datasets.update_dataset = Mock()
|
||||
return client
|
||||
|
||||
def test_import_dataset__flat_format_with_tags__applies_tags(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
self._write_dataset(
|
||||
tmp_path,
|
||||
{"name": "ds", "tags": ["a", "b"], "items": []},
|
||||
)
|
||||
client = self._make_client()
|
||||
|
||||
import_datasets_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.rest_client.datasets.update_dataset.assert_called_once()
|
||||
call = client.rest_client.datasets.update_dataset.call_args
|
||||
assert call.args[0] == "new-ds-id"
|
||||
assert call.kwargs["tags"] == ["a", "b"]
|
||||
|
||||
def test_import_dataset__nested_format_with_tags__applies_tags(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
self._write_dataset(
|
||||
tmp_path,
|
||||
{"dataset": {"name": "ds", "id": "x", "tags": ["c"]}, "items": []},
|
||||
)
|
||||
client = self._make_client()
|
||||
|
||||
import_datasets_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.rest_client.datasets.update_dataset.assert_called_once()
|
||||
assert client.rest_client.datasets.update_dataset.call_args.kwargs["tags"] == [
|
||||
"c"
|
||||
]
|
||||
|
||||
def test_import_dataset__no_tags_field__skips_update(self, tmp_path: Path) -> None:
|
||||
self._write_dataset(tmp_path, {"name": "ds", "items": []})
|
||||
client = self._make_client()
|
||||
|
||||
import_datasets_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.rest_client.datasets.update_dataset.assert_not_called()
|
||||
|
||||
def test_import_dataset__empty_tags_list__clears_tags(self, tmp_path: Path) -> None:
|
||||
# An explicit empty list must still call the REST update so a
|
||||
# re-import can clear pre-existing tags on the destination.
|
||||
self._write_dataset(tmp_path, {"name": "ds", "tags": [], "items": []})
|
||||
client = self._make_client()
|
||||
|
||||
import_datasets_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.rest_client.datasets.update_dataset.assert_called_once()
|
||||
assert client.rest_client.datasets.update_dataset.call_args.kwargs["tags"] == []
|
||||
|
||||
def test_import_dataset__existing_dataset_with_description__preserves_description(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# Tags-only import must not silently null out a description the
|
||||
# destination dataset already had (the backend's update PUT doesn't
|
||||
# COALESCE description against the existing row).
|
||||
self._write_dataset(
|
||||
tmp_path,
|
||||
{"name": "ds", "tags": ["a"], "items": []},
|
||||
)
|
||||
client = self._make_client()
|
||||
# Re-import onto a dataset that already exists (get_dataset succeeds).
|
||||
existing_dataset = Mock()
|
||||
existing_dataset.id = "existing-ds-id"
|
||||
client.get_dataset = Mock(return_value=existing_dataset)
|
||||
existing_remote = Mock()
|
||||
existing_remote.description = "original description"
|
||||
existing_remote.visibility = "private"
|
||||
client.rest_client.datasets.get_dataset_by_id = Mock(
|
||||
return_value=existing_remote
|
||||
)
|
||||
|
||||
import_datasets_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.rest_client.datasets.update_dataset.assert_called_once()
|
||||
call = client.rest_client.datasets.update_dataset.call_args
|
||||
assert call.kwargs["description"] == "original description"
|
||||
assert call.kwargs["visibility"] == "private"
|
||||
assert call.kwargs["tags"] == ["a"]
|
||||
|
||||
def test_import_dataset__update_dataset_raises__continues_and_inserts_items(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# A tag-update failure must be swallowed with a warning so the rest of
|
||||
# the import (item insertion, count, manifest) still proceeds.
|
||||
self._write_dataset(
|
||||
tmp_path,
|
||||
{"name": "ds", "tags": ["a"], "items": [{"id": "1", "input": "x"}]},
|
||||
)
|
||||
client = self._make_client()
|
||||
client.rest_client.datasets.update_dataset = Mock(side_effect=Exception("boom"))
|
||||
|
||||
result = import_datasets_from_directory(
|
||||
client=client,
|
||||
source_dir=tmp_path,
|
||||
project_name="proj",
|
||||
dry_run=False,
|
||||
name_pattern=None,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
# The import did not abort: items were still inserted and the dataset
|
||||
# counted as imported.
|
||||
client.create_dataset.return_value.insert.assert_called_once()
|
||||
assert result["datasets"] == 1
|
||||
assert result["datasets_errors"] == 0
|
||||
|
||||
|
||||
class TestPromptTagsExport:
|
||||
"""Prompt tags are container-level and are dropped by the direct
|
||||
get_prompt() lookup; the exporter must recover them via search_prompts.
|
||||
"""
|
||||
|
||||
def test_export_single_prompt__object_has_tags__skips_search(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
prompt = Mock()
|
||||
prompt.id = "p1"
|
||||
prompt.name = "greeting"
|
||||
prompt.tags = ["prod"] # direct lookup already carries the tags
|
||||
client = Mock()
|
||||
# No history available; _safe_prompt_history swallows the error.
|
||||
client.get_prompt_history = Mock(side_effect=Exception("no history"))
|
||||
|
||||
count = export_single_prompt(
|
||||
client=client,
|
||||
prompt=prompt,
|
||||
output_dir=tmp_path,
|
||||
project_name="proj",
|
||||
max_results=None,
|
||||
force=True,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert count == 1
|
||||
data = json.loads((tmp_path / "prompt_p1.json").read_text())
|
||||
assert data["current_version"]["tags"] == ["prod"]
|
||||
# Tags were already present, so no recovery search is needed.
|
||||
client.search_prompts.assert_not_called()
|
||||
|
||||
def test_export_single_prompt__container_only_tags__recovered_from_search(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
prompt = Mock()
|
||||
prompt.id = "p1"
|
||||
prompt.name = "greeting"
|
||||
prompt.tags = [] # direct lookup dropped the container-level tags
|
||||
|
||||
candidate = Mock()
|
||||
candidate.name = "greeting"
|
||||
candidate.tags = ["prod", "greeting"]
|
||||
client = Mock()
|
||||
client.search_prompts = Mock(return_value=[candidate])
|
||||
# No history available; _safe_prompt_history swallows the error.
|
||||
client.get_prompt_history = Mock(side_effect=Exception("no history"))
|
||||
|
||||
count = export_single_prompt(
|
||||
client=client,
|
||||
prompt=prompt,
|
||||
output_dir=tmp_path,
|
||||
project_name="proj",
|
||||
max_results=None,
|
||||
force=True,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert count == 1
|
||||
data = json.loads((tmp_path / "prompt_p1.json").read_text())
|
||||
assert data["current_version"]["tags"] == ["prod", "greeting"]
|
||||
|
||||
def test_export_single_prompt__search_fails__still_exports_with_fallback(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
# Tag resolution is best-effort: a search_prompts failure is logged but
|
||||
# must not abort the export — the prompt still exports with whatever tags
|
||||
# the object carried.
|
||||
prompt = Mock()
|
||||
prompt.id = "p1"
|
||||
prompt.name = "greeting"
|
||||
prompt.tags = [] # nothing on the object; recovery will be attempted
|
||||
client = Mock()
|
||||
client.search_prompts = Mock(side_effect=Exception("api down"))
|
||||
client.get_prompt_history = Mock(side_effect=Exception("no history"))
|
||||
|
||||
count = export_single_prompt(
|
||||
client=client,
|
||||
prompt=prompt,
|
||||
output_dir=tmp_path,
|
||||
project_name="proj",
|
||||
max_results=None,
|
||||
force=True,
|
||||
debug=False,
|
||||
format="json",
|
||||
)
|
||||
|
||||
assert count == 1
|
||||
data = json.loads((tmp_path / "prompt_p1.json").read_text())
|
||||
assert data["current_version"]["tags"] == []
|
||||
|
||||
|
||||
class TestExperimentTags:
|
||||
def test_export_experiment_structure__with_tags__includes_tags(self) -> None:
|
||||
experiment = Mock()
|
||||
experiment.id = "exp-1"
|
||||
experiment.name = "e"
|
||||
experiment.dataset_name = "ds"
|
||||
data_obj = Mock()
|
||||
data_obj.tags = ["t1", "t2"]
|
||||
experiment.get_experiment_data = Mock(return_value=data_obj)
|
||||
|
||||
structure = create_experiment_data_structure(experiment, [])
|
||||
|
||||
assert structure["experiment"]["tags"] == ["t1", "t2"]
|
||||
|
||||
def test_import_experiment_from_disk__with_tags__forwards_tags(self) -> None:
|
||||
client = Mock()
|
||||
client.flush = Mock(return_value=True)
|
||||
client.get_or_create_dataset = Mock(return_value=Mock(name="ds"))
|
||||
created_experiment = Mock()
|
||||
created_experiment.id = "exp-123"
|
||||
client.create_experiment = Mock(return_value=created_experiment)
|
||||
client._rest_client = Mock()
|
||||
|
||||
experiment_data = ExperimentData(
|
||||
experiment={
|
||||
"id": "exp-123",
|
||||
"name": "test-experiment",
|
||||
"dataset_name": "test-dataset",
|
||||
"type": "regular",
|
||||
"tags": ["golden", "regression"],
|
||||
},
|
||||
items=[],
|
||||
)
|
||||
|
||||
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
|
||||
mock_id_helpers.generate_id = Mock(return_value="generated-id")
|
||||
|
||||
recreate_experiment(
|
||||
client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
{}, # trace_id_map
|
||||
{}, # dataset_item_id_map
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
# target_project_name defaults to None -> import-from-disk path
|
||||
)
|
||||
|
||||
client.create_experiment.assert_called_once()
|
||||
assert client.create_experiment.call_args.kwargs["tags"] == [
|
||||
"golden",
|
||||
"regression",
|
||||
]
|
||||
|
||||
def test_import_experiment_from_disk__empty_tags_list__forwards_empty_tags(
|
||||
self,
|
||||
) -> None:
|
||||
# An explicit empty list must round-trip as tags=[], not collapse to
|
||||
# tags=None (which would drop a deliberately cleared tag set).
|
||||
client = Mock()
|
||||
client.flush = Mock(return_value=True)
|
||||
client.get_or_create_dataset = Mock(return_value=Mock(name="ds"))
|
||||
created_experiment = Mock()
|
||||
created_experiment.id = "exp-123"
|
||||
client.create_experiment = Mock(return_value=created_experiment)
|
||||
client._rest_client = Mock()
|
||||
|
||||
experiment_data = ExperimentData(
|
||||
experiment={
|
||||
"id": "exp-123",
|
||||
"name": "test-experiment",
|
||||
"dataset_name": "test-dataset",
|
||||
"type": "regular",
|
||||
"tags": [],
|
||||
},
|
||||
items=[],
|
||||
)
|
||||
|
||||
with patch("opik.cli.imports.experiment.id_helpers_module") as mock_id_helpers:
|
||||
mock_id_helpers.generate_id = Mock(return_value="generated-id")
|
||||
|
||||
recreate_experiment(
|
||||
client,
|
||||
experiment_data,
|
||||
"test-project",
|
||||
{}, # trace_id_map
|
||||
{}, # dataset_item_id_map
|
||||
dry_run=False,
|
||||
debug=False,
|
||||
)
|
||||
|
||||
client.create_experiment.assert_called_once()
|
||||
assert client.create_experiment.call_args.kwargs["tags"] == []
|
||||
@@ -0,0 +1,557 @@
|
||||
"""Unit tests for matches_trace_filter in exports/trace_filter.py."""
|
||||
|
||||
import sys
|
||||
from types import ModuleType
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
from opik.cli.exports.trace_filter import matches_trace_filter
|
||||
|
||||
|
||||
def _make_oql_module(mock_oql: MagicMock) -> ModuleType:
|
||||
"""Return a fake opik.api_objects.opik_query_language module whose
|
||||
OpikQueryLanguage.for_traces() returns *mock_oql*."""
|
||||
mod = ModuleType("opik.api_objects.opik_query_language")
|
||||
cls = MagicMock()
|
||||
cls.for_traces.return_value = mock_oql
|
||||
mod.OpikQueryLanguage = cls # type: ignore[attr-defined]
|
||||
return mod
|
||||
|
||||
|
||||
class TestMatchesTraceFilter:
|
||||
def test_matches_trace_filter__date_time_gte__matches_trace_after_cutoff(self):
|
||||
"""A trace whose created_at is after the cutoff must pass the filter."""
|
||||
trace = {
|
||||
"created_at": "2024-06-01T12:00:00+00:00",
|
||||
}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "created_at",
|
||||
"operator": ">=",
|
||||
"value": "2024-01-01T00:00:00+00:00",
|
||||
"type": "date_time",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__date_time_gte__excludes_trace_before_cutoff(self):
|
||||
"""A trace whose created_at is before the cutoff must be excluded."""
|
||||
trace = {
|
||||
"created_at": "2023-06-01T12:00:00+00:00",
|
||||
}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "created_at",
|
||||
"operator": ">=",
|
||||
"value": "2024-01-01T00:00:00+00:00",
|
||||
"type": "date_time",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_matches_trace_filter__invalid_filter__returns_true_with_warning(self):
|
||||
"""An unparseable filter string must return True and log a warning."""
|
||||
import opik.cli.exports.trace_filter as trace_filter_module
|
||||
|
||||
trace = {"name": "my-trace"}
|
||||
|
||||
fake_mod = ModuleType("opik.api_objects.opik_query_language")
|
||||
cls = MagicMock()
|
||||
cls.for_traces.side_effect = ValueError("bad filter syntax")
|
||||
fake_mod.OpikQueryLanguage = cls # type: ignore[attr-defined]
|
||||
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
with patch.object(trace_filter_module.logger, "warning") as mock_warn:
|
||||
result = matches_trace_filter(trace, "THIS IS NOT VALID OQL")
|
||||
|
||||
assert result is True
|
||||
mock_warn.assert_called_once()
|
||||
# The warning message should contain both the filter string and the exception
|
||||
warning_args = mock_warn.call_args
|
||||
assert "bad filter syntax" in str(warning_args) or any(
|
||||
"bad filter syntax" in str(a) for a in warning_args.args
|
||||
)
|
||||
|
||||
def test_matches_trace_filter__string_contains__happyflow(self):
|
||||
"""A string 'contains' filter must match when the substring is present."""
|
||||
trace = {"name": "evaluation-run-42"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "contains",
|
||||
"value": "evaluation",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name contains "evaluation"')
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__no_expressions__returns_true(self):
|
||||
"""When the filter parses to zero expressions every trace must pass."""
|
||||
trace = {"name": "any-trace"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = []
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, "")
|
||||
|
||||
assert result is True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# number type
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__number_gte__matches(self):
|
||||
"""A number >= filter must match when the field value satisfies it."""
|
||||
trace = {"usage": {"total_tokens": 500}}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "usage.total_tokens",
|
||||
"operator": ">=",
|
||||
"value": "100",
|
||||
"type": "number",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'usage.total_tokens >= "100"')
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__number_lt__excludes(self):
|
||||
"""A number < filter must exclude a trace that doesn't satisfy it."""
|
||||
trace = {"usage": {"total_tokens": 50}}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "usage.total_tokens",
|
||||
"operator": "<",
|
||||
"value": "10",
|
||||
"type": "number",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'usage.total_tokens < "10"')
|
||||
|
||||
assert result is False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# is_empty / is_not_empty
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__is_empty__none_field__matches(self):
|
||||
"""is_empty must match when the field is absent / None."""
|
||||
trace = {} # 'error' key is absent
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "error",
|
||||
"operator": "is_empty",
|
||||
"value": "",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, "error is_empty")
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__is_empty__non_empty_field__excludes(self):
|
||||
"""is_empty must exclude when the field has a value."""
|
||||
trace = {"error": "something went wrong"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "error",
|
||||
"operator": "is_empty",
|
||||
"value": "",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, "error is_empty")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_matches_trace_filter__is_not_empty__non_empty_field__matches(self):
|
||||
"""is_not_empty must match when the field is present and non-empty."""
|
||||
trace = {"name": "my-trace"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "is_not_empty",
|
||||
"value": "",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, "name is_not_empty")
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__is_not_empty__none_field__excludes(self):
|
||||
"""is_not_empty must exclude when the field is absent."""
|
||||
trace = {}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "is_not_empty",
|
||||
"value": "",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, "name is_not_empty")
|
||||
|
||||
assert result is False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Additional string operators
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__string_not_contains__excludes(self):
|
||||
"""not_contains must exclude when the substring is present."""
|
||||
trace = {"name": "evaluation-run-42"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "not_contains",
|
||||
"value": "evaluation",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name not_contains "evaluation"')
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_matches_trace_filter__string_starts_with__matches(self):
|
||||
"""starts_with must match when the field begins with the prefix."""
|
||||
trace = {"name": "evaluation-run-42"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "starts_with",
|
||||
"value": "evaluation",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name starts_with "evaluation"')
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__string_ends_with__matches(self):
|
||||
"""ends_with must match when the field ends with the suffix."""
|
||||
trace = {"name": "evaluation-run-42"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "ends_with",
|
||||
"value": "42",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name ends_with "42"')
|
||||
|
||||
assert result is True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Nested / dotted field access
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__dotted_field__matches(self):
|
||||
"""Dotted fields like 'usage.total_tokens' should be resolved correctly."""
|
||||
trace = {"usage": {"total_tokens": 200}}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "usage.total_tokens",
|
||||
"operator": "=",
|
||||
"value": "200",
|
||||
"type": "number",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'usage.total_tokens = "200"')
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__key_field_access__matches(self):
|
||||
"""Expressions with a non-empty 'key' resolve via trace_dict[field][key]."""
|
||||
trace = {"metadata": {"env": "production"}}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "metadata",
|
||||
"operator": "=",
|
||||
"value": "production",
|
||||
"type": "string",
|
||||
"key": "env",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'metadata.env = "production"')
|
||||
|
||||
assert result is True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# None field value → False
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__missing_field__returns_false(self):
|
||||
"""When a non-empty-check expression references a missing field the trace must be excluded."""
|
||||
trace = {} # 'name' is absent
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "=",
|
||||
"value": "something",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name = "something"')
|
||||
|
||||
assert result is False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Multiple expressions (AND semantics)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__multiple_expressions__all_must_pass(self):
|
||||
"""When two expressions are given both must be satisfied (AND logic)."""
|
||||
trace = {
|
||||
"name": "evaluation-run-42",
|
||||
"created_at": "2024-06-01T12:00:00+00:00",
|
||||
}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "contains",
|
||||
"value": "evaluation",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
},
|
||||
{
|
||||
"field": "created_at",
|
||||
"operator": ">=",
|
||||
"value": "2024-01-01T00:00:00+00:00",
|
||||
"type": "date_time",
|
||||
"key": "",
|
||||
},
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name contains "evaluation"')
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_matches_trace_filter__multiple_expressions__first_fails__returns_false(
|
||||
self,
|
||||
):
|
||||
"""When the first of two expressions fails the trace must be excluded."""
|
||||
trace = {
|
||||
"name": "other-run",
|
||||
"created_at": "2024-06-01T12:00:00+00:00",
|
||||
}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "name",
|
||||
"operator": "contains",
|
||||
"value": "evaluation",
|
||||
"type": "string",
|
||||
"key": "",
|
||||
},
|
||||
{
|
||||
"field": "created_at",
|
||||
"operator": ">=",
|
||||
"value": "2024-01-01T00:00:00+00:00",
|
||||
"type": "date_time",
|
||||
"key": "",
|
||||
},
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'name contains "evaluation"')
|
||||
|
||||
assert result is False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# date_time: naive datetime field value (no tzinfo)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__date_time_naive__treated_as_utc(self):
|
||||
"""A naive datetime field value must be compared as UTC."""
|
||||
from datetime import datetime
|
||||
|
||||
trace = {"created_at": datetime(2024, 6, 1, 12, 0, 0)} # no tzinfo
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "created_at",
|
||||
"operator": ">=",
|
||||
"value": "2024-01-01T00:00:00+00:00",
|
||||
"type": "date_time",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
|
||||
|
||||
assert result is True
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# date_time: unparseable field value → result=False (not exception)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_matches_trace_filter__date_time_unparseable_value__returns_false(self):
|
||||
"""An unparseable date_time field value must exclude the trace, not raise."""
|
||||
trace = {"created_at": "not-a-date"}
|
||||
|
||||
mock_oql = MagicMock()
|
||||
mock_oql.get_filter_expressions.return_value = [
|
||||
{
|
||||
"field": "created_at",
|
||||
"operator": ">=",
|
||||
"value": "2024-01-01T00:00:00+00:00",
|
||||
"type": "date_time",
|
||||
"key": "",
|
||||
}
|
||||
]
|
||||
|
||||
fake_mod = _make_oql_module(mock_oql)
|
||||
with patch.dict(
|
||||
sys.modules, {"opik.api_objects.opik_query_language": fake_mod}
|
||||
):
|
||||
result = matches_trace_filter(trace, 'created_at >= "2024-01-01T00:00:00Z"')
|
||||
|
||||
assert result is False
|
||||
Reference in New Issue
Block a user