chore: import upstream snapshot with attribution
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:44 +08:00
commit 5a558eb09e
11579 changed files with 1795921 additions and 0 deletions
@@ -0,0 +1,185 @@
"""
Unit tests for AssertionResultsMessageProcessor — the migration-period write
path for OPIK-6054 / OPIK-6048.
Targets the small dedicated class directly so the tests don't depend on the
larger OpikMessageProcessor / replay / permissions plumbing.
"""
from unittest import mock
import pytest
from opik.message_processing import messages
from opik.message_processing.processors.assertion_results_processor import (
AssertionResultsMessageProcessor,
)
from opik.rest_api import core as rest_api_core
from opik.rest_api.types import assertion_result_batch_item, feedback_score_batch_item
@pytest.fixture
def mock_rest_client() -> mock.MagicMock:
client = mock.MagicMock()
# MagicMock blocks attributes starting with "assert" as protection against
# typoed assertion methods; configure assertion_results explicitly.
client.assertion_results = mock.MagicMock(unsafe=True)
return client
@pytest.fixture
def processor(
mock_rest_client: mock.MagicMock,
) -> AssertionResultsMessageProcessor:
return AssertionResultsMessageProcessor(rest_client=mock_rest_client)
def _build_message(
items=None,
entity_type="TRACE",
) -> messages.AddAssertionResultsBatchMessage:
items = items or [
messages.AssertionResultMessage(
entity_id="trace-1",
project_name="proj-A",
name="assertion 1",
status="passed",
source="sdk",
reason="looked good",
),
messages.AssertionResultMessage(
entity_id="trace-1",
project_name="proj-A",
name="assertion 2",
status="failed",
source="sdk",
reason=None,
),
]
msg = messages.AddAssertionResultsBatchMessage(batch=items, entity_type=entity_type)
msg.message_id = 1
return msg
class TestAssertionResultsProcessorHappyPath:
def test_process__happy_path__forwards_to_assertion_results_endpoint(
self,
processor: AssertionResultsMessageProcessor,
mock_rest_client: mock.MagicMock,
):
processor.process(_build_message())
mock_rest_client.assertion_results.store_assertions_batch.assert_called_once()
call_kwargs = (
mock_rest_client.assertion_results.store_assertions_batch.call_args.kwargs
)
assert call_kwargs["entity_type"] == "TRACE"
sent_items = call_kwargs["assertion_results"]
assert len(sent_items) == 2
assert all(
isinstance(item, assertion_result_batch_item.AssertionResultBatchItem)
for item in sent_items
)
assert sent_items[0].entity_id == "trace-1"
assert sent_items[0].name == "assertion 1"
assert sent_items[0].status == "passed"
assert sent_items[0].source == "sdk"
assert sent_items[0].reason == "looked good"
assert sent_items[1].status == "failed"
assert sent_items[1].reason is None
def test_process__happy_path__does_not_use_feedback_scores_endpoint(
self,
processor: AssertionResultsMessageProcessor,
mock_rest_client: mock.MagicMock,
):
processor.process(_build_message())
mock_rest_client.traces.score_batch_of_traces.assert_not_called()
mock_rest_client.spans.score_batch_of_spans.assert_not_called()
class TestAssertionResultsProcessorFallback:
"""
Forward-compat fallback: if the dedicated assertion-results endpoint is
missing on an older self-hosted backend, write through the legacy
feedback-scores piggyback and remember the choice for the rest of the
session.
"""
def test_process__bad_endpoint_404__falls_back_to_feedback_scores(
self,
processor: AssertionResultsMessageProcessor,
mock_rest_client: mock.MagicMock,
):
mock_rest_client.assertion_results.store_assertions_batch.side_effect = (
rest_api_core.ApiError(status_code=404, body="Not Found")
)
processor.process(_build_message())
mock_rest_client.traces.score_batch_of_traces.assert_called_once()
sent = mock_rest_client.traces.score_batch_of_traces.call_args.kwargs["scores"]
assert len(sent) == 2
assert all(
isinstance(item, feedback_score_batch_item.FeedbackScoreBatchItem)
for item in sent
)
assert all(item.category_name == "suite_assertion" for item in sent)
assert sent[0].id == "trace-1"
assert sent[0].name == "assertion 1"
assert sent[0].value == 1.0
assert sent[0].source == "sdk"
assert sent[1].value == 0.0 # status="failed"
assert processor._use_assertion_results_endpoint is False
def test_process__bad_endpoint_405__falls_back_to_feedback_scores(
self,
processor: AssertionResultsMessageProcessor,
mock_rest_client: mock.MagicMock,
):
mock_rest_client.assertion_results.store_assertions_batch.side_effect = (
rest_api_core.ApiError(status_code=405, body="Method Not Allowed")
)
processor.process(_build_message())
mock_rest_client.traces.score_batch_of_traces.assert_called_once()
assert processor._use_assertion_results_endpoint is False
def test_process__non_404_api_error__propagates_and_does_not_set_fallback(
self,
processor: AssertionResultsMessageProcessor,
mock_rest_client: mock.MagicMock,
):
mock_rest_client.assertion_results.store_assertions_batch.side_effect = (
rest_api_core.ApiError(status_code=500, body="Internal Server Error")
)
with pytest.raises(rest_api_core.ApiError):
processor.process(_build_message())
mock_rest_client.traces.score_batch_of_traces.assert_not_called()
assert processor._use_assertion_results_endpoint is True
def test_process__sticky_fallback__second_call_skips_new_endpoint(
self,
processor: AssertionResultsMessageProcessor,
mock_rest_client: mock.MagicMock,
):
mock_rest_client.assertion_results.store_assertions_batch.side_effect = (
rest_api_core.ApiError(status_code=404, body="Not Found")
)
processor.process(_build_message())
# First call: tried new endpoint, hit 404, flipped the flag.
assert mock_rest_client.assertion_results.store_assertions_batch.call_count == 1
assert processor._use_assertion_results_endpoint is False
mock_rest_client.traces.score_batch_of_traces.reset_mock()
processor.process(_build_message())
# Second call must NOT retry the new endpoint.
assert mock_rest_client.assertion_results.store_assertions_batch.call_count == 1
# It went straight to the fallback.
mock_rest_client.traces.score_batch_of_traces.assert_called_once()
@@ -0,0 +1,817 @@
from unittest import mock
import pytest
from opik.api_objects.attachment import attachment, attachment_context
from opik.message_processing import messages
from opik.message_processing.preprocessing import constants
from opik.message_processing.processors import attachments_extraction_processor
@pytest.fixture
def mock_streamer():
"""Create a mock streamer."""
return mock.Mock()
@pytest.fixture
def processor(mock_streamer):
"""Create an AttachmentsExtractionProcessor with default settings."""
return attachments_extraction_processor.AttachmentsExtractionProcessor(
min_attachment_size=20,
messages_streamer=mock_streamer,
url_override="https://example.com",
is_active=True,
)
@pytest.fixture
def inactive_processor(mock_streamer):
"""Create an inactive AttachmentsExtractionProcessor."""
return attachments_extraction_processor.AttachmentsExtractionProcessor(
min_attachment_size=20,
messages_streamer=mock_streamer,
url_override="https://example.com",
is_active=False,
)
def test_process_non_attachment_supporting_message_skips_processing(processor):
"""Test that non-AttachmentSupportingMessage messages are skipped."""
# Create a regular message (not wrapped)
regular_message = messages.CreateSpanMessage(
span_id="span-123",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"key": "value"},
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
# Process should return immediately without doing anything
processor.process(regular_message)
# Streamer should not be called
processor.messages_streamer.put.assert_not_called()
def test_process_attachment_support_message_with_no_attachments(
processor, mock_streamer
):
"""Test processing a message with no extractable attachments."""
# Create an original message with plain text
original_message = messages.CreateSpanMessage(
span_id="span-123",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"text": "just plain text"},
output={"result": "no attachments here"},
metadata={"info": "metadata"},
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
# Wrap in AttachmentSupportingMessage
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Mock extractor to return no attachments
with mock.patch.object(processor.extractor, "extract_and_replace", return_value=[]):
processor.process(wrapped_message)
# The original message should be re-queued with marker
assert mock_streamer.put.call_count == 1
requeued_message = mock_streamer.put.call_args[0][0]
assert requeued_message is original_message
assert hasattr(requeued_message, constants.MARKER_ATTRIBUTE_NAME)
assert getattr(requeued_message, constants.MARKER_ATTRIBUTE_NAME) is True
def test_process_span_message_extracts_from_input(
processor, mock_streamer, temp_file_15kb
):
"""Test extraction of attachments from span input."""
# Create a span message with input containing base64
original_message = messages.CreateSpanMessage(
span_id="span-123",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"image": "fake-base64-data"},
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Mock extracted attachment - use actual temp file
mock_attachment = attachment.Attachment(
data=temp_file_15kb.name,
file_name="input-attachment-123.png",
content_type="image/png",
create_temp_copy=False,
)
mock_attachment_with_context = attachment_context.AttachmentWithContext(
attachment_data=mock_attachment,
entity_type="span",
entity_id="span-123",
project_name="test-project",
context="input",
)
# Mock extractor to return one attachment from input
def mock_extract(data, entity_type, entity_id, project_name, context):
if context == "input":
return [mock_attachment_with_context]
return []
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
# Should have created one CreateAttachmentMessage and re-queued original
assert mock_streamer.put.call_count == 2
# First call should be the attachment message
first_call = mock_streamer.put.call_args_list[0][0][0]
assert isinstance(first_call, messages.CreateAttachmentMessage)
assert first_call.entity_type == "span"
assert first_call.entity_id == "span-123"
assert first_call.project_name == "test-project"
assert first_call.delete_after_upload is True
# Second call should be the original message with marker
second_call = mock_streamer.put.call_args_list[1][0][0]
assert second_call is original_message
assert hasattr(second_call, constants.MARKER_ATTRIBUTE_NAME)
def test_process_span_message_extracts_from_output(
processor, mock_streamer, temp_file_15kb
):
"""Test extraction of attachments from the span output."""
original_message = messages.CreateSpanMessage(
span_id="span-789",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input=None,
output={"image": "fake-base64-data"},
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
mock_attachment = attachment.Attachment(
data=temp_file_15kb.name,
file_name="output-attachment-456.png",
content_type="image/png",
create_temp_copy=False,
)
mock_attachment_with_context = attachment_context.AttachmentWithContext(
attachment_data=mock_attachment,
entity_type="span",
entity_id="span-789",
project_name="test-project",
context="output",
)
def mock_extract(data, entity_type, entity_id, project_name, context):
if context == "output":
return [mock_attachment_with_context]
return []
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
assert mock_streamer.put.call_count == 2
attachment_msg = mock_streamer.put.call_args_list[0][0][0]
assert isinstance(attachment_msg, messages.CreateAttachmentMessage)
def test_process_span_message_extracts_from_metadata(
processor, mock_streamer, temp_file_15kb
):
"""Test extraction of attachments from span metadata."""
original_message = messages.CreateSpanMessage(
span_id="span-999",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input=None,
output=None,
metadata={"doc": "fake-base64-pdf"},
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
mock_attachment = attachment.Attachment(
data=temp_file_15kb.name,
file_name="metadata-attachment-789.pdf",
content_type="application/pdf",
create_temp_copy=False,
)
mock_attachment_with_context = attachment_context.AttachmentWithContext(
attachment_data=mock_attachment,
entity_type="span",
entity_id="span-999",
project_name="test-project",
context="metadata",
)
def mock_extract(data, entity_type, entity_id, project_name, context):
if context == "metadata":
return [mock_attachment_with_context]
return []
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
assert mock_streamer.put.call_count == 2
attachment_msg = mock_streamer.put.call_args_list[0][0][0]
assert isinstance(attachment_msg, messages.CreateAttachmentMessage)
def test_process_trace_message_extracts_attachments(
processor, mock_streamer, temp_file_15kb
):
"""Test extraction from trace messages."""
original_message = messages.CreateTraceMessage(
trace_id="trace-123",
project_name="test-project",
name="test-trace",
start_time=mock.Mock(),
end_time=None,
input={"data": "fake-base64"},
output=None,
metadata=None,
tags=None,
error_info=None,
thread_id=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
mock_attachment = attachment.Attachment(
data=temp_file_15kb.name,
file_name="input-attachment-trace.png",
content_type="image/png",
create_temp_copy=False,
)
mock_attachment_with_context = attachment_context.AttachmentWithContext(
attachment_data=mock_attachment,
entity_type="trace",
entity_id="trace-123",
project_name="test-project",
context="input",
)
def mock_extract(data, entity_type, entity_id, project_name, context):
if context == "input":
return [mock_attachment_with_context]
return []
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
# Should create an attachment message and re-queue the original message
assert mock_streamer.put.call_count == 2
# Verify it's a trace entity type
attachment_msg = mock_streamer.put.call_args_list[0][0][0]
assert isinstance(attachment_msg, messages.CreateAttachmentMessage)
assert attachment_msg.entity_type == "trace"
assert attachment_msg.entity_id == "trace-123"
assert attachment_msg.delete_after_upload is True
# Last call should be the original message with marker
second_call = mock_streamer.put.call_args_list[1][0][0]
assert second_call is original_message
assert hasattr(second_call, constants.MARKER_ATTRIBUTE_NAME)
def test_process_update_span_message(processor, mock_streamer, temp_file_15kb):
"""Test extraction from UpdateSpanMessage."""
original_message = messages.UpdateSpanMessage(
span_id="span-update-123",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
end_time=mock.Mock(),
input={"image": "base64-data"},
output=None,
metadata=None,
tags=None,
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
mock_attachment = attachment.Attachment(
data=temp_file_15kb.name,
file_name="input-attachment.png",
content_type="image/png",
create_temp_copy=False,
)
mock_attachment_with_context = attachment_context.AttachmentWithContext(
attachment_data=mock_attachment,
entity_type="span",
entity_id="span-update-123",
project_name="test-project",
context="input",
)
def mock_extract(data, entity_type, entity_id, project_name, context):
if context == "input":
return [mock_attachment_with_context]
return []
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
assert mock_streamer.put.call_count == 2
attachment_msg = mock_streamer.put.call_args_list[0][0][0]
assert isinstance(attachment_msg, messages.CreateAttachmentMessage)
def test_process_update_trace_message(processor, mock_streamer, temp_file_15kb):
"""Test extraction from UpdateTraceMessage."""
original_message = messages.UpdateTraceMessage(
trace_id="trace-update-123",
project_name="test-project",
end_time=mock.Mock(),
input=None,
output={"result": "base64-data"},
metadata=None,
tags=None,
error_info=None,
thread_id=None,
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
mock_attachment = attachment.Attachment(
data=temp_file_15kb.name,
file_name="output-attachment.png",
content_type="image/png",
create_temp_copy=False,
)
mock_attachment_with_context = attachment_context.AttachmentWithContext(
attachment_data=mock_attachment,
entity_type="trace",
entity_id="trace-update-123",
project_name="test-project",
context="output",
)
def mock_extract(data, entity_type, entity_id, project_name, context):
if context == "output":
return [mock_attachment_with_context]
return []
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
assert mock_streamer.put.call_count == 2
attachment_msg = mock_streamer.put.call_args_list[0][0][0]
assert isinstance(attachment_msg, messages.CreateAttachmentMessage)
def test_process_multiple_attachments_from_different_contexts(
processor, mock_streamer, temp_file_15kb
):
"""Test extraction of multiple attachments from input, output, and metadata."""
original_message = messages.CreateSpanMessage(
span_id="span-multi",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"image1": "base64"},
output={"image2": "base64"},
metadata={"doc": "base64"},
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Create three attachments for different contexts
attachments_map = {
"input": [
attachment_context.AttachmentWithContext(
attachment_data=attachment.Attachment(
data=temp_file_15kb.name,
file_name="input-att.png",
content_type="image/png",
create_temp_copy=False,
),
entity_type="span",
entity_id="span-multi",
project_name="test-project",
context="input",
)
],
"output": [
attachment_context.AttachmentWithContext(
attachment_data=attachment.Attachment(
data=temp_file_15kb.name,
file_name="output-att.png",
content_type="image/png",
create_temp_copy=False,
),
entity_type="span",
entity_id="span-multi",
project_name="test-project",
context="output",
)
],
"metadata": [
attachment_context.AttachmentWithContext(
attachment_data=attachment.Attachment(
data=temp_file_15kb.name,
file_name="meta-att.pdf",
content_type="application/pdf",
create_temp_copy=False,
),
entity_type="span",
entity_id="span-multi",
project_name="test-project",
context="metadata",
)
],
}
def mock_extract(data, entity_type, entity_id, project_name, context):
return attachments_map.get(context, [])
with mock.patch.object(
processor.extractor, "extract_and_replace", side_effect=mock_extract
):
processor.process(wrapped_message)
# Should create 3 attachment messages + 1 re-queued original = 4 total
assert mock_streamer.put.call_count == 4
# First 3 should be attachment messages
for i in range(3):
msg = mock_streamer.put.call_args_list[i][0][0]
assert isinstance(msg, messages.CreateAttachmentMessage)
# Last should be the original message
last_msg = mock_streamer.put.call_args_list[3][0][0]
assert last_msg is original_message
def test_process_with_null_input_output_metadata(processor, mock_streamer):
"""Test processing message with null input/output/metadata fields."""
original_message = messages.CreateSpanMessage(
span_id="span-nulls",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input=None,
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Extractor should never be called since all fields are None
with mock.patch.object(processor.extractor, "extract_and_replace") as mock_extract:
processor.process(wrapped_message)
# Extractor should not be called
mock_extract.assert_not_called()
# Only the original message should be re-queued
assert mock_streamer.put.call_count == 1
def test_process_inactive_processor_skips_extraction(inactive_processor, mock_streamer):
"""Test that the inactive processor skips extraction but still re-queues a message."""
original_message = messages.CreateSpanMessage(
span_id="span-inactive",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"image": "base64-data"},
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Should not call extractor since the processor is inactive
with mock.patch.object(
inactive_processor.extractor, "extract_and_replace"
) as mock_extract:
inactive_processor.process(wrapped_message)
# Extractor should not be called
mock_extract.assert_not_called()
# Original message should still be re-queued with marker
assert mock_streamer.put.call_count == 1
requeued_message = mock_streamer.put.call_args[0][0]
assert requeued_message is original_message
assert hasattr(requeued_message, constants.MARKER_ATTRIBUTE_NAME)
def test_process_handles_extraction_exception_gracefully(processor, mock_streamer):
"""Test that extraction exceptions are caught and logged, but processing continues."""
original_message = messages.CreateSpanMessage(
span_id="span-error",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"data": "causes-error"},
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Mock extractor to raise an exception
with mock.patch.object(
processor.extractor,
"extract_and_replace",
side_effect=Exception("Extraction failed"),
):
# Should not raise exception
processor.process(wrapped_message)
# Original message should still be re-queued despite the error
assert mock_streamer.put.call_count == 1
requeued_message = mock_streamer.put.call_args[0][0]
assert requeued_message is original_message
assert hasattr(requeued_message, constants.MARKER_ATTRIBUTE_NAME)
def test_process_message_already_marked_returns_early(processor):
"""Test that messages already marked with a preprocessing attribute are handled."""
# This test verifies behavior when a message with marker is wrapped
# In practice, preprocess_message() should prevent this, but we test defense
original_message = messages.CreateSpanMessage(
span_id="span-marked",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=mock.Mock(),
end_time=None,
input={"data": "value"},
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
# Mark the message as already processed
setattr(original_message, constants.MARKER_ATTRIBUTE_NAME, True)
wrapped_message = messages.AttachmentSupportingMessage(
original_message=original_message
)
# Mock extractor to verify it's still called (marker is on the original, not wrapper)
with mock.patch.object(
processor.extractor, "extract_and_replace", return_value=[]
) as mock_extract:
processor.process(wrapped_message)
# Extractor should still be called since the wrapper doesn't have the marker
mock_extract.assert_called()
def test_entity_type_from_attachment_message_span():
"""Test entity_type_from_attachment_message extracts correct details for span."""
span_message = messages.CreateSpanMessage(
span_id="span-123",
trace_id="trace-456",
project_name="test-project",
parent_span_id=None,
name="test",
start_time=mock.Mock(),
end_time=None,
input=None,
output=None,
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=mock.Mock(),
source="sdk",
)
result = attachments_extraction_processor.entity_type_from_attachment_message(
span_message
)
assert result is not None
assert result.entity_type == "span"
assert result.entity_id == "span-123"
assert result.project_name == "test-project"
def test_entity_type_from_attachment_message_trace():
"""Test entity_type_from_attachment_message extracts correct details for trace."""
trace_message = messages.CreateTraceMessage(
trace_id="trace-789",
project_name="my-project",
name="test-trace",
start_time=mock.Mock(),
end_time=None,
input=None,
output=None,
metadata=None,
tags=None,
error_info=None,
thread_id=None,
last_updated_at=mock.Mock(),
source="sdk",
)
result = attachments_extraction_processor.entity_type_from_attachment_message(
trace_message
)
assert result is not None
assert result.entity_type == "trace"
assert result.entity_id == "trace-789"
assert result.project_name == "my-project"
def test_entity_type_from_attachment_message_unsupported():
"""Test entity_type_from_attachment_message returns None for unsupported message types."""
unsupported_message = messages.BaseMessage()
result = attachments_extraction_processor.entity_type_from_attachment_message(
unsupported_message
)
assert result is None
@@ -0,0 +1,905 @@
"""
Unit tests for OpikMessageProcessor interaction with ReplayManager.
Tests the three replay-related code paths in OpikMessageProcessor.process():
- register_message: called before every handler invocation
- unregister_message: called after successful handler execution
- message_sent_failed: called on httpx.ConnectError / httpx.TimeoutException
"""
import datetime
from unittest import mock
import httpx
import pydantic
import pytest
import tenacity
from opik.message_processing import messages
from opik.message_processing.processors import online_message_processor
from opik.message_processing.processors.online_message_processor import (
OpikMessageProcessor,
)
from opik.file_upload.s3_multipart_upload import s3_upload_error
from opik.message_processing.replay import replay_manager, db_manager
from opik.rest_api import core as rest_api_core
from opik import exceptions
def _create_trace_message(message_id: int = 1) -> messages.CreateTraceMessage:
msg = messages.CreateTraceMessage(
trace_id="trace-1",
project_name="test-project",
name="test-trace",
start_time=datetime.datetime(2024, 1, 1, 12, 0, 0),
end_time=datetime.datetime(2024, 1, 1, 12, 0, 1),
input={"query": "test"},
output={"answer": "response"},
metadata=None,
tags=None,
error_info=None,
thread_id=None,
last_updated_at=None,
source="sdk",
)
msg.message_id = message_id
return msg
def _create_attachment_message(message_id: int = 1) -> messages.CreateAttachmentMessage:
msg = messages.CreateAttachmentMessage(
file_path="/tmp/test-file.bin",
file_name="test-file.bin",
mime_type="application/octet-stream",
entity_type="trace",
entity_id="trace-1",
project_name="test-project",
encoded_url_override="aHR0cDovL2xvY2FsaG9zdA==",
delete_after_upload=False,
)
msg.message_id = message_id
return msg
def _create_span_message(message_id: int = 1) -> messages.CreateSpanMessage:
msg = messages.CreateSpanMessage(
span_id="span-1",
trace_id="trace-1",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=datetime.datetime(2024, 1, 1, 12, 0, 0),
end_time=datetime.datetime(2024, 1, 1, 12, 0, 1),
input={"prompt": "test"},
output={"response": "result"},
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=None,
source="sdk",
)
msg.message_id = message_id
return msg
@pytest.fixture
def mock_rest_client() -> mock.MagicMock:
# No spec= so nested attributes like .traces.create_trace are auto-created
return mock.MagicMock()
@pytest.fixture
def mock_file_uploader() -> mock.MagicMock:
return mock.MagicMock()
@pytest.fixture
def mock_replay() -> mock.MagicMock:
return mock.MagicMock(spec=replay_manager.ReplayManager)
@pytest.fixture
def processor(
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
) -> OpikMessageProcessor:
return OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=mock.Mock(),
)
class TestRegisterMessage:
def test_process__known_message__registers_before_handler(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""register_message must be called before the REST API handler fires."""
call_order = []
mock_replay.register_message.side_effect = lambda m: call_order.append(
"register"
)
mock_rest_client.traces.create_trace.side_effect = (
lambda **kw: call_order.append("handler")
)
msg = _create_trace_message()
processor.process(msg)
assert "register" in call_order
assert "handler" in call_order
assert call_order.index("register") < call_order.index("handler")
def test_process__trace_message__registers_with_correct_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""register_message is called with the exact message object."""
msg = _create_trace_message(message_id=42)
processor.process(msg)
mock_replay.register_message.assert_called_once_with(msg)
def test_process__unknown_message_type__does_not_register(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""Unknown message types should be skipped without registration."""
msg = mock.MagicMock(spec=messages.BaseMessage)
# Use a type not in the handlers dict
msg.__class__ = type("UnknownMessage", (messages.BaseMessage,), {})
processor.process(msg)
mock_replay.register_message.assert_not_called()
def test_process__inactive_processor__does_not_register(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
):
"""When the processor is inactive, no registration should occur."""
inactive_processor = OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
active=False,
unauthorized_message_types_registry=mock.Mock(),
)
msg = _create_trace_message()
inactive_processor.process(msg)
mock_replay.register_message.assert_not_called()
class TestNoServerConnection:
"""Tests for when has_server_connection is False — messages should be
registered as failed and the handler should NOT execute."""
@pytest.fixture
def offline_replay(self) -> mock.MagicMock:
m = mock.MagicMock(spec=replay_manager.ReplayManager)
m.has_server_connection = False
return m
@pytest.fixture
def offline_processor(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
offline_replay: mock.MagicMock,
) -> OpikMessageProcessor:
return OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=offline_replay,
unauthorized_message_types_registry=mock.Mock(),
)
def test_process__no_connection__registers_message_as_failed(
self,
offline_processor: OpikMessageProcessor,
offline_replay: mock.MagicMock,
):
"""When there is no server connection, register_message must be called
with status=MessageStatus.failed."""
msg = _create_trace_message(message_id=10)
offline_processor.process(msg)
offline_replay.register_message.assert_called_once_with(
msg, status=db_manager.MessageStatus.failed
)
offline_replay.unregister_message.assert_not_called()
def test_process__no_connection__handler_not_called(
self,
offline_processor: OpikMessageProcessor,
offline_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""When there is no server connection, the REST API handler must NOT
be invoked."""
msg = _create_trace_message()
offline_processor.process(msg)
mock_rest_client.traces.create_trace.assert_not_called()
offline_replay.unregister_message.assert_not_called()
def test_process__no_connection__unregister_not_called(
self,
offline_processor: OpikMessageProcessor,
offline_replay: mock.MagicMock,
):
"""When there is no server connection, unregister_message must NOT be
called since the handler is skipped."""
msg = _create_trace_message()
offline_processor.process(msg)
offline_replay.unregister_message.assert_not_called()
def test_process__no_connection__message_sent_failed_not_called(
self,
offline_processor: OpikMessageProcessor,
offline_replay: mock.MagicMock,
):
"""message_sent_failed should NOT be called — it is only for connection
errors that occur during handler execution."""
msg = _create_trace_message()
offline_processor.process(msg)
offline_replay.message_sent_failed.assert_not_called()
class TestUnregisterMessage:
def test_process__successful_handler__unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""After successful handler execution, unregister_message is called."""
msg = _create_trace_message(message_id=10)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(10)
def test_process__successful_handler__unregisters_after_handler(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""unregister_message must be called after the handler completes."""
call_order = []
mock_rest_client.traces.create_trace.side_effect = (
lambda **kw: call_order.append("handler")
)
mock_replay.unregister_message.side_effect = lambda mid: call_order.append(
"unregister"
)
msg = _create_trace_message()
processor.process(msg)
assert call_order.index("handler") < call_order.index("unregister")
def test_process__api_error_409__unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""409 Conflict triggers an explicit unregister_message call and early
return (the message is considered delivered)."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=409, body="conflict"
)
msg = _create_trace_message(message_id=11)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(11)
def test_process__api_error_500__unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""Non-connection API errors (e.g. 500) still unregister the message."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=500, body="internal error"
)
msg = _create_trace_message(message_id=12)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(12)
def test_process__retry_error__unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""tenacity.RetryError unregisters the message (not a connection error)."""
future = tenacity.Future(attempt_number=1)
future.set_exception(RuntimeError("underlying cause"))
mock_rest_client.traces.create_trace.side_effect = tenacity.RetryError(
last_attempt=future,
)
msg = _create_trace_message(message_id=13)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(13)
def test_process__validation_error__unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""pydantic.ValidationError unregisters the message."""
# Generate a real ValidationError via pydantic
validation_error: pydantic.ValidationError
try:
pydantic.TypeAdapter(int).validate_python("not-an-int")
except pydantic.ValidationError as e:
validation_error = e
mock_rest_client.traces.create_trace.side_effect = validation_error
msg = _create_trace_message(message_id=14)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(14)
def test_process__generic_exception__unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""A generic Exception still unregisters the message."""
mock_rest_client.traces.create_trace.side_effect = RuntimeError("unexpected")
msg = _create_trace_message(message_id=15)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(15)
@pytest.mark.parametrize(
"connection_error",
[httpx.ConnectError("refused"), httpx.ReadTimeout("timed out")],
)
def test_process__connection_error__does_not_unregister(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
connection_error: Exception,
):
"""httpx connection/timeout errors must NOT unregister — the message
stays registered for replay."""
mock_rest_client.traces.create_trace.side_effect = connection_error
msg = _create_trace_message(message_id=16)
processor.process(msg)
mock_replay.unregister_message.assert_not_called()
def test_process__api_error_429_with_rate_limit__does_not_unregister(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""429 with valid rate-limit headers raises and leaves the message
registered for retry — unregister must NOT be called."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=429, body="rate limited", headers={"RateLimit-Reset": "10"}
)
msg = _create_trace_message(message_id=17)
with pytest.raises(exceptions.OpikCloudRequestsRateLimited):
processor.process(msg)
mock_replay.unregister_message.assert_not_called()
class TestMessageSentFailed:
@pytest.mark.parametrize(
"connection_error",
[httpx.ConnectError("connection refused"), httpx.ReadTimeout("read timed out")],
)
def test_process__connection_error__marks_message_as_failed(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
connection_error: Exception,
):
"""httpx.TimeoutException should trigger message_sent_failed."""
mock_rest_client.traces.create_trace.side_effect = connection_error
msg = _create_trace_message(message_id=8)
processor.process(msg)
mock_replay.message_sent_failed.assert_called_once_with(
8, failure_reason=str(connection_error)
)
# assert that unregister_message was not called
mock_replay.unregister_message.assert_not_called()
class TestErrorPathsNoFailedMark:
"""Verify that non-connection errors do NOT call message_sent_failed.
The message stays registered (not unregistered, not marked failed)."""
def test_process__api_error_409__no_failed_mark(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""409 Conflict is silently ignored and a message unregistered."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=409, body="conflict"
)
msg = _create_trace_message(message_id=1)
processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
mock_replay.unregister_message.assert_called()
def test_process__api_error_500__no_failed_mark(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""500 Server Error logs but does not mark as failed."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=500, body="internal error"
)
msg = _create_trace_message(message_id=2)
processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
mock_replay.unregister_message.assert_called()
def test_process__generic_exception__no_failed_mark(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""A generic Exception does not mark as failed."""
mock_rest_client.traces.create_trace.side_effect = RuntimeError("unexpected")
msg = _create_trace_message(message_id=3)
processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
mock_replay.unregister_message.assert_called()
class TestApiError429WithRateLimiter:
"""Verify that a 429 ApiError with valid rate-limit headers raises
OpikCloudRequestsRateLimited, leaving the message registered for retry."""
def test_process__api_error_429_with_rate_limit_headers__no_unregister_no_failed(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""Neither unregister nor message_sent_failed is called — a message stays
registered so the queue consumer can retry it after the rate-limit window."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=429, body="rate limited", headers={"RateLimit-Reset": "10"}
)
msg = _create_trace_message(message_id=3)
with pytest.raises(exceptions.OpikCloudRequestsRateLimited):
processor.process(msg)
mock_replay.register_message.assert_called_once_with(msg)
mock_replay.unregister_message.assert_not_called()
mock_replay.message_sent_failed.assert_not_called()
def test_process__api_error_429_without_rate_limit_headers__no_raise(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""429 without parseable rate-limit headers falls through to generic
ApiError logging, unregisters the message, and does not raise."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=429, body="rate limited", headers={"Other-Header": "value"}
)
msg = _create_trace_message(message_id=4)
# Should NOT raise — falls through to the generic ApiError branch
processor.process(msg)
mock_replay.register_message.assert_called_once_with(msg)
# Not a connection error, so a message is unregistered
mock_replay.unregister_message.assert_called_once_with(4)
mock_replay.message_sent_failed.assert_not_called()
def test_process__api_error_429_with_none_headers__no_raise(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""429 with headers=None skips rate-limit parsing, unregisters the
message, and does not raise."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=429, body="rate limited", headers=None
)
msg = _create_trace_message(message_id=5)
processor.process(msg)
mock_replay.register_message.assert_called_once_with(msg)
# Not a connection error, so a message is unregistered
mock_replay.unregister_message.assert_called_once_with(5)
mock_replay.message_sent_failed.assert_not_called()
class TestMultipleMessagesSequence:
def test_process__success_then_connection_error__correct_replay_calls(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""The first message succeeds (register+unregister), the second fails with
ConnectError (register+message_sent_failed)."""
# The first call succeeds, the second raises ConnectError
mock_rest_client.traces.create_trace.side_effect = [
None,
httpx.ConnectError("connection refused"),
]
msg1 = _create_trace_message(message_id=1)
msg2 = _create_trace_message(message_id=2)
processor.process(msg1)
processor.process(msg2)
# Both registered
assert mock_replay.register_message.call_count == 2
# Only first unregistered
mock_replay.unregister_message.assert_called_once_with(1)
# Only second marked as failed
mock_replay.message_sent_failed.assert_called_once_with(
2, failure_reason="connection refused"
)
def test_process__connection_error_then_success__correct_replay_calls(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""The first message fails with a connection error, the second succeeds."""
mock_rest_client.traces.create_trace.side_effect = [
httpx.ConnectError("connection refused"),
None,
]
msg1 = _create_trace_message(message_id=1)
msg2 = _create_trace_message(message_id=2)
processor.process(msg1)
processor.process(msg2)
assert mock_replay.register_message.call_count == 2
mock_replay.message_sent_failed.assert_called_once_with(
1, failure_reason="connection refused"
)
mock_replay.unregister_message.assert_called_once_with(2)
class TestAttachmentUploadCallbacks:
"""Tests for CreateAttachmentMessage upload callback interactions with
the replay manager.
The attachment handler passes on_upload_success / on_upload_failed callbacks
to the file uploader. These callbacks are responsible for calling
unregister_message or message_sent_failed on the replay manager."""
def test_process__attachment__registers_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""CreateAttachmentMessage should be registered before the upload handler."""
msg = _create_attachment_message(message_id=50)
processor.process(msg)
mock_replay.register_message.assert_called_once_with(msg)
def test_process__attachment__passes_callbacks_to_uploader(
self,
processor: OpikMessageProcessor,
mock_file_uploader: mock.MagicMock,
):
"""The file uploader must receive on_upload_success and on_upload_failed
callbacks."""
msg = _create_attachment_message(message_id=50)
processor.process(msg)
mock_file_uploader.upload.assert_called_once()
call_kwargs = mock_file_uploader.upload.call_args
assert call_kwargs.kwargs["on_upload_success"] is not None
assert call_kwargs.kwargs["on_upload_failed"] is not None
def test_process__attachment__success_callback_unregisters_message(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When the on_upload_success callback fires, it should call
unregister_message with the correct message_id."""
msg = _create_attachment_message(message_id=60)
# Capture the callback by invoking it inside the mock upload
def fake_upload(message, on_upload_success=None, on_upload_failed=None):
on_upload_success()
mock_file_uploader.upload.side_effect = fake_upload
mock_replay.reset_mock()
processor.process(msg)
mock_replay.unregister_message.assert_any_call(60)
def test_process__attachment__failed_callback_connect_error__marks_failed(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When on_upload_failed fires with httpx.ConnectError, it should call
message_sent_failed."""
msg = _create_attachment_message(message_id=70)
error = httpx.ConnectError("connection refused")
def fake_upload(message, on_upload_success=None, on_upload_failed=None):
on_upload_failed(error)
mock_file_uploader.upload.side_effect = fake_upload
mock_replay.reset_mock()
processor.process(msg)
mock_replay.message_sent_failed.assert_called_once_with(
70, failure_reason=str(error)
)
def test_process__attachment__failed_callback_timeout__marks_failed(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When on_upload_failed fires with httpx.TimeoutException, it should call
message_sent_failed."""
msg = _create_attachment_message(message_id=71)
error = httpx.ReadTimeout("read timed out")
def fake_upload(message, on_upload_success=None, on_upload_failed=None):
on_upload_failed(error)
mock_file_uploader.upload.side_effect = fake_upload
mock_replay.reset_mock()
processor.process(msg)
mock_replay.message_sent_failed.assert_called_once_with(
71, failure_reason=str(error)
)
def test_process__attachment__failed_callback_s3_connection_error__marks_failed(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When on_upload_failed fires with S3UploadError(connection_error=True),
it should call message_sent_failed."""
msg = _create_attachment_message(message_id=72)
error = s3_upload_error.S3UploadError(
reason="connection lost", connection_error=True
)
def fake_upload(message, on_upload_success=None, on_upload_failed=None):
on_upload_failed(error)
mock_file_uploader.upload.side_effect = fake_upload
mock_replay.reset_mock()
processor.process(msg)
mock_replay.message_sent_failed.assert_called_once_with(
72, failure_reason=str(error)
)
def test_process__attachment__failed_callback_s3_non_connection_error__no_failed_mark(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When on_upload_failed fires with S3UploadError(connection_error=False),
message_sent_failed should NOT be called."""
msg = _create_attachment_message(message_id=73)
error = s3_upload_error.S3UploadError(
reason="file too large", connection_error=False
)
def fake_upload(message, on_upload_success=None, on_upload_failed=None):
on_upload_failed(error)
mock_file_uploader.upload.side_effect = fake_upload
mock_replay.reset_mock()
processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
mock_replay.unregister_message.assert_called_once_with(73)
def test_process__attachment__failed_callback_generic_error__no_failed_mark(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When on_upload_failed fires with a non-connection error (e.g. ValueError),
message_sent_failed should NOT be called."""
msg = _create_attachment_message(message_id=74)
error = ValueError("bad data")
def fake_upload(message, on_upload_success=None, on_upload_failed=None):
on_upload_failed(error)
mock_file_uploader.upload.side_effect = fake_upload
mock_replay.reset_mock()
processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
mock_replay.unregister_message.assert_called_once_with(74)
def test_process__attachment_no_connection__registers_as_failed_and_skips_upload(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""When has_server_connection is False, the attachment message should be
registered as failed and the file uploader should NOT be called."""
offline_replay = mock.MagicMock(spec=replay_manager.ReplayManager)
offline_replay.has_server_connection = False
offline_processor = OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=offline_replay,
unauthorized_message_types_registry=mock.Mock(),
)
msg = _create_attachment_message(message_id=80)
offline_processor.process(msg)
offline_replay.register_message.assert_called_once_with(
msg, status=db_manager.MessageStatus.failed
)
mock_file_uploader.upload.assert_not_called()
offline_replay.unregister_message.assert_not_called()
def _create_attachment_supporting_message(
message_id: int = 1,
) -> messages.AttachmentSupportingMessage:
inner = _create_trace_message(message_id)
msg = messages.AttachmentSupportingMessage(original_message=inner)
msg.message_id = message_id
return msg
class TestIgnoredMessageTypes:
"""Tests for messages listed in _ignored_message_types_for_replay.
AttachmentSupportingMessage is the sole type on this list. Such messages
bypass replay registration entirely — the handler runs directly regardless
of connection state, and replay manager methods are never invoked.
"""
def test_process__ignored_type__no_replay_on_online_connection__handler_still_called(
self,
processor: OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""All three replay manager methods are untouched for ignored types when online."""
noop_handler = mock.MagicMock(
spec=online_message_processor.MessageProcessingHandler
)
processor.register_message_handler(
handler=noop_handler, message_type=messages.AttachmentSupportingMessage
)
msg = _create_attachment_supporting_message(message_id=106)
processor.process(msg)
# check that the noop handler was called
noop_handler.assert_called_once_with(msg)
# No replay interaction at all
mock_replay.register_message.assert_not_called()
mock_replay.unregister_message.assert_not_called()
mock_replay.message_sent_failed.assert_not_called()
def test_process__ignored_type__no_replay_called_on_offline_connection__handler_still_called(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
):
"""The handler (noop) must be invoked for ignored types even when in offline mode —
replay state does not block execution for these message types."""
offline_replay = mock.MagicMock(spec=replay_manager.ReplayManager)
offline_replay.has_server_connection = False
offline_processor = OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=offline_replay,
unauthorized_message_types_registry=mock.Mock(),
)
noop_handler = mock.MagicMock(
spec=online_message_processor.MessageProcessingHandler
)
offline_processor.register_message_handler(
handler=noop_handler, message_type=messages.AttachmentSupportingMessage
)
# Process completes without error — the noop handler runs
msg = _create_attachment_supporting_message(message_id=105)
offline_processor.process(msg)
# check that the noop handler was called
noop_handler.assert_called_once_with(msg)
# No replay interaction at all
offline_replay.register_message.assert_not_called()
offline_replay.unregister_message.assert_not_called()
offline_replay.message_sent_failed.assert_not_called()
@@ -0,0 +1,409 @@
"""
Unit tests for OpikMessageProcessor interaction with UnauthorizedMessageTypeRegistry.
Tests cover:
- Unauthorized message types (is_authorized returns False) are skipped:
handler not called, replay manager not touched.
- 401 ApiError triggers registry.add() with the message type.
- 401 ApiError still causes message unregistration (not a connection error).
- Non-401 errors do NOT add to the registry.
- End-to-end: after a 401 causes a type to be registered, later messages
of that type are blocked; messages of other types are unaffected.
"""
import datetime
from unittest import mock
import pytest
from opik.message_processing import messages
from opik.message_processing import permissions
from opik.message_processing.processors import online_message_processor
from opik.message_processing.replay import replay_manager
from opik.rest_api import core as rest_api_core
def _create_trace_message(message_id: int = 1) -> messages.CreateTraceMessage:
msg = messages.CreateTraceMessage(
trace_id="trace-1",
project_name="test-project",
name="test-trace",
start_time=datetime.datetime(
2024, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc
),
end_time=datetime.datetime(2024, 1, 1, 12, 0, 1, tzinfo=datetime.timezone.utc),
input={"query": "test"},
output={"answer": "response"},
metadata=None,
tags=None,
error_info=None,
thread_id=None,
last_updated_at=None,
source="sdk",
)
msg.message_id = message_id
return msg
def _create_span_message(message_id: int = 1) -> messages.CreateSpanMessage:
msg = messages.CreateSpanMessage(
span_id="span-1",
trace_id="trace-1",
project_name="test-project",
parent_span_id=None,
name="test-span",
start_time=datetime.datetime(
2024, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc
),
end_time=datetime.datetime(2024, 1, 1, 12, 0, 1, tzinfo=datetime.timezone.utc),
input={"prompt": "test"},
output={"response": "result"},
metadata=None,
tags=None,
type="general",
usage=None,
model=None,
provider=None,
error_info=None,
total_cost=None,
last_updated_at=None,
source="sdk",
)
msg.message_id = message_id
return msg
@pytest.fixture
def mock_rest_client() -> mock.MagicMock:
return mock.MagicMock()
@pytest.fixture
def mock_file_uploader() -> mock.MagicMock:
return mock.MagicMock()
@pytest.fixture
def mock_replay() -> mock.MagicMock:
return mock.MagicMock(spec=replay_manager.ReplayManager)
@pytest.fixture
def mock_registry() -> mock.MagicMock:
registry = mock.MagicMock(spec=permissions.UnauthorizedMessageTypeRegistry)
registry.is_authorized.return_value = True
return registry
@pytest.fixture
def processor(
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
mock_registry: mock.MagicMock,
) -> online_message_processor.OpikMessageProcessor:
return online_message_processor.OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=mock_registry,
)
def test_process__authorized_message_type__handler_called(
processor: online_message_processor.OpikMessageProcessor,
mock_rest_client: mock.MagicMock,
):
"""Control: when is_authorized returns True, the handler must be invoked."""
msg = _create_trace_message()
processor.process(msg)
mock_rest_client.traces.create_trace.assert_called_once()
class TestUnauthorizedMessageTypeBlocking:
"""Tests for the early-exit path when is_authorized() returns False.
When is_authorized returns False, process() should return immediately
without invoking the handler or any replay manager methods.
"""
@pytest.fixture
def unauthorized_processor(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
) -> online_message_processor.OpikMessageProcessor:
registry = mock.MagicMock(spec=permissions.UnauthorizedMessageTypeRegistry)
registry.is_authorized.return_value = False
return online_message_processor.OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=registry,
)
def test_process__unauthorized_message_type__handler_not_called(
self,
unauthorized_processor: online_message_processor.OpikMessageProcessor,
mock_rest_client: mock.MagicMock,
):
"""When the message type is unauthorized, no REST handler should fire."""
msg = _create_trace_message()
unauthorized_processor.process(msg)
mock_rest_client.traces.create_trace.assert_not_called()
def test_process__unauthorized_message_type__register_not_called(
self,
unauthorized_processor: online_message_processor.OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""When the message type is unauthorized, register_message must not be called."""
msg = _create_trace_message()
unauthorized_processor.process(msg)
mock_replay.register_message.assert_not_called()
def test_process__unauthorized_message_type__unregister_not_called(
self,
unauthorized_processor: online_message_processor.OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""When the message type is unauthorized, unregister_message must not be called."""
msg = _create_trace_message()
unauthorized_processor.process(msg)
mock_replay.unregister_message.assert_not_called()
def test_process__unauthorized_message_type__message_sent_failed_not_called(
self,
unauthorized_processor: online_message_processor.OpikMessageProcessor,
mock_replay: mock.MagicMock,
):
"""When the message type is unauthorized, message_sent_failed must not be called."""
msg = _create_trace_message()
unauthorized_processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
def test_process__unauthorized_message_type__is_authorized_called_with_message_type(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
):
"""is_authorized must be invoked with message.message_type."""
registry = mock.MagicMock(spec=permissions.UnauthorizedMessageTypeRegistry)
registry.is_authorized.return_value = False
proc = online_message_processor.OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=registry,
)
msg = _create_trace_message()
proc.process(msg)
registry.is_authorized.assert_called_once_with(msg.message_type)
class TestUnauthorizedOnApiError401:
"""Tests for the 401 ApiError path.
A 401 response must cause the message type to be added to the unauthorized
registry. It should also unregister the message (not a connection error) but
must not call message_sent_failed.
"""
def test_process__api_error_401__adds_message_type_to_registry(
self,
processor: online_message_processor.OpikMessageProcessor,
mock_registry: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""On 401, registry.add must be called with message.message_type."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=401, body="Unauthorized"
)
msg = _create_trace_message()
processor.process(msg)
mock_registry.add.assert_called_once_with(msg.message_type)
def test_process__api_error_401__unregisters_message(
self,
processor: online_message_processor.OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""On 401, unregister_message must be called (not a connection error)."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=401, body="Unauthorized"
)
msg = _create_trace_message(message_id=20)
processor.process(msg)
mock_replay.unregister_message.assert_called_once_with(20)
def test_process__api_error_401__message_sent_failed_not_called(
self,
processor: online_message_processor.OpikMessageProcessor,
mock_replay: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""On 401, message_sent_failed must NOT be called."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=401, body="Unauthorized"
)
msg = _create_trace_message()
processor.process(msg)
mock_replay.message_sent_failed.assert_not_called()
@pytest.mark.parametrize(
"status_code,body",
[
(500, "Internal Server Error"),
(409, "Conflict"),
],
)
def test_process__api_error_not_401__does_not_add_to_registry(
self,
status_code,
body,
processor: online_message_processor.OpikMessageProcessor,
mock_registry: mock.MagicMock,
mock_rest_client: mock.MagicMock,
):
"""A non-401 error must NOT call registry.add — only 401 triggers this."""
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=status_code, body=body
)
msg = _create_trace_message()
processor.process(msg)
mock_registry.add.assert_not_called()
class TestUnauthorizedMessageTypeEndToEnd:
"""Integration tests using a real UnauthorizedMessageTypeRegistry.
These tests verify that after a 401 causes a type to be registered,
later messages of that type are blocked, while other types continue
to be processed normally.
"""
def test_process__after_401_error__same_type_blocked(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
capture_log,
):
"""After a 401 error registers a type, the next message of that type must
be skipped — handler not called, replay not touched for the second message."""
real_registry = permissions.UnauthorizedMessageTypeRegistry(
retry_interval_seconds=3600 # long interval — type stays blocked
)
proc = online_message_processor.OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=real_registry,
)
# The first call gets a 401 → type is registered as unauthorized
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=401, body="Unauthorized"
)
proc.process(_create_trace_message(message_id=1))
# Reset call tracking for the second call
mock_rest_client.reset_mock()
mock_replay.reset_mock()
# The second call — type is now blocked, handler should not fire
proc.process(_create_trace_message(message_id=2))
mock_rest_client.traces.create_trace.assert_not_called()
mock_replay.register_message.assert_not_called()
assert (
capture_log.records[0].message
== "Unauthorized message type 'CreateTraceMessage' processing request: Unauthorized"
)
assert capture_log.records[0].levelname == "ERROR"
def test_process__after_401_error__different_type_not_blocked(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
):
"""After 401 blocks CreateTraceMessage, CreateSpanMessage must still
be processed normally."""
real_registry = permissions.UnauthorizedMessageTypeRegistry(
retry_interval_seconds=3600
)
proc = online_message_processor.OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=real_registry,
)
# Cause CreateTraceMessage to be blocked via a 401
mock_rest_client.traces.create_trace.side_effect = rest_api_core.ApiError(
status_code=401, body="Unauthorized"
)
proc.process(_create_trace_message(message_id=1))
mock_rest_client.reset_mock()
mock_replay.reset_mock()
# CreateSpanMessage is a different type — should still go through
proc.process(_create_span_message(message_id=2))
mock_rest_client.spans.create_span.assert_called_once()
def test_process__after_retry_interval__type_authorized_again(
self,
mock_rest_client: mock.MagicMock,
mock_file_uploader: mock.MagicMock,
mock_replay: mock.MagicMock,
):
"""After the retry interval has elapsed, an unauthorized type should be
re-authorized and its handler invoked."""
real_registry = permissions.UnauthorizedMessageTypeRegistry(
retry_interval_seconds=60
)
proc = online_message_processor.OpikMessageProcessor(
rest_client=mock_rest_client,
file_upload_manager=mock_file_uploader,
fallback_replay_manager=mock_replay,
unauthorized_message_types_registry=real_registry,
)
# Manually add the type with a timestamp far in the past so the
# retry interval (60 s) has already elapsed
past_time = 0.0
real_registry.add(
messages.CreateTraceMessage.message_type, attempt_time=past_time
)
mock_rest_client.reset_mock()
mock_replay.reset_mock()
# The handler should be called because the interval has elapsed
mock_rest_client.traces.create_trace.side_effect = None # success
proc.process(_create_trace_message(message_id=3))
mock_rest_client.traces.create_trace.assert_called_once()