chore: import upstream snapshot with attribution
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:44 +08:00
commit 5a558eb09e
11579 changed files with 1795921 additions and 0 deletions
@@ -0,0 +1,67 @@
ExperimentItemModel
===================
.. currentmodule:: opik.message_processing.emulation.models
.. autoclass:: ExperimentItemModel
:special-members: __init__
Description
-----------
``ExperimentItemModel`` links a trace produced during evaluation to the dataset item
and experiment run that generated it. The SDK instantiates these records for you
while :func:`opik.evaluate` or experiment reruns execute; most users interact with
them through ``ScoreResult.metadata`` rather than constructing instances manually.
Metrics that analyse evaluation outputs can rely on this structure to connect
results back to source data.
Attributes
----------
.. attribute:: id
:type: str
:noindex:
Unique identifier for the experiment item record.
.. attribute:: experiment_id
:type: str
:noindex:
Identifier of the experiment that produced this item.
.. attribute:: trace_id
:type: str
:noindex:
Identifier of the trace logged during the evaluation run.
.. attribute:: dataset_item_id
:type: str
:noindex:
Identifier of the dataset item evaluated in this experiment result.
Usage Example
-------------
The SDK populates ``ExperimentItemModel`` instances automatically while running evaluations:
.. code-block:: python
from opik.message_processing.emulation.models import ExperimentItemModel
experiment_item = ExperimentItemModel(
id="exp_item_001",
experiment_id="exp_123",
trace_id="trace_abc",
dataset_item_id="dataset_item_xyz",
)
See Also
--------
- :class:`TraceModel` - Stores the trace referenced by ``trace_id``.
- :class:`SpanModel` - Contains spans that reference the same experiment item.
- :doc:`../evaluation/evaluate` - How experiments produce trace results.
@@ -0,0 +1,95 @@
FeedbackScoreModel
==================
.. currentmodule:: opik.message_processing.emulation.models
.. autoclass:: FeedbackScoreModel
:special-members: __init__
Description
-----------
The ``FeedbackScoreModel`` class represents a feedback score used to evaluate specific spans or traces in the Opik system. It stores and manages feedback scores linked to defined criteria, including identifiers, names, values, categories, and explanations for each score.
This model is typically used in evaluation contexts where you need to score or rate the performance of traces and spans based on various metrics.
Attributes
----------
.. attribute:: id
:type: str
:noindex:
Unique identifier for the feedback score.
.. attribute:: name
:type: str
:noindex:
Name associated with the feedback score, typically describing the metric being measured.
.. attribute:: value
:type: float
:noindex:
The numerical value of the feedback score. This represents the actual score or rating assigned.
.. attribute:: category_name
:type: Optional[str]
:value: None
:noindex:
Category to which the feedback score belongs, if any. This can be used to group related feedback scores together.
.. attribute:: reason
:type: Optional[str]
:value: None
:noindex:
Reason or explanation for the feedback score, if available. This provides context for why a particular score was assigned.
Examples
--------
Creating a basic feedback score:
.. code-block:: python
from opik.message_processing.emulation.models import FeedbackScoreModel
# Create a feedback score for a quality metric
feedback_score = FeedbackScoreModel(
id="score_123",
name="response_quality",
value=0.85,
category_name="quality",
reason="Response was accurate and well-structured"
)
Creating a feedback score with minimal information:
.. code-block:: python
# Create a simple feedback score
simple_score = FeedbackScoreModel(
id="score_456",
name="accuracy",
value=1.0
)
Usage in Evaluation
-------------------
``FeedbackScoreModel`` objects are commonly used in:
- **Evaluation Metrics**: Storing results from custom evaluation metrics
- **Span Scoring**: Associating quality scores with specific spans
- **Trace Evaluation**: Rating overall trace performance
- **A/B Testing**: Comparing different model outputs with scored feedback
See Also
--------
- :class:`SpanModel` - Contains lists of feedback scores
- :class:`TraceModel` - Also contains lists of feedback scores
- :doc:`../evaluation/evaluate` - For information about evaluation metrics that generate these scores
@@ -0,0 +1,247 @@
SpanModel
=========
.. currentmodule:: opik.message_processing.emulation.models
.. autoclass:: SpanModel
:special-members: __init__
Description
-----------
The ``SpanModel`` class represents a span model used to describe specific points in a process, their metadata, and associated data. This class is used to store and manipulate structured data for events or spans, including metadata, time markers, associated input/output, tags, and additional properties.
It serves as a representative structure for recording and organizing event-specific information, often used in applications like logging, distributed tracing, or data processing pipelines. In the context of Opik, spans represent individual operations or function calls within a larger trace.
Attributes
----------
Required Attributes
~~~~~~~~~~~~~~~~~~~
.. attribute:: id
:type: str
:noindex:
Unique identifier for the span.
.. attribute:: start_time
:type: datetime.datetime
:noindex:
Start time of the span, marking when the operation began.
Optional Attributes
~~~~~~~~~~~~~~~~~~~
.. attribute:: name
:type: Optional[str]
:noindex:
:value: None
Name of the span, if provided. This typically describes the operation being performed.
.. attribute:: input
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Input data associated with the span, if any. This contains the parameters or data passed to the operation.
.. attribute:: output
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Output data associated with the span, if any. This contains the results or return values from the operation.
.. attribute:: tags
:type: Optional[List[str]]
:noindex:
:value: None
List of tags linked to the span. Tags are used for categorization and filtering.
.. attribute:: metadata
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Additional metadata for the span. This can contain any custom information about the operation.
.. attribute:: type
:type: str
:noindex:
:value: "general"
Type of the span, defaulting to "general". Common types include "llm", "general", "tool", etc.
.. attribute:: usage
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Usage-related information for the span, such as token counts, API usage statistics, etc.
.. attribute:: end_time
:type: Optional[datetime.datetime]
:noindex:
:value: None
End time of the span, if available. This marks when the operation completed.
.. attribute:: project_name
:type: str
:noindex:
:value: OPIK_PROJECT_DEFAULT_NAME
Name of the project the span is associated with, defaulting to a predefined project name.
.. attribute:: model
:type: Optional[str]
:noindex:
:value: None
Model identification used, if applicable. This is commonly used for LLM spans to track which model was used.
.. attribute:: provider
:type: Optional[str]
:noindex:
:value: None
Provider of the span or associated services, if any. Examples include "openai", "anthropic", etc.
.. attribute:: error_info
:type: Optional[ErrorInfoDict]
:noindex:
:value: None
Error information or diagnostics for the span, if applicable. Contains details about any errors that occurred.
.. attribute:: total_cost
:type: Optional[float]
:noindex:
:value: None
Total cost incurred associated with this span, if relevant. This is useful for tracking API costs.
.. attribute:: last_updated_at
:type: Optional[datetime.datetime]
:noindex:
:value: None
Timestamp of when the span was last updated, if available.
Collection Attributes
~~~~~~~~~~~~~~~~~~~~~
.. attribute:: spans
:type: List[SpanModel]
:noindex:
List of nested spans related to this span. This creates a hierarchical structure where spans can contain child spans.
.. attribute:: feedback_scores
:type: List[FeedbackScoreModel]
:noindex:
List of feedback scores associated with the span. These scores are used for evaluation and quality assessment.
Examples
--------
Creating a basic span:
.. code-block:: python
import datetime
from opik.message_processing.emulation.models import SpanModel
# Create a simple span
span = SpanModel(
id="span_123",
start_time=datetime.datetime.now(),
name="llm_call",
type="llm",
input={"prompt": "What is the capital of France?"},
output={"response": "Paris is the capital of France."},
model="gpt-4",
provider="openai"
)
Creating a span with nested spans:
.. code-block:: python
# Create a parent span with child spans
parent_span = SpanModel(
id="parent_123",
start_time=datetime.datetime.now(),
name="complex_operation"
)
child_span = SpanModel(
id="child_456",
start_time=datetime.datetime.now(),
name="preprocessing_step"
)
parent_span.spans.append(child_span)
Adding feedback scores to a span:
.. code-block:: python
from opik.message_processing.emulation.models import FeedbackScoreModel
# Add evaluation scores to the span
quality_score = FeedbackScoreModel(
id="score_789",
name="response_quality",
value=0.92,
reason="High quality response with accurate information"
)
span.feedback_scores.append(quality_score)
Usage in Task Span Evaluation
-----------------------------
``SpanModel`` objects are particularly important in task span evaluation, where custom metrics can analyze the span data:
.. code-block:: python
from opik.evaluation.metrics import BaseMetric, score_result
class CustomSpanMetric(BaseMetric):
def score(self, task_span: SpanModel) -> score_result.ScoreResult:
# Access span properties for evaluation
input_data = task_span.input
output_data = task_span.output
# Perform custom evaluation logic
score_value = self.evaluate_span_quality(input_data, output_data)
return score_result.ScoreResult(
value=score_value,
name=self.name,
reason=f"Evaluated span '{task_span.name}'"
)
Common Use Cases
----------------
``SpanModel`` is commonly used for:
- **Function Tracking**: Recording individual function or method calls
- **LLM Operations**: Tracking language model API calls with usage and cost information
- **Pipeline Steps**: Representing steps in data processing pipelines
- **Evaluation**: Providing detailed execution data for custom evaluation metrics
- **Debugging**: Analyzing the structure and performance of complex operations
See Also
--------
- :class:`TraceModel` - The parent container that holds spans
- :class:`FeedbackScoreModel` - For attaching evaluation scores to spans
- :doc:`../evaluation/evaluate` - For information about evaluating spans with custom metrics
@@ -0,0 +1,254 @@
TraceModel
==========
.. currentmodule:: opik.message_processing.emulation.models
.. autoclass:: TraceModel
:special-members: __init__
Description
-----------
The ``TraceModel`` class represents a trace model that encapsulates data about a trace, its related metadata, and associated spans. It is used for tracking and analyzing data during execution or processing tasks.
This class provides a structure to represent trace information, including the start and end times, associated project details, input/output data, feedback scores, error information, and thread association. It is designed to handle optional fields for flexible use across various scenarios.
A trace represents the complete execution path of a request or operation, containing one or more spans that represent individual steps or components within that execution.
Attributes
----------
Required Attributes
~~~~~~~~~~~~~~~~~~~
.. attribute:: id
:type: str
:noindex:
Unique identifier for the trace.
.. attribute:: start_time
:type: datetime.datetime
:noindex:
Timestamp representing the start of the trace.
.. attribute:: name
:type: Optional[str]
:noindex:
Optional name for the trace, which can provide a descriptive label for the operation being traced.
.. attribute:: project_name
:type: str
:noindex:
Name of the project associated with the trace.
Optional Attributes
~~~~~~~~~~~~~~~~~~~
.. attribute:: input
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Optional dictionary containing the input data associated with the trace. This represents the initial parameters or data that started the trace.
.. attribute:: output
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Optional dictionary containing the output data generated by the trace. This represents the final results or return values.
.. attribute:: tags
:type: Optional[List[str]]
:noindex:
:value: None
Optional list of tags associated with the trace for classification or filtering purposes.
.. attribute:: metadata
:type: Optional[Dict[str, Any]]
:noindex:
:value: None
Optional metadata providing additional information about the trace.
.. attribute:: end_time
:type: Optional[datetime.datetime]
:noindex:
:value: None
Timestamp representing the end of the trace. When set, this marks when the operation completed.
.. attribute:: error_info
:type: Optional[ErrorInfoDict]
:noindex:
:value: None
Optional dictionary containing information about errors encountered during the trace.
.. attribute:: thread_id
:type: Optional[str]
:noindex:
:value: None
Optional identifier of the thread associated with the trace. Useful for concurrent operations.
.. attribute:: last_updated_at
:type: Optional[datetime.datetime]
:noindex:
:value: None
Timestamp for when the trace was last updated.
Collection Attributes
~~~~~~~~~~~~~~~~~~~~~
.. attribute:: spans
:type: List[SpanModel]
:noindex:
List of spans associated with the trace, representing individual processing parts or segments within the trace. Each span represents a specific operation or step in the overall execution.
.. attribute:: feedback_scores
:type: List[FeedbackScoreModel]
:noindex:
List of feedback scores associated with the trace. These are used for overall trace evaluation and quality assessment.
Examples
--------
Creating a basic trace:
.. code-block:: python
import datetime
from opik.message_processing.emulation.models import TraceModel
# Create a simple trace
trace = TraceModel(
id="trace_123",
start_time=datetime.datetime.now(),
name="user_query_processing",
project_name="my_project",
input={"user_query": "What is machine learning?"},
output={"response": "Machine learning is a subset of AI..."}
)
Creating a trace with spans:
.. code-block:: python
from opik.message_processing.emulation.models import SpanModel
# Create a trace with associated spans
trace = TraceModel(
id="trace_456",
start_time=datetime.datetime.now(),
name="complex_operation",
project_name="ai_project"
)
# Add spans to represent different steps
preprocessing_span = SpanModel(
id="span_1",
start_time=datetime.datetime.now(),
name="data_preprocessing"
)
llm_span = SpanModel(
id="span_2",
start_time=datetime.datetime.now(),
name="llm_call",
type="llm"
)
trace.spans.extend([preprocessing_span, llm_span])
Adding feedback scores to a trace:
.. code-block:: python
from opik.message_processing.emulation.models import FeedbackScoreModel
# Add overall evaluation scores to the trace
overall_quality = FeedbackScoreModel(
id="score_123",
name="overall_quality",
value=0.88,
reason="Good response quality with minor improvements needed"
)
trace.feedback_scores.append(overall_quality)
Working with trace hierarchies:
.. code-block:: python
# Access nested spans within a trace
for span in trace.spans:
print(f"Span: {span.name}")
# Each span can have nested spans too
for nested_span in span.spans:
print(f" Nested: {nested_span.name}")
Usage in Evaluation Context
---------------------------
``TraceModel`` objects are commonly used in evaluation scenarios where you need to analyze the complete execution:
.. code-block:: python
# Example of accessing trace data in evaluation
def analyze_trace(trace: TraceModel):
# Analyze overall trace performance
duration = trace.end_time - trace.start_time if trace.end_time else None
# Count different types of spans
llm_spans = [s for s in trace.spans if s.type == "llm"]
# Analyze input/output
input_complexity = len(str(trace.input)) if trace.input else 0
output_quality = evaluate_output_quality(trace.output)
return {
"duration": duration,
"llm_calls": len(llm_spans),
"complexity": input_complexity,
"quality": output_quality
}
Common Use Cases
----------------
``TraceModel`` is commonly used for:
- **Request Tracking**: Tracking complete user requests from start to finish
- **Performance Analysis**: Analyzing the performance of complex operations
- **Evaluation**: Providing complete context for evaluation metrics
- **Debugging**: Understanding the full execution path and identifying issues
- **Cost Tracking**: Aggregating costs across all spans in a trace
- **A/B Testing**: Comparing different execution paths and their outcomes
Relationship with Spans
-----------------------
A trace acts as a container for spans, creating a hierarchical structure:
- **Trace**: The top-level container representing the complete operation
- **Spans**: Individual steps or operations within the trace
- **Nested Spans**: Spans can contain other spans, creating a tree structure
This hierarchy allows for detailed tracking of complex operations while maintaining the overall context.
See Also
--------
- :class:`SpanModel` - Individual operations within a trace
- :class:`FeedbackScoreModel` - For attaching evaluation scores to traces
- :doc:`../evaluation/evaluate` - For information about evaluating traces with custom metrics
@@ -0,0 +1,158 @@
Message Processing Emulation Models
====================================
.. currentmodule:: opik.message_processing.emulation.models
This module provides data models used for message processing emulation in Opik. These models represent the core data structures for traces, spans, and feedback scores that are used internally by the Opik SDK during evaluation.
Overview
--------
The message processing emulation models are primarily used in evaluation contexts, particularly for task span evaluation where custom metrics need access to detailed execution information. These models provide a structured representation of:
- **Traces**: Complete execution paths of requests or operations
- **Spans**: Individual steps or operations within a trace
- **Feedback Scores**: Evaluation results attached to traces and spans
- **Experiment Items**: Links between traces, datasets, and experiment runs
Key Classes
-----------
.. toctree::
:maxdepth: 1
FeedbackScoreModel
SpanModel
TraceModel
ExperimentItemModel
local_recording
Class Hierarchy
---------------
The models form a hierarchical relationship:
.. code-block:: text
TraceModel
├── spans: List[SpanModel]
│ ├── spans: List[SpanModel] (nested spans)
│ └── feedback_scores: List[FeedbackScoreModel]
└── feedback_scores: List[FeedbackScoreModel]
Quick Start
-----------
Import the models:
.. code-block:: python
from opik.message_processing.emulation.models import (
TraceModel,
SpanModel,
FeedbackScoreModel,
ExperimentItemModel
)
Common Usage Patterns
---------------------
Task Span Evaluation
~~~~~~~~~~~~~~~~~~~~~
The primary use case for these models is in task span evaluation, where custom metrics analyze span data:
.. code-block:: python
from opik.evaluation.metrics import BaseMetric, score_result
from opik.message_processing.emulation.models import SpanModel
class CustomSpanMetric(BaseMetric):
def score(self, task_span: SpanModel) -> score_result.ScoreResult:
# Access span properties
span_name = task_span.name
input_data = task_span.input
output_data = task_span.output
# Perform evaluation logic
score_value = self.evaluate_span(span_name, input_data, output_data)
return score_result.ScoreResult(
value=score_value,
name=self.name,
reason=f"Evaluated span: {span_name}"
)
Analyzing Trace Structure
~~~~~~~~~~~~~~~~~~~~~~~~~
You can traverse and analyze the hierarchical structure of traces:
.. code-block:: python
def analyze_trace_structure(trace: TraceModel):
print(f"Trace: {trace.name}")
print(f"Total spans: {len(trace.spans)}")
for span in trace.spans:
print(f" Span: {span.name} (type: {span.type})")
# Analyze nested spans
for nested_span in span.spans:
print(f" Nested: {nested_span.name}")
Working with Feedback Scores
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Both traces and spans can contain feedback scores from evaluations:
.. code-block:: python
def collect_all_scores(trace: TraceModel):
all_scores = []
# Collect trace-level scores
all_scores.extend(trace.feedback_scores)
# Collect span-level scores
for span in trace.spans:
all_scores.extend(span.feedback_scores)
# Recursively collect from nested spans
for nested_span in span.spans:
all_scores.extend(nested_span.feedback_scores)
return all_scores
Integration with Evaluation System
----------------------------------
These models are automatically populated and used by the Opik evaluation system:
1. **Trace Creation**: When you run ``opik.evaluate()``, traces are automatically created
2. **Span Population**: Individual function calls become spans within the trace
3. **Task Span Evaluation**: Metrics with ``task_span`` parameters receive ``SpanModel`` objects
4. **Score Attachment**: Feedback scores are automatically attached to the appropriate traces and spans
You typically don't need to create these models manually - they're generated automatically during evaluation. However, understanding their structure is essential for writing effective task span evaluation metrics.
Use Cases
---------
These models are commonly used for:
- **Custom Evaluation Metrics**: Analyzing detailed execution data in custom metrics
- **Performance Analysis**: Understanding execution patterns and performance characteristics
- **Debugging**: Investigating issues in complex operations
- **Cost Tracking**: Aggregating usage and cost information across operations
- **Quality Assessment**: Evaluating the quality of individual steps and overall operations
Module Reference
----------------
For detailed API documentation, see the following class reference pages:
- :doc:`TraceModel <../message_processing_emulation/TraceModel>`
- :doc:`SpanModel <../message_processing_emulation/SpanModel>`
- :doc:`FeedbackScoreModel <../message_processing_emulation/FeedbackScoreModel>`
- :doc:`ExperimentItemModel <../message_processing_emulation/ExperimentItemModel>`
@@ -0,0 +1,49 @@
Local Recording Context Manager
===============================
.. currentmodule:: opik
`record_traces_locally`
-----------------------
The ``record_traces_locally`` context manager enables local, in-memory recording of any traces and spans created inside its block. This is useful for testing, debugging, or for programmatically inspecting your span/trace trees without sending data to the backend.
Basic usage
~~~~~~~~~~~
.. code-block:: python
import opik
with opik.record_traces_locally() as storage:
# Your instrumented code that creates traces/spans
# e.g., functions decorated with @opik.track, manual opik.Opik().span()/trace(), integrations, etc.
...
# Access in-memory results (automatically flushed before reading)
span_models = storage.span_trees
trace_models = storage.trace_trees
What it returns
~~~~~~~~~~~~~~~
The context yields a lightweight handle having these properties:
- ``span_trees``: List of :class:`opik.message_processing.emulation.models.SpanModel`
- ``trace_trees``: List of :class:`opik.message_processing.emulation.models.TraceModel`
Each accessor flushes the Opik client to ensure all in-flight messages are processed before reading the local state.
No nested usage
~~~~~~~~~~~~~~~~
Nested or concurrent usages within the same process are not supported. If a local recording is already active, entering another ``record_traces_locally`` block raises ``RuntimeError``.
Notes
~~~~~
- Uses the SDK's local emulator to mirror what would be sent to the backend.
- Data is kept in memory only for the life of the context. On exit, the local recorder is disabled and state is reset.
- Ideal for `task_span` metrics validation, writing tests or ad-hoc scripts that need access to the span/trace tree structure.