chore: import upstream snapshot with attribution
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
This commit is contained in:
+67
@@ -0,0 +1,67 @@
|
||||
ExperimentItemModel
|
||||
===================
|
||||
|
||||
.. currentmodule:: opik.message_processing.emulation.models
|
||||
|
||||
.. autoclass:: ExperimentItemModel
|
||||
:special-members: __init__
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
``ExperimentItemModel`` links a trace produced during evaluation to the dataset item
|
||||
and experiment run that generated it. The SDK instantiates these records for you
|
||||
while :func:`opik.evaluate` or experiment reruns execute; most users interact with
|
||||
them through ``ScoreResult.metadata`` rather than constructing instances manually.
|
||||
Metrics that analyse evaluation outputs can rely on this structure to connect
|
||||
results back to source data.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. attribute:: id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Unique identifier for the experiment item record.
|
||||
|
||||
.. attribute:: experiment_id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Identifier of the experiment that produced this item.
|
||||
|
||||
.. attribute:: trace_id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Identifier of the trace logged during the evaluation run.
|
||||
|
||||
.. attribute:: dataset_item_id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Identifier of the dataset item evaluated in this experiment result.
|
||||
|
||||
Usage Example
|
||||
-------------
|
||||
|
||||
The SDK populates ``ExperimentItemModel`` instances automatically while running evaluations:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import ExperimentItemModel
|
||||
|
||||
experiment_item = ExperimentItemModel(
|
||||
id="exp_item_001",
|
||||
experiment_id="exp_123",
|
||||
trace_id="trace_abc",
|
||||
dataset_item_id="dataset_item_xyz",
|
||||
)
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
- :class:`TraceModel` - Stores the trace referenced by ``trace_id``.
|
||||
- :class:`SpanModel` - Contains spans that reference the same experiment item.
|
||||
- :doc:`../evaluation/evaluate` - How experiments produce trace results.
|
||||
+95
@@ -0,0 +1,95 @@
|
||||
FeedbackScoreModel
|
||||
==================
|
||||
|
||||
.. currentmodule:: opik.message_processing.emulation.models
|
||||
|
||||
.. autoclass:: FeedbackScoreModel
|
||||
:special-members: __init__
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
The ``FeedbackScoreModel`` class represents a feedback score used to evaluate specific spans or traces in the Opik system. It stores and manages feedback scores linked to defined criteria, including identifiers, names, values, categories, and explanations for each score.
|
||||
|
||||
This model is typically used in evaluation contexts where you need to score or rate the performance of traces and spans based on various metrics.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. attribute:: id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Unique identifier for the feedback score.
|
||||
|
||||
.. attribute:: name
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Name associated with the feedback score, typically describing the metric being measured.
|
||||
|
||||
.. attribute:: value
|
||||
:type: float
|
||||
:noindex:
|
||||
|
||||
The numerical value of the feedback score. This represents the actual score or rating assigned.
|
||||
|
||||
.. attribute:: category_name
|
||||
:type: Optional[str]
|
||||
:value: None
|
||||
:noindex:
|
||||
|
||||
Category to which the feedback score belongs, if any. This can be used to group related feedback scores together.
|
||||
|
||||
.. attribute:: reason
|
||||
:type: Optional[str]
|
||||
:value: None
|
||||
:noindex:
|
||||
|
||||
Reason or explanation for the feedback score, if available. This provides context for why a particular score was assigned.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Creating a basic feedback score:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import FeedbackScoreModel
|
||||
|
||||
# Create a feedback score for a quality metric
|
||||
feedback_score = FeedbackScoreModel(
|
||||
id="score_123",
|
||||
name="response_quality",
|
||||
value=0.85,
|
||||
category_name="quality",
|
||||
reason="Response was accurate and well-structured"
|
||||
)
|
||||
|
||||
Creating a feedback score with minimal information:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Create a simple feedback score
|
||||
simple_score = FeedbackScoreModel(
|
||||
id="score_456",
|
||||
name="accuracy",
|
||||
value=1.0
|
||||
)
|
||||
|
||||
Usage in Evaluation
|
||||
-------------------
|
||||
|
||||
``FeedbackScoreModel`` objects are commonly used in:
|
||||
|
||||
- **Evaluation Metrics**: Storing results from custom evaluation metrics
|
||||
- **Span Scoring**: Associating quality scores with specific spans
|
||||
- **Trace Evaluation**: Rating overall trace performance
|
||||
- **A/B Testing**: Comparing different model outputs with scored feedback
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
- :class:`SpanModel` - Contains lists of feedback scores
|
||||
- :class:`TraceModel` - Also contains lists of feedback scores
|
||||
- :doc:`../evaluation/evaluate` - For information about evaluation metrics that generate these scores
|
||||
+247
@@ -0,0 +1,247 @@
|
||||
SpanModel
|
||||
=========
|
||||
|
||||
.. currentmodule:: opik.message_processing.emulation.models
|
||||
|
||||
.. autoclass:: SpanModel
|
||||
:special-members: __init__
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
The ``SpanModel`` class represents a span model used to describe specific points in a process, their metadata, and associated data. This class is used to store and manipulate structured data for events or spans, including metadata, time markers, associated input/output, tags, and additional properties.
|
||||
|
||||
It serves as a representative structure for recording and organizing event-specific information, often used in applications like logging, distributed tracing, or data processing pipelines. In the context of Opik, spans represent individual operations or function calls within a larger trace.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
Required Attributes
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Unique identifier for the span.
|
||||
|
||||
.. attribute:: start_time
|
||||
:type: datetime.datetime
|
||||
:noindex:
|
||||
|
||||
Start time of the span, marking when the operation began.
|
||||
|
||||
Optional Attributes
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: name
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Name of the span, if provided. This typically describes the operation being performed.
|
||||
|
||||
.. attribute:: input
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Input data associated with the span, if any. This contains the parameters or data passed to the operation.
|
||||
|
||||
.. attribute:: output
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Output data associated with the span, if any. This contains the results or return values from the operation.
|
||||
|
||||
.. attribute:: tags
|
||||
:type: Optional[List[str]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
List of tags linked to the span. Tags are used for categorization and filtering.
|
||||
|
||||
.. attribute:: metadata
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Additional metadata for the span. This can contain any custom information about the operation.
|
||||
|
||||
.. attribute:: type
|
||||
:type: str
|
||||
:noindex:
|
||||
:value: "general"
|
||||
|
||||
Type of the span, defaulting to "general". Common types include "llm", "general", "tool", etc.
|
||||
|
||||
.. attribute:: usage
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Usage-related information for the span, such as token counts, API usage statistics, etc.
|
||||
|
||||
.. attribute:: end_time
|
||||
:type: Optional[datetime.datetime]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
End time of the span, if available. This marks when the operation completed.
|
||||
|
||||
.. attribute:: project_name
|
||||
:type: str
|
||||
:noindex:
|
||||
:value: OPIK_PROJECT_DEFAULT_NAME
|
||||
|
||||
Name of the project the span is associated with, defaulting to a predefined project name.
|
||||
|
||||
.. attribute:: model
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Model identification used, if applicable. This is commonly used for LLM spans to track which model was used.
|
||||
|
||||
.. attribute:: provider
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Provider of the span or associated services, if any. Examples include "openai", "anthropic", etc.
|
||||
|
||||
.. attribute:: error_info
|
||||
:type: Optional[ErrorInfoDict]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Error information or diagnostics for the span, if applicable. Contains details about any errors that occurred.
|
||||
|
||||
.. attribute:: total_cost
|
||||
:type: Optional[float]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Total cost incurred associated with this span, if relevant. This is useful for tracking API costs.
|
||||
|
||||
.. attribute:: last_updated_at
|
||||
:type: Optional[datetime.datetime]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Timestamp of when the span was last updated, if available.
|
||||
|
||||
Collection Attributes
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: spans
|
||||
:type: List[SpanModel]
|
||||
:noindex:
|
||||
|
||||
List of nested spans related to this span. This creates a hierarchical structure where spans can contain child spans.
|
||||
|
||||
.. attribute:: feedback_scores
|
||||
:type: List[FeedbackScoreModel]
|
||||
:noindex:
|
||||
|
||||
List of feedback scores associated with the span. These scores are used for evaluation and quality assessment.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Creating a basic span:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import datetime
|
||||
from opik.message_processing.emulation.models import SpanModel
|
||||
|
||||
# Create a simple span
|
||||
span = SpanModel(
|
||||
id="span_123",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="llm_call",
|
||||
type="llm",
|
||||
input={"prompt": "What is the capital of France?"},
|
||||
output={"response": "Paris is the capital of France."},
|
||||
model="gpt-4",
|
||||
provider="openai"
|
||||
)
|
||||
|
||||
Creating a span with nested spans:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Create a parent span with child spans
|
||||
parent_span = SpanModel(
|
||||
id="parent_123",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="complex_operation"
|
||||
)
|
||||
|
||||
child_span = SpanModel(
|
||||
id="child_456",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="preprocessing_step"
|
||||
)
|
||||
|
||||
parent_span.spans.append(child_span)
|
||||
|
||||
Adding feedback scores to a span:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import FeedbackScoreModel
|
||||
|
||||
# Add evaluation scores to the span
|
||||
quality_score = FeedbackScoreModel(
|
||||
id="score_789",
|
||||
name="response_quality",
|
||||
value=0.92,
|
||||
reason="High quality response with accurate information"
|
||||
)
|
||||
|
||||
span.feedback_scores.append(quality_score)
|
||||
|
||||
Usage in Task Span Evaluation
|
||||
-----------------------------
|
||||
|
||||
``SpanModel`` objects are particularly important in task span evaluation, where custom metrics can analyze the span data:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.evaluation.metrics import BaseMetric, score_result
|
||||
|
||||
class CustomSpanMetric(BaseMetric):
|
||||
def score(self, task_span: SpanModel) -> score_result.ScoreResult:
|
||||
# Access span properties for evaluation
|
||||
input_data = task_span.input
|
||||
output_data = task_span.output
|
||||
|
||||
# Perform custom evaluation logic
|
||||
score_value = self.evaluate_span_quality(input_data, output_data)
|
||||
|
||||
return score_result.ScoreResult(
|
||||
value=score_value,
|
||||
name=self.name,
|
||||
reason=f"Evaluated span '{task_span.name}'"
|
||||
)
|
||||
|
||||
Common Use Cases
|
||||
----------------
|
||||
|
||||
``SpanModel`` is commonly used for:
|
||||
|
||||
- **Function Tracking**: Recording individual function or method calls
|
||||
- **LLM Operations**: Tracking language model API calls with usage and cost information
|
||||
- **Pipeline Steps**: Representing steps in data processing pipelines
|
||||
- **Evaluation**: Providing detailed execution data for custom evaluation metrics
|
||||
- **Debugging**: Analyzing the structure and performance of complex operations
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
- :class:`TraceModel` - The parent container that holds spans
|
||||
- :class:`FeedbackScoreModel` - For attaching evaluation scores to spans
|
||||
- :doc:`../evaluation/evaluate` - For information about evaluating spans with custom metrics
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
TraceModel
|
||||
==========
|
||||
|
||||
.. currentmodule:: opik.message_processing.emulation.models
|
||||
|
||||
.. autoclass:: TraceModel
|
||||
:special-members: __init__
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
The ``TraceModel`` class represents a trace model that encapsulates data about a trace, its related metadata, and associated spans. It is used for tracking and analyzing data during execution or processing tasks.
|
||||
|
||||
This class provides a structure to represent trace information, including the start and end times, associated project details, input/output data, feedback scores, error information, and thread association. It is designed to handle optional fields for flexible use across various scenarios.
|
||||
|
||||
A trace represents the complete execution path of a request or operation, containing one or more spans that represent individual steps or components within that execution.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
Required Attributes
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Unique identifier for the trace.
|
||||
|
||||
.. attribute:: start_time
|
||||
:type: datetime.datetime
|
||||
:noindex:
|
||||
|
||||
Timestamp representing the start of the trace.
|
||||
|
||||
.. attribute:: name
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
|
||||
Optional name for the trace, which can provide a descriptive label for the operation being traced.
|
||||
|
||||
.. attribute:: project_name
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Name of the project associated with the trace.
|
||||
|
||||
Optional Attributes
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: input
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional dictionary containing the input data associated with the trace. This represents the initial parameters or data that started the trace.
|
||||
|
||||
.. attribute:: output
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional dictionary containing the output data generated by the trace. This represents the final results or return values.
|
||||
|
||||
.. attribute:: tags
|
||||
:type: Optional[List[str]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional list of tags associated with the trace for classification or filtering purposes.
|
||||
|
||||
.. attribute:: metadata
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional metadata providing additional information about the trace.
|
||||
|
||||
.. attribute:: end_time
|
||||
:type: Optional[datetime.datetime]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Timestamp representing the end of the trace. When set, this marks when the operation completed.
|
||||
|
||||
.. attribute:: error_info
|
||||
:type: Optional[ErrorInfoDict]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional dictionary containing information about errors encountered during the trace.
|
||||
|
||||
.. attribute:: thread_id
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional identifier of the thread associated with the trace. Useful for concurrent operations.
|
||||
|
||||
.. attribute:: last_updated_at
|
||||
:type: Optional[datetime.datetime]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Timestamp for when the trace was last updated.
|
||||
|
||||
Collection Attributes
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: spans
|
||||
:type: List[SpanModel]
|
||||
:noindex:
|
||||
|
||||
List of spans associated with the trace, representing individual processing parts or segments within the trace. Each span represents a specific operation or step in the overall execution.
|
||||
|
||||
.. attribute:: feedback_scores
|
||||
:type: List[FeedbackScoreModel]
|
||||
:noindex:
|
||||
|
||||
List of feedback scores associated with the trace. These are used for overall trace evaluation and quality assessment.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Creating a basic trace:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import datetime
|
||||
from opik.message_processing.emulation.models import TraceModel
|
||||
|
||||
# Create a simple trace
|
||||
trace = TraceModel(
|
||||
id="trace_123",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="user_query_processing",
|
||||
project_name="my_project",
|
||||
input={"user_query": "What is machine learning?"},
|
||||
output={"response": "Machine learning is a subset of AI..."}
|
||||
)
|
||||
|
||||
Creating a trace with spans:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import SpanModel
|
||||
|
||||
# Create a trace with associated spans
|
||||
trace = TraceModel(
|
||||
id="trace_456",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="complex_operation",
|
||||
project_name="ai_project"
|
||||
)
|
||||
|
||||
# Add spans to represent different steps
|
||||
preprocessing_span = SpanModel(
|
||||
id="span_1",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="data_preprocessing"
|
||||
)
|
||||
|
||||
llm_span = SpanModel(
|
||||
id="span_2",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="llm_call",
|
||||
type="llm"
|
||||
)
|
||||
|
||||
trace.spans.extend([preprocessing_span, llm_span])
|
||||
|
||||
Adding feedback scores to a trace:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import FeedbackScoreModel
|
||||
|
||||
# Add overall evaluation scores to the trace
|
||||
overall_quality = FeedbackScoreModel(
|
||||
id="score_123",
|
||||
name="overall_quality",
|
||||
value=0.88,
|
||||
reason="Good response quality with minor improvements needed"
|
||||
)
|
||||
|
||||
trace.feedback_scores.append(overall_quality)
|
||||
|
||||
Working with trace hierarchies:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Access nested spans within a trace
|
||||
for span in trace.spans:
|
||||
print(f"Span: {span.name}")
|
||||
|
||||
# Each span can have nested spans too
|
||||
for nested_span in span.spans:
|
||||
print(f" Nested: {nested_span.name}")
|
||||
|
||||
Usage in Evaluation Context
|
||||
---------------------------
|
||||
|
||||
``TraceModel`` objects are commonly used in evaluation scenarios where you need to analyze the complete execution:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Example of accessing trace data in evaluation
|
||||
def analyze_trace(trace: TraceModel):
|
||||
# Analyze overall trace performance
|
||||
duration = trace.end_time - trace.start_time if trace.end_time else None
|
||||
|
||||
# Count different types of spans
|
||||
llm_spans = [s for s in trace.spans if s.type == "llm"]
|
||||
|
||||
# Analyze input/output
|
||||
input_complexity = len(str(trace.input)) if trace.input else 0
|
||||
output_quality = evaluate_output_quality(trace.output)
|
||||
|
||||
return {
|
||||
"duration": duration,
|
||||
"llm_calls": len(llm_spans),
|
||||
"complexity": input_complexity,
|
||||
"quality": output_quality
|
||||
}
|
||||
|
||||
Common Use Cases
|
||||
----------------
|
||||
|
||||
``TraceModel`` is commonly used for:
|
||||
|
||||
- **Request Tracking**: Tracking complete user requests from start to finish
|
||||
- **Performance Analysis**: Analyzing the performance of complex operations
|
||||
- **Evaluation**: Providing complete context for evaluation metrics
|
||||
- **Debugging**: Understanding the full execution path and identifying issues
|
||||
- **Cost Tracking**: Aggregating costs across all spans in a trace
|
||||
- **A/B Testing**: Comparing different execution paths and their outcomes
|
||||
|
||||
Relationship with Spans
|
||||
-----------------------
|
||||
|
||||
A trace acts as a container for spans, creating a hierarchical structure:
|
||||
|
||||
- **Trace**: The top-level container representing the complete operation
|
||||
- **Spans**: Individual steps or operations within the trace
|
||||
- **Nested Spans**: Spans can contain other spans, creating a tree structure
|
||||
|
||||
This hierarchy allows for detailed tracking of complex operations while maintaining the overall context.
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
- :class:`SpanModel` - Individual operations within a trace
|
||||
- :class:`FeedbackScoreModel` - For attaching evaluation scores to traces
|
||||
- :doc:`../evaluation/evaluate` - For information about evaluating traces with custom metrics
|
||||
+158
@@ -0,0 +1,158 @@
|
||||
Message Processing Emulation Models
|
||||
====================================
|
||||
|
||||
.. currentmodule:: opik.message_processing.emulation.models
|
||||
|
||||
This module provides data models used for message processing emulation in Opik. These models represent the core data structures for traces, spans, and feedback scores that are used internally by the Opik SDK during evaluation.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The message processing emulation models are primarily used in evaluation contexts, particularly for task span evaluation where custom metrics need access to detailed execution information. These models provide a structured representation of:
|
||||
|
||||
- **Traces**: Complete execution paths of requests or operations
|
||||
- **Spans**: Individual steps or operations within a trace
|
||||
- **Feedback Scores**: Evaluation results attached to traces and spans
|
||||
- **Experiment Items**: Links between traces, datasets, and experiment runs
|
||||
|
||||
Key Classes
|
||||
-----------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
FeedbackScoreModel
|
||||
SpanModel
|
||||
TraceModel
|
||||
ExperimentItemModel
|
||||
local_recording
|
||||
|
||||
Class Hierarchy
|
||||
---------------
|
||||
|
||||
The models form a hierarchical relationship:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
TraceModel
|
||||
├── spans: List[SpanModel]
|
||||
│ ├── spans: List[SpanModel] (nested spans)
|
||||
│ └── feedback_scores: List[FeedbackScoreModel]
|
||||
└── feedback_scores: List[FeedbackScoreModel]
|
||||
|
||||
Quick Start
|
||||
-----------
|
||||
|
||||
Import the models:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import (
|
||||
TraceModel,
|
||||
SpanModel,
|
||||
FeedbackScoreModel,
|
||||
ExperimentItemModel
|
||||
)
|
||||
|
||||
Common Usage Patterns
|
||||
---------------------
|
||||
|
||||
Task Span Evaluation
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The primary use case for these models is in task span evaluation, where custom metrics analyze span data:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.evaluation.metrics import BaseMetric, score_result
|
||||
from opik.message_processing.emulation.models import SpanModel
|
||||
|
||||
class CustomSpanMetric(BaseMetric):
|
||||
def score(self, task_span: SpanModel) -> score_result.ScoreResult:
|
||||
# Access span properties
|
||||
span_name = task_span.name
|
||||
input_data = task_span.input
|
||||
output_data = task_span.output
|
||||
|
||||
# Perform evaluation logic
|
||||
score_value = self.evaluate_span(span_name, input_data, output_data)
|
||||
|
||||
return score_result.ScoreResult(
|
||||
value=score_value,
|
||||
name=self.name,
|
||||
reason=f"Evaluated span: {span_name}"
|
||||
)
|
||||
|
||||
Analyzing Trace Structure
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can traverse and analyze the hierarchical structure of traces:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def analyze_trace_structure(trace: TraceModel):
|
||||
print(f"Trace: {trace.name}")
|
||||
print(f"Total spans: {len(trace.spans)}")
|
||||
|
||||
for span in trace.spans:
|
||||
print(f" Span: {span.name} (type: {span.type})")
|
||||
|
||||
# Analyze nested spans
|
||||
for nested_span in span.spans:
|
||||
print(f" Nested: {nested_span.name}")
|
||||
|
||||
Working with Feedback Scores
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Both traces and spans can contain feedback scores from evaluations:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def collect_all_scores(trace: TraceModel):
|
||||
all_scores = []
|
||||
|
||||
# Collect trace-level scores
|
||||
all_scores.extend(trace.feedback_scores)
|
||||
|
||||
# Collect span-level scores
|
||||
for span in trace.spans:
|
||||
all_scores.extend(span.feedback_scores)
|
||||
|
||||
# Recursively collect from nested spans
|
||||
for nested_span in span.spans:
|
||||
all_scores.extend(nested_span.feedback_scores)
|
||||
|
||||
return all_scores
|
||||
|
||||
Integration with Evaluation System
|
||||
----------------------------------
|
||||
|
||||
These models are automatically populated and used by the Opik evaluation system:
|
||||
|
||||
1. **Trace Creation**: When you run ``opik.evaluate()``, traces are automatically created
|
||||
2. **Span Population**: Individual function calls become spans within the trace
|
||||
3. **Task Span Evaluation**: Metrics with ``task_span`` parameters receive ``SpanModel`` objects
|
||||
4. **Score Attachment**: Feedback scores are automatically attached to the appropriate traces and spans
|
||||
|
||||
You typically don't need to create these models manually - they're generated automatically during evaluation. However, understanding their structure is essential for writing effective task span evaluation metrics.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
These models are commonly used for:
|
||||
|
||||
- **Custom Evaluation Metrics**: Analyzing detailed execution data in custom metrics
|
||||
- **Performance Analysis**: Understanding execution patterns and performance characteristics
|
||||
- **Debugging**: Investigating issues in complex operations
|
||||
- **Cost Tracking**: Aggregating usage and cost information across operations
|
||||
- **Quality Assessment**: Evaluating the quality of individual steps and overall operations
|
||||
|
||||
Module Reference
|
||||
----------------
|
||||
|
||||
For detailed API documentation, see the following class reference pages:
|
||||
|
||||
- :doc:`TraceModel <../message_processing_emulation/TraceModel>`
|
||||
- :doc:`SpanModel <../message_processing_emulation/SpanModel>`
|
||||
- :doc:`FeedbackScoreModel <../message_processing_emulation/FeedbackScoreModel>`
|
||||
- :doc:`ExperimentItemModel <../message_processing_emulation/ExperimentItemModel>`
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
Local Recording Context Manager
|
||||
===============================
|
||||
|
||||
.. currentmodule:: opik
|
||||
|
||||
`record_traces_locally`
|
||||
-----------------------
|
||||
|
||||
The ``record_traces_locally`` context manager enables local, in-memory recording of any traces and spans created inside its block. This is useful for testing, debugging, or for programmatically inspecting your span/trace trees without sending data to the backend.
|
||||
|
||||
Basic usage
|
||||
~~~~~~~~~~~
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import opik
|
||||
|
||||
with opik.record_traces_locally() as storage:
|
||||
# Your instrumented code that creates traces/spans
|
||||
# e.g., functions decorated with @opik.track, manual opik.Opik().span()/trace(), integrations, etc.
|
||||
...
|
||||
|
||||
# Access in-memory results (automatically flushed before reading)
|
||||
span_models = storage.span_trees
|
||||
trace_models = storage.trace_trees
|
||||
|
||||
What it returns
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
The context yields a lightweight handle having these properties:
|
||||
|
||||
- ``span_trees``: List of :class:`opik.message_processing.emulation.models.SpanModel`
|
||||
- ``trace_trees``: List of :class:`opik.message_processing.emulation.models.TraceModel`
|
||||
|
||||
Each accessor flushes the Opik client to ensure all in-flight messages are processed before reading the local state.
|
||||
|
||||
No nested usage
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
Nested or concurrent usages within the same process are not supported. If a local recording is already active, entering another ``record_traces_locally`` block raises ``RuntimeError``.
|
||||
|
||||
Notes
|
||||
~~~~~
|
||||
|
||||
- Uses the SDK's local emulator to mirror what would be sent to the backend.
|
||||
- Data is kept in memory only for the life of the context. On exit, the local recorder is disabled and state is reset.
|
||||
- Ideal for `task_span` metrics validation, writing tests or ad-hoc scripts that need access to the span/trace tree structure.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user