chore: import upstream snapshot with attribution
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / TypeScript SDK Compatibility V1.x E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
TypeScript SDK E2E Tests / TypeScript SDK E2E Tests Node ${{matrix.node_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
Python SDK E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK E2E Tests / Python SDK E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK E2E Tests / build-opik (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Select Python version matrix (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Python SDK Compatibility V1.x E2E Tests / build-opik (push) Has been cancelled
TypeScript SDK E2E Tests / Select Node version matrix (push) Has been cancelled
TypeScript SDK E2E Tests / build-opik (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer E2E Tests Python ${{matrix.python_version}} (push) Has been cancelled
Opik Optimizer - E2E Tests / Opik Optimizer Integration Smoke Tests (push) Has been cancelled
🐙 Code Quality / detect (push) Has been cancelled
🐙 Code Quality / lint (${{ matrix.leg.name }}) (push) Has been cancelled
🐙 Code Quality / summary (push) Has been cancelled
TypeScript SDK Library Integration Tests / Check Secrets (push) Has been cancelled
TypeScript SDK Library Integration Tests / opik-vercel (Vercel AI SDK / eve) (push) Has been cancelled
SDK Library Integration Tests Runner / Check Secrets (push) Has been cancelled
SDK Library Integration Tests Runner / Missed OpenAI API Key Warning (push) Has been cancelled
SDK Library Integration Tests Runner / Build (push) Has been cancelled
SDK Library Integration Tests Runner / openai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_tests (push) Has been cancelled
SDK Library Integration Tests Runner / langchain_legacy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / llama_index_tests (push) Has been cancelled
SDK Library Integration Tests Runner / anthropic_tests (push) Has been cancelled
SDK Library Integration Tests Runner / mistral_tests (push) Has been cancelled
SDK Library Integration Tests Runner / groq_tests (push) Has been cancelled
SDK Library Integration Tests Runner / aisuite_tests (push) Has been cancelled
SDK Library Integration Tests Runner / haystack_tests (push) Has been cancelled
SDK Library Integration Tests Runner / dspy_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / crewai_v1_tests (push) Has been cancelled
SDK Library Integration Tests Runner / genai_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_tests (push) Has been cancelled
SDK Library Integration Tests Runner / adk_legacy_1_3_0_tests (push) Has been cancelled
SDK Library Integration Tests Runner / evaluation_metrics_tests (push) Has been cancelled
SDK Library Integration Tests Runner / bedrock_tests (push) Has been cancelled
SDK Library Integration Tests Runner / litellm_tests (push) Has been cancelled
SDK Library Integration Tests Runner / harbor_tests (push) Has been cancelled
SDK Library Integration Tests Runner / Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / render-equality (push) Has been cancelled
Opik Optimizer - Unit Tests / Opik Optimizer Unit Tests Python ${{matrix.python_version}} (push) Has been cancelled
Python BE E2E Tests / Python BE E2E (push) Has been cancelled
Python Backend Tests / run-python-backend-tests (push) Has been cancelled
Python SDK Unit Tests / Python SDK Unit Tests ${{matrix.python_version}} (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
SDK E2E Libraries Integration Tests / Check Secrets (push) Has been cancelled
SDK E2E Libraries Integration Tests / Missed OpenAI API Key Warning (push) Has been cancelled
SDK E2E Libraries Integration Tests / build-opik (push) Has been cancelled
SDK E2E Libraries Integration Tests / E2E Lib Integration Python ${{matrix.python_version}} (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-gemini) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-langchain) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-openai) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-otel) (push) Has been cancelled
TypeScript SDK Integration Build & Publish / build-and-publish (opik-vercel) (push) Has been cancelled
TypeScript SDK Build & Publish / build-and-publish (push) Has been cancelled
TypeScript SDK Unit Tests / Test on Node ${{ matrix.node-version }} (push) Has been cancelled
Backend Tests / discover-tests (push) Has been cancelled
Backend Tests / ${{ matrix.name }} (push) Has been cancelled
Build and Publish SDK / build-and-publish (push) Has been cancelled
Build Opik Docker Images / set-version (push) Has been cancelled
Build Opik Docker Images / build-backend (push) Has been cancelled
Build Opik Docker Images / build-sandbox-executor-python (push) Has been cancelled
Build Opik Docker Images / build-python-backend (push) Has been cancelled
Build Opik Docker Images / build-frontend (push) Has been cancelled
Build Opik Docker Images / create-git-tag (push) Has been cancelled
ClickHouse Migration Cluster Check / validate-clickhouse-migrations (push) Has been cancelled
Docs - Publish / run (push) Has been cancelled
E2E Tests - Post Merge (v2) / 🧪 E2E v2 Tests (${{ github.event.inputs.tier || 't1' }}) (push) Has been cancelled
E2E Tests - Post Merge (v2) / 📢 Slack Notification (push) Has been cancelled
Frontend Unit Tests / Test on Node 20 (push) Has been cancelled
Guardrails E2E Tests / Select Python version matrix (push) Has been cancelled
Guardrails E2E Tests / Guardrails E2E Tests ${{matrix.python_version}} (push) Has been cancelled
Guardrails E2E Tests / 📢 Slack Notification (push) Has been cancelled
Guardrails Backend Unit Tests / Guardrails Backend Unit Tests (push) Has been cancelled
Guardrails Backend Unit Tests / 📢 Slack Notification (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v3.21.0) (push) Has been cancelled
Lint Opik Helm Chart / lint-helm-chart (Helm v4.2.0) (push) Has been cancelled
Lint Opik Helm Chart / unittest-helm-chart (push) Has been cancelled
This commit is contained in:
+254
@@ -0,0 +1,254 @@
|
||||
TraceModel
|
||||
==========
|
||||
|
||||
.. currentmodule:: opik.message_processing.emulation.models
|
||||
|
||||
.. autoclass:: TraceModel
|
||||
:special-members: __init__
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
The ``TraceModel`` class represents a trace model that encapsulates data about a trace, its related metadata, and associated spans. It is used for tracking and analyzing data during execution or processing tasks.
|
||||
|
||||
This class provides a structure to represent trace information, including the start and end times, associated project details, input/output data, feedback scores, error information, and thread association. It is designed to handle optional fields for flexible use across various scenarios.
|
||||
|
||||
A trace represents the complete execution path of a request or operation, containing one or more spans that represent individual steps or components within that execution.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
Required Attributes
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: id
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Unique identifier for the trace.
|
||||
|
||||
.. attribute:: start_time
|
||||
:type: datetime.datetime
|
||||
:noindex:
|
||||
|
||||
Timestamp representing the start of the trace.
|
||||
|
||||
.. attribute:: name
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
|
||||
Optional name for the trace, which can provide a descriptive label for the operation being traced.
|
||||
|
||||
.. attribute:: project_name
|
||||
:type: str
|
||||
:noindex:
|
||||
|
||||
Name of the project associated with the trace.
|
||||
|
||||
Optional Attributes
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: input
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional dictionary containing the input data associated with the trace. This represents the initial parameters or data that started the trace.
|
||||
|
||||
.. attribute:: output
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional dictionary containing the output data generated by the trace. This represents the final results or return values.
|
||||
|
||||
.. attribute:: tags
|
||||
:type: Optional[List[str]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional list of tags associated with the trace for classification or filtering purposes.
|
||||
|
||||
.. attribute:: metadata
|
||||
:type: Optional[Dict[str, Any]]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional metadata providing additional information about the trace.
|
||||
|
||||
.. attribute:: end_time
|
||||
:type: Optional[datetime.datetime]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Timestamp representing the end of the trace. When set, this marks when the operation completed.
|
||||
|
||||
.. attribute:: error_info
|
||||
:type: Optional[ErrorInfoDict]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional dictionary containing information about errors encountered during the trace.
|
||||
|
||||
.. attribute:: thread_id
|
||||
:type: Optional[str]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Optional identifier of the thread associated with the trace. Useful for concurrent operations.
|
||||
|
||||
.. attribute:: last_updated_at
|
||||
:type: Optional[datetime.datetime]
|
||||
:noindex:
|
||||
:value: None
|
||||
|
||||
Timestamp for when the trace was last updated.
|
||||
|
||||
Collection Attributes
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. attribute:: spans
|
||||
:type: List[SpanModel]
|
||||
:noindex:
|
||||
|
||||
List of spans associated with the trace, representing individual processing parts or segments within the trace. Each span represents a specific operation or step in the overall execution.
|
||||
|
||||
.. attribute:: feedback_scores
|
||||
:type: List[FeedbackScoreModel]
|
||||
:noindex:
|
||||
|
||||
List of feedback scores associated with the trace. These are used for overall trace evaluation and quality assessment.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Creating a basic trace:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import datetime
|
||||
from opik.message_processing.emulation.models import TraceModel
|
||||
|
||||
# Create a simple trace
|
||||
trace = TraceModel(
|
||||
id="trace_123",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="user_query_processing",
|
||||
project_name="my_project",
|
||||
input={"user_query": "What is machine learning?"},
|
||||
output={"response": "Machine learning is a subset of AI..."}
|
||||
)
|
||||
|
||||
Creating a trace with spans:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import SpanModel
|
||||
|
||||
# Create a trace with associated spans
|
||||
trace = TraceModel(
|
||||
id="trace_456",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="complex_operation",
|
||||
project_name="ai_project"
|
||||
)
|
||||
|
||||
# Add spans to represent different steps
|
||||
preprocessing_span = SpanModel(
|
||||
id="span_1",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="data_preprocessing"
|
||||
)
|
||||
|
||||
llm_span = SpanModel(
|
||||
id="span_2",
|
||||
start_time=datetime.datetime.now(),
|
||||
name="llm_call",
|
||||
type="llm"
|
||||
)
|
||||
|
||||
trace.spans.extend([preprocessing_span, llm_span])
|
||||
|
||||
Adding feedback scores to a trace:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from opik.message_processing.emulation.models import FeedbackScoreModel
|
||||
|
||||
# Add overall evaluation scores to the trace
|
||||
overall_quality = FeedbackScoreModel(
|
||||
id="score_123",
|
||||
name="overall_quality",
|
||||
value=0.88,
|
||||
reason="Good response quality with minor improvements needed"
|
||||
)
|
||||
|
||||
trace.feedback_scores.append(overall_quality)
|
||||
|
||||
Working with trace hierarchies:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Access nested spans within a trace
|
||||
for span in trace.spans:
|
||||
print(f"Span: {span.name}")
|
||||
|
||||
# Each span can have nested spans too
|
||||
for nested_span in span.spans:
|
||||
print(f" Nested: {nested_span.name}")
|
||||
|
||||
Usage in Evaluation Context
|
||||
---------------------------
|
||||
|
||||
``TraceModel`` objects are commonly used in evaluation scenarios where you need to analyze the complete execution:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Example of accessing trace data in evaluation
|
||||
def analyze_trace(trace: TraceModel):
|
||||
# Analyze overall trace performance
|
||||
duration = trace.end_time - trace.start_time if trace.end_time else None
|
||||
|
||||
# Count different types of spans
|
||||
llm_spans = [s for s in trace.spans if s.type == "llm"]
|
||||
|
||||
# Analyze input/output
|
||||
input_complexity = len(str(trace.input)) if trace.input else 0
|
||||
output_quality = evaluate_output_quality(trace.output)
|
||||
|
||||
return {
|
||||
"duration": duration,
|
||||
"llm_calls": len(llm_spans),
|
||||
"complexity": input_complexity,
|
||||
"quality": output_quality
|
||||
}
|
||||
|
||||
Common Use Cases
|
||||
----------------
|
||||
|
||||
``TraceModel`` is commonly used for:
|
||||
|
||||
- **Request Tracking**: Tracking complete user requests from start to finish
|
||||
- **Performance Analysis**: Analyzing the performance of complex operations
|
||||
- **Evaluation**: Providing complete context for evaluation metrics
|
||||
- **Debugging**: Understanding the full execution path and identifying issues
|
||||
- **Cost Tracking**: Aggregating costs across all spans in a trace
|
||||
- **A/B Testing**: Comparing different execution paths and their outcomes
|
||||
|
||||
Relationship with Spans
|
||||
-----------------------
|
||||
|
||||
A trace acts as a container for spans, creating a hierarchical structure:
|
||||
|
||||
- **Trace**: The top-level container representing the complete operation
|
||||
- **Spans**: Individual steps or operations within the trace
|
||||
- **Nested Spans**: Spans can contain other spans, creating a tree structure
|
||||
|
||||
This hierarchy allows for detailed tracking of complex operations while maintaining the overall context.
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
- :class:`SpanModel` - Individual operations within a trace
|
||||
- :class:`FeedbackScoreModel` - For attaching evaluation scores to traces
|
||||
- :doc:`../evaluation/evaluate` - For information about evaluating traces with custom metrics
|
||||
Reference in New Issue
Block a user