comet-ml--opik/apps/opik-documentation/documentation/fern/docs/tracing/integrations/litellm.mdx

---
description: Start here to integrate Opik into your LiteLLM-based genai application
  for end-to-end LLM observability, unit testing, and optimization.
headline: LiteLLM | Opik Documentation
og:description: Learn to call LLM APIs with LiteLLM using OpenAI format. Configure
  using the Python SDK or Proxy Server for seamless integration.
og:site_name: Opik Documentation
og:title: LiteLLM Gateways - Opik Integration
title: Observability for LiteLLM with Opik
canonical-url: https://www.comet.com/docs/opik/integrations/litellm
---

[LiteLLM](https://github.com/BerriAI/litellm) allows you to call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]. There are two main ways to use LiteLLM:

1. Using the [LiteLLM Python SDK](https://docs.litellm.ai/docs/#litellm-python-sdk)
2. Using the [LiteLLM Proxy Server (LLM Gateway)](https://docs.litellm.ai/docs/#litellm-proxy-server-llm-gateway)

## Account Setup

[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=litellm&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=litellm&utm_campaign=opik) and grab your API Key.

> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=litellm&utm_campaign=opik) for more information.

## Getting Started

### Installation

First, ensure you have both `opik` and `litellm` packages installed:

```bash
pip install opik litellm
```

### Configuring Opik

Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/v1/tracing/sdk_configuration) for detailed instructions on:

- **CLI configuration**: `opik configure`
- **Code configuration**: `opik.configure()`
- **Self-hosted vs Cloud vs Enterprise** setup
- **Configuration files** and environment variables

### Configuring LiteLLM

In order to use LiteLLM, you will need to configure your LLM provider API keys. For this example, we'll use OpenAI. You can [find or create your API keys in these pages](https://platform.openai.com/settings/organization/api-keys):

You can set them as environment variables:

```bash
export OPENAI_API_KEY="YOUR_API_KEY"
```

Or set them programmatically:

```python
import os
import getpass

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```

## Using Opik with the LiteLLM Python SDK

### Logging LLM calls

In order to log the LLM calls to Opik, you will need to create the OpikLogger callback. Once the OpikLogger callback is created and added to LiteLLM, you can make calls to LiteLLM as you normally would:

```python
from litellm.integrations.opik.opik import OpikLogger
import litellm
import os

# Set project name for better organization
os.environ["OPIK_PROJECT_NAME"] = "litellm-integration-demo"

opik_logger = OpikLogger()
litellm.callbacks = [opik_logger]

response = litellm.completion(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
    ]
)

print(response.choices[0].message.content)
```

<Frame>
  <img src="/img/cookbook/litellm_cookbook.png" />
</Frame>

### Logging LLM calls within a tracked function

If you are using LiteLLM within a function tracked with the [`@track`](/v1/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:

```python
from opik import track
from opik.opik_context import get_current_span_data
from litellm.integrations.opik.opik import OpikLogger
import litellm

opik_logger = OpikLogger()
litellm.callbacks = [opik_logger]

@track
def streaming_function(input):
    messages = [{"role": "user", "content": input}]
    response = litellm.completion(
        model="gpt-3.5-turbo",
        messages=messages,
        metadata = {
            "opik": {
                "current_span_data": get_current_span_data(),
                "tags": ["streaming-test"],
            },
        }
    )
    return response

response = streaming_function("Why is tracking and evaluation of LLMs important?")
chunks = list(response)
```

## Using Opik with the LiteLLM Proxy Server

<Info>
  **Opik Agent Optimizer & LiteLLM**: Beyond tracing, the Opik Agent Optimizer SDK also leverages LiteLLM for comprehensive model support within its optimization algorithms. This allows you to use a wide range of LLMs (including local ones) for prompt optimization tasks.
</Info>

### Configuring the LiteLLM Proxy Server

In order to configure the Opik logging, you will need to update the `litellm_settings` section in the LiteLLM `config.yaml` config file:

```yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
litellm_settings:
  success_callback: ["opik"]
```

You can now start the LiteLLM Proxy Server and all LLM calls will be logged to Opik:

```bash
litellm --config config.yaml
```

### Using the LiteLLM Proxy Server

Each API call made to the LiteLLM Proxy server will now be logged to Opik:

```bash
curl -X POST http://localhost:4000/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H "Content-Type: application/json" -d '{
    "model": "gpt-4o",
    "messages": [
        {
            "role": "user",
            "content": "Hello!"
        }
    ]
}'
```