tracer-cloud--opensre/docs/tempo.mdx

---
title: "Grafana Tempo"
description: "Connect Grafana Tempo so OpenSRE can query distributed traces during investigations"
---

OpenSRE uses Grafana Tempo to investigate trace-related alerts — searching spans by service, fetching full traces by ID, listing instrumented services, and filtering by error status or latency.

This integration talks to Tempo **directly** via its HTTP API. It does not require a Grafana instance or datasource proxy. If you run the full Grafana stack, the [Grafana](/grafana) integration already surfaces Tempo through the datasource proxy — use this integration when you run Tempo standalone.

## Prerequisites

- Grafana Tempo 1.4+
- Network access from the OpenSRE environment to your Tempo instance
- Auth credentials only if your deployment requires them (many run without auth behind a gateway)

## Setup

### Option 1: Interactive CLI

```bash
opensre integrations setup tempo
```

You will be prompted for the Tempo URL and optional auth. Leave auth fields blank if your Tempo runs without authentication.

### Option 2: Environment variables

Add to your `.env`:

```bash
TEMPO_URL=http://localhost:3200
TEMPO_API_KEY=<bearer-token>       # optional
TEMPO_USERNAME=<username>          # optional (basic auth)
TEMPO_PASSWORD=<token>             # optional (basic auth)
TEMPO_ORG_ID=<tenant-id>           # optional (X-Scope-OrgID for multi-tenant)
```

| Variable | Default | Description |
| --- | --- | --- |
| `TEMPO_URL` | — | **Required.** Tempo HTTP API base URL |
| `TEMPO_API_KEY` | _(empty)_ | Bearer token for auth |
| `TEMPO_USERNAME` | _(empty)_ | Username for basic auth |
| `TEMPO_PASSWORD` | _(empty)_ | Password for basic auth |
| `TEMPO_ORG_ID` | _(empty)_ | Tenant ID sent as `X-Scope-OrgID` for multi-tenant deployments |

### Option 3: Persistent store

Integrations are automatically persisted to `~/.opensre/integrations.json`:

```json
{
  "version": 1,
  "integrations": [
    {
      "id": "tempo-prod",
      "service": "tempo",
      "status": "active",
      "credentials": {
        "url": "http://localhost:3200",
        "api_key": ""
      }
    }
  ]
}
```

## Investigation tools

OpenSRE exposes a single `query_tempo` tool with an `action` parameter:

### search

Searches traces by service, span name, duration, and tags using TraceQL. Returns one summary row per trace.

```text
query_tempo(action="search", service="checkout-service", min_duration_ms=500)
query_tempo(action="search", tags={"http.status_code": "500"})
```

| Parameter | Description |
| --- | --- |
| `service` | Filter by `resource.service.name` |
| `span_name` | Filter by span name |
| `min_duration_ms` | Minimum trace duration |
| `max_duration_ms` | Maximum trace duration |
| `tags` | Key/value span attributes |
| `time_range_minutes` | Lookback window (default 60) |
| `limit` | Max traces to return (default 20) |

### get_trace

Fetches a full trace by ID and flattens its spans.

```text
query_tempo(action="get_trace", trace_id="4f8c...e21")
```

### list_services

Lists all services registered in Tempo.

```text
query_tempo(action="list_services")
```

### list_span_names

Lists all span names registered in Tempo.

```text
query_tempo(action="list_span_names")
```

## Verify

```bash
opensre integrations verify tempo
```

Expected output:

```
Service: tempo
Status:  passed
Detail:  Connected to Grafana Tempo HTTP API (/api/search, /api/traces).
```

## Troubleshooting

| Symptom | Fix |
| --- | --- |
| **Connection refused** | Verify the URL and port. Default Tempo HTTP port is `3200`. |
| **HTTP 401** | Set `TEMPO_API_KEY` or `TEMPO_USERNAME`/`TEMPO_PASSWORD` if your deployment requires auth. |
| **HTTP 404 on verify** | Check your Tempo version — the `/api/search/tags` endpoint requires Tempo 1.4+. |
| **No traces returned** | Confirm your services are sending traces to Tempo and the time range covers the period of interest. |
| **Multi-tenant 400** | Set `TEMPO_ORG_ID` to the correct tenant ID. |

## Security best practices

- Use a **read-only** token or service account if your Tempo deployment supports auth.
- Store credentials in `.env`, never in code.
- Restrict network access to Tempo — OpenSRE only needs the HTTP API port (`3200` by default).