tracer-cloud--opensre/docs/splunk.mdx

---
title: "Splunk"
description: "Connect Splunk so OpenSRE can search logs using SPL during investigations"
---

OpenSRE queries Splunk using the REST API to surface relevant log evidence during alert investigations — searching indexes with SPL, correlating error patterns with incidents, and identifying root causes.

## Prerequisites

- Splunk Enterprise or Splunk Cloud instance (version 8.x or later)
- REST API access on port 8089
- A bearer token with search capability (see [Generating a Bearer Token](#generating-a-bearer-token))

## Setup

### Option 1: Interactive CLI

```bash
opensre integrations setup
```

Select **Splunk** when prompted and provide your REST API base URL and bearer token.

### Option 2: Environment variables

Add to your `.env`:

```bash
SPLUNK_URL=https://splunk.corp.com:8089    # REST API base URL (port 8089 default)
SPLUNK_TOKEN=your-bearer-token             # API bearer token (NOT an HEC token)
SPLUNK_INDEX=main                          # Default index to search (optional)
SPLUNK_VERIFY_SSL=true                     # Set false to skip SSL verification (optional)
SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem  # Path to custom CA bundle (optional)
```

| Variable | Default | Description |
| --- | --- | --- |
| `SPLUNK_URL` | — | **Required.** REST API base URL including port |
| `SPLUNK_TOKEN` | — | **Required.** Bearer token with search capability |
| `SPLUNK_INDEX` | `main` | Default index searched when no index is specified in the alert |
| `SPLUNK_VERIFY_SSL` | `true` | Set to `false` to disable SSL verification (dev/local only) |
| `SPLUNK_CA_BUNDLE` | — | Path to a PEM CA bundle for enterprise self-signed certificates |

### Option 3: Persistent store

```json
{
  "version": 1,
  "integrations": [
    {
      "id": "splunk-prod",
      "service": "splunk",
      "status": "active",
      "credentials": {
        "base_url": "https://splunk.corp.com:8089",
        "token": "your-bearer-token",
        "index": "main",
        "verify_ssl": true,
        "ca_bundle": "/etc/ssl/certs/corp-ca.pem"
      }
    }
  ]
}
```

### Multi-instance setup

To connect multiple Splunk instances (e.g. separate prod and staging clusters):

```bash
SPLUNK_INSTANCES='[
  {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}},
  {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}}
]'
```

When `SPLUNK_INSTANCES` is set it overrides the single-instance `SPLUNK_URL` / `SPLUNK_TOKEN` variables.

## Generating a bearer token

OpenSRE uses bearer tokens — not basic auth and not HEC tokens. To generate one:

**Via the Splunk UI:**

1. Go to **Settings** → **Tokens**
2. Click **New Token**
3. Set a name (e.g. `opensre`) and an expiry date
4. Copy the generated token

**Via the REST API** (replace `<PASSWORD>` with your admin password):

```bash
curl -sk -u admin:<PASSWORD> \
  https://splunk.corp.com:8089/services/authorization/tokens \
  -X POST \
  --data-urlencode "name=opensre" \
  --data-urlencode "expires_on=+90d" \
  --data-urlencode "output_mode=json" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])"
```

The token needs the `search` capability. The `admin` role includes this by default. For a dedicated service account, ensure the role includes:

- `search`
- `read_splunkd_private_settings` (needed for the verify call against `/services/server/info`)

## Verify

```bash
opensre integrations verify splunk
```

Expected output:

```
Service: splunk
Status: passed
Detail: Connected to Splunk 9.x.x
```

## How queries are generated

OpenSRE builds SPL queries deterministically from the alert payload — the LLM selects which tool to call but never writes the query itself. This keeps investigations reproducible and auditable.

Query construction priority:

| Priority | Source | Example |
| --- | --- | --- |
| 1 | `annotations.splunk_query` — verbatim SPL from your alert | `index=prod "PaymentTimeout" \| head 50` |
| 2 | `annotations.query` or `annotations.log_query` | Any pre-populated query field |
| 3 | `error_message` field — keyword search built automatically | `search index=main "NullPointerException" \| head 50` |
| 4 | `alert_name` — last-resort keyword search | `search index=main "payments-error-spike" \| head 50` |
| 5 | Fallback — index scan | `search index=main \| head 50` |

To pass a specific SPL query through an alert, set `commonAnnotations.splunk_query`:

```json
{
  "alert_name": "Payment service errors",
  "commonAnnotations": {
    "splunk_query": "index=prod sourcetype=app_logs \"NullPointerException\" | head 50"
  }
}
```

## Test with the built-in alert template

```bash
opensre investigate --template splunk
```

This runs a synthetic investigation using a pre-built alert fixture — no live alert infrastructure needed.

## Troubleshooting

| Symptom | Fix |
| --- | --- |
| `SSL: CERTIFICATE_VERIFY_FAILED` | Set `SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem` (preferred) or `SPLUNK_VERIFY_SSL=false` (dev only) |
| `HTTP 401 Unauthorized` | Token expired or was generated with the wrong account — regenerate |
| `HTTP 403 Forbidden` | Token lacks `search` capability — check the role assigned to the token |
| Empty search results | Data may not have been ingested yet, or the index name is wrong |
| `Connection refused` on port 8089 | Splunk management port may be firewalled; confirm network access |
| `opensre integrations verify` fails | Check `SPLUNK_URL` includes the protocol and port (`https://host:8089`) |

## Security best practices

- Use a **read-only bearer token** — never use an admin token in production.
- Store `SPLUNK_TOKEN` in `.env` or the credential store, not in source code or CI logs.
- Prefer a dedicated `opensre` service account with only the `search` capability.
- For enterprise self-signed certificates, set `SPLUNK_CA_BUNDLE` to the CA bundle path rather than disabling verification entirely.
- Set `SPLUNK_VERIFY_SSL=false` only in local or dev environments when you cannot supply a CA bundle.
- Rotate tokens on a schedule and revoke them when no longer needed.