tracer-cloud--opensre/docs/opensearch.mdx

---
title: "OpenSearch / Elasticsearch"
description: "Connect OpenSearch or Elasticsearch so OpenSRE can search application logs and analytics indices during investigations"
---

OpenSRE queries OpenSearch (or Elasticsearch) to retrieve application logs, error events, and analytics records — pulling concrete log lines into the investigation alongside metrics and traces from your other observability tools.

## Prerequisites

- OpenSearch 1.x/2.x or Elasticsearch 7.x/8.x reachable from the machine running OpenSRE
- The cluster URL (e.g. `https://my-cluster.us-east-1.es.amazonaws.com`)
- Credentials for one of:
  - HTTP Basic Auth (username and password) — typical for self-hosted OpenSearch
  - API key — typical for Elastic Cloud
  - No auth — only when the cluster has the security plugin disabled

OpenSearch authenticates clients via Basic Auth by default; the security plugin does not natively issue API keys ([opensearch-project/security#4009](https://github.com/opensearch-project/security/issues/4009)). Most self-hosted clusters use Basic Auth.

## Setup

### Option 1: Interactive onboarding wizard

```bash
opensre onboard
```

Pick **OpenSearch / Elasticsearch** from the integration menu. The wizard asks for:

1. **OpenSearch URL** — the base URL of your cluster
2. **Authentication method** — choose one of:
   - **Username + Password** (default) — for self-hosted OpenSearch with the security plugin enabled
   - **API key** — for Elastic Cloud or clusters with API-key auth configured
   - **None (security disabled)** — for clusters running with the security plugin disabled

The wizard validates the configuration by calling `GET /_cluster/health` before saving, so you'll see immediate feedback if the URL or credentials are wrong.

### Option 2: Legacy CLI

```bash
opensre integrations setup opensearch
```

Same prompts as the wizard, in a smaller standalone form.

### Option 3: Environment variables

Add to your `.env`:

```bash
OPENSEARCH_URL=https://my-cluster.example.com

# Basic Auth (typical for self-hosted OpenSearch)
OPENSEARCH_USERNAME=admin
OPENSEARCH_PASSWORD=secret

# API key (typical for Elastic Cloud — use instead of username/password)
OPENSEARCH_API_KEY=your-api-key
```

| Variable | Default | Description |
| --- | --- | --- |
| `OPENSEARCH_URL` | — | **Required.** Base URL of your OpenSearch / Elasticsearch cluster |
| `OPENSEARCH_USERNAME` | — | Basic auth username |
| `OPENSEARCH_PASSWORD` | — | Basic auth password |
| `OPENSEARCH_API_KEY` | — | API key (used in `Authorization: ApiKey ...` header) |

When both `OPENSEARCH_API_KEY` and Basic Auth credentials are set, the API key takes precedence. If only one of `OPENSEARCH_USERNAME` / `OPENSEARCH_PASSWORD` is set, no `Authorization` header is emitted (the cluster will reject the request, surfacing the misconfiguration).

### Option 4: Persistent store

```json
{
  "version": 1,
  "integrations": [
    {
      "id": "opensearch-prod",
      "service": "opensearch",
      "status": "active",
      "credentials": {
        "url": "https://my-cluster.example.com",
        "username": "admin",
        "password": "secret",
        "api_key": "",
        "index_pattern": "logs-*"
      }
    }
  ]
}
```

## Verify

```bash
opensre integrations verify opensearch
```

Expected output on success:

```
Service: opensearch
Status:  passed
Detail:  Connected to OpenSearch cluster 'my-cluster' (green, 3 node(s)).
```


## How it works in investigations

When an OpenSearch integration is configured, OpenSRE automatically includes it as an evidence source during every investigation. Two tools become available to the investigation agent:

### `query_elasticsearch_logs`

Searches log indices for messages matching a Lucene/KQL query within a bounded time window. The agent uses this to:

- Pull error and exception messages around the alert timestamp
- Filter logs by service, container, or correlation ID extracted from alert annotations
- Surface stack traces and panic messages that explain a metric anomaly
- Cross-reference logs against alerts firing in Datadog, Grafana, or Alertmanager

The tool returns both the raw matching logs and a separate `error_logs` slice pre-filtered for keywords like `error`, `exception`, `traceback`, and `panic`, so the agent can prioritize signal over noise.

### `query_opensearch_analytics`

Runs bounded analytics queries against any index pattern (default `*`). The agent uses this for non-log data stored in OpenSearch — APM events, audit trails, business metrics, or any custom analytics index your team maintains.

Both tools share the same configured credentials, so configuring the integration once enables both query paths.

## Troubleshooting

| Symptom | Fix |
| --- | --- |
| **Status: missing** | Set `OPENSEARCH_URL` or run `opensre onboard` and pick OpenSearch |
| **Connection refused** | Verify the URL is reachable from this host; check firewall and VPC rules |
| **401 Unauthorized** | Verify credentials are correct. For self-hosted OpenSearch, use Basic Auth (`OPENSEARCH_USERNAME` + `OPENSEARCH_PASSWORD`), not API key — most installs do not have API keys enabled |
| **SSL certificate errors** | For self-signed certificates, ensure the CA cert is in the system trust store |
| **Empty results from a known-good query** | Confirm the index pattern matches your data; the default `*` matches every index but specific patterns like `logs-*` are faster and more reliable |
| **`security_exception: no permissions`** | The user needs at least `read` and `view_index_metadata` permissions on the queried indices |

## Security best practices

- Create a **read-only** OpenSearch user for OpenSRE — the agent only reads logs and analytics indices and never writes to the cluster during investigations.
- Limit the role to the indices OpenSRE needs (typically logs and APM indices), not cluster-wide.
- Store credentials in `.env` or the integration store, not in source code.
- For Elastic Cloud, generate a scoped API key and rotate it on a schedule rather than using the master deployment credentials.
- For self-hosted clusters behind a reverse proxy, prefer Basic Auth over disabling security entirely — even on internal networks.