tracer-cloud--opensre/docs/pagerduty.mdx

---
title: "PagerDuty"
description: "Connect PagerDuty so OpenSRE can read incidents, on-call schedules, and service topology during investigations"
---

OpenSRE queries PagerDuty to retrieve active incidents, their timelines, on-call responders, and service/escalation-policy configuration — correlating incident management data with infrastructure events during root cause analysis.

## Prerequisites

- PagerDuty account
- REST API key with read access

## Setup

### Option 1: Interactive CLI

```bash
opensre integrations setup
```

Select **PagerDuty** when prompted and provide your API key.

### Option 2: Environment variables

Add to your `.env`:

```bash
PAGERDUTY_API_KEY=your-api-key
```

| Variable | Default | Description |
| --- | --- | --- |
| `PAGERDUTY_API_KEY` | — | **Required.** PagerDuty REST API v2 key |
| `PAGERDUTY_BASE_URL` | `https://api.pagerduty.com` | API base URL (override only for EU or custom instances) |

### Option 3: Persistent store

```json
{
  "version": 1,
  "integrations": [
    {
      "id": "pagerduty-prod",
      "service": "pagerduty",
      "status": "active",
      "credentials": {
        "api_key": "your-api-key"
      }
    }
  ]
}
```

## Creating an API key

1. In PagerDuty, go to **Integrations** → **Developer Tools** → **API Access Keys**
2. Click **Create New API Key**
3. Add a description (e.g. `opensre read-only`)
4. Copy the key immediately — it won't be shown again

<Info>
PagerDuty API keys are account-level. Use a read-only key to follow the principle of least privilege.
</Info>

## Verify

```bash
opensre integrations verify pagerduty
```

Expected output:

```
Service: pagerduty
Status: passed
Detail: Connected to PagerDuty; API key accepted
```

## Available tools

| Tool | Description |
| --- | --- |
| `pagerduty_incidents` | List and search incidents (filter by status, urgency, service, time range) |
| `pagerduty_incident_detail` | Fetch full incident details and activity timeline (log entries) |
| `pagerduty_oncall` | Fetch current on-call responders by escalation policy |
| `pagerduty_services` | List services with escalation policies, integrations, and alert routing rules |

## Example investigation context

During an investigation, OpenSRE may use PagerDuty tools to:

- Find which incidents were triggered around the alert time window
- Identify who was on-call and how quickly incidents were acknowledged
- Map which services and escalation policies were involved
- Correlate PagerDuty incident timelines with Grafana/Datadog metrics

## Troubleshooting

| Symptom | Fix |
| --- | --- |
| **401 Unauthorized** | Check the API key — ensure it's a valid REST API v2 key |
| **403 Forbidden** | The key may lack required permissions; recreate with full read access |
| **429 Rate Limited** | PagerDuty enforces rate limits (960 requests/min); reduce query frequency |
| **No incidents returned** | Check time range filters — default returns recent incidents only |

## Security best practices

- Use a **read-only API key** — OpenSRE never needs to modify PagerDuty resources.
- Store the API key in `.env`, not in source code.
- Rotate API keys periodically via the PagerDuty Developer Tools page.