wehub-resource-sync
67 lines
2.4 KiB
Plaintext
67 lines
2.4 KiB
Plaintext
---
|
|
description: Guidelines for creating OpenTelemetry metrics to avoid cardinality issues
|
|
globs:
|
|
- "**/*.ts"
|
|
---
|
|
|
|
# OpenTelemetry Metrics Guidelines
|
|
|
|
When creating or editing OTEL metrics (counters, histograms, gauges), always ensure metric attributes have **low cardinality**.
|
|
|
|
## What is Cardinality?
|
|
|
|
Cardinality refers to the number of unique values an attribute can have. Each unique combination of attribute values creates a new time series, which consumes memory and storage in your metrics backend.
|
|
|
|
## Rules
|
|
|
|
### DO use low-cardinality attributes:
|
|
- **Enums**: `environment_type` (PRODUCTION, STAGING, DEVELOPMENT, PREVIEW)
|
|
- **Booleans**: `hasFailures`, `streaming`, `success`
|
|
- **Bounded error codes**: A finite, controlled set of error types
|
|
- **Shard IDs**: When sharding is bounded (e.g., 0-15)
|
|
|
|
### DO NOT use high-cardinality attributes:
|
|
- **UUIDs/IDs**: `envId`, `userId`, `runId`, `projectId`, `organizationId`
|
|
- **Unbounded integers**: `itemCount`, `batchSize`, `retryCount`
|
|
- **Timestamps**: `createdAt`, `startTime`
|
|
- **Free-form strings**: `errorMessage`, `taskName`, `queueName`
|
|
|
|
## Example
|
|
|
|
```typescript
|
|
// BAD - High cardinality
|
|
this.counter.add(1, {
|
|
envId: options.environmentId, // UUID - unbounded
|
|
itemCount: options.runCount, // Integer - unbounded
|
|
});
|
|
|
|
// GOOD - Low cardinality
|
|
this.counter.add(1, {
|
|
environment_type: options.environmentType, // Enum - 4 values
|
|
streaming: true, // Boolean - 2 values
|
|
});
|
|
```
|
|
|
|
## Prometheus Metric Naming
|
|
|
|
When metrics are exported via OTLP to Prometheus, the exporter automatically adds unit suffixes to metric names:
|
|
|
|
| OTel Metric Name | Unit | Prometheus Name |
|
|
|------------------|------|-----------------|
|
|
| `my_duration_ms` | `ms` | `my_duration_ms_milliseconds` |
|
|
| `my_counter` | counter | `my_counter_total` |
|
|
| `items_inserted` | counter | `items_inserted_inserts_total` |
|
|
| `batch_size` | histogram | `batch_size_items_bucket` |
|
|
|
|
Keep this in mind when writing Grafana dashboards or Prometheus queries—the metric names in Prometheus will differ from the names defined in code.
|
|
|
|
## Reference
|
|
|
|
See the schedule engine (`internal-packages/schedule-engine/src/engine/index.ts`) for a good example of low-cardinality metric attributes.
|
|
|
|
High cardinality metrics can cause:
|
|
- Memory bloat in metrics backends (Axiom, Prometheus, etc.)
|
|
- Slow queries and dashboard timeouts
|
|
- Increased costs (many backends charge per time series)
|
|
- Potential data loss or crashes at scale
|