# Observability: Monitoring, Logging & Tracing

## The Three Pillars of Observability

### 1. Metrics (What is happening?)
- **Definition**: Numeric measurements over time
- **Examples**: CPU usage, request rate, error rate, latency
- **Tools**: Prometheus, Datadog, CloudWatch, New Relic

### 2. Logs (Why is it happening?)
- **Definition**: Timestamped event records
- **Examples**: Application logs, access logs, error logs
- **Tools**: ELK Stack, Splunk, CloudWatch Logs, Loki

### 3. Traces (Where is it happening?)
- **Definition**: Request journey through distributed system
- **Examples**: Service call chains, database queries, external API calls
- **Tools**: Jaeger, Zipkin, AWS X-Ray, Datadog APM

## SLI/SLO/SLA Framework

### Service Level Indicators (SLIs)
**Quantitative measurements of service quality**

```yaml
# Common SLIs
availability:
  definition: "Percentage of successful requests"
  measurement: "(successful_requests / total_requests) * 100"

latency:
  definition: "Time to process request"
  measurement: "p95 response time < 200ms"

error_rate:
  definition: "Percentage of failed requests"
  measurement: "(failed_requests / total_requests) * 100"

throughput:
  definition: "Requests processed per second"
  measurement: "requests_per_second"
```

### Service Level Objectives (SLOs)
**Target values for SLIs**

```yaml
# Example SLOs
availability_slo:
  target: 99.9%
  measurement_window: 30 days
  error_budget: 0.1% (43 minutes per month)

latency_slo:
  target: "95% of requests < 200ms"
  measurement_window: 7 days

error_rate_slo:
  target: "< 0.1%"
  measurement_window: 24 hours
```

### Service Level Agreements (SLAs)
**Business contracts with consequences**

```yaml
# Example SLA
web_application_sla:
  availability: 99.9%
  latency_p95: 300ms
  consequences:
    - availability < 99.9%: 10% service credit
    - availability < 99.0%: 25% service credit
    - availability < 95.0%: 50% service credit
```

## Prometheus Setup

### Prometheus Configuration
```yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    environment: 'prod'

# Alert manager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load rules
rule_files:
  - "/etc/prometheus/rules/*.yml"

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Kubernetes pods
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

  # Node exporter
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
```

### Alert Rules
```yaml
# alert-rules.yml
groups:
  - name: application_alerts
    interval: 30s
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.job }}"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"

      # High latency
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value | humanizeDuration }} for {{ $labels.job }}"

      # Low availability
      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
          team: sre
        annotations:
          summary: "Service is down"
          description: "{{ $labels.job }} has been down for more than 2 minutes"

  - name: kubernetes_alerts
    interval: 30s
    rules:
      # Pod crash looping
      - alert: PodCrashLooping
        expr: |
          rate(kube_pod_container_status_restarts_total[15m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Pod crash looping"
          description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

      # High memory usage
      - alert: HighMemoryUsage
        expr: |
          (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Container {{ $labels.container }} in pod {{ $labels.pod }} is using {{ $value | humanizePercentage }} of memory"

      # Node disk space
      - alert: NodeDiskSpaceLow
        expr: |
          (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node disk space low"
          description: "Node {{ $labels.node }} has less than 10% disk space available"
```

## Structured Logging

### Best Practices
```json
{
  "timestamp": "2025-10-17T10:30:45.123Z",
  "level": "ERROR",
  "service": "api-gateway",
  "version": "v1.2.3",
  "trace_id": "abc123def456",
  "span_id": "789ghi012jkl",
  "user_id": "user-12345",
  "request_id": "req-67890",
  "method": "POST",
  "path": "/api/v1/orders",
  "status_code": 500,
  "duration_ms": 245,
  "error": {
    "type": "DatabaseConnectionError",
    "message": "Failed to connect to database",
    "stack_trace": "..."
  },
  "context": {
    "order_id": "order-98765",
    "customer_id": "cust-54321"
  }
}
```

### Logging Configuration (Node.js Example)
```javascript
const winston = require('winston');

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: {
    service: process.env.SERVICE_NAME,
    version: process.env.SERVICE_VERSION,
    environment: process.env.ENVIRONMENT
  },
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({
      filename: 'error.log',
      level: 'error'
    }),
    new winston.transports.File({
      filename: 'combined.log'
    })
  ]
});

// Usage with correlation ID
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] || uuidv4();
  req.logger = logger.child({
    request_id: req.id,
    trace_id: req.headers['x-trace-id']
  });
  next();
});

app.post('/api/orders', async (req, res) => {
  req.logger.info('Creating order', {
    customer_id: req.body.customer_id
  });

  try {
    const order = await createOrder(req.body);
    req.logger.info('Order created successfully', {
      order_id: order.id
    });
    res.json(order);
  } catch (error) {
    req.logger.error('Failed to create order', {
      error: error.message,
      stack: error.stack
    });
    res.status(500).json({ error: 'Internal server error' });
  }
});
```

## Distributed Tracing

### OpenTelemetry Configuration
```yaml
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

  memory_limiter:
    check_interval: 1s
    limit_mib: 512

  resource:
    attributes:
      - key: environment
        value: production
        action: insert

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889

  logging:
    loglevel: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [jaeger, logging]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [prometheus, logging]
```

### Application Instrumentation (Python Example)
```python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

# Set up tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter
otlp_exporter = OTLPSpanExporter(
    endpoint="otel-collector:4317",
    insecure=True
)

# Add span processor
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Instrument Flask and requests library
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

# Manual span creation
@app.route('/api/order/<order_id>')
def get_order(order_id):
    with tracer.start_as_current_span("get_order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("user.id", request.headers.get('X-User-ID'))

        # Add events
        span.add_event("Fetching order from database")
        order = fetch_order_from_db(order_id)

        if not order:
            span.set_status(Status(StatusCode.ERROR, "Order not found"))
            return {"error": "Order not found"}, 404

        span.add_event("Order retrieved successfully")
        return order
```

## Dashboards & Visualization

### Grafana Dashboard JSON (Example)
```json
{
  "dashboard": {
    "title": "Application Performance",
    "panels": [
      {
        "title": "Request Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{status}}"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m])",
            "legendFormat": "Error Rate"
          }
        ],
        "type": "graph"
      },
      {
        "title": "P95 Latency",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "P95 Latency"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Active Connections",
        "targets": [
          {
            "expr": "sum(up{job=\"myapp\"})",
            "legendFormat": "Active Instances"
          }
        ],
        "type": "stat"
      }
    ]
  }
}
```

## On-Call & Incident Response

### Runbook Template
```markdown
# Runbook: High Error Rate Alert

## Alert Details
- **Alert Name**: HighErrorRate
- **Severity**: Critical
- **Team**: Backend Engineering
- **On-Call**: See PagerDuty schedule

## Symptoms
- Error rate exceeds 5% for 5 minutes
- Users experiencing 5xx errors
- Elevated p95 latency

## Investigation Steps

1. **Check service health**
   ```bash
   kubectl get pods -n production -l app=myapp
   kubectl logs -n production -l app=myapp --tail=100
   ```

2. **Review error logs**
   - Check Grafana dashboard
   - Review application logs in Kibana
   - Check CloudWatch metrics

3. **Identify error patterns**
   - What endpoints are failing?
   - Are errors consistent across all pods?
   - Is there a pattern in timing?

4. **Check dependencies**
   - Database connectivity
   - External API availability
   - Redis/cache status

## Common Causes & Solutions

### Database Connection Issues
- **Symptoms**: Connection timeout errors
- **Solution**:
  ```bash
  # Check database connectivity
  kubectl exec -it <pod-name> -- nc -zv database-host 5432

  # Check connection pool
  kubectl logs <pod-name> | grep "connection pool"
  ```

### Memory Leaks
- **Symptoms**: Increasing memory usage, OOM kills
- **Solution**: Restart affected pods, investigate memory usage

### Deployment Issues
- **Symptoms**: Errors started after deployment
- **Solution**: Rollback deployment
  ```bash
  kubectl rollout undo deployment/myapp -n production
  ```

## Escalation
- If unresolved after 15 minutes, escalate to Senior Engineer
- If service degradation > 30 minutes, notify VP Engineering

## Post-Incident
- Create incident report
- Schedule post-mortem
- Update runbook with findings
```

## Observability Best Practices

1. **Use consistent naming**: Follow naming conventions for metrics, logs, traces
2. **Add context**: Include correlation IDs in logs and traces
3. **Set meaningful alerts**: Avoid alert fatigue with actionable alerts
4. **Define SLOs**: Measure what matters to users
5. **Practice incident response**: Regular game days and fire drills
6. **Automate runbooks**: Convert manual steps to automated remediation
7. **Monitor the monitors**: Ensure observability stack is reliable
8. **Continuous improvement**: Review and refine based on incidents

---

## Tools Comparison

| Feature | Prometheus | Datadog | New Relic | CloudWatch |
|---------|-----------|---------|-----------|------------|
| Metrics | ✓✓✓ | ✓✓✓ | ✓✓✓ | ✓✓ |
| Logs | via Loki | ✓✓✓ | ✓✓✓ | ✓✓✓ |
| Traces | via Tempo | ✓✓✓ | ✓✓✓ | ✓✓ |
| Cost | Free (self-hosted) | $$$ | $$$ | $$ |
| Learning Curve | Medium | Low | Low | Low |
| Kubernetes Native | ✓✓✓ | ✓✓ | ✓✓ | ✓ |