wehub-resource-sync
14 KiB
14 KiB
Observability: Monitoring, Logging & Tracing
The Three Pillars of Observability
1. Metrics (What is happening?)
- Definition: Numeric measurements over time
- Examples: CPU usage, request rate, error rate, latency
- Tools: Prometheus, Datadog, CloudWatch, New Relic
2. Logs (Why is it happening?)
- Definition: Timestamped event records
- Examples: Application logs, access logs, error logs
- Tools: ELK Stack, Splunk, CloudWatch Logs, Loki
3. Traces (Where is it happening?)
- Definition: Request journey through distributed system
- Examples: Service call chains, database queries, external API calls
- Tools: Jaeger, Zipkin, AWS X-Ray, Datadog APM
SLI/SLO/SLA Framework
Service Level Indicators (SLIs)
Quantitative measurements of service quality
# Common SLIs
availability:
definition: "Percentage of successful requests"
measurement: "(successful_requests / total_requests) * 100"
latency:
definition: "Time to process request"
measurement: "p95 response time < 200ms"
error_rate:
definition: "Percentage of failed requests"
measurement: "(failed_requests / total_requests) * 100"
throughput:
definition: "Requests processed per second"
measurement: "requests_per_second"
Service Level Objectives (SLOs)
Target values for SLIs
# Example SLOs
availability_slo:
target: 99.9%
measurement_window: 30 days
error_budget: 0.1% (43 minutes per month)
latency_slo:
target: "95% of requests < 200ms"
measurement_window: 7 days
error_rate_slo:
target: "< 0.1%"
measurement_window: 24 hours
Service Level Agreements (SLAs)
Business contracts with consequences
# Example SLA
web_application_sla:
availability: 99.9%
latency_p95: 300ms
consequences:
- availability < 99.9%: 10% service credit
- availability < 99.0%: 25% service credit
- availability < 95.0%: 50% service credit
Prometheus Setup
Prometheus Configuration
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: 'production'
environment: 'prod'
# Alert manager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
# Load rules
rule_files:
- "/etc/prometheus/rules/*.yml"
# Scrape configurations
scrape_configs:
# Prometheus self-monitoring
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Kubernetes pods
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
# Node exporter
- job_name: 'node-exporter'
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
Alert Rules
# alert-rules.yml
groups:
- name: application_alerts
interval: 30s
rules:
# High error rate
- alert: HighErrorRate
expr: |
rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 5m
labels:
severity: critical
team: backend
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.job }}"
runbook: "https://wiki.example.com/runbooks/high-error-rate"
# High latency
- alert: HighLatency
expr: |
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
for: 10m
labels:
severity: warning
team: backend
annotations:
summary: "High latency detected"
description: "P95 latency is {{ $value | humanizeDuration }} for {{ $labels.job }}"
# Low availability
- alert: ServiceDown
expr: up == 0
for: 2m
labels:
severity: critical
team: sre
annotations:
summary: "Service is down"
description: "{{ $labels.job }} has been down for more than 2 minutes"
- name: kubernetes_alerts
interval: 30s
rules:
# Pod crash looping
- alert: PodCrashLooping
expr: |
rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Pod crash looping"
description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
# High memory usage
- alert: HighMemoryUsage
expr: |
(container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "High memory usage"
description: "Container {{ $labels.container }} in pod {{ $labels.pod }} is using {{ $value | humanizePercentage }} of memory"
# Node disk space
- alert: NodeDiskSpaceLow
expr: |
(node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "Node disk space low"
description: "Node {{ $labels.node }} has less than 10% disk space available"
Structured Logging
Best Practices
{
"timestamp": "2025-10-17T10:30:45.123Z",
"level": "ERROR",
"service": "api-gateway",
"version": "v1.2.3",
"trace_id": "abc123def456",
"span_id": "789ghi012jkl",
"user_id": "user-12345",
"request_id": "req-67890",
"method": "POST",
"path": "/api/v1/orders",
"status_code": 500,
"duration_ms": 245,
"error": {
"type": "DatabaseConnectionError",
"message": "Failed to connect to database",
"stack_trace": "..."
},
"context": {
"order_id": "order-98765",
"customer_id": "cust-54321"
}
}
Logging Configuration (Node.js Example)
const winston = require('winston');
const logger = winston.createLogger({
level: process.env.LOG_LEVEL || 'info',
format: winston.format.combine(
winston.format.timestamp(),
winston.format.errors({ stack: true }),
winston.format.json()
),
defaultMeta: {
service: process.env.SERVICE_NAME,
version: process.env.SERVICE_VERSION,
environment: process.env.ENVIRONMENT
},
transports: [
new winston.transports.Console(),
new winston.transports.File({
filename: 'error.log',
level: 'error'
}),
new winston.transports.File({
filename: 'combined.log'
})
]
});
// Usage with correlation ID
app.use((req, res, next) => {
req.id = req.headers['x-request-id'] || uuidv4();
req.logger = logger.child({
request_id: req.id,
trace_id: req.headers['x-trace-id']
});
next();
});
app.post('/api/orders', async (req, res) => {
req.logger.info('Creating order', {
customer_id: req.body.customer_id
});
try {
const order = await createOrder(req.body);
req.logger.info('Order created successfully', {
order_id: order.id
});
res.json(order);
} catch (error) {
req.logger.error('Failed to create order', {
error: error.message,
stack: error.stack
});
res.status(500).json({ error: 'Internal server error' });
}
});
Distributed Tracing
OpenTelemetry Configuration
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 10s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 512
resource:
attributes:
- key: environment
value: production
action: insert
exporters:
jaeger:
endpoint: jaeger:14250
tls:
insecure: true
prometheus:
endpoint: 0.0.0.0:8889
logging:
loglevel: info
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [jaeger, logging]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [prometheus, logging]
Application Instrumentation (Python Example)
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
# Set up tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# Configure OTLP exporter
otlp_exporter = OTLPSpanExporter(
endpoint="otel-collector:4317",
insecure=True
)
# Add span processor
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
# Instrument Flask and requests library
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
# Manual span creation
@app.route('/api/order/<order_id>')
def get_order(order_id):
with tracer.start_as_current_span("get_order") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("user.id", request.headers.get('X-User-ID'))
# Add events
span.add_event("Fetching order from database")
order = fetch_order_from_db(order_id)
if not order:
span.set_status(Status(StatusCode.ERROR, "Order not found"))
return {"error": "Order not found"}, 404
span.add_event("Order retrieved successfully")
return order
Dashboards & Visualization
Grafana Dashboard JSON (Example)
{
"dashboard": {
"title": "Application Performance",
"panels": [
{
"title": "Request Rate",
"targets": [
{
"expr": "rate(http_requests_total[5m])",
"legendFormat": "{{method}} {{status}}"
}
],
"type": "graph"
},
{
"title": "Error Rate",
"targets": [
{
"expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m])",
"legendFormat": "Error Rate"
}
],
"type": "graph"
},
{
"title": "P95 Latency",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
"legendFormat": "P95 Latency"
}
],
"type": "graph"
},
{
"title": "Active Connections",
"targets": [
{
"expr": "sum(up{job=\"myapp\"})",
"legendFormat": "Active Instances"
}
],
"type": "stat"
}
]
}
}
On-Call & Incident Response
Runbook Template
# Runbook: High Error Rate Alert
## Alert Details
- **Alert Name**: HighErrorRate
- **Severity**: Critical
- **Team**: Backend Engineering
- **On-Call**: See PagerDuty schedule
## Symptoms
- Error rate exceeds 5% for 5 minutes
- Users experiencing 5xx errors
- Elevated p95 latency
## Investigation Steps
1. **Check service health**
```bash
kubectl get pods -n production -l app=myapp
kubectl logs -n production -l app=myapp --tail=100
-
Review error logs
- Check Grafana dashboard
- Review application logs in Kibana
- Check CloudWatch metrics
-
Identify error patterns
- What endpoints are failing?
- Are errors consistent across all pods?
- Is there a pattern in timing?
-
Check dependencies
- Database connectivity
- External API availability
- Redis/cache status
Common Causes & Solutions
Database Connection Issues
- Symptoms: Connection timeout errors
- Solution:
# Check database connectivity kubectl exec -it <pod-name> -- nc -zv database-host 5432 # Check connection pool kubectl logs <pod-name> | grep "connection pool"
Memory Leaks
- Symptoms: Increasing memory usage, OOM kills
- Solution: Restart affected pods, investigate memory usage
Deployment Issues
- Symptoms: Errors started after deployment
- Solution: Rollback deployment
kubectl rollout undo deployment/myapp -n production
Escalation
- If unresolved after 15 minutes, escalate to Senior Engineer
- If service degradation > 30 minutes, notify VP Engineering
Post-Incident
- Create incident report
- Schedule post-mortem
- Update runbook with findings
## Observability Best Practices
1. **Use consistent naming**: Follow naming conventions for metrics, logs, traces
2. **Add context**: Include correlation IDs in logs and traces
3. **Set meaningful alerts**: Avoid alert fatigue with actionable alerts
4. **Define SLOs**: Measure what matters to users
5. **Practice incident response**: Regular game days and fire drills
6. **Automate runbooks**: Convert manual steps to automated remediation
7. **Monitor the monitors**: Ensure observability stack is reliable
8. **Continuous improvement**: Review and refine based on incidents
---
## Tools Comparison
| Feature | Prometheus | Datadog | New Relic | CloudWatch |
|---------|-----------|---------|-----------|------------|
| Metrics | ✓✓✓ | ✓✓✓ | ✓✓✓ | ✓✓ |
| Logs | via Loki | ✓✓✓ | ✓✓✓ | ✓✓✓ |
| Traces | via Tempo | ✓✓✓ | ✓✓✓ | ✓✓ |
| Cost | Free (self-hosted) | $$$ | $$$ | $$ |
| Learning Curve | Medium | Low | Low | Low |
| Kubernetes Native | ✓✓✓ | ✓✓ | ✓✓ | ✓ |