wehub-resource-sync
e071084ebe
govulncheck / govulncheck (push) Waiting to run
Harness (E2E) / Harnesses (mock LLM) (push) Waiting to run
Harness (E2E) / Provider harnesses (live LLM conformance) (push) Waiting to run
Lint / golangci-lint (push) Waiting to run
Run Tests / Unit Tests (push) Waiting to run
Run Tests / Etcd Integration Tests (push) Waiting to run
96 lines
2.7 KiB
Markdown
96 lines
2.7 KiB
Markdown
---
|
||
layout: default
|
||
---
|
||
|
||
# Observability
|
||
|
||
<img src="/images/generated/observability.jpg" alt="Observability" style="width: 100%; border-radius: 8px; margin: 1rem 0 1.5rem;" />
|
||
|
||
Observability in Go Micro spans logs, metrics, and traces. The goal is rapid insight into service behavior with minimal configuration.
|
||
|
||
## Core Principles
|
||
|
||
1. Structured Logs – Machine-parsable, leveled output
|
||
2. Metrics – Quantitative trends (counters, gauges, histograms)
|
||
3. Traces – Request flows across service boundaries
|
||
4. Correlation – IDs flowing through all three signals
|
||
|
||
## Logging
|
||
|
||
The default logger can be replaced. Use env vars to adjust level:
|
||
|
||
```bash
|
||
MICRO_LOG_LEVEL=debug go run main.go
|
||
```
|
||
|
||
Recommended fields:
|
||
- `service` – service name
|
||
- `version` – release identifier
|
||
- `trace_id` – propagated context id
|
||
- `span_id` – current operation id
|
||
|
||
## Metrics
|
||
|
||
Patterns:
|
||
- Emit counters for request totals
|
||
- Use histograms for latency
|
||
- Track error rates per endpoint
|
||
|
||
Example (pseudo-code):
|
||
|
||
```go
|
||
// Wrap handler to record metrics
|
||
func MetricsWrapper(fn micro.HandlerFunc) micro.HandlerFunc {
|
||
return func(ctx context.Context, req micro.Request, rsp interface{}) error {
|
||
start := time.Now()
|
||
err := fn(ctx, req, rsp)
|
||
latency := time.Since(start)
|
||
metrics.Inc("requests_total", req.Endpoint(), errorLabel(err))
|
||
metrics.Observe("request_latency_seconds", latency, req.Endpoint())
|
||
return err
|
||
}
|
||
}
|
||
```
|
||
|
||
## Tracing
|
||
|
||
Distributed tracing links calls across services.
|
||
|
||
Propagation strategy:
|
||
- Extract trace context from incoming headers
|
||
- Inject into outgoing RPC calls/broker messages
|
||
- Create spans per handler and client call
|
||
|
||
## Local Development Strategy
|
||
|
||
Start with only structured logs. Add metrics when operating multiple services. Introduce tracing once debugging multi-hop latency or failures.
|
||
|
||
## Roadmap (Planned Enhancements)
|
||
|
||
- Native OpenTelemetry exporter helpers
|
||
- Automatic handler/client wrapping for spans
|
||
- Default correlation IDs across broker messages
|
||
|
||
## Deployment Recommendations
|
||
|
||
| Scale | Suggested Stack |
|
||
|-------|-----------------|
|
||
| Dev | Console logs only |
|
||
| Staging | Logs + basic metrics (Prometheus) |
|
||
| Prod (basic) | Logs + metrics + sampling traces |
|
||
| Prod (complex) | Full tracing + profiling + anomaly detection |
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---------|-------|-----|
|
||
| Missing trace IDs in logs | Context not propagated | Ensure wrappers add IDs |
|
||
| Metrics server empty | Endpoint not scraped | Verify Prometheus config |
|
||
| High cardinality metrics | Dynamic labels | Reduce labeled dimensions |
|
||
|
||
## Related
|
||
|
||
- [Getting Started](getting-started.html)
|
||
- [Plugins](plugins.html)
|
||
- [Architecture Decisions](architecture/index.md)
|