wehub-resource-sync
4b6817381b
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
36 lines
2.4 KiB
YAML
36 lines
2.4 KiB
YAML
root_cause_category: replication_lag
|
|
equivalent_root_cause_categories:
|
|
- replication_lag_wal_volume
|
|
required_keywords:
|
|
- WAL
|
|
- replica
|
|
required_evidence_sources:
|
|
- aws_rds_events
|
|
- aws_performance_insights
|
|
optimal_trajectory:
|
|
- query_grafana_logs
|
|
- query_grafana_metrics
|
|
- query_grafana_alert_rules
|
|
- describe_rds_instance
|
|
- describe_rds_events
|
|
max_investigation_loops: 4
|
|
model_response: |
|
|
ROOT_CAUSE: Replication lag on reporting-prod-replica-1 caused by a sustained bulk INSERT ETL job generating WAL faster than the replica can replay. The ReplicaLag CloudWatch metric is unavailable (collection gap on a recently provisioned replica), but the diagnosis is confirmed indirectly by: (1) RDS events explicitly reporting lag exceeding 900s then 1800s, (2) TransactionLogsGeneration sustaining ~195 MB/s on the primary, and (3) Performance Insights showing WAL:Lock as the dominant wait event at 4.2 AAS.
|
|
ROOT_CAUSE_CATEGORY: replication_lag
|
|
|
|
VALIDATED_CLAIMS:
|
|
- WriteIOPS spiked from 410 to 19,530 at 07:01Z and sustained, consistent with a large bulk write workload. [evidence: aws_cloudwatch_metrics]
|
|
- TransactionLogsGeneration reached ~200 MB/s, indicating WAL production far exceeding typical replica replay capacity. [evidence: aws_cloudwatch_metrics]
|
|
- ReplicaLag metric is absent from CloudWatch (not published for this replica instance). Root cause is inferred from indirect signals. [evidence: aws_cloudwatch_metrics]
|
|
- RDS events confirm: replication lag exceeded 900s at 07:06Z and 1800s at 07:11Z on reporting-prod-replica-1. [evidence: aws_rds_events]
|
|
- Performance Insights identifies an ETL INSERT statement as the top load consumer (6.1 AAS) with WAL:Lock as the dominant wait event — confirming WAL-heavy write workload. [evidence: aws_performance_insights]
|
|
|
|
NON_VALIDATED_CLAIMS:
|
|
- Throttling the ETL batch size or scheduling it outside of reporting hours would reduce WAL generation and allow the replica to catch up.
|
|
|
|
CAUSAL_CHAIN:
|
|
- An ETL bulk INSERT job began loading reporting_events at 07:01Z, generating ~195 MB/s of WAL on the primary.
|
|
- The read replica (reporting-prod-replica-1) cannot replay WAL at this rate, accumulating lag.
|
|
- The ReplicaLag CloudWatch metric is not being published for this replica (provisioned too recently for the metric to appear in CloudWatch).
|
|
- RDS control-plane events and PI WAL:Lock dominance confirm the lag is caused by WAL replay falling behind, not by a replica failure.
|