wehub-resource-sync
1 line
12 KiB
JSON
1 line
12 KiB
JSON
{"content": "---\nallowed-tools: Read, Edit, Bash\nargument-hint: [target-version] | --previous | --emergency | --validate-first | --with-db\ndescription: Rollback deployment to previous version with safety checks, database considerations, and monitoring\n---\n\n# Deployment Rollback\n\nRollback deployment to previous version: $ARGUMENTS\n\n## Current Deployment State\n\n- Current version: !`curl -s https://api.example.com/version 2>/dev/null || kubectl get deployments -o wide 2>/dev/null | head -3 || echo \"Version detection needed\"`\n- Available versions: !`git tag --sort=-version:refname | head -5`\n- Container status: !`docker ps --format \"table {{.Names}}\\t{{.Image}}\\t{{.Status}}\" 2>/dev/null | head -5 || echo \"No containers\"`\n- K8s deployments: !`kubectl get deployments 2>/dev/null || echo \"No K8s access\"`\n- Health status: !`curl -sf https://api.example.com/health 2>/dev/null && echo \"✅ Healthy\" || echo \"❌ Unhealthy\"`\n\n## Emergency Rollback Protocol\n\nSystematic rollback procedure: $ARGUMENTS\n\n1. **Incident Assessment and Decision**\n - Assess the severity and impact of the current deployment issues\n - Determine if rollback is necessary or if forward fix is better\n - Identify affected systems, users, and business functions\n - Consider data integrity and consistency implications\n - Document the decision rationale and timeline\n\n2. **Emergency Response Setup**\n ```bash\n # Activate incident response team\n # Set up communication channels\n # Notify stakeholders immediately\n \n # Example emergency notification\n echo \"🚨 ROLLBACK INITIATED\n Issue: Critical performance degradation after v1.3.0 deployment\n Action: Rolling back to v1.2.9\n ETA: 15 minutes\n Impact: Temporary service interruption possible\n Status channel: #incident-rollback-202401\"\n ```\n\n3. **Pre-Rollback Safety Checks**\n ```bash\n # Verify current production version\n curl -s https://api.example.com/version\n kubectl get deployments -o wide\n \n # Check system status\n curl -s https://api.example.com/health | jq .\n \n # Identify target rollback version\n git tag --sort=-version:refname | head -5\n \n # Verify rollback target exists and is deployable\n git show v1.2.9 --stat\n ```\n\n4. **Database Considerations**\n ```bash\n # Check for database migrations since last version\n ./check-migrations.sh v1.2.9 v1.3.0\n \n # If migrations exist, plan database rollback\n # WARNING: Database rollbacks can cause data loss\n # Consider forward fix instead if migrations are present\n \n # Create database backup before rollback\n ./backup-database.sh \"pre-rollback-$(date +%Y%m%d-%H%M%S)\"\n ```\n\n5. **Traffic Management Preparation**\n ```bash\n # Prepare to redirect traffic\n # Option 1: Maintenance page\n ./enable-maintenance-mode.sh\n \n # Option 2: Load balancer management\n ./drain-traffic.sh --gradual\n \n # Option 3: Circuit breaker activation\n ./activate-circuit-breaker.sh\n ```\n\n6. **Container/Kubernetes Rollback**\n ```bash\n # Kubernetes rollback\n kubectl rollout history deployment/app-deployment\n kubectl rollout undo deployment/app-deployment\n \n # Or rollback to specific revision\n kubectl rollout undo deployment/app-deployment --to-revision=3\n \n # Monitor rollback progress\n kubectl rollout status deployment/app-deployment --timeout=300s\n \n # Verify pods are running\n kubectl get pods -l app=your-app\n ```\n\n7. **Docker Swarm Rollback**\n ```bash\n # List service history\n docker service ps app-service --no-trunc\n \n # Rollback to previous version\n docker service update --rollback app-service\n \n # Or update to specific image\n docker service update --image app:v1.2.9 app-service\n \n # Monitor rollback\n docker service ps app-service\n ```\n\n8. **Traditional Deployment Rollback**\n ```bash\n # Blue-Green deployment rollback\n ./switch-to-blue.sh # or green, depending on current\n \n # Rolling deployment rollback\n ./deploy-version.sh v1.2.9 --rolling\n \n # Symlink-based rollback\n ln -sfn /releases/v1.2.9 /current\n sudo systemctl restart app-service\n ```\n\n9. **Load Balancer and CDN Updates**\n ```bash\n # Update load balancer to point to old version\n aws elbv2 modify-target-group --target-group-arn $TG_ARN --targets Id=old-instance\n \n # Clear CDN cache if needed\n aws cloudfront create-invalidation --distribution-id $DIST_ID --paths \\\"/*\\\"\n \n # Update DNS if necessary (last resort, has propagation delay)\n # aws route53 change-resource-record-sets ...\n ```\n\n10. **Configuration Rollback**\n ```bash\\n # Rollback configuration files\\n git checkout v1.2.9 -- config/\\n \\n # Restart services with old configuration\\n sudo systemctl restart nginx\\n sudo systemctl restart app-service\\n \\n # Rollback environment variables\\n ./restore-env-vars.sh v1.2.9\\n \\n # Update feature flags\\n ./update-feature-flags.sh --disable-new-features\\n ```\\n\\n11. **Database Rollback (if necessary)**\\n ```sql\\n -- EXTREME CAUTION: Can cause data loss\\n \\n -- Check migration status\\n SELECT * FROM schema_migrations ORDER BY version DESC LIMIT 5;\\n \\n -- Rollback specific migrations (framework dependent)\\n -- Rails: rake db:migrate:down VERSION=20240115120000\\n -- Django: python manage.py migrate app_name 0001\\n -- Node.js: npm run migrate:down\\n \\n -- Verify database state\\n SHOW TABLES;\\n DESCRIBE critical_table;\\n ```\\n\\n12. **Service Health Validation**\\n ```bash\\n # Health check script\\n #!/bin/bash\\n \\n echo \\\"Validating rollback...\\\"\\n \\n # Check application health\\n if curl -f -s https://api.example.com/health > /dev/null; then\\n echo \\\"✅ Health check passed\\\"\\n else\\n echo \\\"❌ Health check failed\\\"\\n exit 1\\n fi\\n \\n # Check critical endpoints\\n endpoints=(\\n \\\"/api/users/me\\\"\\n \\\"/api/auth/status\\\"\\n \\\"/api/data/latest\\\"\\n )\\n \\n for endpoint in \\\"${endpoints[@]}\\\"; do\\n if curl -f -s \\\"https://api.example.com$endpoint\\\" > /dev/null; then\\n echo \\\"✅ $endpoint working\\\"\\n else\\n echo \\\"❌ $endpoint failed\\\"\\n fi\\n done\\n ```\\n\\n13. **Performance and Metrics Validation**\\n ```bash\\n # Check response times\\n curl -w \\\"Response time: %{time_total}s\\\\n\\\" -s -o /dev/null https://api.example.com/\\n \\n # Monitor error rates\\n tail -f /var/log/app/error.log | head -20\\n \\n # Check system resources\\n top -bn1 | head -10\\n free -h\\n df -h\\n \\n # Validate database connectivity\\n mysql -u app -p -e \\\"SELECT 1;\\\"\\n ```\\n\\n14. **Traffic Restoration**\\n ```bash\\n # Gradually restore traffic\\n ./restore-traffic.sh --gradual\\n \\n # Disable maintenance mode\\n ./disable-maintenance-mode.sh\\n \\n # Re-enable circuit breakers\\n ./deactivate-circuit-breaker.sh\\n \\n # Monitor traffic patterns\\n ./monitor-traffic.sh --duration 300\\n ```\\n\\n15. **Monitoring and Alerting**\\n ```bash\\n # Enable enhanced monitoring during rollback\\n ./enable-enhanced-monitoring.sh\\n \\n # Watch key metrics\\n watch -n 10 'curl -s https://api.example.com/metrics | jq .'\\n \\n # Monitor logs in real-time\\n tail -f /var/log/app/*.log | grep -E \\\"ERROR|WARN|EXCEPTION\\\"\\n \\n # Check application metrics\\n # - Response times\\n # - Error rates\\n # - User sessions\\n # - Database performance\\n ```\\n\\n16. **User Communication**\\n ```markdown\\n ## Service Update - Rollback Completed\\n \\n **Status:** ✅ Service Restored\\n **Time:** 2024-01-15 15:45 UTC\\n **Duration:** 12 minutes of degraded performance\\n \\n **What Happened:**\\n We identified performance issues with our latest release and \\n performed a rollback to ensure optimal service quality.\\n \\n **Current Status:**\\n - All services operating normally\\n - Performance metrics back to baseline\\n - No data loss occurred\\n \\n **Next Steps:**\\n We're investigating the root cause and will provide updates \\n on our status page.\\n ```\\n\\n17. **Post-Rollback Validation**\\n ```bash\\n # Extended monitoring period\\n ./monitor-extended.sh --duration 3600 # 1 hour\\n \\n # Run integration tests\\n npm run test:integration:production\\n \\n # Check user-reported issues\\n ./check-support-tickets.sh --since \\\"1 hour ago\\\"\\n \\n # Validate business metrics\\n ./check-business-metrics.sh\\n ```\\n\\n18. **Documentation and Reporting**\\n ```markdown\\n # Rollback Incident Report\\n \\n **Incident ID:** INC-2024-0115-001\\n **Rollback Version:** v1.2.9 (from v1.3.0)\\n **Start Time:** 2024-01-15 15:30 UTC\\n **End Time:** 2024-01-15 15:42 UTC\\n **Total Duration:** 12 minutes\\n \\n **Timeline:**\\n - 15:25 - Performance degradation detected\\n - 15:30 - Rollback decision made\\n - 15:32 - Traffic drained\\n - 15:35 - Rollback initiated\\n - 15:38 - Rollback completed\\n - 15:42 - Traffic fully restored\\n \\n **Impact:**\\n - 12 minutes of degraded performance\\n - ~5% of users experienced slow responses\\n - No data loss or corruption\\n - No security implications\\n \\n **Root Cause:**\\n Memory leak in new feature causing performance degradation\\n \\n **Lessons Learned:**\\n - Need better performance testing in staging\\n - Improve monitoring for memory usage\\n - Consider canary deployments for major releases\\n ```\\n\\n19. **Cleanup and Follow-up**\\n ```bash\\n # Clean up failed deployment artifacts\\n docker image rm app:v1.3.0\\n \\n # Update deployment status\\n ./update-deployment-status.sh \\\"rollback-completed\\\"\\n \\n # Reset feature flags if needed\\n ./reset-feature-flags.sh\\n \\n # Schedule post-incident review\\n ./schedule-postmortem.sh --date \\\"2024-01-16 10:00\\\"\\n ```\\n\\n20. **Prevention and Improvement**\\n - Analyze what went wrong with the deployment\\n - Improve testing and validation procedures\\n - Enhance monitoring and alerting\\n - Update rollback procedures based on learnings\\n - Consider implementing canary deployments\\n\\n**Rollback Decision Matrix:**\\n\\n| Issue Severity | Data Impact | Time to Fix | Decision |\\n|---------------|-------------|-------------|----------|\\n| Critical | None | > 30 min | Rollback |\\n| High | Minor | > 60 min | Rollback |\\n| Medium | None | > 2 hours | Consider rollback |\\n| Low | None | Any | Forward fix |\\n\\n**Emergency Rollback Script Template:**\\n```bash\\n#!/bin/bash\\nset -e\\n\\n# Emergency rollback script\\nPREVIOUS_VERSION=\\\"${1:-v1.2.9}\\\"\\nCURRENT_VERSION=$(curl -s https://api.example.com/version)\\n\\necho \\\"🚨 EMERGENCY ROLLBACK\\\"\\necho \\\"From: $CURRENT_VERSION\\\"\\necho \\\"To: $PREVIOUS_VERSION\\\"\\necho \\\"\\\"\\n\\n# Confirm rollback\\nread -p \\\"Proceed with rollback? (yes/no): \\\" confirm\\nif [ \\\"$confirm\\\" != \\\"yes\\\" ]; then\\n echo \\\"Rollback cancelled\\\"\\n exit 1\\nfi\\n\\n# Execute rollback\\necho \\\"Starting rollback...\\\"\\nkubectl set image deployment/app-deployment app=app:$PREVIOUS_VERSION\\nkubectl rollout status deployment/app-deployment --timeout=300s\\n\\n# Validate\\necho \\\"Validating rollback...\\\"\\nsleep 30\\ncurl -f https://api.example.com/health\\n\\necho \\\"✅ Rollback completed successfully\\\"\\n```\\n\\nRemember: Rollbacks should be a last resort. Always consider forward fixes first, especially when database migrations are involved."} |