wehub-resource-sync
1 line
9.3 KiB
JSON
1 line
9.3 KiB
JSON
{"content": "---\nallowed-tools: Read, Edit, Bash\nargument-hint: [hotfix-type] | --security | --critical | --rollback-ready | --emergency\ndescription: Deploy critical hotfixes with emergency procedures, validation, and rollback capabilities\n---\n\n# Emergency Hotfix Deployment\n\nDeploy critical hotfix: $ARGUMENTS\n\n## Current Production State\n\n- Current version: !`git describe --tags --abbrev=0 2>/dev/null || echo \"No tags found\"`\n- Production branch: !`git branch --show-current`\n- Recent commits: !`git log --oneline -5`\n- Deployment status: !`curl -s https://api.example.com/health 2>/dev/null | jq -r '.version // \"Unknown\"' || echo \"Health check failed\"`\n- Staging environment: Check for staging deployment capabilities\n\n## Emergency Response Protocol\n\nExecute emergency hotfix deployment: $ARGUMENTS\n\n1. **Emergency Assessment and Triage**\n - Assess the severity and impact of the issue\n - Determine if a hotfix is necessary or if it can wait\n - Identify affected systems and user impact\n - Estimate time sensitivity and business impact\n - Document the incident and decision rationale\n\n2. **Incident Response Setup**\n - Create incident tracking in your incident management system\n - Set up war room or communication channel\n - Notify stakeholders and on-call team members\n - Establish clear communication protocols\n - Document initial incident details and timeline\n\n3. **Branch and Environment Setup**\n ```bash\n # Create hotfix branch from production tag\n git fetch --tags\n git checkout tags/v1.2.3 # Latest production version\n git checkout -b hotfix/critical-auth-fix\n \n # Alternative: Branch from main if using trunk-based development\n git checkout main\n git pull origin main\n git checkout -b hotfix/critical-auth-fix\n ```\n\n4. **Rapid Development Process**\n - Keep changes minimal and focused on the critical issue only\n - Avoid refactoring, optimization, or unrelated improvements\n - Use well-tested patterns and established approaches\n - Add minimal logging for troubleshooting purposes\n - Follow existing code conventions and patterns\n\n5. **Accelerated Testing**\n ```bash\n # Run focused tests related to the fix\n npm test -- --testPathPattern=auth\n npm run test:security\n \n # Manual testing checklist\n # [ ] Core functionality works correctly\n # [ ] Hotfix resolves the critical issue\n # [ ] No new issues introduced\n # [ ] Critical user flows remain functional\n ```\n\n6. **Fast-Track Code Review**\n - Get expedited review from senior team member\n - Focus review on security and correctness\n - Use pair programming if available and time permits\n - Document review decisions and rationale quickly\n - Ensure proper approval process even under time pressure\n\n7. **Version and Tagging**\n ```bash\n # Update version for hotfix\n # 1.2.3 -> 1.2.4 (patch version)\n # or 1.2.3 -> 1.2.3-hotfix.1 (hotfix identifier)\n \n # Commit with detailed message\n git add .\n git commit -m \"hotfix: fix critical authentication vulnerability\n \n - Fix password validation logic\n - Resolve security issue allowing bypass\n - Minimal change to reduce deployment risk\n \n Fixes: #1234\"\n \n # Tag the hotfix version\n git tag -a v1.2.4 -m \"Hotfix v1.2.4: Critical auth security fix\"\n git push origin hotfix/critical-auth-fix\n git push origin v1.2.4\n ```\n\n8. **Staging Deployment and Validation**\n ```bash\n # Deploy to staging environment for final validation\n ./deploy-staging.sh v1.2.4\n \n # Critical path testing\n curl -X POST staging.example.com/api/auth/login \\\n -H \"Content-Type: application/json\" \\\n -d '{\"email\":\"test@example.com\",\"password\":\"testpass\"}'\n \n # Run smoke tests\n npm run test:smoke:staging\n ```\n\n9. **Production Deployment Strategy**\n \n **Blue-Green Deployment:**\n ```bash\n # Deploy to blue environment\n ./deploy-blue.sh v1.2.4\n \n # Validate blue environment health\n ./health-check-blue.sh\n \n # Switch traffic to blue environment\n ./switch-to-blue.sh\n \n # Monitor deployment metrics\n ./monitor-deployment.sh\n ```\n \n **Rolling Deployment:**\n ```bash\n # Deploy to subset of servers first\n ./deploy-rolling.sh v1.2.4 --batch-size 1\n \n # Monitor each batch deployment\n ./monitor-batch.sh\n \n # Continue with next batch if healthy\n ./deploy-next-batch.sh\n ```\n\n10. **Pre-Deployment Checklist**\n ```bash\n # Verify all prerequisites are met\n # [ ] Database backup completed successfully\n # [ ] Rollback plan documented and ready\n # [ ] Monitoring alerts configured and active\n # [ ] Team members standing by for support\n # [ ] Communication channels established\n \n # Execute production deployment\n ./deploy-production.sh v1.2.4\n \n # Run immediate post-deployment validation\n ./validate-hotfix.sh\n ```\n\n11. **Real-Time Monitoring**\n ```bash\n # Monitor key application metrics\n watch -n 10 'curl -s https://api.example.com/health | jq .'\n \n # Monitor error rates and logs\n tail -f /var/log/app/error.log | grep -i \"auth\"\n \n # Track critical metrics:\n # - Response times and latency\n # - Error rates and exception counts\n # - User authentication success rates\n # - System resource usage (CPU, memory)\n ```\n\n12. **Post-Deployment Validation**\n ```bash\n # Run comprehensive validation tests\n ./test-critical-paths.sh\n \n # Test user authentication functionality\n curl -X POST https://api.example.com/auth/login \\\n -H \"Content-Type: application/json\" \\\n -d '{\"email\":\"test@example.com\",\"password\":\"testpass\"}'\n \n # Validate security fix effectiveness\n ./security-validation.sh\n \n # Check overall system performance\n ./performance-check.sh\n ```\n\n13. **Communication and Status Updates**\n - Provide regular status updates to stakeholders\n - Use consistent communication channels\n - Document deployment progress and results\n - Update incident tracking systems\n - Notify relevant teams of deployment completion\n\n14. **Rollback Procedures**\n ```bash\n # Automated rollback script\n #!/bin/bash\n PREVIOUS_VERSION=\"v1.2.3\"\n \n if [ \"$1\" = \"rollback\" ]; then\n echo \"Rolling back to $PREVIOUS_VERSION\"\n ./deploy-production.sh $PREVIOUS_VERSION\n ./validate-rollback.sh\n echo \"Rollback completed successfully\"\n fi\n \n # Manual rollback steps if automation fails:\n # 1. Switch load balancer back to previous version\n # 2. Validate previous version health and functionality\n # 3. Monitor system stability after rollback\n # 4. Communicate rollback status to team\n ```\n\n15. **Post-Deployment Monitoring Period**\n - Monitor system for 2-4 hours after deployment\n - Watch error rates and performance metrics closely\n - Check user feedback and support ticket volume\n - Validate that the hotfix resolves the original issue\n - Document any issues or unexpected behaviors\n\n16. **Documentation and Incident Reporting**\n - Document the complete hotfix process and timeline\n - Record lessons learned and process improvements\n - Update incident management systems with resolution\n - Create post-incident review materials\n - Share knowledge with team for future reference\n\n17. **Merge Back to Main Branch**\n ```bash\n # After successful hotfix deployment and validation\n git checkout main\n git pull origin main\n git merge hotfix/critical-auth-fix\n git push origin main\n \n # Clean up hotfix branch\n git branch -d hotfix/critical-auth-fix\n git push origin --delete hotfix/critical-auth-fix\n ```\n\n18. **Post-Incident Activities**\n - Schedule and conduct post-incident review meeting\n - Update runbooks and emergency procedures\n - Identify and implement process improvements\n - Update monitoring and alerting configurations\n - Plan preventive measures to avoid similar issues\n\n**Hotfix Best Practices:**\n\n- **Keep It Simple:** Make minimal changes focused only on the critical issue\n- **Test Thoroughly:** Maintain testing standards even under time pressure\n- **Communicate Clearly:** Keep all stakeholders informed throughout the process\n- **Monitor Closely:** Watch the fix carefully in production environment\n- **Document Everything:** Record all decisions and actions for post-incident review\n- **Plan for Rollback:** Always have a tested way to revert changes quickly\n- **Learn and Improve:** Use each incident to strengthen processes and procedures\n\n**Emergency Escalation Guidelines:**\n\n```bash\n# Emergency contact information\nON_CALL_ENGINEER=\"+1-555-0123\"\nSENIOR_ENGINEER=\"+1-555-0124\"\nENGINEERING_MANAGER=\"+1-555-0125\"\nINCIDENT_COMMANDER=\"+1-555-0126\"\n\n# Escalation timeline thresholds:\n# 15 minutes: Escalate to senior engineer\n# 30 minutes: Escalate to engineering manager\n# 60 minutes: Escalate to incident commander\n```\n\n**Important Reminders:**\n\n- Hotfixes should only be used for genuine production emergencies\n- When in doubt about severity, follow the normal release process\n- Always prioritize system stability over speed of deployment\n- Maintain clear audit trails for all emergency changes\n- Regular drills help ensure team readiness for real emergencies"} |