wehub/davila7--claude-code-templates
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
davila7--claude-code-templates/cli-tool/components/skills/development/it-operations/reference/automation.md
T
wehub-resource-sync bb5c75ce05
Component Security Validation / Security Audit (push) Has been cancelled
Details
Deploy to Cloudflare Pages / deploy (push) Has been cancelled
Details
chore: import upstream snapshot with attribution
2026-07-13 12:38:58 +08:00

1483 lines
39 KiB
Markdown
Raw Permalink Blame History

# Automation and Orchestration
Comprehensive guide to IT automation, scripting, configuration management, orchestration tools, and reducing operational toil.
## Table of Contents
- [Automation Strategy](#automation-strategy)
- [Scripting Best Practices](#scripting-best-practices)
- [Configuration Management](#configuration-management)
- [Orchestration Tools](#orchestration-tools)
- [CI/CD for Infrastructure](#cicd-for-infrastructure)
- [Runbook Automation](#runbook-automation)
- [Self-Healing Systems](#self-healing-systems)
- [Toil Reduction](#toil-reduction)
## Automation Strategy
### When to Automate
```yaml
Automation Decision Framework:
High Priority (Automate Immediately):
- Task performed > 10 times per week
- Task is error-prone when done manually
- Task requires precision (e.g., exact timing, complex steps)
- Task performed outside business hours
- Task blocks other work
Examples:
- Server provisioning
- Deployment process
- Backup operations
- Certificate renewal
- Log rotation
Medium Priority (Automate Within 3 Months):
- Task performed 2-10 times per week
- Task is time-consuming (> 30 minutes)
- Task has multiple steps that could be forgotten
- Task involves multiple systems
Examples:
- User account creation
- Monthly reporting
- Security patching
- Capacity planning data collection
Low Priority (Consider Automation):
- Task performed < 2 times per week
- Task is simple and quick (< 5 minutes)
- Task requires human judgment
- Automation effort > 10x manual effort
Examples:
- One-off investigations
- Ad-hoc queries
- Executive requests
Don't Automate:
- Task requires critical thinking
- Task changes frequently (automation would break)
- Task is performed once per year
- Legal/compliance requires human oversight
Examples:
- Incident response decision-making
- Architecture reviews
- Contract negotiations
```
### Automation Maturity Model
```yaml
Level 0: Manual
- All tasks performed manually
- Documentation may exist
- High error rate, inconsistent results
- Tribal knowledge
Level 1: Documented
- Detailed runbooks exist
- Step-by-step procedures
- Anyone can follow documentation
- Still fully manual execution
Level 2: Scripted
- Scripts automate individual tasks
- Scripts run on-demand
- Reduces errors, saves time
- Scripts may live on individual machines
Example: backup_database.sh
Level 3: Scheduled Automation
- Scripts run on schedule (cron, systemd timers)
- Central script repository (Git)
- Logging and error handling
- Notifications on failure
Example: Daily backups via cron
Level 4: Event-Driven Automation
- Automation triggered by events (alerts, webhooks)
- Self-healing (auto-restart failed services)
- Integration with monitoring systems
- Closed-loop automation
Example: Auto-scale on high CPU alert
Level 5: Intelligent Automation
- Machine learning predicts issues
- Proactive remediation before user impact
- Continuous optimization
- Full observability and feedback loops
Example: Predictive auto-scaling based on traffic patterns
```
### ROI Calculation for Automation
```python
# Automation ROI Calculator
def calculate_automation_roi(
manual_time_hours,
frequency_per_month,
hourly_cost,
automation_development_hours,
automation_maintenance_hours_per_month
):
"""
Calculate ROI for automation project
Args:
manual_time_hours: Time to perform task manually (hours)
frequency_per_month: How often task is performed per month
hourly_cost: Cost per hour of engineer time ($)
automation_development_hours: Time to develop automation (hours)
automation_maintenance_hours_per_month: Ongoing maintenance (hours/month)
Returns:
dict with ROI metrics
"""
# Monthly cost without automation
monthly_manual_cost = manual_time_hours * frequency_per_month * hourly_cost
# Annual manual cost
annual_manual_cost = monthly_manual_cost * 12
# Upfront automation cost
automation_dev_cost = automation_development_hours * hourly_cost
# Monthly automation maintenance cost
monthly_automation_cost = automation_maintenance_hours_per_month * hourly_cost
# Annual automation cost
annual_automation_cost = monthly_automation_cost * 12
# Annual savings
annual_savings = annual_manual_cost - annual_automation_cost
# Break-even time (months)
if annual_savings > 0:
break_even_months = automation_dev_cost / (annual_savings / 12)
else:
break_even_months = float('inf')
# 3-year ROI
three_year_savings = (annual_savings * 3) - automation_dev_cost
roi_percentage = (three_year_savings / automation_dev_cost) * 100 if automation_dev_cost > 0 else 0
return {
'monthly_manual_cost': monthly_manual_cost,
'annual_manual_cost': annual_manual_cost,
'automation_dev_cost': automation_dev_cost,
'annual_automation_cost': annual_automation_cost,
'annual_savings': annual_savings,
'break_even_months': break_even_months,
'three_year_savings': three_year_savings,
'roi_percentage': roi_percentage,
'recommendation': 'Automate' if break_even_months < 12 else 'Consider alternatives'
}
# Example: Database backup automation
result = calculate_automation_roi(
manual_time_hours=2, # 2 hours to backup manually
frequency_per_month=30, # Daily backups
hourly_cost=75, # Engineer costs $75/hour
automation_development_hours=16, # 2 days to develop automation
automation_maintenance_hours_per_month=1 # 1 hour/month maintenance
)
print("=== Automation ROI Analysis ===")
print(f"Monthly Manual Cost: ${result['monthly_manual_cost']:,.2f}")
print(f"Annual Manual Cost: ${result['annual_manual_cost']:,.2f}")
print(f"Automation Development Cost: ${result['automation_dev_cost']:,.2f}")
print(f"Annual Automation Cost: ${result['annual_automation_cost']:,.2f}")
print(f"Annual Savings: ${result['annual_savings']:,.2f}")
print(f"Break-even Time: {result['break_even_months']:.1f} months")
print(f"3-Year Savings: ${result['three_year_savings']:,.2f}")
print(f"3-Year ROI: {result['roi_percentage']:.0f}%")
print(f"Recommendation: {result['recommendation']}")
# Output:
# === Automation ROI Analysis ===
# Monthly Manual Cost: $4,500.00
# Annual Manual Cost: $54,000.00
# Automation Development Cost: $1,200.00
# Annual Automation Cost: $900.00
# Annual Savings: $53,100.00
# Break-even Time: 0.3 months
# 3-Year Savings: $158,100.00
# 3-Year ROI: 13175%
# Recommendation: Automate
```
## Scripting Best Practices
### Shell Scripting (Bash)
```bash
#!/bin/bash
#
# Script: server_health_check.sh
# Description: Perform comprehensive server health check
# Author: Ops Team
# Date: 2025-01-15
# Version: 1.0
# Strict error handling
set -euo pipefail # Exit on error, undefined var, pipe failure
IFS=$'\n\t' # Set internal field separator
# Configuration
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly LOG_FILE="/var/log/health_check.log"
readonly EMAIL_TO="ops-team@example.com"
readonly DISK_THRESHOLD=80
readonly MEMORY_THRESHOLD=85
readonly CPU_THRESHOLD=90
# Colors for output
readonly RED='\033[0;31m'
readonly GREEN='\033[0;32m'
readonly YELLOW='\033[1;33m'
readonly NC='\033[0m' # No Color
# Logging function
log() {
local level=$1
shift
local message="$@"
local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
echo "[${timestamp}] [${level}] ${message}" | tee -a "${LOG_FILE}"
}
# Error handling
error_exit() {
log "ERROR" "$1"
exit 1
}
# Cleanup on exit
cleanup() {
log "INFO" "Cleaning up..."
# Remove temp files, etc.
}
trap cleanup EXIT
# Check if running as root
check_root() {
if [[ $EUID -ne 0 ]]; then
error_exit "This script must be run as root"
fi
}
# Check disk space
check_disk_space() {
log "INFO" "Checking disk space..."
local status=0
while read -r line; do
local usage=$(echo "$line" | awk '{print $5}' | sed 's/%//')
local mount=$(echo "$line" | awk '{print $6}')
if [[ $usage -ge $DISK_THRESHOLD ]]; then
log "WARN" "Disk usage on ${mount} is ${usage}% (threshold: ${DISK_THRESHOLD}%)"
status=1
else
log "INFO" "Disk usage on ${mount} is ${usage}% - OK"
fi
done < <(df -h | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{print $0}')
return $status
}
# Check memory usage
check_memory() {
log "INFO" "Checking memory usage..."
local total_mem=$(free -m | awk '/^Mem:/{print $2}')
local used_mem=$(free -m | awk '/^Mem:/{print $3}')
local usage_pct=$((used_mem * 100 / total_mem))
if [[ $usage_pct -ge $MEMORY_THRESHOLD ]]; then
log "WARN" "Memory usage is ${usage_pct}% (threshold: ${MEMORY_THRESHOLD}%)"
return 1
else
log "INFO" "Memory usage is ${usage_pct}% - OK"
return 0
fi
}
# Check CPU load
check_cpu() {
log "INFO" "Checking CPU load..."
local load_avg=$(uptime | awk -F'load average:' '{print $2}' | awk '{print $1}' | sed 's/,//')
local cpu_count=$(nproc)
local load_pct=$(echo "scale=2; ($load_avg / $cpu_count) * 100" | bc)
local load_pct_int=${load_pct%.*}
if [[ $load_pct_int -ge $CPU_THRESHOLD ]]; then
log "WARN" "CPU load is ${load_pct}% (threshold: ${CPU_THRESHOLD}%)"
return 1
else
log "INFO" "CPU load is ${load_pct}% - OK"
return 0
fi
}
# Check critical services
check_services() {
log "INFO" "Checking critical services..."
local services=("sshd" "nginx" "postgresql")
local status=0
for service in "${services[@]}"; do
if systemctl is-active --quiet "$service"; then
log "INFO" "Service ${service} is running - OK"
else
log "ERROR" "Service ${service} is NOT running"
status=1
fi
done
return $status
}
# Check network connectivity
check_network() {
log "INFO" "Checking network connectivity..."
local hosts=("8.8.8.8" "1.1.1.1")
local status=0
for host in "${hosts[@]}"; do
if ping -c 3 -W 5 "$host" &>/dev/null; then
log "INFO" "Network connectivity to ${host} - OK"
else
log "ERROR" "Cannot reach ${host}"
status=1
fi
done
return $status
}
# Generate report
generate_report() {
local overall_status=$1
cat <<EOF > /tmp/health_report.txt
Server Health Check Report
==========================
Hostname: $(hostname)
Date: $(date)
Status: $([[ $overall_status -eq 0 ]] && echo "HEALTHY" || echo "ISSUES DETECTED")
Details:
$(tail -n 50 "${LOG_FILE}")
EOF
# Email report if issues detected
if [[ $overall_status -ne 0 ]]; then
mail -s "ALERT: Health Check Failed on $(hostname)" "${EMAIL_TO}" < /tmp/health_report.txt
fi
}
# Main execution
main() {
log "INFO" "Starting server health check..."
local overall_status=0
check_root
check_disk_space || overall_status=1
check_memory || overall_status=1
check_cpu || overall_status=1
check_services || overall_status=1
check_network || overall_status=1
if [[ $overall_status -eq 0 ]]; then
log "INFO" "All health checks passed"
else
log "WARN" "Some health checks failed"
fi
generate_report $overall_status
log "INFO" "Health check complete"
exit $overall_status
}
# Run main function
main "$@"
```
### Python Automation Scripts
```python
#!/usr/bin/env python3
"""
User account management automation
Automates user provisioning across multiple systems
"""
import argparse
import logging
import sys
import subprocess
import json
from typing import Dict, List, Optional
from pathlib import Path
import yaml
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('/var/log/user_management.log'),
logging.StreamHandler(sys.stdout)
]
)
logger = logging.getLogger(__name__)
class UserManagementError(Exception):
"""Custom exception for user management errors"""
pass
class UserManager:
"""Manages user accounts across systems"""
def __init__(self, config_file: str):
self.config = self._load_config(config_file)
self.dry_run = False
def _load_config(self, config_file: str) -> Dict:
"""Load configuration from YAML file"""
try:
with open(config_file, 'r') as f:
return yaml.safe_load(f)
except Exception as e:
logger.error(f"Failed to load config: {e}")
raise UserManagementError(f"Config load failed: {e}")
def _run_command(self, cmd: List[str], check: bool = True) -> subprocess.CompletedProcess:
"""Execute shell command with error handling"""
logger.debug(f"Executing command: {' '.join(cmd)}")
if self.dry_run:
logger.info(f"[DRY RUN] Would execute: {' '.join(cmd)}")
return subprocess.CompletedProcess(cmd, 0, '', '')
try:
result = subprocess.run(
cmd,
check=check,
capture_output=True,
text=True,
timeout=30
)
return result
except subprocess.CalledProcessError as e:
logger.error(f"Command failed: {e.stderr}")
raise UserManagementError(f"Command failed: {e}")
except subprocess.TimeoutExpired:
logger.error(f"Command timed out: {' '.join(cmd)}")
raise UserManagementError("Command timeout")
def user_exists(self, username: str) -> bool:
"""Check if user exists"""
result = self._run_command(['id', username], check=False)
return result.returncode == 0
def create_user(
self,
username: str,
full_name: str,
email: str,
groups: Optional[List[str]] = None,
ssh_public_key: Optional[str] = None
) -> None:
"""Create user account with specified attributes"""
logger.info(f"Creating user: {username}")
if self.user_exists(username):
logger.warning(f"User {username} already exists")
return
# Create user
cmd = ['useradd', '-m', '-s', '/bin/bash']
if full_name:
cmd.extend(['-c', full_name])
if groups:
cmd.extend(['-G', ','.join(groups)])
cmd.append(username)
self._run_command(cmd)
# Set up SSH key
if ssh_public_key:
self._setup_ssh_key(username, ssh_public_key)
# Send welcome email (pseudo-code)
# self._send_welcome_email(username, email)
logger.info(f"User {username} created successfully")
def _setup_ssh_key(self, username: str, public_key: str) -> None:
"""Set up SSH public key for user"""
logger.info(f"Setting up SSH key for {username}")
# Create .ssh directory
ssh_dir = Path(f"/home/{username}/.ssh")
ssh_dir.mkdir(mode=0o700, exist_ok=True)
# Write authorized_keys
authorized_keys = ssh_dir / "authorized_keys"
authorized_keys.write_text(public_key + "\n")
authorized_keys.chmod(0o600)
# Set ownership
self._run_command(['chown', '-R', f'{username}:{username}', str(ssh_dir)])
def delete_user(self, username: str, remove_home: bool = False) -> None:
"""Delete user account"""
logger.info(f"Deleting user: {username}")
if not self.user_exists(username):
logger.warning(f"User {username} does not exist")
return
cmd = ['userdel']
if remove_home:
cmd.append('-r')
cmd.append(username)
self._run_command(cmd)
logger.info(f"User {username} deleted successfully")
def modify_user_groups(self, username: str, groups: List[str]) -> None:
"""Modify user's group membership"""
logger.info(f"Modifying groups for {username}")
if not self.user_exists(username):
raise UserManagementError(f"User {username} does not exist")
# Set supplementary groups
cmd = ['usermod', '-G', ','.join(groups), username]
self._run_command(cmd)
logger.info(f"Groups updated for {username}")
def list_users(self) -> List[Dict[str, str]]:
"""List all users on the system"""
logger.info("Listing all users")
users = []
with open('/etc/passwd', 'r') as f:
for line in f:
parts = line.strip().split(':')
if int(parts[2]) >= 1000: # Only regular users
users.append({
'username': parts[0],
'uid': parts[2],
'home': parts[5],
'shell': parts[6]
})
return users
def bulk_create_users(self, users_file: str) -> None:
"""Create multiple users from CSV file"""
logger.info(f"Bulk creating users from {users_file}")
import csv
with open(users_file, 'r') as f:
reader = csv.DictReader(f)
for row in reader:
try:
self.create_user(
username=row['username'],
full_name=row.get('full_name', ''),
email=row.get('email', ''),
groups=row.get('groups', '').split(',') if row.get('groups') else None,
ssh_public_key=row.get('ssh_public_key')
)
except Exception as e:
logger.error(f"Failed to create user {row['username']}: {e}")
def main():
parser = argparse.ArgumentParser(description='User account management automation')
parser.add_argument('--config', default='/etc/user_mgmt/config.yaml', help='Config file path')
parser.add_argument('--dry-run', action='store_true', help='Dry run mode')
subparsers = parser.add_subparsers(dest='command', help='Command to execute')
# Create user
create_parser = subparsers.add_parser('create', help='Create user')
create_parser.add_argument('username', help='Username')
create_parser.add_argument('--full-name', help='Full name')
create_parser.add_argument('--email', help='Email address')
create_parser.add_argument('--groups', help='Comma-separated groups')
create_parser.add_argument('--ssh-key', help='SSH public key')
# Delete user
delete_parser = subparsers.add_parser('delete', help='Delete user')
delete_parser.add_argument('username', help='Username')
delete_parser.add_argument('--remove-home', action='store_true', help='Remove home directory')
# List users
list_parser = subparsers.add_parser('list', help='List users')
# Bulk create
bulk_parser = subparsers.add_parser('bulk-create', help='Bulk create users')
bulk_parser.add_argument('csv_file', help='CSV file with user data')
args = parser.parse_args()
# Initialize manager
manager = UserManager(args.config)
manager.dry_run = args.dry_run
# Execute command
try:
if args.command == 'create':
groups = args.groups.split(',') if args.groups else None
manager.create_user(
args.username,
args.full_name or '',
args.email or '',
groups,
args.ssh_key
)
elif args.command == 'delete':
manager.delete_user(args.username, args.remove_home)
elif args.command == 'list':
users = manager.list_users()
for user in users:
print(f"{user['username']}\t{user['uid']}\t{user['home']}")
elif args.command == 'bulk-create':
manager.bulk_create_users(args.csv_file)
else:
parser.print_help()
sys.exit(1)
except UserManagementError as e:
logger.error(f"Operation failed: {e}")
sys.exit(1)
if __name__ == '__main__':
main()
```
## Configuration Management
### Ansible Playbooks
```yaml
# playbook.yml - Web server setup playbook
---
- name: Configure web servers
hosts: webservers
become: yes
vars:
nginx_port: 80
app_user: webapp
app_path: /var/www/myapp
tasks:
# System updates
- name: Update apt cache
apt:
update_cache: yes
cache_valid_time: 3600
when: ansible_os_family == "Debian"
# Install packages
- name: Install required packages
apt:
name:
- nginx
- python3-pip
- git
- ufw
state: present
# Create application user
- name: Create application user
user:
name: "{{ app_user }}"
shell: /bin/bash
create_home: yes
system: no
# Configure firewall
- name: Configure UFW
ufw:
rule: allow
port: "{{ item }}"
proto: tcp
loop:
- 22
- 80
- 443
- name: Enable UFW
ufw:
state: enabled
policy: deny
# Deploy application
- name: Create application directory
file:
path: "{{ app_path }}"
state: directory
owner: "{{ app_user }}"
group: "{{ app_user }}"
mode: '0755'
- name: Clone application repository
git:
repo: 'https://github.com/example/myapp.git'
dest: "{{ app_path }}"
version: main
become_user: "{{ app_user }}"
notify: restart nginx
# Configure Nginx
- name: Deploy Nginx configuration
template:
src: templates/nginx.conf.j2
dest: /etc/nginx/sites-available/myapp
owner: root
group: root
mode: '0644'
notify: restart nginx
- name: Enable Nginx site
file:
src: /etc/nginx/sites-available/myapp
dest: /etc/nginx/sites-enabled/myapp
state: link
notify: restart nginx
- name: Remove default Nginx site
file:
path: /etc/nginx/sites-enabled/default
state: absent
notify: restart nginx
# Install application dependencies
- name: Install Python dependencies
pip:
requirements: "{{ app_path }}/requirements.txt"
virtualenv: "{{ app_path }}/venv"
virtualenv_command: python3 -m venv
become_user: "{{ app_user }}"
# Configure systemd service
- name: Deploy systemd service file
template:
src: templates/myapp.service.j2
dest: /etc/systemd/system/myapp.service
owner: root
group: root
mode: '0644'
notify: restart myapp
- name: Enable and start application service
systemd:
name: myapp
enabled: yes
state: started
daemon_reload: yes
# Monitoring
- name: Install node_exporter
include_role:
name: node_exporter
handlers:
- name: restart nginx
service:
name: nginx
state: restarted
- name: restart myapp
service:
name: myapp
state: restarted
# Nginx template (templates/nginx.conf.j2)
---
server {
listen {{ nginx_port }};
server_name {{ ansible_hostname }};
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /static {
alias {{ app_path }}/static;
}
}
```
### Ansible Inventory
```ini
# inventory/production.ini
[webservers]
web01.example.com ansible_host=10.0.10.20
web02.example.com ansible_host=10.0.10.21
web03.example.com ansible_host=10.0.10.22
[databases]
db01.example.com ansible_host=10.0.30.10
db02.example.com ansible_host=10.0.30.11
[monitoring]
mon01.example.com ansible_host=10.0.99.10
[all:vars]
ansible_user=ansible
ansible_become=yes
ansible_python_interpreter=/usr/bin/python3
[webservers:vars]
nginx_port=80
environment=production
```
### Running Ansible
```bash
# Check syntax
ansible-playbook playbook.yml --syntax-check
# Dry run (check mode)
ansible-playbook -i inventory/production.ini playbook.yml --check
# Run playbook
ansible-playbook -i inventory/production.ini playbook.yml
# Run specific tags
ansible-playbook -i inventory/production.ini playbook.yml --tags "nginx,app"
# Limit to specific hosts
ansible-playbook -i inventory/production.ini playbook.yml --limit web01.example.com
# Run ad-hoc command
ansible webservers -i inventory/production.ini -m shell -a "uptime"
ansible webservers -i inventory/production.ini -m apt -a "name=nginx state=latest" --become
```
## Orchestration Tools
### Terraform for Infrastructure Orchestration
See [infrastructure.md](infrastructure.md#infrastructure-as-code) for detailed Terraform examples.
### Kubernetes Operators (Advanced Automation)
```yaml
# Custom Kubernetes Operator for database backups
# backup-operator.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: backup-operator
namespace: operators
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: backup-operator
rules:
- apiGroups: [""]
resources: ["pods", "secrets"]
verbs: ["get", "list", "create"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list", "create", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: backup-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: backup-operator
subjects:
- kind: ServiceAccount
name: backup-operator
namespace: operators
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backup-operator
namespace: operators
spec:
replicas: 1
selector:
matchLabels:
app: backup-operator
template:
metadata:
labels:
app: backup-operator
spec:
serviceAccountName: backup-operator
containers:
- name: operator
image: myregistry/backup-operator:latest
env:
- name: WATCH_NAMESPACE
value: ""
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "backup-operator"
```
## CI/CD for Infrastructure
### GitOps Workflow
```yaml
# GitOps workflow for infrastructure changes
Step 1: Developer makes change
- Edit Terraform/Ansible files
- Commit to feature branch
- Open pull request
Step 2: Automated validation (CI)
- terraform fmt -check (code formatting)
- terraform validate (syntax check)
- tflint (linting)
- terraform plan (dry run)
- Security scan (Checkov, tfsec)
- Cost estimation (Infracost)
Step 3: Peer review
- Code review by team member
- Review terraform plan output
- Approve or request changes
Step 4: Merge to main
- PR approved and merged
- Triggers deployment pipeline
Step 5: Automated deployment (CD)
- terraform apply (auto-approved for specific changes)
- Or manual approval for high-risk changes
- Post-deployment validation
- Notify team in Slack
Step 6: Monitoring
- Monitor for issues
- Auto-rollback on failure
- Update status page
```
### GitHub Actions for Infrastructure
```yaml
# .github/workflows/terraform.yml
name: Terraform CI/CD
on:
pull_request:
branches: [main]
paths:
- 'terraform/**'
push:
branches: [main]
paths:
- 'terraform/**'
env:
TF_VERSION: 1.6.0
AWS_REGION: us-east-1
jobs:
validate:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
with:
terraform_version: ${{ env.TF_VERSION }}
- name: Terraform Format Check
run: terraform fmt -check -recursive
working-directory: ./terraform
- name: Terraform Init
run: terraform init
working-directory: ./terraform
- name: Terraform Validate
run: terraform validate
working-directory: ./terraform
- name: TFLint
uses: terraform-linters/setup-tflint@v3
with:
tflint_version: latest
- name: Run TFLint
run: tflint --recursive
working-directory: ./terraform
plan:
runs-on: ubuntu-latest
needs: validate
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
with:
terraform_version: ${{ env.TF_VERSION }}
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
- name: Terraform Init
run: terraform init
working-directory: ./terraform
- name: Terraform Plan
run: terraform plan -out=tfplan
working-directory: ./terraform
- name: Comment PR with Plan
uses: actions/github-script@v6
with:
script: |
const fs = require('fs');
const plan = fs.readFileSync('./terraform/tfplan', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## Terraform Plan\n\`\`\`\n${plan}\n\`\`\``
});
apply:
runs-on: ubuntu-latest
needs: validate
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
with:
terraform_version: ${{ env.TF_VERSION }}
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
- name: Terraform Init
run: terraform init
working-directory: ./terraform
- name: Terraform Apply
run: terraform apply -auto-approve
working-directory: ./terraform
- name: Notify Slack
uses: 8398a7/action-slack@v3
with:
status: ${{ job.status }}
text: 'Terraform applied successfully'
webhook_url: ${{ secrets.SLACK_WEBHOOK }}
if: always()
```
## Runbook Automation
### ChatOps Integration
```python
# Slack bot for runbook automation
from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler
import subprocess
import logging
app = App(token="xoxb-your-token")
logging.basicConfig(level=logging.INFO)
# Command: restart service
@app.command("/restart-service")
def restart_service_command(ack, command, respond):
ack()
service_name = command['text']
# Validate service name (whitelist)
allowed_services = ['nginx', 'postgresql', 'redis']
if service_name not in allowed_services:
respond(f"❌ Service '{service_name}' is not allowed. Allowed: {', '.join(allowed_services)}")
return
# Confirm with user
respond(f"Restarting service: {service_name}...")
try:
# Execute restart command
result = subprocess.run(
['ansible', 'webservers', '-m', 'service', '-a', f'name={service_name} state=restarted'],
capture_output=True,
text=True,
timeout=30
)
if result.returncode == 0:
respond(f"✅ Service {service_name} restarted successfully\n```{result.stdout}```")
else:
respond(f"❌ Failed to restart {service_name}\n```{result.stderr}```")
except Exception as e:
respond(f"❌ Error: {str(e)}")
# Command: check server health
@app.command("/health-check")
def health_check_command(ack, command, respond):
ack()
hostname = command['text'] or 'all'
respond(f"Running health check on {hostname}...")
try:
result = subprocess.run(
['ansible', hostname, '-m', 'shell', '-a', '/opt/scripts/health_check.sh'],
capture_output=True,
text=True,
timeout=60
)
respond(f"```{result.stdout}```")
except Exception as e:
respond(f"❌ Error: {str(e)}")
# Interactive approval flow
@app.action("approve_deployment")
def handle_approval(ack, body, client):
ack()
user = body["user"]["id"]
deployment_id = body["actions"][0]["value"]
# Execute deployment
result = execute_deployment(deployment_id)
client.chat_postMessage(
channel=body["channel"]["id"],
text=f"<@{user}> approved deployment {deployment_id}. Status: {result}"
)
def execute_deployment(deployment_id):
# Deployment logic here
return "Success"
if __name__ == "__main__":
handler = SocketModeHandler(app, "xapp-your-app-token")
handler.start()
```
## Self-Healing Systems
### Auto-Remediation Framework
```python
# Self-healing automation framework
import time
import logging
from prometheus_client import start_http_server, Counter, Gauge
import requests
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Metrics
remediation_counter = Counter('remediations_total', 'Total remediations', ['type', 'status'])
system_health = Gauge('system_health', 'System health status (1=healthy, 0=unhealthy)')
class SelfHealingAgent:
"""Auto-remediation agent that monitors and fixes common issues"""
def __init__(self):
self.checks = [
self.check_disk_space,
self.check_service_health,
self.check_memory_usage,
]
def check_disk_space(self):
"""Check disk space and clean up if needed"""
import shutil
stat = shutil.disk_usage('/')
usage_pct = (stat.used / stat.total) * 100
if usage_pct > 85:
logger.warning(f"Disk usage high: {usage_pct:.1f}%")
self.cleanup_disk()
remediation_counter.labels(type='disk_cleanup', status='executed').inc()
return False
return True
def cleanup_disk(self):
"""Clean up disk space"""
import subprocess
logger.info("Cleaning up disk space...")
# Clean apt cache
subprocess.run(['apt-get', 'clean'], check=False)
# Clean old logs
subprocess.run(['find', '/var/log', '-name', '*.gz', '-mtime', '+30', '-delete'], check=False)
# Clean temp files
subprocess.run(['find', '/tmp', '-mtime', '+7', '-delete'], check=False)
logger.info("Disk cleanup complete")
def check_service_health(self):
"""Check if critical services are running"""
import subprocess
services = ['nginx', 'postgresql']
all_healthy = True
for service in services:
result = subprocess.run(
['systemctl', 'is-active', service],
capture_output=True,
text=True
)
if result.returncode != 0:
logger.warning(f"Service {service} is not running")
self.restart_service(service)
all_healthy = False
return all_healthy
def restart_service(self, service_name):
"""Restart a failed service"""
import subprocess
logger.info(f"Restarting service: {service_name}")
try:
subprocess.run(['systemctl', 'restart', service_name], check=True, timeout=30)
logger.info(f"Service {service_name} restarted successfully")
remediation_counter.labels(type='service_restart', status='success').inc()
except Exception as e:
logger.error(f"Failed to restart {service_name}: {e}")
remediation_counter.labels(type='service_restart', status='failed').inc()
def check_memory_usage(self):
"""Check memory usage and kill memory hogs if needed"""
import psutil
mem = psutil.virtual_memory()
usage_pct = mem.percent
if usage_pct > 90:
logger.warning(f"Memory usage critical: {usage_pct:.1f}%")
self.kill_memory_hogs()
remediation_counter.labels(type='memory_cleanup', status='executed').inc()
return False
return True
def kill_memory_hogs(self):
"""Kill processes using excessive memory"""
import psutil
logger.info("Identifying memory hogs...")
# Get processes sorted by memory usage
processes = []
for proc in psutil.process_iter(['pid', 'name', 'memory_percent']):
try:
processes.append(proc.info)
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
processes.sort(key=lambda x: x['memory_percent'], reverse=True)
# Kill top memory consumer (with safety checks)
for proc in processes[:3]: # Check top 3
# Don't kill critical processes
if proc['name'] in ['systemd', 'init', 'sshd']:
continue
if proc['memory_percent'] > 20: # Using more than 20% memory
logger.warning(f"Killing memory hog: {proc['name']} (PID {proc['pid']})")
try:
psutil.Process(proc['pid']).kill()
break
except Exception as e:
logger.error(f"Failed to kill process: {e}")
def run(self, interval=60):
"""Run continuous health checks"""
logger.info("Starting self-healing agent...")
while True:
try:
all_healthy = True
for check in self.checks:
if not check():
all_healthy = False
system_health.set(1 if all_healthy else 0)
time.sleep(interval)
except Exception as e:
logger.error(f"Error in health check loop: {e}")
time.sleep(interval)
if __name__ == '__main__':
# Start Prometheus metrics server
start_http_server(9101)
# Run agent
agent = SelfHealingAgent()
agent.run(interval=60)
```
## Toil Reduction
### Measuring Toil
```yaml
Toil Definition (Google SRE):
- Manual: Requires human intervention
- Repetitive: Done over and over
- Automatable: Can be automated
- Tactical: Interrupt-driven, reactive
- No enduring value: Doesn't improve system
- Scales linearly: Grows with service
Examples of Toil:
- Manually restarting services
- Copying files between servers
- Running deployment scripts manually
- Responding to recurring alerts
- Manual user provisioning
- Generating reports manually
Target: < 50% of time on toil (Google SRE recommendation)
```
### Toil Reduction Strategy
```yaml
Step 1: Identify Toil (Weekly)
- Track time spent on tasks
- Categorize: Toil vs Engineering work
- Prioritize by time impact
Tool: Time tracking spreadsheet
Columns: Task, Time Spent, Frequency, Automatable?, Priority
Step 2: Quantify Impact
- Calculate time spent per month
- Estimate automation effort
- Calculate ROI (see ROI calculator above)
Step 3: Automate High-Impact Toil
- Start with highest ROI
- Build automation incrementally
- Test thoroughly
- Document automation
Step 4: Measure Improvement
- Track toil percentage monthly
- Celebrate wins
- Share automations across team
Step 5: Prevent New Toil
- Question new manual processes
- Design for automation from start
- Code review includes automation check
```
This comprehensive automation guide provides everything needed to reduce toil and improve operational efficiency through automation.
Reference in New Issue View Git Blame Copy Permalink