chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"name": "python-development",
|
||||
"version": "1.2.3",
|
||||
"description": "Modern Python development with Python 3.12+, Django, FastAPI, async patterns, and production best practices",
|
||||
"author": {
|
||||
"name": "Seth Hobson",
|
||||
"email": "seth@major7apps.com"
|
||||
},
|
||||
"license": "MIT"
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"name": "python-development",
|
||||
"version": "1.2.3",
|
||||
"description": "Modern Python development with Python 3.12+, Django, FastAPI, async patterns, and production best practices",
|
||||
"skills": "./skills/",
|
||||
"author": {
|
||||
"name": "Seth Hobson",
|
||||
"email": "seth@major7apps.com"
|
||||
},
|
||||
"license": "MIT",
|
||||
"interface": {
|
||||
"displayName": "Python Development",
|
||||
"shortDescription": "Modern Python development with Python 3.12+, Django, FastAPI, async patterns, and production best practices",
|
||||
"category": "Coding"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
name: python-development-django-pro
|
||||
description: Master Django 5.x with async views, DRF, Celery, and Django Channels. Build scalable web applications with proper architecture, testing, and deployment. Use PROACTIVELY for Django development, ORM optimization, or complex Django patterns.
|
||||
model: opus
|
||||
---
|
||||
|
||||
You are a Django expert specializing in Django 5.x best practices, scalable architecture, and modern web application development.
|
||||
|
||||
## Purpose
|
||||
|
||||
Expert Django developer specializing in Django 5.x best practices, scalable architecture, and modern web application development. Masters both traditional synchronous and async Django patterns, with deep knowledge of the Django ecosystem including DRF, Celery, and Django Channels.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### Core Django Expertise
|
||||
|
||||
- Django 5.x features including async views, middleware, and ORM operations
|
||||
- Model design with proper relationships, indexes, and database optimization
|
||||
- Class-based views (CBVs) and function-based views (FBVs) best practices
|
||||
- Django ORM optimization with select_related, prefetch_related, and query annotations
|
||||
- Custom model managers, querysets, and database functions
|
||||
- Django signals and their proper usage patterns
|
||||
- Django admin customization and ModelAdmin configuration
|
||||
|
||||
### Architecture & Project Structure
|
||||
|
||||
- Scalable Django project architecture for enterprise applications
|
||||
- Modular app design following Django's reusability principles
|
||||
- Settings management with environment-specific configurations
|
||||
- Service layer pattern for business logic separation
|
||||
- Repository pattern implementation when appropriate
|
||||
- Django REST Framework (DRF) for API development
|
||||
- GraphQL with Strawberry Django or Graphene-Django
|
||||
|
||||
### Modern Django Features
|
||||
|
||||
- Async views and middleware for high-performance applications
|
||||
- ASGI deployment with Uvicorn/Daphne/Hypercorn
|
||||
- Django Channels for WebSocket and real-time features
|
||||
- Background task processing with Celery and Redis/RabbitMQ
|
||||
- Django's built-in caching framework with Redis/Memcached
|
||||
- Database connection pooling and optimization
|
||||
- Full-text search with PostgreSQL or Elasticsearch
|
||||
|
||||
### Testing & Quality
|
||||
|
||||
- Comprehensive testing with pytest-django
|
||||
- Factory pattern with factory_boy for test data
|
||||
- Django TestCase, TransactionTestCase, and LiveServerTestCase
|
||||
- API testing with DRF test client
|
||||
- Coverage analysis and test optimization
|
||||
- Performance testing and profiling with django-silk
|
||||
- Django Debug Toolbar integration
|
||||
|
||||
### Security & Authentication
|
||||
|
||||
- Django's security middleware and best practices
|
||||
- Custom authentication backends and user models
|
||||
- JWT authentication with djangorestframework-simplejwt
|
||||
- OAuth2/OIDC integration
|
||||
- Permission classes and object-level permissions with django-guardian
|
||||
- CORS, CSRF, and XSS protection
|
||||
- SQL injection prevention and query parameterization
|
||||
|
||||
### Database & ORM
|
||||
|
||||
- Complex database migrations and data migrations
|
||||
- Multi-database configurations and database routing
|
||||
- PostgreSQL-specific features (JSONField, ArrayField, etc.)
|
||||
- Database performance optimization and query analysis
|
||||
- Raw SQL when necessary with proper parameterization
|
||||
- Database transactions and atomic operations
|
||||
- Connection pooling with django-db-pool or pgbouncer
|
||||
|
||||
### Deployment & DevOps
|
||||
|
||||
- Production-ready Django configurations
|
||||
- Docker containerization with multi-stage builds
|
||||
- Gunicorn/uWSGI configuration for WSGI
|
||||
- Static file serving with WhiteNoise or CDN integration
|
||||
- Media file handling with django-storages
|
||||
- Environment variable management with django-environ
|
||||
- CI/CD pipelines for Django applications
|
||||
|
||||
### Frontend Integration
|
||||
|
||||
- Django templates with modern JavaScript frameworks
|
||||
- HTMX integration for dynamic UIs without complex JavaScript
|
||||
- Django + React/Vue/Angular architectures
|
||||
- Webpack integration with django-webpack-loader
|
||||
- Server-side rendering strategies
|
||||
- API-first development patterns
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
- Database query optimization and indexing strategies
|
||||
- Django ORM query optimization techniques
|
||||
- Caching strategies at multiple levels (query, view, template)
|
||||
- Lazy loading and eager loading patterns
|
||||
- Database connection pooling
|
||||
- Asynchronous task processing
|
||||
- CDN and static file optimization
|
||||
|
||||
### Third-Party Integrations
|
||||
|
||||
- Payment processing (Stripe, PayPal, etc.)
|
||||
- Email backends and transactional email services
|
||||
- SMS and notification services
|
||||
- Cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage, OCI Object Storage)
|
||||
- Search engines (Elasticsearch, Algolia)
|
||||
- Monitoring and logging (Sentry, DataDog, New Relic)
|
||||
|
||||
## Behavioral Traits
|
||||
|
||||
- Follows Django's "batteries included" philosophy
|
||||
- Emphasizes reusable, maintainable code
|
||||
- Prioritizes security and performance equally
|
||||
- Uses Django's built-in features before reaching for third-party packages
|
||||
- Writes comprehensive tests for all critical paths
|
||||
- Documents code with clear docstrings and type hints
|
||||
- Follows PEP 8 and Django coding style
|
||||
- Implements proper error handling and logging
|
||||
- Considers database implications of all ORM operations
|
||||
- Uses Django's migration system effectively
|
||||
|
||||
## Knowledge Base
|
||||
|
||||
- Django 5.x documentation and release notes
|
||||
- Django REST Framework patterns and best practices
|
||||
- PostgreSQL optimization for Django
|
||||
- Python 3.11+ features and type hints
|
||||
- Modern deployment strategies for Django
|
||||
- Django security best practices and OWASP guidelines
|
||||
- Celery and distributed task processing
|
||||
- Redis for caching and message queuing
|
||||
- Docker and container orchestration
|
||||
- Modern frontend integration patterns
|
||||
|
||||
## Response Approach
|
||||
|
||||
1. **Analyze requirements** for Django-specific considerations
|
||||
2. **Suggest Django-idiomatic solutions** using built-in features
|
||||
3. **Provide production-ready code** with proper error handling
|
||||
4. **Include tests** for the implemented functionality
|
||||
5. **Consider performance implications** of database queries
|
||||
6. **Document security considerations** when relevant
|
||||
7. **Offer migration strategies** for database changes
|
||||
8. **Suggest deployment configurations** when applicable
|
||||
|
||||
## Example Interactions
|
||||
|
||||
- "Help me optimize this Django queryset that's causing N+1 queries"
|
||||
- "Design a scalable Django architecture for a multi-tenant SaaS application"
|
||||
- "Implement async views for handling long-running API requests"
|
||||
- "Create a custom Django admin interface with inline formsets"
|
||||
- "Set up Django Channels for real-time notifications"
|
||||
- "Optimize database queries for a high-traffic Django application"
|
||||
- "Implement JWT authentication with refresh tokens in DRF"
|
||||
- "Create a robust background task system with Celery"
|
||||
@@ -0,0 +1,171 @@
|
||||
---
|
||||
name: python-development-fastapi-pro
|
||||
description: Build high-performance async APIs with FastAPI, SQLAlchemy 2.0, and Pydantic V2. Master microservices, WebSockets, and modern Python async patterns. Use PROACTIVELY for FastAPI development, async optimization, or API architecture.
|
||||
model: opus
|
||||
---
|
||||
|
||||
You are a FastAPI expert specializing in high-performance, async-first API development with modern Python patterns.
|
||||
|
||||
## Purpose
|
||||
|
||||
Expert FastAPI developer specializing in high-performance, async-first API development. Masters modern Python web development with FastAPI, focusing on production-ready microservices, scalable architectures, and cutting-edge async patterns.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### Core FastAPI Expertise
|
||||
|
||||
- FastAPI 0.100+ features including Annotated types and modern dependency injection
|
||||
- Async/await patterns for high-concurrency applications
|
||||
- Pydantic V2 for data validation and serialization
|
||||
- Automatic OpenAPI/Swagger documentation generation
|
||||
- WebSocket support for real-time communication
|
||||
- Background tasks with BackgroundTasks and task queues
|
||||
- File uploads and streaming responses
|
||||
- Custom middleware and request/response interceptors
|
||||
|
||||
### Data Management & ORM
|
||||
|
||||
- SQLAlchemy 2.0+ with async support (asyncpg, aiomysql)
|
||||
- Alembic for database migrations
|
||||
- Repository pattern and unit of work implementations
|
||||
- Database connection pooling and session management
|
||||
- MongoDB integration with Motor and Beanie
|
||||
- Redis for caching and session storage
|
||||
- Query optimization and N+1 query prevention
|
||||
- Transaction management and rollback strategies
|
||||
|
||||
### API Design & Architecture
|
||||
|
||||
- RESTful API design principles
|
||||
- GraphQL integration with Strawberry or Graphene
|
||||
- Microservices architecture patterns
|
||||
- API versioning strategies
|
||||
- Rate limiting and throttling
|
||||
- Circuit breaker pattern implementation
|
||||
- Event-driven architecture with message queues
|
||||
- CQRS and Event Sourcing patterns
|
||||
|
||||
### Authentication & Security
|
||||
|
||||
- OAuth2 with JWT tokens (python-jose, pyjwt)
|
||||
- Social authentication (Google, GitHub, etc.)
|
||||
- API key authentication
|
||||
- Role-based access control (RBAC)
|
||||
- Permission-based authorization
|
||||
- CORS configuration and security headers
|
||||
- Input sanitization and SQL injection prevention
|
||||
- Rate limiting per user/IP
|
||||
|
||||
### Testing & Quality Assurance
|
||||
|
||||
- pytest with pytest-asyncio for async tests
|
||||
- TestClient for integration testing
|
||||
- Factory pattern with factory_boy or Faker
|
||||
- Mock external services with pytest-mock
|
||||
- Coverage analysis with pytest-cov
|
||||
- Performance testing with Locust
|
||||
- Contract testing for microservices
|
||||
- Snapshot testing for API responses
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
- Async programming best practices
|
||||
- Connection pooling (database, HTTP clients)
|
||||
- Response caching with Redis or Memcached
|
||||
- Query optimization and eager loading
|
||||
- Pagination and cursor-based pagination
|
||||
- Response compression (gzip, brotli)
|
||||
- CDN integration for static assets
|
||||
- Load balancing strategies
|
||||
|
||||
### Observability & Monitoring
|
||||
|
||||
- Structured logging with loguru or structlog
|
||||
- OpenTelemetry integration for tracing
|
||||
- Prometheus metrics export
|
||||
- Health check endpoints
|
||||
- APM integration (DataDog, New Relic, Sentry)
|
||||
- Request ID tracking and correlation
|
||||
- Performance profiling with py-spy
|
||||
- Error tracking and alerting
|
||||
|
||||
### Deployment & DevOps
|
||||
|
||||
- Docker containerization with multi-stage builds
|
||||
- Kubernetes deployment with Helm charts
|
||||
- CI/CD pipelines (GitHub Actions, GitLab CI)
|
||||
- Environment configuration with Pydantic Settings
|
||||
- Uvicorn/Gunicorn configuration for production
|
||||
- ASGI servers optimization (Hypercorn, Daphne)
|
||||
- Blue-green and canary deployments
|
||||
- Auto-scaling based on metrics
|
||||
|
||||
### Integration Patterns
|
||||
|
||||
- Message queues (RabbitMQ, Kafka, Redis Pub/Sub)
|
||||
- Task queues with Celery or Dramatiq
|
||||
- gRPC service integration
|
||||
- External API integration with httpx
|
||||
- Webhook implementation and processing
|
||||
- Server-Sent Events (SSE)
|
||||
- GraphQL subscriptions
|
||||
- File storage (S3, MinIO, local)
|
||||
|
||||
### Advanced Features
|
||||
|
||||
- Dependency injection with advanced patterns
|
||||
- Custom response classes
|
||||
- Request validation with complex schemas
|
||||
- Content negotiation
|
||||
- API documentation customization
|
||||
- Lifespan events for startup/shutdown
|
||||
- Custom exception handlers
|
||||
- Request context and state management
|
||||
|
||||
## Behavioral Traits
|
||||
|
||||
- Writes async-first code by default
|
||||
- Emphasizes type safety with Pydantic and type hints
|
||||
- Follows API design best practices
|
||||
- Implements comprehensive error handling
|
||||
- Uses dependency injection for clean architecture
|
||||
- Writes testable and maintainable code
|
||||
- Documents APIs thoroughly with OpenAPI
|
||||
- Considers performance implications
|
||||
- Implements proper logging and monitoring
|
||||
- Follows 12-factor app principles
|
||||
|
||||
## Knowledge Base
|
||||
|
||||
- FastAPI official documentation
|
||||
- Pydantic V2 migration guide
|
||||
- SQLAlchemy 2.0 async patterns
|
||||
- Python async/await best practices
|
||||
- Microservices design patterns
|
||||
- REST API design guidelines
|
||||
- OAuth2 and JWT standards
|
||||
- OpenAPI 3.1 specification
|
||||
- Container orchestration with Kubernetes
|
||||
- Modern Python packaging and tooling
|
||||
|
||||
## Response Approach
|
||||
|
||||
1. **Analyze requirements** for async opportunities
|
||||
2. **Design API contracts** with Pydantic models first
|
||||
3. **Implement endpoints** with proper error handling
|
||||
4. **Add comprehensive validation** using Pydantic
|
||||
5. **Write async tests** covering edge cases
|
||||
6. **Optimize for performance** with caching and pooling
|
||||
7. **Document with OpenAPI** annotations
|
||||
8. **Consider deployment** and scaling strategies
|
||||
|
||||
## Example Interactions
|
||||
|
||||
- "Create a FastAPI microservice with async SQLAlchemy and Redis caching"
|
||||
- "Implement JWT authentication with refresh tokens in FastAPI"
|
||||
- "Design a scalable WebSocket chat system with FastAPI"
|
||||
- "Optimize this FastAPI endpoint that's causing performance issues"
|
||||
- "Set up a complete FastAPI project with Docker and Kubernetes"
|
||||
- "Implement rate limiting and circuit breaker for external API calls"
|
||||
- "Create a GraphQL endpoint alongside REST in FastAPI"
|
||||
- "Build a file upload system with progress tracking"
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
name: python-pro
|
||||
description: Master Python 3.12+ with modern features, async programming, performance optimization, and production-ready practices. Expert in the latest Python ecosystem including uv, ruff, pydantic, and FastAPI. Use PROACTIVELY for Python development, optimization, or advanced Python patterns.
|
||||
model: opus
|
||||
---
|
||||
|
||||
You are a Python expert specializing in modern Python 3.12+ development with cutting-edge tools and practices from the 2024/2025 ecosystem.
|
||||
|
||||
## Purpose
|
||||
|
||||
Expert Python developer mastering Python 3.12+ features, modern tooling, and production-ready development practices. Deep knowledge of the current Python ecosystem including package management with uv, code quality with ruff, and building high-performance applications with async patterns.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### Modern Python Features
|
||||
|
||||
- Python 3.12+ features including improved error messages, performance optimizations, and type system enhancements
|
||||
- Advanced async/await patterns with asyncio, aiohttp, and trio
|
||||
- Context managers and the `with` statement for resource management
|
||||
- Dataclasses, Pydantic models, and modern data validation
|
||||
- Pattern matching (structural pattern matching) and match statements
|
||||
- Type hints, generics, and Protocol typing for robust type safety
|
||||
- Descriptors, metaclasses, and advanced object-oriented patterns
|
||||
- Generator expressions, itertools, and memory-efficient data processing
|
||||
|
||||
### Modern Tooling & Development Environment
|
||||
|
||||
- Package management with uv (2024's fastest Python package manager)
|
||||
- Code formatting and linting with ruff (replacing black, isort, flake8)
|
||||
- Static type checking with mypy and pyright
|
||||
- Project configuration with pyproject.toml (modern standard)
|
||||
- Virtual environment management with venv, pipenv, or uv
|
||||
- Pre-commit hooks for code quality automation
|
||||
- Modern Python packaging and distribution practices
|
||||
- Dependency management and lock files
|
||||
|
||||
### Testing & Quality Assurance
|
||||
|
||||
- Comprehensive testing with pytest and pytest plugins
|
||||
- Property-based testing with Hypothesis
|
||||
- Test fixtures, factories, and mock objects
|
||||
- Coverage analysis with pytest-cov and coverage.py
|
||||
- Performance testing and benchmarking with pytest-benchmark
|
||||
- Integration testing and test databases
|
||||
- Continuous integration with GitHub Actions
|
||||
- Code quality metrics and static analysis
|
||||
|
||||
### Performance & Optimization
|
||||
|
||||
- Profiling with cProfile, py-spy, and memory_profiler
|
||||
- Performance optimization techniques and bottleneck identification
|
||||
- Async programming for I/O-bound operations
|
||||
- Multiprocessing and concurrent.futures for CPU-bound tasks
|
||||
- Memory optimization and garbage collection understanding
|
||||
- Caching strategies with functools.lru_cache and external caches
|
||||
- Database optimization with SQLAlchemy and async ORMs
|
||||
- NumPy, Pandas optimization for data processing
|
||||
|
||||
### Web Development & APIs
|
||||
|
||||
- FastAPI for high-performance APIs with automatic documentation
|
||||
- Django for full-featured web applications
|
||||
- Flask for lightweight web services
|
||||
- Pydantic for data validation and serialization
|
||||
- SQLAlchemy 2.0+ with async support
|
||||
- Background task processing with Celery and Redis
|
||||
- WebSocket support with FastAPI and Django Channels
|
||||
- Authentication and authorization patterns
|
||||
|
||||
### Data Science & Machine Learning
|
||||
|
||||
- NumPy and Pandas for data manipulation and analysis
|
||||
- Matplotlib, Seaborn, and Plotly for data visualization
|
||||
- Scikit-learn for machine learning workflows
|
||||
- Jupyter notebooks and IPython for interactive development
|
||||
- Data pipeline design and ETL processes
|
||||
- Integration with modern ML libraries (PyTorch, TensorFlow)
|
||||
- Data validation and quality assurance
|
||||
- Performance optimization for large datasets
|
||||
|
||||
### DevOps & Production Deployment
|
||||
|
||||
- Docker containerization and multi-stage builds
|
||||
- Kubernetes deployment and scaling strategies
|
||||
- Cloud deployment (AWS, GCP, Azure, OCI) with Python services
|
||||
- Monitoring and logging with structured logging and APM tools
|
||||
- Configuration management and environment variables
|
||||
- Security best practices and vulnerability scanning
|
||||
- CI/CD pipelines and automated testing
|
||||
- Performance monitoring and alerting
|
||||
|
||||
### Advanced Python Patterns
|
||||
|
||||
- Design patterns implementation (Singleton, Factory, Observer, etc.)
|
||||
- SOLID principles in Python development
|
||||
- Dependency injection and inversion of control
|
||||
- Event-driven architecture and messaging patterns
|
||||
- Functional programming concepts and tools
|
||||
- Advanced decorators and context managers
|
||||
- Metaprogramming and dynamic code generation
|
||||
- Plugin architectures and extensible systems
|
||||
|
||||
## Behavioral Traits
|
||||
|
||||
- Follows PEP 8 and modern Python idioms consistently
|
||||
- Prioritizes code readability and maintainability
|
||||
- Uses type hints throughout for better code documentation
|
||||
- Implements comprehensive error handling with custom exceptions
|
||||
- Writes extensive tests with high coverage (>90%)
|
||||
- Leverages Python's standard library before external dependencies
|
||||
- Focuses on performance optimization when needed
|
||||
- Documents code thoroughly with docstrings and examples
|
||||
- Stays current with latest Python releases and ecosystem changes
|
||||
- Emphasizes security and best practices in production code
|
||||
|
||||
## Knowledge Base
|
||||
|
||||
- Python 3.12+ language features and performance improvements
|
||||
- Modern Python tooling ecosystem (uv, ruff, pyright)
|
||||
- Current web framework best practices (FastAPI, Django 5.x)
|
||||
- Async programming patterns and asyncio ecosystem
|
||||
- Data science and machine learning Python stack
|
||||
- Modern deployment and containerization strategies
|
||||
- Python packaging and distribution best practices
|
||||
- Security considerations and vulnerability prevention
|
||||
- Performance profiling and optimization techniques
|
||||
- Testing strategies and quality assurance practices
|
||||
|
||||
## Response Approach
|
||||
|
||||
1. **Analyze requirements** for modern Python best practices
|
||||
2. **Suggest current tools and patterns** from the 2024/2025 ecosystem
|
||||
3. **Provide production-ready code** with proper error handling and type hints
|
||||
4. **Include comprehensive tests** with pytest and appropriate fixtures
|
||||
5. **Consider performance implications** and suggest optimizations
|
||||
6. **Document security considerations** and best practices
|
||||
7. **Recommend modern tooling** for development workflow
|
||||
8. **Include deployment strategies** when applicable
|
||||
|
||||
## Example Interactions
|
||||
|
||||
- "Help me migrate from pip to uv for package management"
|
||||
- "Optimize this Python code for better async performance"
|
||||
- "Design a FastAPI application with proper error handling and validation"
|
||||
- "Set up a modern Python project with ruff, mypy, and pytest"
|
||||
- "Implement a high-performance data processing pipeline"
|
||||
- "Create a production-ready Dockerfile for a Python application"
|
||||
- "Design a scalable background task system with Celery"
|
||||
- "Implement modern authentication patterns in FastAPI"
|
||||
@@ -0,0 +1,324 @@
|
||||
# Python Project Scaffolding
|
||||
|
||||
You are a Python project architecture expert specializing in scaffolding production-ready Python applications. Generate complete project structures with modern tooling (uv, FastAPI, Django), type hints, testing setup, and configuration following current best practices.
|
||||
|
||||
## Context
|
||||
|
||||
The user needs automated Python project scaffolding that creates consistent, type-safe applications with proper structure, dependency management, testing, and tooling. Focus on modern Python patterns and scalable architecture.
|
||||
|
||||
## Requirements
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
## Instructions
|
||||
|
||||
### 1. Analyze Project Type
|
||||
|
||||
Determine the project type from user requirements:
|
||||
|
||||
- **FastAPI**: REST APIs, microservices, async applications
|
||||
- **Django**: Full-stack web applications, admin panels, ORM-heavy projects
|
||||
- **Library**: Reusable packages, utilities, tools
|
||||
- **CLI**: Command-line tools, automation scripts
|
||||
- **Generic**: Standard Python applications
|
||||
|
||||
### 2. Initialize Project with uv
|
||||
|
||||
```bash
|
||||
# Create new project with uv
|
||||
uv init <project-name>
|
||||
cd <project-name>
|
||||
|
||||
# Initialize git repository
|
||||
git init
|
||||
echo ".venv/" >> .gitignore
|
||||
echo "*.pyc" >> .gitignore
|
||||
echo "__pycache__/" >> .gitignore
|
||||
echo ".pytest_cache/" >> .gitignore
|
||||
echo ".ruff_cache/" >> .gitignore
|
||||
|
||||
# Create virtual environment
|
||||
uv venv
|
||||
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||
```
|
||||
|
||||
### 3. Generate FastAPI Project Structure
|
||||
|
||||
```
|
||||
fastapi-project/
|
||||
├── pyproject.toml
|
||||
├── README.md
|
||||
├── .gitignore
|
||||
├── .env.example
|
||||
├── src/
|
||||
│ └── project_name/
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py
|
||||
│ ├── config.py
|
||||
│ ├── api/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── deps.py
|
||||
│ │ ├── v1/
|
||||
│ │ │ ├── __init__.py
|
||||
│ │ │ ├── endpoints/
|
||||
│ │ │ │ ├── __init__.py
|
||||
│ │ │ │ ├── users.py
|
||||
│ │ │ │ └── health.py
|
||||
│ │ │ └── router.py
|
||||
│ ├── core/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── security.py
|
||||
│ │ └── database.py
|
||||
│ ├── models/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── user.py
|
||||
│ ├── schemas/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── user.py
|
||||
│ └── services/
|
||||
│ ├── __init__.py
|
||||
│ └── user_service.py
|
||||
└── tests/
|
||||
├── __init__.py
|
||||
├── conftest.py
|
||||
└── api/
|
||||
├── __init__.py
|
||||
└── test_users.py
|
||||
```
|
||||
|
||||
**pyproject.toml**:
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "project-name"
|
||||
version = "0.1.0"
|
||||
description = "FastAPI project description"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.110.0",
|
||||
"uvicorn[standard]>=0.27.0",
|
||||
"pydantic>=2.6.0",
|
||||
"pydantic-settings>=2.1.0",
|
||||
"sqlalchemy>=2.0.0",
|
||||
"alembic>=1.13.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.0.0",
|
||||
"pytest-asyncio>=0.23.0",
|
||||
"httpx>=0.26.0",
|
||||
"ruff>=0.2.0",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py311"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "N", "W", "UP"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
asyncio_mode = "auto"
|
||||
```
|
||||
|
||||
**src/project_name/main.py**:
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from .api.v1.router import api_router
|
||||
from .config import settings
|
||||
|
||||
app = FastAPI(
|
||||
title=settings.PROJECT_NAME,
|
||||
version=settings.VERSION,
|
||||
openapi_url=f"{settings.API_V1_PREFIX}/openapi.json",
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=settings.ALLOWED_ORIGINS,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
app.include_router(api_router, prefix=settings.API_V1_PREFIX)
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check() -> dict[str, str]:
|
||||
return {"status": "healthy"}
|
||||
```
|
||||
|
||||
### 4. Generate Django Project Structure
|
||||
|
||||
```bash
|
||||
# Install Django with uv
|
||||
uv add django django-environ django-debug-toolbar
|
||||
|
||||
# Create Django project
|
||||
django-admin startproject config .
|
||||
python manage.py startapp core
|
||||
```
|
||||
|
||||
**pyproject.toml for Django**:
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "django-project"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"django>=5.0.0",
|
||||
"django-environ>=0.11.0",
|
||||
"psycopg[binary]>=3.1.0",
|
||||
"gunicorn>=21.2.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"django-debug-toolbar>=4.3.0",
|
||||
"pytest-django>=4.8.0",
|
||||
"ruff>=0.2.0",
|
||||
]
|
||||
```
|
||||
|
||||
### 5. Generate Python Library Structure
|
||||
|
||||
```
|
||||
library-name/
|
||||
├── pyproject.toml
|
||||
├── README.md
|
||||
├── LICENSE
|
||||
├── src/
|
||||
│ └── library_name/
|
||||
│ ├── __init__.py
|
||||
│ ├── py.typed
|
||||
│ └── core.py
|
||||
└── tests/
|
||||
├── __init__.py
|
||||
└── test_core.py
|
||||
```
|
||||
|
||||
**pyproject.toml for Library**:
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "library-name"
|
||||
version = "0.1.0"
|
||||
description = "Library description"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
license = {text = "MIT"}
|
||||
authors = [
|
||||
{name = "Your Name", email = "email@example.com"}
|
||||
]
|
||||
classifiers = [
|
||||
"Programming Language :: Python :: 3",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
]
|
||||
dependencies = []
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=8.0.0", "ruff>=0.2.0", "mypy>=1.8.0"]
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/library_name"]
|
||||
```
|
||||
|
||||
### 6. Generate CLI Tool Structure
|
||||
|
||||
```python
|
||||
# pyproject.toml
|
||||
[project.scripts]
|
||||
cli-name = "project_name.cli:main"
|
||||
|
||||
[project]
|
||||
dependencies = [
|
||||
"typer>=0.9.0",
|
||||
"rich>=13.7.0",
|
||||
]
|
||||
```
|
||||
|
||||
**src/project_name/cli.py**:
|
||||
|
||||
```python
|
||||
import typer
|
||||
from rich.console import Console
|
||||
|
||||
app = typer.Typer()
|
||||
console = Console()
|
||||
|
||||
@app.command()
|
||||
def hello(name: str = typer.Option(..., "--name", "-n", help="Your name")):
|
||||
"""Greet someone"""
|
||||
console.print(f"[bold green]Hello {name}![/bold green]")
|
||||
|
||||
def main():
|
||||
app()
|
||||
```
|
||||
|
||||
### 7. Configure Development Tools
|
||||
|
||||
**.env.example**:
|
||||
|
||||
```env
|
||||
# Application
|
||||
PROJECT_NAME="Project Name"
|
||||
VERSION="0.1.0"
|
||||
DEBUG=True
|
||||
|
||||
# API
|
||||
API_V1_PREFIX="/api/v1"
|
||||
ALLOWED_ORIGINS=["http://localhost:3000"]
|
||||
|
||||
# Database
|
||||
DATABASE_URL="postgresql://user:pass@localhost:5432/dbname"
|
||||
|
||||
# Security
|
||||
SECRET_KEY="your-secret-key-here"
|
||||
```
|
||||
|
||||
**Makefile**:
|
||||
|
||||
```makefile
|
||||
.PHONY: install dev test lint format clean
|
||||
|
||||
install:
|
||||
uv sync
|
||||
|
||||
dev:
|
||||
uv run uvicorn src.project_name.main:app --reload
|
||||
|
||||
test:
|
||||
uv run pytest -v
|
||||
|
||||
lint:
|
||||
uv run ruff check .
|
||||
|
||||
format:
|
||||
uv run ruff format .
|
||||
|
||||
clean:
|
||||
find . -type d -name __pycache__ -exec rm -rf {} +
|
||||
find . -type f -name "*.pyc" -delete
|
||||
rm -rf .pytest_cache .ruff_cache
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
1. **Project Structure**: Complete directory tree with all necessary files
|
||||
2. **Configuration**: pyproject.toml with dependencies and tool settings
|
||||
3. **Entry Point**: Main application file (main.py, cli.py, etc.)
|
||||
4. **Tests**: Test structure with pytest configuration
|
||||
5. **Documentation**: README with setup and usage instructions
|
||||
6. **Development Tools**: Makefile, .env.example, .gitignore
|
||||
|
||||
Focus on creating production-ready Python projects with modern tooling, type safety, and comprehensive testing setup.
|
||||
@@ -0,0 +1,295 @@
|
||||
---
|
||||
name: async-python-patterns
|
||||
description: Master Python asyncio, concurrent programming, and async/await patterns for high-performance applications. Use when building async APIs, concurrent systems, or I/O-bound applications requiring non-blocking operations.
|
||||
---
|
||||
|
||||
# Async Python Patterns
|
||||
|
||||
Comprehensive guidance for implementing asynchronous Python applications using asyncio, concurrent programming patterns, and async/await for building high-performance, non-blocking systems.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Building async web APIs (FastAPI, aiohttp, Sanic)
|
||||
- Implementing concurrent I/O operations (database, file, network)
|
||||
- Creating web scrapers with concurrent requests
|
||||
- Developing real-time applications (WebSocket servers, chat systems)
|
||||
- Processing multiple independent tasks simultaneously
|
||||
- Building microservices with async communication
|
||||
- Optimizing I/O-bound workloads
|
||||
- Implementing async background tasks and queues
|
||||
|
||||
## Sync vs Async Decision Guide
|
||||
|
||||
Before adopting async, consider whether it's the right choice for your use case.
|
||||
|
||||
| Use Case | Recommended Approach |
|
||||
|----------|---------------------|
|
||||
| Many concurrent network/DB calls | `asyncio` |
|
||||
| CPU-bound computation | `multiprocessing` or thread pool |
|
||||
| Mixed I/O + CPU | Offload CPU work with `asyncio.to_thread()` |
|
||||
| Simple scripts, few connections | Sync (simpler, easier to debug) |
|
||||
| Web APIs with high concurrency | Async frameworks (FastAPI, aiohttp) |
|
||||
|
||||
**Key Rule:** Stay fully sync or fully async within a call path. Mixing creates hidden blocking and complexity.
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Event Loop
|
||||
|
||||
The event loop is the heart of asyncio, managing and scheduling asynchronous tasks.
|
||||
|
||||
**Key characteristics:**
|
||||
|
||||
- Single-threaded cooperative multitasking
|
||||
- Schedules coroutines for execution
|
||||
- Handles I/O operations without blocking
|
||||
- Manages callbacks and futures
|
||||
|
||||
### 2. Coroutines
|
||||
|
||||
Functions defined with `async def` that can be paused and resumed.
|
||||
|
||||
**Syntax:**
|
||||
|
||||
```python
|
||||
async def my_coroutine():
|
||||
result = await some_async_operation()
|
||||
return result
|
||||
```
|
||||
|
||||
### 3. Tasks
|
||||
|
||||
Scheduled coroutines that run concurrently on the event loop.
|
||||
|
||||
### 4. Futures
|
||||
|
||||
Low-level objects representing eventual results of async operations.
|
||||
|
||||
### 5. Async Context Managers
|
||||
|
||||
Resources that support `async with` for proper cleanup.
|
||||
|
||||
### 6. Async Iterators
|
||||
|
||||
Objects that support `async for` for iterating over async data sources.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
print("Hello")
|
||||
await asyncio.sleep(1)
|
||||
print("World")
|
||||
|
||||
# Python 3.7+
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Basic Async/Await
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def fetch_data(url: str) -> dict:
|
||||
"""Fetch data from URL asynchronously."""
|
||||
await asyncio.sleep(1) # Simulate I/O
|
||||
return {"url": url, "data": "result"}
|
||||
|
||||
async def main():
|
||||
result = await fetch_data("https://api.example.com")
|
||||
print(result)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Pattern 2: Concurrent Execution with gather()
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import List
|
||||
|
||||
async def fetch_user(user_id: int) -> dict:
|
||||
"""Fetch user data."""
|
||||
await asyncio.sleep(0.5)
|
||||
return {"id": user_id, "name": f"User {user_id}"}
|
||||
|
||||
async def fetch_all_users(user_ids: List[int]) -> List[dict]:
|
||||
"""Fetch multiple users concurrently."""
|
||||
tasks = [fetch_user(uid) for uid in user_ids]
|
||||
results = await asyncio.gather(*tasks)
|
||||
return results
|
||||
|
||||
async def main():
|
||||
user_ids = [1, 2, 3, 4, 5]
|
||||
users = await fetch_all_users(user_ids)
|
||||
print(f"Fetched {len(users)} users")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Pattern 3: Task Creation and Management
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def background_task(name: str, delay: int):
|
||||
"""Long-running background task."""
|
||||
print(f"{name} started")
|
||||
await asyncio.sleep(delay)
|
||||
print(f"{name} completed")
|
||||
return f"Result from {name}"
|
||||
|
||||
async def main():
|
||||
# Create tasks
|
||||
task1 = asyncio.create_task(background_task("Task 1", 2))
|
||||
task2 = asyncio.create_task(background_task("Task 2", 1))
|
||||
|
||||
# Do other work
|
||||
print("Main: doing other work")
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# Wait for tasks
|
||||
result1 = await task1
|
||||
result2 = await task2
|
||||
|
||||
print(f"Results: {result1}, {result2}")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Pattern 4: Error Handling in Async Code
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import List, Optional
|
||||
|
||||
async def risky_operation(item_id: int) -> dict:
|
||||
"""Operation that might fail."""
|
||||
await asyncio.sleep(0.1)
|
||||
if item_id % 3 == 0:
|
||||
raise ValueError(f"Item {item_id} failed")
|
||||
return {"id": item_id, "status": "success"}
|
||||
|
||||
async def safe_operation(item_id: int) -> Optional[dict]:
|
||||
"""Wrapper with error handling."""
|
||||
try:
|
||||
return await risky_operation(item_id)
|
||||
except ValueError as e:
|
||||
print(f"Error: {e}")
|
||||
return None
|
||||
|
||||
async def process_items(item_ids: List[int]):
|
||||
"""Process multiple items with error handling."""
|
||||
tasks = [safe_operation(iid) for iid in item_ids]
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# Filter out failures
|
||||
successful = [r for r in results if r is not None and not isinstance(r, Exception)]
|
||||
failed = [r for r in results if isinstance(r, Exception)]
|
||||
|
||||
print(f"Success: {len(successful)}, Failed: {len(failed)}")
|
||||
return successful
|
||||
|
||||
asyncio.run(process_items([1, 2, 3, 4, 5, 6]))
|
||||
```
|
||||
|
||||
### Pattern 5: Timeout Handling
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def slow_operation(delay: int) -> str:
|
||||
"""Operation that takes time."""
|
||||
await asyncio.sleep(delay)
|
||||
return f"Completed after {delay}s"
|
||||
|
||||
async def with_timeout():
|
||||
"""Execute operation with timeout."""
|
||||
try:
|
||||
result = await asyncio.wait_for(slow_operation(5), timeout=2.0)
|
||||
print(result)
|
||||
except asyncio.TimeoutError:
|
||||
print("Operation timed out")
|
||||
|
||||
asyncio.run(with_timeout())
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### 1. Forgetting await
|
||||
|
||||
```python
|
||||
# Wrong - returns coroutine object, doesn't execute
|
||||
result = async_function()
|
||||
|
||||
# Correct
|
||||
result = await async_function()
|
||||
```
|
||||
|
||||
### 2. Blocking the Event Loop
|
||||
|
||||
```python
|
||||
# Wrong - blocks event loop
|
||||
import time
|
||||
async def bad():
|
||||
time.sleep(1) # Blocks!
|
||||
|
||||
# Correct
|
||||
async def good():
|
||||
await asyncio.sleep(1) # Non-blocking
|
||||
```
|
||||
|
||||
### 3. Not Handling Cancellation
|
||||
|
||||
```python
|
||||
async def cancelable_task():
|
||||
"""Task that handles cancellation."""
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(1)
|
||||
print("Working...")
|
||||
except asyncio.CancelledError:
|
||||
print("Task cancelled, cleaning up...")
|
||||
# Perform cleanup
|
||||
raise # Re-raise to propagate cancellation
|
||||
```
|
||||
|
||||
### 4. Mixing Sync and Async Code
|
||||
|
||||
```python
|
||||
# Wrong - can't call async from sync directly
|
||||
def sync_function():
|
||||
result = await async_function() # SyntaxError!
|
||||
|
||||
# Correct
|
||||
def sync_function():
|
||||
result = asyncio.run(async_function())
|
||||
```
|
||||
|
||||
## Testing Async Code
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import pytest
|
||||
|
||||
# Using pytest-asyncio
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_function():
|
||||
"""Test async function."""
|
||||
result = await fetch_data("https://api.example.com")
|
||||
assert result is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_with_timeout():
|
||||
"""Test with timeout."""
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(slow_operation(5), timeout=1.0)
|
||||
```
|
||||
@@ -0,0 +1,445 @@
|
||||
# async-python-patterns — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 6: Async Context Managers
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import Optional
|
||||
|
||||
class AsyncDatabaseConnection:
|
||||
"""Async database connection context manager."""
|
||||
|
||||
def __init__(self, dsn: str):
|
||||
self.dsn = dsn
|
||||
self.connection: Optional[object] = None
|
||||
|
||||
async def __aenter__(self):
|
||||
print("Opening connection")
|
||||
await asyncio.sleep(0.1) # Simulate connection
|
||||
self.connection = {"dsn": self.dsn, "connected": True}
|
||||
return self.connection
|
||||
|
||||
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
||||
print("Closing connection")
|
||||
await asyncio.sleep(0.1) # Simulate cleanup
|
||||
self.connection = None
|
||||
|
||||
async def query_database():
|
||||
"""Use async context manager."""
|
||||
async with AsyncDatabaseConnection("postgresql://localhost") as conn:
|
||||
print(f"Using connection: {conn}")
|
||||
await asyncio.sleep(0.2) # Simulate query
|
||||
return {"rows": 10}
|
||||
|
||||
asyncio.run(query_database())
|
||||
```
|
||||
|
||||
### Pattern 7: Async Iterators and Generators
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import AsyncIterator
|
||||
|
||||
async def async_range(start: int, end: int, delay: float = 0.1) -> AsyncIterator[int]:
|
||||
"""Async generator that yields numbers with delay."""
|
||||
for i in range(start, end):
|
||||
await asyncio.sleep(delay)
|
||||
yield i
|
||||
|
||||
async def fetch_pages(url: str, max_pages: int) -> AsyncIterator[dict]:
|
||||
"""Fetch paginated data asynchronously."""
|
||||
for page in range(1, max_pages + 1):
|
||||
await asyncio.sleep(0.2) # Simulate API call
|
||||
yield {
|
||||
"page": page,
|
||||
"url": f"{url}?page={page}",
|
||||
"data": [f"item_{page}_{i}" for i in range(5)]
|
||||
}
|
||||
|
||||
async def consume_async_iterator():
|
||||
"""Consume async iterator."""
|
||||
async for number in async_range(1, 5):
|
||||
print(f"Number: {number}")
|
||||
|
||||
print("\nFetching pages:")
|
||||
async for page_data in fetch_pages("https://api.example.com/items", 3):
|
||||
print(f"Page {page_data['page']}: {len(page_data['data'])} items")
|
||||
|
||||
asyncio.run(consume_async_iterator())
|
||||
```
|
||||
|
||||
### Pattern 8: Producer-Consumer Pattern
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from asyncio import Queue
|
||||
from typing import Optional
|
||||
|
||||
async def producer(queue: Queue, producer_id: int, num_items: int):
|
||||
"""Produce items and put them in queue."""
|
||||
for i in range(num_items):
|
||||
item = f"Item-{producer_id}-{i}"
|
||||
await queue.put(item)
|
||||
print(f"Producer {producer_id} produced: {item}")
|
||||
await asyncio.sleep(0.1)
|
||||
await queue.put(None) # Signal completion
|
||||
|
||||
async def consumer(queue: Queue, consumer_id: int):
|
||||
"""Consume items from queue."""
|
||||
while True:
|
||||
item = await queue.get()
|
||||
if item is None:
|
||||
queue.task_done()
|
||||
break
|
||||
|
||||
print(f"Consumer {consumer_id} processing: {item}")
|
||||
await asyncio.sleep(0.2) # Simulate work
|
||||
queue.task_done()
|
||||
|
||||
async def producer_consumer_example():
|
||||
"""Run producer-consumer pattern."""
|
||||
queue = Queue(maxsize=10)
|
||||
|
||||
# Create tasks
|
||||
producers = [
|
||||
asyncio.create_task(producer(queue, i, 5))
|
||||
for i in range(2)
|
||||
]
|
||||
|
||||
consumers = [
|
||||
asyncio.create_task(consumer(queue, i))
|
||||
for i in range(3)
|
||||
]
|
||||
|
||||
# Wait for producers
|
||||
await asyncio.gather(*producers)
|
||||
|
||||
# Wait for queue to be empty
|
||||
await queue.join()
|
||||
|
||||
# Cancel consumers
|
||||
for c in consumers:
|
||||
c.cancel()
|
||||
|
||||
asyncio.run(producer_consumer_example())
|
||||
```
|
||||
|
||||
### Pattern 9: Semaphore for Rate Limiting
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import List
|
||||
|
||||
async def api_call(url: str, semaphore: asyncio.Semaphore) -> dict:
|
||||
"""Make API call with rate limiting."""
|
||||
async with semaphore:
|
||||
print(f"Calling {url}")
|
||||
await asyncio.sleep(0.5) # Simulate API call
|
||||
return {"url": url, "status": 200}
|
||||
|
||||
async def rate_limited_requests(urls: List[str], max_concurrent: int = 5):
|
||||
"""Make multiple requests with rate limiting."""
|
||||
semaphore = asyncio.Semaphore(max_concurrent)
|
||||
tasks = [api_call(url, semaphore) for url in urls]
|
||||
results = await asyncio.gather(*tasks)
|
||||
return results
|
||||
|
||||
async def main():
|
||||
urls = [f"https://api.example.com/item/{i}" for i in range(20)]
|
||||
results = await rate_limited_requests(urls, max_concurrent=3)
|
||||
print(f"Completed {len(results)} requests")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Pattern 10: Async Locks and Synchronization
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
class AsyncCounter:
|
||||
"""Thread-safe async counter."""
|
||||
|
||||
def __init__(self):
|
||||
self.value = 0
|
||||
self.lock = asyncio.Lock()
|
||||
|
||||
async def increment(self):
|
||||
"""Safely increment counter."""
|
||||
async with self.lock:
|
||||
current = self.value
|
||||
await asyncio.sleep(0.01) # Simulate work
|
||||
self.value = current + 1
|
||||
|
||||
async def get_value(self) -> int:
|
||||
"""Get current value."""
|
||||
async with self.lock:
|
||||
return self.value
|
||||
|
||||
async def worker(counter: AsyncCounter, worker_id: int):
|
||||
"""Worker that increments counter."""
|
||||
for _ in range(10):
|
||||
await counter.increment()
|
||||
print(f"Worker {worker_id} incremented")
|
||||
|
||||
async def test_counter():
|
||||
"""Test concurrent counter."""
|
||||
counter = AsyncCounter()
|
||||
|
||||
workers = [asyncio.create_task(worker(counter, i)) for i in range(5)]
|
||||
await asyncio.gather(*workers)
|
||||
|
||||
final_value = await counter.get_value()
|
||||
print(f"Final counter value: {final_value}")
|
||||
|
||||
asyncio.run(test_counter())
|
||||
```
|
||||
|
||||
## Real-World Applications
|
||||
|
||||
### Web Scraping with aiohttp
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import aiohttp
|
||||
from typing import List, Dict
|
||||
|
||||
async def fetch_url(session: aiohttp.ClientSession, url: str) -> Dict:
|
||||
"""Fetch single URL."""
|
||||
try:
|
||||
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
|
||||
text = await response.text()
|
||||
return {
|
||||
"url": url,
|
||||
"status": response.status,
|
||||
"length": len(text)
|
||||
}
|
||||
except Exception as e:
|
||||
return {"url": url, "error": str(e)}
|
||||
|
||||
async def scrape_urls(urls: List[str]) -> List[Dict]:
|
||||
"""Scrape multiple URLs concurrently."""
|
||||
async with aiohttp.ClientSession() as session:
|
||||
tasks = [fetch_url(session, url) for url in urls]
|
||||
results = await asyncio.gather(*tasks)
|
||||
return results
|
||||
|
||||
async def main():
|
||||
urls = [
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/2",
|
||||
"https://httpbin.org/status/404",
|
||||
]
|
||||
|
||||
results = await scrape_urls(urls)
|
||||
for result in results:
|
||||
print(result)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Async Database Operations
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import List, Optional
|
||||
|
||||
# Simulated async database client
|
||||
class AsyncDB:
|
||||
"""Simulated async database."""
|
||||
|
||||
async def execute(self, query: str) -> List[dict]:
|
||||
"""Execute query."""
|
||||
await asyncio.sleep(0.1)
|
||||
return [{"id": 1, "name": "Example"}]
|
||||
|
||||
async def fetch_one(self, query: str) -> Optional[dict]:
|
||||
"""Fetch single row."""
|
||||
await asyncio.sleep(0.1)
|
||||
return {"id": 1, "name": "Example"}
|
||||
|
||||
async def get_user_data(db: AsyncDB, user_id: int) -> dict:
|
||||
"""Fetch user and related data concurrently."""
|
||||
user_task = db.fetch_one(f"SELECT * FROM users WHERE id = {user_id}")
|
||||
orders_task = db.execute(f"SELECT * FROM orders WHERE user_id = {user_id}")
|
||||
profile_task = db.fetch_one(f"SELECT * FROM profiles WHERE user_id = {user_id}")
|
||||
|
||||
user, orders, profile = await asyncio.gather(user_task, orders_task, profile_task)
|
||||
|
||||
return {
|
||||
"user": user,
|
||||
"orders": orders,
|
||||
"profile": profile
|
||||
}
|
||||
|
||||
async def main():
|
||||
db = AsyncDB()
|
||||
user_data = await get_user_data(db, 1)
|
||||
print(user_data)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### WebSocket Server
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from typing import Set
|
||||
|
||||
# Simulated WebSocket connection
|
||||
class WebSocket:
|
||||
"""Simulated WebSocket."""
|
||||
|
||||
def __init__(self, client_id: str):
|
||||
self.client_id = client_id
|
||||
|
||||
async def send(self, message: str):
|
||||
"""Send message."""
|
||||
print(f"Sending to {self.client_id}: {message}")
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
async def recv(self) -> str:
|
||||
"""Receive message."""
|
||||
await asyncio.sleep(1)
|
||||
return f"Message from {self.client_id}"
|
||||
|
||||
class WebSocketServer:
|
||||
"""Simple WebSocket server."""
|
||||
|
||||
def __init__(self):
|
||||
self.clients: Set[WebSocket] = set()
|
||||
|
||||
async def register(self, websocket: WebSocket):
|
||||
"""Register new client."""
|
||||
self.clients.add(websocket)
|
||||
print(f"Client {websocket.client_id} connected")
|
||||
|
||||
async def unregister(self, websocket: WebSocket):
|
||||
"""Unregister client."""
|
||||
self.clients.remove(websocket)
|
||||
print(f"Client {websocket.client_id} disconnected")
|
||||
|
||||
async def broadcast(self, message: str):
|
||||
"""Broadcast message to all clients."""
|
||||
if self.clients:
|
||||
tasks = [client.send(message) for client in self.clients]
|
||||
await asyncio.gather(*tasks)
|
||||
|
||||
async def handle_client(self, websocket: WebSocket):
|
||||
"""Handle individual client connection."""
|
||||
await self.register(websocket)
|
||||
try:
|
||||
async for message in self.message_iterator(websocket):
|
||||
await self.broadcast(f"{websocket.client_id}: {message}")
|
||||
finally:
|
||||
await self.unregister(websocket)
|
||||
|
||||
async def message_iterator(self, websocket: WebSocket):
|
||||
"""Iterate over messages from client."""
|
||||
for _ in range(3): # Simulate 3 messages
|
||||
yield await websocket.recv()
|
||||
```
|
||||
|
||||
## Performance Best Practices
|
||||
|
||||
### 1. Use Connection Pools
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import aiohttp
|
||||
|
||||
async def with_connection_pool():
|
||||
"""Use connection pool for efficiency."""
|
||||
connector = aiohttp.TCPConnector(limit=100, limit_per_host=10)
|
||||
|
||||
async with aiohttp.ClientSession(connector=connector) as session:
|
||||
tasks = [session.get(f"https://api.example.com/item/{i}") for i in range(50)]
|
||||
responses = await asyncio.gather(*tasks)
|
||||
return responses
|
||||
```
|
||||
|
||||
### 2. Batch Operations
|
||||
|
||||
```python
|
||||
async def batch_process(items: List[str], batch_size: int = 10):
|
||||
"""Process items in batches."""
|
||||
for i in range(0, len(items), batch_size):
|
||||
batch = items[i:i + batch_size]
|
||||
tasks = [process_item(item) for item in batch]
|
||||
await asyncio.gather(*tasks)
|
||||
print(f"Processed batch {i // batch_size + 1}")
|
||||
|
||||
async def process_item(item: str):
|
||||
"""Process single item."""
|
||||
await asyncio.sleep(0.1)
|
||||
return f"Processed: {item}"
|
||||
```
|
||||
|
||||
### 3. Avoid Blocking Operations
|
||||
|
||||
Never block the event loop with synchronous operations. A single blocking call stalls all concurrent tasks.
|
||||
|
||||
```python
|
||||
# BAD - blocks the entire event loop
|
||||
async def fetch_data_bad():
|
||||
import time
|
||||
import requests
|
||||
time.sleep(1) # Blocks!
|
||||
response = requests.get(url) # Also blocks!
|
||||
|
||||
# GOOD - use async-native libraries (e.g., httpx for async HTTP)
|
||||
import httpx
|
||||
|
||||
async def fetch_data_good(url: str):
|
||||
await asyncio.sleep(1)
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get(url)
|
||||
```
|
||||
|
||||
**Wrapping Blocking Code with `asyncio.to_thread()` (Python 3.9+):**
|
||||
|
||||
When you must use synchronous libraries, offload to a thread pool:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
async def read_file_async(path: str) -> str:
|
||||
"""Read file without blocking event loop."""
|
||||
# asyncio.to_thread() runs sync code in a thread pool
|
||||
return await asyncio.to_thread(Path(path).read_text)
|
||||
|
||||
async def call_sync_library(data: dict) -> dict:
|
||||
"""Wrap a synchronous library call."""
|
||||
# Useful for sync database drivers, file I/O, CPU work
|
||||
return await asyncio.to_thread(sync_library.process, data)
|
||||
```
|
||||
|
||||
**Lower-level approach with `run_in_executor()`:**
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import concurrent.futures
|
||||
from typing import Any
|
||||
|
||||
def blocking_operation(data: Any) -> Any:
|
||||
"""CPU-intensive blocking operation."""
|
||||
import time
|
||||
time.sleep(1)
|
||||
return data * 2
|
||||
|
||||
async def run_in_executor(data: Any) -> Any:
|
||||
"""Run blocking operation in thread pool."""
|
||||
loop = asyncio.get_running_loop()
|
||||
with concurrent.futures.ThreadPoolExecutor() as pool:
|
||||
result = await loop.run_in_executor(pool, blocking_operation, data)
|
||||
return result
|
||||
|
||||
async def main():
|
||||
results = await asyncio.gather(*[run_in_executor(i) for i in range(5)])
|
||||
print(results)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
@@ -0,0 +1,349 @@
|
||||
---
|
||||
name: python-anti-patterns
|
||||
description: Use this skill when reviewing Python code for common anti-patterns to avoid. Use as a checklist when reviewing code, before finalizing implementations, or when debugging issues that might stem from known bad practices.
|
||||
---
|
||||
|
||||
# Python Anti-Patterns Checklist
|
||||
|
||||
A reference checklist of common mistakes and anti-patterns in Python code. Review this before finalizing implementations to catch issues early.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Reviewing code before merge
|
||||
- Debugging mysterious issues
|
||||
- Teaching or learning Python best practices
|
||||
- Establishing team coding standards
|
||||
- Refactoring legacy code
|
||||
|
||||
**Note:** This skill focuses on what to avoid. For guidance on positive patterns and architecture, see the `python-design-patterns` skill.
|
||||
|
||||
## Infrastructure Anti-Patterns
|
||||
|
||||
### Scattered Timeout/Retry Logic
|
||||
|
||||
```python
|
||||
# BAD: Timeout logic duplicated everywhere
|
||||
def fetch_user(user_id):
|
||||
try:
|
||||
return requests.get(url, timeout=30)
|
||||
except Timeout:
|
||||
logger.warning("Timeout fetching user")
|
||||
return None
|
||||
|
||||
def fetch_orders(user_id):
|
||||
try:
|
||||
return requests.get(url, timeout=30)
|
||||
except Timeout:
|
||||
logger.warning("Timeout fetching orders")
|
||||
return None
|
||||
```
|
||||
|
||||
**Fix:** Centralize in decorators or client wrappers.
|
||||
|
||||
```python
|
||||
# GOOD: Centralized retry logic
|
||||
@retry(stop=stop_after_attempt(3), wait=wait_exponential())
|
||||
def http_get(url: str) -> Response:
|
||||
return requests.get(url, timeout=30)
|
||||
```
|
||||
|
||||
### Double Retry
|
||||
|
||||
```python
|
||||
# BAD: Retrying at multiple layers
|
||||
@retry(max_attempts=3) # Application retry
|
||||
def call_service():
|
||||
return client.request() # Client also has retry configured!
|
||||
```
|
||||
|
||||
**Fix:** Retry at one layer only. Know your infrastructure's retry behavior.
|
||||
|
||||
### Hard-Coded Configuration
|
||||
|
||||
```python
|
||||
# BAD: Secrets and config in code
|
||||
DB_HOST = "prod-db.example.com"
|
||||
API_KEY = "sk-12345"
|
||||
|
||||
def connect():
|
||||
return psycopg.connect(f"host={DB_HOST}...")
|
||||
```
|
||||
|
||||
**Fix:** Use environment variables with typed settings.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
class Settings(BaseSettings):
|
||||
db_host: str = Field(alias="DB_HOST")
|
||||
api_key: str = Field(alias="API_KEY")
|
||||
|
||||
settings = Settings()
|
||||
```
|
||||
|
||||
## Architecture Anti-Patterns
|
||||
|
||||
### Exposed Internal Types
|
||||
|
||||
```python
|
||||
# BAD: Leaking ORM model to API
|
||||
@app.get("/users/{id}")
|
||||
def get_user(id: str) -> UserModel: # SQLAlchemy model
|
||||
return db.query(UserModel).get(id)
|
||||
```
|
||||
|
||||
**Fix:** Use DTOs/response models.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
@app.get("/users/{id}")
|
||||
def get_user(id: str) -> UserResponse:
|
||||
user = db.query(UserModel).get(id)
|
||||
return UserResponse.from_orm(user)
|
||||
```
|
||||
|
||||
### Mixed I/O and Business Logic
|
||||
|
||||
```python
|
||||
# BAD: SQL embedded in business logic
|
||||
def calculate_discount(user_id: str) -> float:
|
||||
user = db.query("SELECT * FROM users WHERE id = ?", user_id)
|
||||
orders = db.query("SELECT * FROM orders WHERE user_id = ?", user_id)
|
||||
# Business logic mixed with data access
|
||||
if len(orders) > 10:
|
||||
return 0.15
|
||||
return 0.0
|
||||
```
|
||||
|
||||
**Fix:** Repository pattern. Keep business logic pure.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def calculate_discount(user: User, orders: list[Order]) -> float:
|
||||
# Pure business logic, easily testable
|
||||
if len(orders) > 10:
|
||||
return 0.15
|
||||
return 0.0
|
||||
```
|
||||
|
||||
## Error Handling Anti-Patterns
|
||||
|
||||
### Bare Exception Handling
|
||||
|
||||
```python
|
||||
# BAD: Swallowing all exceptions
|
||||
try:
|
||||
process()
|
||||
except Exception:
|
||||
pass # Silent failure - bugs hidden forever
|
||||
```
|
||||
|
||||
**Fix:** Catch specific exceptions. Log or handle appropriately.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
try:
|
||||
process()
|
||||
except ConnectionError as e:
|
||||
logger.warning("Connection failed, will retry", error=str(e))
|
||||
raise
|
||||
except ValueError as e:
|
||||
logger.error("Invalid input", error=str(e))
|
||||
raise BadRequestError(str(e))
|
||||
```
|
||||
|
||||
### Ignored Partial Failures
|
||||
|
||||
```python
|
||||
# BAD: Stops on first error
|
||||
def process_batch(items):
|
||||
results = []
|
||||
for item in items:
|
||||
result = process(item) # Raises on error - batch aborted
|
||||
results.append(result)
|
||||
return results
|
||||
```
|
||||
|
||||
**Fix:** Capture both successes and failures.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def process_batch(items) -> BatchResult:
|
||||
succeeded = {}
|
||||
failed = {}
|
||||
for idx, item in enumerate(items):
|
||||
try:
|
||||
succeeded[idx] = process(item)
|
||||
except Exception as e:
|
||||
failed[idx] = e
|
||||
return BatchResult(succeeded, failed)
|
||||
```
|
||||
|
||||
### Missing Input Validation
|
||||
|
||||
```python
|
||||
# BAD: No validation
|
||||
def create_user(data: dict):
|
||||
return User(**data) # Crashes deep in code on bad input
|
||||
```
|
||||
|
||||
**Fix:** Validate early at API boundaries.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def create_user(data: dict) -> User:
|
||||
validated = CreateUserInput.model_validate(data)
|
||||
return User.from_input(validated)
|
||||
```
|
||||
|
||||
## Resource Anti-Patterns
|
||||
|
||||
### Unclosed Resources
|
||||
|
||||
```python
|
||||
# BAD: File never closed
|
||||
def read_file(path):
|
||||
f = open(path)
|
||||
return f.read() # What if this raises?
|
||||
```
|
||||
|
||||
**Fix:** Use context managers.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def read_file(path):
|
||||
with open(path) as f:
|
||||
return f.read()
|
||||
```
|
||||
|
||||
### Blocking in Async
|
||||
|
||||
```python
|
||||
# BAD: Blocks the entire event loop
|
||||
async def fetch_data():
|
||||
time.sleep(1) # Blocks everything!
|
||||
response = requests.get(url) # Also blocks!
|
||||
```
|
||||
|
||||
**Fix:** Use async-native libraries.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
async def fetch_data():
|
||||
await asyncio.sleep(1)
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get(url)
|
||||
```
|
||||
|
||||
## Type Safety Anti-Patterns
|
||||
|
||||
### Missing Type Hints
|
||||
|
||||
```python
|
||||
# BAD: No types
|
||||
def process(data):
|
||||
return data["value"] * 2
|
||||
```
|
||||
|
||||
**Fix:** Annotate all public functions.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def process(data: dict[str, int]) -> int:
|
||||
return data["value"] * 2
|
||||
```
|
||||
|
||||
### Untyped Collections
|
||||
|
||||
```python
|
||||
# BAD: Generic list without type parameter
|
||||
def get_users() -> list:
|
||||
...
|
||||
```
|
||||
|
||||
**Fix:** Use type parameters.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def get_users() -> list[User]:
|
||||
...
|
||||
```
|
||||
|
||||
## Testing Anti-Patterns
|
||||
|
||||
### Only Testing Happy Paths
|
||||
|
||||
```python
|
||||
# BAD: Only tests success case
|
||||
def test_create_user():
|
||||
user = service.create_user(valid_data)
|
||||
assert user.id is not None
|
||||
```
|
||||
|
||||
**Fix:** Test error conditions and edge cases.
|
||||
|
||||
```python
|
||||
# GOOD
|
||||
def test_create_user_success():
|
||||
user = service.create_user(valid_data)
|
||||
assert user.id is not None
|
||||
|
||||
def test_create_user_invalid_email():
|
||||
with pytest.raises(ValueError, match="Invalid email"):
|
||||
service.create_user(invalid_email_data)
|
||||
|
||||
def test_create_user_duplicate_email():
|
||||
service.create_user(valid_data)
|
||||
with pytest.raises(ConflictError):
|
||||
service.create_user(valid_data)
|
||||
```
|
||||
|
||||
### Over-Mocking
|
||||
|
||||
```python
|
||||
# BAD: Mocking everything
|
||||
def test_user_service():
|
||||
mock_repo = Mock()
|
||||
mock_cache = Mock()
|
||||
mock_logger = Mock()
|
||||
mock_metrics = Mock()
|
||||
# Test doesn't verify real behavior
|
||||
```
|
||||
|
||||
**Fix:** Use integration tests for critical paths. Mock only external services.
|
||||
|
||||
## Quick Review Checklist
|
||||
|
||||
Before finalizing code, verify:
|
||||
|
||||
- [ ] No scattered timeout/retry logic (centralized)
|
||||
- [ ] No double retry (app + infrastructure)
|
||||
- [ ] No hard-coded configuration or secrets
|
||||
- [ ] No exposed internal types (ORM models, protobufs)
|
||||
- [ ] No mixed I/O and business logic
|
||||
- [ ] No bare `except Exception: pass`
|
||||
- [ ] No ignored partial failures in batches
|
||||
- [ ] No missing input validation
|
||||
- [ ] No unclosed resources (using context managers)
|
||||
- [ ] No blocking calls in async code
|
||||
- [ ] All public functions have type hints
|
||||
- [ ] Collections have type parameters
|
||||
- [ ] Error paths are tested
|
||||
- [ ] Edge cases are covered
|
||||
|
||||
## Common Fixes Summary
|
||||
|
||||
| Anti-Pattern | Fix |
|
||||
|-------------|-----|
|
||||
| Scattered retry logic | Centralized decorators |
|
||||
| Hard-coded config | Environment variables + pydantic-settings |
|
||||
| Exposed ORM models | DTO/response schemas |
|
||||
| Mixed I/O + logic | Repository pattern |
|
||||
| Bare except | Catch specific exceptions |
|
||||
| Batch stops on error | Return BatchResult with successes/failures |
|
||||
| No validation | Validate at boundaries with Pydantic |
|
||||
| Unclosed resources | Context managers |
|
||||
| Blocking in async | Async-native libraries |
|
||||
| Missing types | Type annotations on all public APIs |
|
||||
| Only happy path tests | Test errors and edge cases |
|
||||
@@ -0,0 +1,241 @@
|
||||
---
|
||||
name: python-background-jobs
|
||||
description: Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles.
|
||||
---
|
||||
|
||||
# Python Background Jobs & Task Queues
|
||||
|
||||
Decouple long-running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Processing tasks that take longer than a few seconds
|
||||
- Sending emails, notifications, or webhooks
|
||||
- Generating reports or exporting data
|
||||
- Processing uploads or media transformations
|
||||
- Integrating with unreliable external services
|
||||
- Building event-driven architectures
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Task Queue Pattern
|
||||
|
||||
API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously.
|
||||
|
||||
### 2. Idempotency
|
||||
|
||||
Tasks may be retried on failure. Design for safe re-execution.
|
||||
|
||||
### 3. Job State Machine
|
||||
|
||||
Jobs transition through states: pending → running → succeeded/failed.
|
||||
|
||||
### 4. At-Least-Once Delivery
|
||||
|
||||
Most queues guarantee at-least-once delivery. Your code must handle duplicates.
|
||||
|
||||
## Quick Start
|
||||
|
||||
This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud-native solutions (AWS SQS, GCP Tasks) are equally valid choices.
|
||||
|
||||
```python
|
||||
from celery import Celery
|
||||
|
||||
app = Celery("tasks", broker="redis://localhost:6379")
|
||||
|
||||
@app.task
|
||||
def send_email(to: str, subject: str, body: str) -> None:
|
||||
# This runs in a background worker
|
||||
email_client.send(to, subject, body)
|
||||
|
||||
# In your API handler
|
||||
send_email.delay("user@example.com", "Welcome!", "Thanks for signing up")
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Return Job ID Immediately
|
||||
|
||||
For operations exceeding a few seconds, return a job ID and process asynchronously.
|
||||
|
||||
```python
|
||||
from uuid import uuid4
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from datetime import datetime
|
||||
|
||||
class JobStatus(Enum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
SUCCEEDED = "succeeded"
|
||||
FAILED = "failed"
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
id: str
|
||||
status: JobStatus
|
||||
created_at: datetime
|
||||
started_at: datetime | None = None
|
||||
completed_at: datetime | None = None
|
||||
result: dict | None = None
|
||||
error: str | None = None
|
||||
|
||||
# API endpoint
|
||||
async def start_export(request: ExportRequest) -> JobResponse:
|
||||
"""Start export job and return job ID."""
|
||||
job_id = str(uuid4())
|
||||
|
||||
# Persist job record
|
||||
await jobs_repo.create(Job(
|
||||
id=job_id,
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.utcnow(),
|
||||
))
|
||||
|
||||
# Enqueue task for background processing
|
||||
await task_queue.enqueue(
|
||||
"export_data",
|
||||
job_id=job_id,
|
||||
params=request.model_dump(),
|
||||
)
|
||||
|
||||
# Return immediately with job ID
|
||||
return JobResponse(
|
||||
job_id=job_id,
|
||||
status="pending",
|
||||
poll_url=f"/jobs/{job_id}",
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 2: Celery Task Configuration
|
||||
|
||||
Configure Celery tasks with proper retry and timeout settings.
|
||||
|
||||
```python
|
||||
from celery import Celery
|
||||
|
||||
app = Celery("tasks", broker="redis://localhost:6379")
|
||||
|
||||
# Global configuration
|
||||
app.conf.update(
|
||||
task_time_limit=3600, # Hard limit: 1 hour
|
||||
task_soft_time_limit=3000, # Soft limit: 50 minutes
|
||||
task_acks_late=True, # Acknowledge after completion
|
||||
task_reject_on_worker_lost=True,
|
||||
worker_prefetch_multiplier=1, # Don't prefetch too many tasks
|
||||
)
|
||||
|
||||
@app.task(
|
||||
bind=True,
|
||||
max_retries=3,
|
||||
default_retry_delay=60,
|
||||
autoretry_for=(ConnectionError, TimeoutError),
|
||||
)
|
||||
def process_payment(self, payment_id: str) -> dict:
|
||||
"""Process payment with automatic retry on transient errors."""
|
||||
try:
|
||||
result = payment_gateway.charge(payment_id)
|
||||
return {"status": "success", "transaction_id": result.id}
|
||||
except PaymentDeclinedError as e:
|
||||
# Don't retry permanent failures
|
||||
return {"status": "declined", "reason": str(e)}
|
||||
except TransientError as e:
|
||||
# Retry with exponential backoff
|
||||
raise self.retry(exc=e, countdown=2 ** self.request.retries * 60)
|
||||
```
|
||||
|
||||
### Pattern 3: Make Tasks Idempotent
|
||||
|
||||
Workers may retry on crash or timeout. Design for safe re-execution.
|
||||
|
||||
```python
|
||||
@app.task(bind=True)
|
||||
def process_order(self, order_id: str) -> None:
|
||||
"""Process order idempotently."""
|
||||
order = orders_repo.get(order_id)
|
||||
|
||||
# Already processed? Return early
|
||||
if order.status == OrderStatus.COMPLETED:
|
||||
logger.info("Order already processed", order_id=order_id)
|
||||
return
|
||||
|
||||
# Already in progress? Check if we should continue
|
||||
if order.status == OrderStatus.PROCESSING:
|
||||
# Use idempotency key to avoid double-charging
|
||||
pass
|
||||
|
||||
# Process with idempotency key
|
||||
result = payment_provider.charge(
|
||||
amount=order.total,
|
||||
idempotency_key=f"order-{order_id}", # Critical!
|
||||
)
|
||||
|
||||
orders_repo.update(order_id, status=OrderStatus.COMPLETED)
|
||||
```
|
||||
|
||||
**Idempotency Strategies:**
|
||||
|
||||
1. **Check-before-write**: Verify state before action
|
||||
2. **Idempotency keys**: Use unique tokens with external services
|
||||
3. **Upsert patterns**: `INSERT ... ON CONFLICT UPDATE`
|
||||
4. **Deduplication window**: Track processed IDs for N hours
|
||||
|
||||
### Pattern 4: Job State Management
|
||||
|
||||
Persist job state transitions for visibility and debugging.
|
||||
|
||||
```python
|
||||
class JobRepository:
|
||||
"""Repository for managing job state."""
|
||||
|
||||
async def create(self, job: Job) -> Job:
|
||||
"""Create new job record."""
|
||||
await self._db.execute(
|
||||
"""INSERT INTO jobs (id, status, created_at)
|
||||
VALUES ($1, $2, $3)""",
|
||||
job.id, job.status.value, job.created_at,
|
||||
)
|
||||
return job
|
||||
|
||||
async def update_status(
|
||||
self,
|
||||
job_id: str,
|
||||
status: JobStatus,
|
||||
**fields,
|
||||
) -> None:
|
||||
"""Update job status with timestamp."""
|
||||
updates = {"status": status.value, **fields}
|
||||
|
||||
if status == JobStatus.RUNNING:
|
||||
updates["started_at"] = datetime.utcnow()
|
||||
elif status in (JobStatus.SUCCEEDED, JobStatus.FAILED):
|
||||
updates["completed_at"] = datetime.utcnow()
|
||||
|
||||
await self._db.execute(
|
||||
"UPDATE jobs SET status = $1, ... WHERE id = $2",
|
||||
updates, job_id,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Job status updated",
|
||||
job_id=job_id,
|
||||
status=status.value,
|
||||
)
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Return immediately** - Don't block requests for long operations
|
||||
2. **Persist job state** - Enable status polling and debugging
|
||||
3. **Make tasks idempotent** - Safe to retry on any failure
|
||||
4. **Use idempotency keys** - For external service calls
|
||||
5. **Set timeouts** - Both soft and hard limits
|
||||
6. **Implement DLQ** - Capture permanently failed tasks
|
||||
7. **Log transitions** - Track job state changes
|
||||
8. **Retry appropriately** - Exponential backoff for transient errors
|
||||
9. **Don't retry permanent failures** - Validation errors, invalid credentials
|
||||
10. **Monitor queue depth** - Alert on backlog growth
|
||||
@@ -0,0 +1,128 @@
|
||||
# python-background-jobs — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Dead Letter Queue
|
||||
|
||||
Handle permanently failed tasks for manual inspection.
|
||||
|
||||
```python
|
||||
@app.task(bind=True, max_retries=3)
|
||||
def process_webhook(self, webhook_id: str, payload: dict) -> None:
|
||||
"""Process webhook with DLQ for failures."""
|
||||
try:
|
||||
result = send_webhook(payload)
|
||||
if not result.success:
|
||||
raise WebhookFailedError(result.error)
|
||||
except Exception as e:
|
||||
if self.request.retries >= self.max_retries:
|
||||
# Move to dead letter queue for manual inspection
|
||||
dead_letter_queue.send({
|
||||
"task": "process_webhook",
|
||||
"webhook_id": webhook_id,
|
||||
"payload": payload,
|
||||
"error": str(e),
|
||||
"attempts": self.request.retries + 1,
|
||||
"failed_at": datetime.utcnow().isoformat(),
|
||||
})
|
||||
logger.error(
|
||||
"Webhook moved to DLQ after max retries",
|
||||
webhook_id=webhook_id,
|
||||
error=str(e),
|
||||
)
|
||||
return
|
||||
|
||||
# Exponential backoff retry
|
||||
raise self.retry(exc=e, countdown=2 ** self.request.retries * 60)
|
||||
```
|
||||
|
||||
### Pattern 6: Status Polling Endpoint
|
||||
|
||||
Provide an endpoint for clients to check job status.
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, HTTPException
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
@app.get("/jobs/{job_id}")
|
||||
async def get_job_status(job_id: str) -> JobStatusResponse:
|
||||
"""Get current status of a background job."""
|
||||
job = await jobs_repo.get(job_id)
|
||||
|
||||
if job is None:
|
||||
raise HTTPException(404, f"Job {job_id} not found")
|
||||
|
||||
return JobStatusResponse(
|
||||
job_id=job.id,
|
||||
status=job.status.value,
|
||||
created_at=job.created_at,
|
||||
started_at=job.started_at,
|
||||
completed_at=job.completed_at,
|
||||
result=job.result if job.status == JobStatus.SUCCEEDED else None,
|
||||
error=job.error if job.status == JobStatus.FAILED else None,
|
||||
# Helpful for clients
|
||||
is_terminal=job.status in (JobStatus.SUCCEEDED, JobStatus.FAILED),
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 7: Task Chaining and Workflows
|
||||
|
||||
Compose complex workflows from simple tasks.
|
||||
|
||||
```python
|
||||
from celery import chain, group, chord
|
||||
|
||||
# Simple chain: A → B → C
|
||||
workflow = chain(
|
||||
extract_data.s(source_id),
|
||||
transform_data.s(),
|
||||
load_data.s(destination_id),
|
||||
)
|
||||
|
||||
# Parallel execution: A, B, C all at once
|
||||
parallel = group(
|
||||
send_email.s(user_email),
|
||||
send_sms.s(user_phone),
|
||||
update_analytics.s(event_data),
|
||||
)
|
||||
|
||||
# Chord: Run tasks in parallel, then a callback
|
||||
# Process all items, then send completion notification
|
||||
workflow = chord(
|
||||
[process_item.s(item_id) for item_id in item_ids],
|
||||
send_completion_notification.s(batch_id),
|
||||
)
|
||||
|
||||
workflow.apply_async()
|
||||
```
|
||||
|
||||
### Pattern 8: Alternative Task Queues
|
||||
|
||||
Choose the right tool for your needs.
|
||||
|
||||
**RQ (Redis Queue)**: Simple, Redis-based
|
||||
```python
|
||||
from rq import Queue
|
||||
from redis import Redis
|
||||
|
||||
queue = Queue(connection=Redis())
|
||||
job = queue.enqueue(send_email, "user@example.com", "Subject", "Body")
|
||||
```
|
||||
|
||||
**Dramatiq**: Modern Celery alternative
|
||||
```python
|
||||
import dramatiq
|
||||
from dramatiq.brokers.redis import RedisBroker
|
||||
|
||||
dramatiq.set_broker(RedisBroker())
|
||||
|
||||
@dramatiq.actor
|
||||
def send_email(to: str, subject: str, body: str) -> None:
|
||||
email_client.send(to, subject, body)
|
||||
```
|
||||
|
||||
**Cloud-native options:**
|
||||
- AWS SQS + Lambda
|
||||
- Google Cloud Tasks
|
||||
- Azure Functions
|
||||
@@ -0,0 +1,360 @@
|
||||
---
|
||||
name: python-code-style
|
||||
description: Python code style, linting, formatting, naming conventions, and documentation standards. Use when writing new code, reviewing style, configuring linters, writing docstrings, or establishing project standards.
|
||||
---
|
||||
|
||||
# Python Code Style & Documentation
|
||||
|
||||
Consistent code style and clear documentation make codebases maintainable and collaborative. This skill covers modern Python tooling, naming conventions, and documentation standards.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Setting up linting and formatting for a new project
|
||||
- Writing or reviewing docstrings
|
||||
- Establishing team coding standards
|
||||
- Configuring ruff, mypy, or pyright
|
||||
- Reviewing code for style consistency
|
||||
- Creating project documentation
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Automated Formatting
|
||||
|
||||
Let tools handle formatting debates. Configure once, enforce automatically.
|
||||
|
||||
### 2. Consistent Naming
|
||||
|
||||
Follow PEP 8 conventions with meaningful, descriptive names.
|
||||
|
||||
### 3. Documentation as Code
|
||||
|
||||
Docstrings should be maintained alongside the code they describe.
|
||||
|
||||
### 4. Type Annotations
|
||||
|
||||
Modern Python code should include type hints for all public APIs.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install modern tooling
|
||||
pip install ruff mypy
|
||||
|
||||
# Configure in pyproject.toml
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
target-version = "py312" # Adjust based on your project's minimum Python version
|
||||
|
||||
[tool.mypy]
|
||||
strict = true
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Modern Python Tooling
|
||||
|
||||
Use `ruff` as an all-in-one linter and formatter. It replaces flake8, isort, and black with a single fast tool.
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
target-version = "py312" # Adjust based on your project's minimum Python version
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = [
|
||||
"E", # pycodestyle errors
|
||||
"W", # pycodestyle warnings
|
||||
"F", # pyflakes
|
||||
"I", # isort
|
||||
"B", # flake8-bugbear
|
||||
"C4", # flake8-comprehensions
|
||||
"UP", # pyupgrade
|
||||
"SIM", # flake8-simplify
|
||||
]
|
||||
ignore = ["E501"] # Line length handled by formatter
|
||||
|
||||
[tool.ruff.format]
|
||||
quote-style = "double"
|
||||
indent-style = "space"
|
||||
```
|
||||
|
||||
Run with:
|
||||
|
||||
```bash
|
||||
ruff check --fix . # Lint and auto-fix
|
||||
ruff format . # Format code
|
||||
```
|
||||
|
||||
### Pattern 2: Type Checking Configuration
|
||||
|
||||
Configure strict type checking for production code.
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[tool.mypy]
|
||||
python_version = "3.12"
|
||||
strict = true
|
||||
warn_return_any = true
|
||||
warn_unused_ignores = true
|
||||
disallow_untyped_defs = true
|
||||
disallow_incomplete_defs = true
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = "tests.*"
|
||||
disallow_untyped_defs = false
|
||||
```
|
||||
|
||||
Alternative: Use `pyright` for faster checking.
|
||||
|
||||
```toml
|
||||
[tool.pyright]
|
||||
pythonVersion = "3.12"
|
||||
typeCheckingMode = "strict"
|
||||
```
|
||||
|
||||
### Pattern 3: Naming Conventions
|
||||
|
||||
Follow PEP 8 with emphasis on clarity over brevity.
|
||||
|
||||
**Files and Modules:**
|
||||
|
||||
```python
|
||||
# Good: Descriptive snake_case
|
||||
user_repository.py
|
||||
order_processing.py
|
||||
http_client.py
|
||||
|
||||
# Avoid: Abbreviations
|
||||
usr_repo.py
|
||||
ord_proc.py
|
||||
http_cli.py
|
||||
```
|
||||
|
||||
**Classes and Functions:**
|
||||
|
||||
```python
|
||||
# Classes: PascalCase
|
||||
class UserRepository:
|
||||
pass
|
||||
|
||||
class HTTPClientFactory: # Acronyms stay uppercase
|
||||
pass
|
||||
|
||||
# Functions and variables: snake_case
|
||||
def get_user_by_email(email: str) -> User | None:
|
||||
retry_count = 3
|
||||
max_connections = 100
|
||||
```
|
||||
|
||||
**Constants:**
|
||||
|
||||
```python
|
||||
# Module-level constants: SCREAMING_SNAKE_CASE
|
||||
MAX_RETRY_ATTEMPTS = 3
|
||||
DEFAULT_TIMEOUT_SECONDS = 30
|
||||
API_BASE_URL = "https://api.example.com"
|
||||
```
|
||||
|
||||
### Pattern 4: Import Organization
|
||||
|
||||
Group imports in a consistent order: standard library, third-party, local.
|
||||
|
||||
```python
|
||||
# Standard library
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
# Third-party packages
|
||||
import httpx
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import Column
|
||||
|
||||
# Local imports
|
||||
from myproject.models import User
|
||||
from myproject.services import UserService
|
||||
```
|
||||
|
||||
Use absolute imports exclusively:
|
||||
|
||||
```python
|
||||
# Preferred
|
||||
from myproject.utils import retry_decorator
|
||||
|
||||
# Avoid relative imports
|
||||
from ..utils import retry_decorator
|
||||
```
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Google-Style Docstrings
|
||||
|
||||
Write docstrings for all public classes, methods, and functions.
|
||||
|
||||
**Simple Function:**
|
||||
|
||||
```python
|
||||
def get_user(user_id: str) -> User:
|
||||
"""Retrieve a user by their unique identifier."""
|
||||
...
|
||||
```
|
||||
|
||||
**Complex Function:**
|
||||
|
||||
```python
|
||||
def process_batch(
|
||||
items: list[Item],
|
||||
max_workers: int = 4,
|
||||
on_progress: Callable[[int, int], None] | None = None,
|
||||
) -> BatchResult:
|
||||
"""Process items concurrently using a worker pool.
|
||||
|
||||
Processes each item in the batch using the configured number of
|
||||
workers. Progress can be monitored via the optional callback.
|
||||
|
||||
Args:
|
||||
items: The items to process. Must not be empty.
|
||||
max_workers: Maximum concurrent workers. Defaults to 4.
|
||||
on_progress: Optional callback receiving (completed, total) counts.
|
||||
|
||||
Returns:
|
||||
BatchResult containing succeeded items and any failures with
|
||||
their associated exceptions.
|
||||
|
||||
Raises:
|
||||
ValueError: If items is empty.
|
||||
ProcessingError: If the batch cannot be processed.
|
||||
|
||||
Example:
|
||||
>>> result = process_batch(items, max_workers=8)
|
||||
>>> print(f"Processed {len(result.succeeded)} items")
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
**Class Docstring:**
|
||||
|
||||
```python
|
||||
class UserService:
|
||||
"""Service for managing user operations.
|
||||
|
||||
Provides methods for creating, retrieving, updating, and
|
||||
deleting users with proper validation and error handling.
|
||||
|
||||
Attributes:
|
||||
repository: The data access layer for user persistence.
|
||||
logger: Logger instance for operation tracking.
|
||||
|
||||
Example:
|
||||
>>> service = UserService(repository, logger)
|
||||
>>> user = service.create_user(CreateUserInput(...))
|
||||
"""
|
||||
|
||||
def __init__(self, repository: UserRepository, logger: Logger) -> None:
|
||||
"""Initialize the user service.
|
||||
|
||||
Args:
|
||||
repository: Data access layer for users.
|
||||
logger: Logger for tracking operations.
|
||||
"""
|
||||
self.repository = repository
|
||||
self.logger = logger
|
||||
```
|
||||
|
||||
### Pattern 6: Line Length and Formatting
|
||||
|
||||
Set line length to 120 characters for modern displays while maintaining readability.
|
||||
|
||||
```python
|
||||
# Good: Readable line breaks
|
||||
def create_user(
|
||||
email: str,
|
||||
name: str,
|
||||
role: UserRole = UserRole.MEMBER,
|
||||
notify: bool = True,
|
||||
) -> User:
|
||||
...
|
||||
|
||||
# Good: Chain method calls clearly
|
||||
result = (
|
||||
db.query(User)
|
||||
.filter(User.active == True)
|
||||
.order_by(User.created_at.desc())
|
||||
.limit(10)
|
||||
.all()
|
||||
)
|
||||
|
||||
# Good: Format long strings
|
||||
error_message = (
|
||||
f"Failed to process user {user_id}: "
|
||||
f"received status {response.status_code} "
|
||||
f"with body {response.text[:100]}"
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 7: Project Documentation
|
||||
|
||||
**README Structure:**
|
||||
|
||||
```markdown
|
||||
# Project Name
|
||||
|
||||
Brief description of what the project does.
|
||||
|
||||
## Installation
|
||||
|
||||
\`\`\`bash
|
||||
pip install myproject
|
||||
\`\`\`
|
||||
|
||||
## Quick Start
|
||||
|
||||
\`\`\`python
|
||||
from myproject import Client
|
||||
|
||||
client = Client(api_key="...")
|
||||
result = client.process(data)
|
||||
\`\`\`
|
||||
|
||||
## Configuration
|
||||
|
||||
Document environment variables and configuration options.
|
||||
|
||||
## Development
|
||||
|
||||
\`\`\`bash
|
||||
pip install -e ".[dev]"
|
||||
pytest
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
**CHANGELOG Format (Keep a Changelog):**
|
||||
|
||||
```markdown
|
||||
# Changelog
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- New feature X
|
||||
|
||||
### Changed
|
||||
- Modified behavior of Y
|
||||
|
||||
### Fixed
|
||||
- Bug in Z
|
||||
```
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Use ruff** - Single tool for linting and formatting
|
||||
2. **Enable strict mypy** - Catch type errors before runtime
|
||||
3. **120 character lines** - Modern standard for readability
|
||||
4. **Descriptive names** - Clarity over brevity
|
||||
5. **Absolute imports** - More maintainable than relative
|
||||
6. **Google-style docstrings** - Consistent, readable documentation
|
||||
7. **Document public APIs** - Every public function needs a docstring
|
||||
8. **Keep docs updated** - Treat documentation as code
|
||||
9. **Automate in CI** - Run linters on every commit
|
||||
10. **Target Python 3.10+** - For new projects, Python 3.12+ is recommended for modern language features
|
||||
@@ -0,0 +1,211 @@
|
||||
---
|
||||
name: python-configuration
|
||||
description: Python configuration management via environment variables and typed settings. Use when externalizing config, setting up pydantic-settings, managing secrets, or implementing environment-specific behavior.
|
||||
---
|
||||
|
||||
# Python Configuration Management
|
||||
|
||||
Externalize configuration from code using environment variables and typed settings. Well-managed configuration enables the same code to run in any environment without modification.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Setting up a new project's configuration system
|
||||
- Migrating from hardcoded values to environment variables
|
||||
- Implementing pydantic-settings for typed configuration
|
||||
- Managing secrets and sensitive values
|
||||
- Creating environment-specific settings (dev/staging/prod)
|
||||
- Validating configuration at application startup
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Externalized Configuration
|
||||
|
||||
All environment-specific values (URLs, secrets, feature flags) come from environment variables, not code.
|
||||
|
||||
### 2. Typed Settings
|
||||
|
||||
Parse and validate configuration into typed objects at startup, not scattered throughout code.
|
||||
|
||||
### 3. Fail Fast
|
||||
|
||||
Validate all required configuration at application boot. Missing config should crash immediately with a clear message.
|
||||
|
||||
### 4. Sensible Defaults
|
||||
|
||||
Provide reasonable defaults for local development while requiring explicit values for sensitive settings.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field
|
||||
|
||||
class Settings(BaseSettings):
|
||||
database_url: str = Field(alias="DATABASE_URL")
|
||||
api_key: str = Field(alias="API_KEY")
|
||||
debug: bool = Field(default=False, alias="DEBUG")
|
||||
|
||||
settings = Settings() # Loads from environment
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Typed Settings with Pydantic
|
||||
|
||||
Create a central settings class that loads and validates all configuration.
|
||||
|
||||
```python
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field, PostgresDsn, ValidationError
|
||||
import sys
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application configuration loaded from environment variables."""
|
||||
|
||||
# Database
|
||||
db_host: str = Field(alias="DB_HOST")
|
||||
db_port: int = Field(default=5432, alias="DB_PORT")
|
||||
db_name: str = Field(alias="DB_NAME")
|
||||
db_user: str = Field(alias="DB_USER")
|
||||
db_password: str = Field(alias="DB_PASSWORD")
|
||||
|
||||
# Redis
|
||||
redis_url: str = Field(default="redis://localhost:6379", alias="REDIS_URL")
|
||||
|
||||
# API Keys
|
||||
api_secret_key: str = Field(alias="API_SECRET_KEY")
|
||||
|
||||
# Feature flags
|
||||
enable_new_feature: bool = Field(default=False, alias="ENABLE_NEW_FEATURE")
|
||||
|
||||
model_config = {
|
||||
"env_file": ".env",
|
||||
"env_file_encoding": "utf-8",
|
||||
}
|
||||
|
||||
# Create singleton instance at module load
|
||||
try:
|
||||
settings = Settings()
|
||||
except ValidationError as e:
|
||||
print(f"Configuration error:\n{e}")
|
||||
sys.exit(1)
|
||||
```
|
||||
|
||||
Import `settings` throughout your application:
|
||||
|
||||
```python
|
||||
from myapp.config import settings
|
||||
|
||||
def get_database_connection():
|
||||
return connect(
|
||||
host=settings.db_host,
|
||||
port=settings.db_port,
|
||||
database=settings.db_name,
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 2: Fail Fast on Missing Configuration
|
||||
|
||||
Required settings should crash the application immediately with a clear error.
|
||||
|
||||
```python
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field, ValidationError
|
||||
import sys
|
||||
|
||||
class Settings(BaseSettings):
|
||||
# Required - no default means it must be set
|
||||
api_key: str = Field(alias="API_KEY")
|
||||
database_url: str = Field(alias="DATABASE_URL")
|
||||
|
||||
# Optional with defaults
|
||||
log_level: str = Field(default="INFO", alias="LOG_LEVEL")
|
||||
|
||||
try:
|
||||
settings = Settings()
|
||||
except ValidationError as e:
|
||||
print("=" * 60)
|
||||
print("CONFIGURATION ERROR")
|
||||
print("=" * 60)
|
||||
for error in e.errors():
|
||||
field = error["loc"][0]
|
||||
print(f" - {field}: {error['msg']}")
|
||||
print("\nPlease set the required environment variables.")
|
||||
sys.exit(1)
|
||||
```
|
||||
|
||||
A clear error at startup is better than a cryptic `None` failure mid-request.
|
||||
|
||||
### Pattern 3: Local Development Defaults
|
||||
|
||||
Provide sensible defaults for local development while requiring explicit values for secrets.
|
||||
|
||||
```python
|
||||
class Settings(BaseSettings):
|
||||
# Has local default, but prod will override
|
||||
db_host: str = Field(default="localhost", alias="DB_HOST")
|
||||
db_port: int = Field(default=5432, alias="DB_PORT")
|
||||
|
||||
# Always required - no default for secrets
|
||||
db_password: str = Field(alias="DB_PASSWORD")
|
||||
api_secret_key: str = Field(alias="API_SECRET_KEY")
|
||||
|
||||
# Development convenience
|
||||
debug: bool = Field(default=False, alias="DEBUG")
|
||||
|
||||
model_config = {"env_file": ".env"}
|
||||
```
|
||||
|
||||
Create a `.env` file for local development (never commit this):
|
||||
|
||||
```bash
|
||||
# .env (add to .gitignore)
|
||||
DB_PASSWORD=local_dev_password
|
||||
API_SECRET_KEY=dev-secret-key
|
||||
DEBUG=true
|
||||
```
|
||||
|
||||
### Pattern 4: Namespaced Environment Variables
|
||||
|
||||
Prefix related variables for clarity and easy debugging.
|
||||
|
||||
```bash
|
||||
# Database configuration
|
||||
DB_HOST=localhost
|
||||
DB_PORT=5432
|
||||
DB_NAME=myapp
|
||||
DB_USER=admin
|
||||
DB_PASSWORD=secret
|
||||
|
||||
# Redis configuration
|
||||
REDIS_URL=redis://localhost:6379
|
||||
REDIS_MAX_CONNECTIONS=10
|
||||
|
||||
# Authentication
|
||||
AUTH_SECRET_KEY=your-secret-key
|
||||
AUTH_TOKEN_EXPIRY_SECONDS=3600
|
||||
AUTH_ALGORITHM=HS256
|
||||
|
||||
# Feature flags
|
||||
FEATURE_NEW_CHECKOUT=true
|
||||
FEATURE_BETA_UI=false
|
||||
```
|
||||
|
||||
Makes `env | grep DB_` useful for debugging.
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Never hardcode config** - All environment-specific values from env vars
|
||||
2. **Use typed settings** - Pydantic-settings with validation
|
||||
3. **Fail fast** - Crash on missing required config at startup
|
||||
4. **Provide dev defaults** - Make local development easy
|
||||
5. **Never commit secrets** - Use `.env` files (gitignored) or secret managers
|
||||
6. **Namespace variables** - `DB_HOST`, `REDIS_URL` for clarity
|
||||
7. **Import settings singleton** - Don't call `os.getenv()` throughout code
|
||||
8. **Document all variables** - README should list required env vars
|
||||
9. **Validate early** - Check config correctness at boot time
|
||||
10. **Use secrets_dir** - Support mounted secrets in containers
|
||||
@@ -0,0 +1,162 @@
|
||||
# python-configuration — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Type Coercion
|
||||
|
||||
Pydantic handles common conversions automatically.
|
||||
|
||||
```python
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field, field_validator
|
||||
|
||||
class Settings(BaseSettings):
|
||||
# Automatically converts "true", "1", "yes" to True
|
||||
debug: bool = False
|
||||
|
||||
# Automatically converts string to int
|
||||
max_connections: int = 100
|
||||
|
||||
# Parse comma-separated string to list
|
||||
allowed_hosts: list[str] = Field(default_factory=list)
|
||||
|
||||
@field_validator("allowed_hosts", mode="before")
|
||||
@classmethod
|
||||
def parse_allowed_hosts(cls, v: str | list[str]) -> list[str]:
|
||||
if isinstance(v, str):
|
||||
return [host.strip() for host in v.split(",") if host.strip()]
|
||||
return v
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```bash
|
||||
ALLOWED_HOSTS=example.com,api.example.com,localhost
|
||||
MAX_CONNECTIONS=50
|
||||
DEBUG=true
|
||||
```
|
||||
|
||||
### Pattern 6: Environment-Specific Configuration
|
||||
|
||||
Use an environment enum to switch behavior.
|
||||
|
||||
```python
|
||||
from enum import Enum
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field, computed_field
|
||||
|
||||
class Environment(str, Enum):
|
||||
LOCAL = "local"
|
||||
STAGING = "staging"
|
||||
PRODUCTION = "production"
|
||||
|
||||
class Settings(BaseSettings):
|
||||
environment: Environment = Field(
|
||||
default=Environment.LOCAL,
|
||||
alias="ENVIRONMENT",
|
||||
)
|
||||
|
||||
# Settings that vary by environment
|
||||
log_level: str = Field(default="DEBUG", alias="LOG_LEVEL")
|
||||
|
||||
@computed_field
|
||||
@property
|
||||
def is_production(self) -> bool:
|
||||
return self.environment == Environment.PRODUCTION
|
||||
|
||||
@computed_field
|
||||
@property
|
||||
def is_local(self) -> bool:
|
||||
return self.environment == Environment.LOCAL
|
||||
|
||||
# Usage
|
||||
if settings.is_production:
|
||||
configure_production_logging()
|
||||
else:
|
||||
configure_debug_logging()
|
||||
```
|
||||
|
||||
### Pattern 7: Nested Configuration Groups
|
||||
|
||||
Organize related settings into nested models.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
class DatabaseSettings(BaseModel):
|
||||
host: str = "localhost"
|
||||
port: int = 5432
|
||||
name: str
|
||||
user: str
|
||||
password: str
|
||||
|
||||
class RedisSettings(BaseModel):
|
||||
url: str = "redis://localhost:6379"
|
||||
max_connections: int = 10
|
||||
|
||||
class Settings(BaseSettings):
|
||||
database: DatabaseSettings
|
||||
redis: RedisSettings
|
||||
debug: bool = False
|
||||
|
||||
model_config = {
|
||||
"env_nested_delimiter": "__",
|
||||
"env_file": ".env",
|
||||
}
|
||||
```
|
||||
|
||||
Environment variables use double underscore for nesting:
|
||||
|
||||
```bash
|
||||
DATABASE__HOST=db.example.com
|
||||
DATABASE__PORT=5432
|
||||
DATABASE__NAME=myapp
|
||||
DATABASE__USER=admin
|
||||
DATABASE__PASSWORD=secret
|
||||
REDIS__URL=redis://redis.example.com:6379
|
||||
```
|
||||
|
||||
### Pattern 8: Secrets from Files
|
||||
|
||||
For container environments, read secrets from mounted files.
|
||||
|
||||
```python
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field
|
||||
from pathlib import Path
|
||||
|
||||
class Settings(BaseSettings):
|
||||
# Read from environment variable or file
|
||||
db_password: str = Field(alias="DB_PASSWORD")
|
||||
|
||||
model_config = {
|
||||
"secrets_dir": "/run/secrets", # Docker secrets location
|
||||
}
|
||||
```
|
||||
|
||||
Pydantic will look for `/run/secrets/db_password` if the env var isn't set.
|
||||
|
||||
### Pattern 9: Configuration Validation
|
||||
|
||||
Add custom validation for complex requirements.
|
||||
|
||||
```python
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import Field, model_validator
|
||||
|
||||
class Settings(BaseSettings):
|
||||
db_host: str = Field(alias="DB_HOST")
|
||||
db_port: int = Field(alias="DB_PORT")
|
||||
read_replica_host: str | None = Field(default=None, alias="READ_REPLICA_HOST")
|
||||
read_replica_port: int = Field(default=5432, alias="READ_REPLICA_PORT")
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_replica_settings(self):
|
||||
if self.read_replica_host and self.read_replica_port == self.db_port:
|
||||
if self.read_replica_host == self.db_host:
|
||||
raise ValueError(
|
||||
"Read replica cannot be the same as primary database"
|
||||
)
|
||||
return self
|
||||
```
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: python-design-patterns
|
||||
description: Python design patterns including KISS, Separation of Concerns, Single Responsibility, and composition over inheritance. Use this skill when designing a new service or component from scratch and choosing how to layer responsibilities, when refactoring a God class or monolithic function that has grown too large, when deciding whether to add a new abstraction or live with duplication, when evaluating a pull request for structural issues like tight coupling or leaking internal types, when choosing between inheritance and composition for a new class hierarchy, or when a codebase is becoming hard to test because of entangled I/O and business logic.
|
||||
---
|
||||
|
||||
# Python Design Patterns
|
||||
|
||||
Write maintainable Python code using fundamental design principles. These patterns help you build systems that are easy to understand, test, and modify.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Designing new components or services
|
||||
- Refactoring complex or tangled code
|
||||
- Deciding whether to create an abstraction
|
||||
- Choosing between inheritance and composition
|
||||
- Evaluating code complexity and coupling
|
||||
- Planning modular architectures
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. KISS (Keep It Simple)
|
||||
|
||||
Choose the simplest solution that works. Complexity must be justified by concrete requirements.
|
||||
|
||||
### 2. Single Responsibility (SRP)
|
||||
|
||||
Each unit should have one reason to change. Separate concerns into focused components.
|
||||
|
||||
### 3. Composition Over Inheritance
|
||||
|
||||
Build behavior by combining objects, not extending classes.
|
||||
|
||||
### 4. Rule of Three
|
||||
|
||||
Wait until you have three instances before abstracting. Duplication is often better than premature abstraction.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
# Simple beats clever
|
||||
# Instead of a factory/registry pattern:
|
||||
FORMATTERS = {"json": JsonFormatter, "csv": CsvFormatter}
|
||||
|
||||
def get_formatter(name: str) -> Formatter:
|
||||
return FORMATTERS[name]()
|
||||
```
|
||||
|
||||
## Detailed patterns and worked examples
|
||||
|
||||
Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Keep it simple** - Choose the simplest solution that works
|
||||
2. **Single responsibility** - Each unit has one reason to change
|
||||
3. **Separate concerns** - Distinct layers with clear purposes
|
||||
4. **Compose, don't inherit** - Combine objects for flexibility
|
||||
5. **Rule of three** - Wait before abstracting
|
||||
6. **Keep functions small** - 20-50 lines (varies by complexity), one purpose
|
||||
7. **Inject dependencies** - Constructor injection for testability
|
||||
8. **Delete before abstracting** - Remove dead code, then consider patterns
|
||||
9. **Test each layer** - Isolated tests for each concern
|
||||
10. **Explicit over clever** - Readable code beats elegant code
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**A class is growing and seems to have multiple responsibilities, but splitting it feels wrong.**
|
||||
Apply the "reason to change" test: list every change that could require editing this class. If the list has items from different domains (e.g., HTTP parsing AND business rules AND formatting), split it. If all changes stem from the same domain concern, the class may be appropriately sized.
|
||||
|
||||
**Injecting all dependencies through the constructor is producing constructors with 7+ parameters.**
|
||||
This is a sign of too many responsibilities in one class, not a problem with dependency injection. Split the class into smaller units first, then each constructor naturally becomes smaller.
|
||||
|
||||
**Composition is producing deeply nested wrapper objects that are hard to trace.**
|
||||
Keep the composition shallow (2-3 levels). If wrapping is the only mechanism, consider whether a Protocol-based approach or simple function composition would be cleaner than a chain of decorator objects.
|
||||
|
||||
**The rule of three says not to abstract yet, but the duplication is causing bugs when one copy is updated but not the other.**
|
||||
Duplication that diverges in dangerous ways should be abstracted sooner. The rule of three is a heuristic, not a law. If the copies are already diverging incorrectly, extract immediately and add a test that exercises the shared behavior.
|
||||
|
||||
**A service layer is importing from the API layer, breaking the dependency direction.**
|
||||
This is a layering violation. The service layer must not import from handlers. Introduce a shared types/models layer that both can import from, keeping the dependency arrow pointing downward (API → Service → Repository).
|
||||
|
||||
## Related Skills
|
||||
|
||||
- [python-testing-patterns](../python-testing-patterns/SKILL.md) — Test each layer in isolation using the dependency injection structure established here
|
||||
- [python-project-setup](../python-project-setup/SKILL.md) — Set up project structure and tooling that enforces layer boundaries from the start
|
||||
@@ -0,0 +1,353 @@
|
||||
# python-design-patterns — detailed patterns and worked examples
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: KISS - Keep It Simple
|
||||
|
||||
Before adding complexity, ask: does a simpler solution work?
|
||||
|
||||
```python
|
||||
# Over-engineered: Factory with registration
|
||||
class OutputFormatterFactory:
|
||||
_formatters: dict[str, type[Formatter]] = {}
|
||||
|
||||
@classmethod
|
||||
def register(cls, name: str):
|
||||
def decorator(formatter_cls):
|
||||
cls._formatters[name] = formatter_cls
|
||||
return formatter_cls
|
||||
return decorator
|
||||
|
||||
@classmethod
|
||||
def create(cls, name: str) -> Formatter:
|
||||
return cls._formatters[name]()
|
||||
|
||||
@OutputFormatterFactory.register("json")
|
||||
class JsonFormatter(Formatter):
|
||||
...
|
||||
|
||||
# Simple: Just use a dictionary
|
||||
FORMATTERS = {
|
||||
"json": JsonFormatter,
|
||||
"csv": CsvFormatter,
|
||||
"xml": XmlFormatter,
|
||||
}
|
||||
|
||||
def get_formatter(name: str) -> Formatter:
|
||||
"""Get formatter by name."""
|
||||
if name not in FORMATTERS:
|
||||
raise ValueError(f"Unknown format: {name}")
|
||||
return FORMATTERS[name]()
|
||||
```
|
||||
|
||||
The factory pattern adds code without adding value here. Save patterns for when they solve real problems.
|
||||
|
||||
### Pattern 2: Single Responsibility Principle
|
||||
|
||||
Each class or function should have one reason to change.
|
||||
|
||||
```python
|
||||
# BAD: Handler does everything
|
||||
class UserHandler:
|
||||
async def create_user(self, request: Request) -> Response:
|
||||
# HTTP parsing
|
||||
data = await request.json()
|
||||
|
||||
# Validation
|
||||
if not data.get("email"):
|
||||
return Response({"error": "email required"}, status=400)
|
||||
|
||||
# Database access
|
||||
user = await db.execute(
|
||||
"INSERT INTO users (email, name) VALUES ($1, $2) RETURNING *",
|
||||
data["email"], data["name"]
|
||||
)
|
||||
|
||||
# Response formatting
|
||||
return Response({"id": user.id, "email": user.email}, status=201)
|
||||
|
||||
# GOOD: Separated concerns
|
||||
class UserService:
|
||||
"""Business logic only."""
|
||||
|
||||
def __init__(self, repo: UserRepository) -> None:
|
||||
self._repo = repo
|
||||
|
||||
async def create_user(self, data: CreateUserInput) -> User:
|
||||
# Only business rules here
|
||||
user = User(email=data.email, name=data.name)
|
||||
return await self._repo.save(user)
|
||||
|
||||
class UserHandler:
|
||||
"""HTTP concerns only."""
|
||||
|
||||
def __init__(self, service: UserService) -> None:
|
||||
self._service = service
|
||||
|
||||
async def create_user(self, request: Request) -> Response:
|
||||
data = CreateUserInput(**(await request.json()))
|
||||
user = await self._service.create_user(data)
|
||||
return Response(user.to_dict(), status=201)
|
||||
```
|
||||
|
||||
Now HTTP changes don't affect business logic, and vice versa.
|
||||
|
||||
### Pattern 3: Separation of Concerns
|
||||
|
||||
Organize code into distinct layers with clear responsibilities.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ API Layer (handlers) │
|
||||
│ - Parse requests │
|
||||
│ - Call services │
|
||||
│ - Format responses │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Service Layer (business logic) │
|
||||
│ - Domain rules and validation │
|
||||
│ - Orchestrate operations │
|
||||
│ - Pure functions where possible │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Repository Layer (data access) │
|
||||
│ - SQL queries │
|
||||
│ - External API calls │
|
||||
│ - Cache operations │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Each layer depends only on layers below it:
|
||||
|
||||
```python
|
||||
# Repository: Data access
|
||||
class UserRepository:
|
||||
async def get_by_id(self, user_id: str) -> User | None:
|
||||
row = await self._db.fetchrow(
|
||||
"SELECT * FROM users WHERE id = $1", user_id
|
||||
)
|
||||
return User(**row) if row else None
|
||||
|
||||
# Service: Business logic
|
||||
class UserService:
|
||||
def __init__(self, repo: UserRepository) -> None:
|
||||
self._repo = repo
|
||||
|
||||
async def get_user(self, user_id: str) -> User:
|
||||
user = await self._repo.get_by_id(user_id)
|
||||
if user is None:
|
||||
raise UserNotFoundError(user_id)
|
||||
return user
|
||||
|
||||
# Handler: HTTP concerns
|
||||
@app.get("/users/{user_id}")
|
||||
async def get_user(user_id: str) -> UserResponse:
|
||||
user = await user_service.get_user(user_id)
|
||||
return UserResponse.from_user(user)
|
||||
```
|
||||
|
||||
### Pattern 4: Composition Over Inheritance
|
||||
|
||||
Build behavior by combining objects rather than inheriting.
|
||||
|
||||
```python
|
||||
# Inheritance: Rigid and hard to test
|
||||
class EmailNotificationService(NotificationService):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._smtp = SmtpClient() # Hard to mock
|
||||
|
||||
def notify(self, user: User, message: str) -> None:
|
||||
self._smtp.send(user.email, message)
|
||||
|
||||
# Composition: Flexible and testable
|
||||
class NotificationService:
|
||||
"""Send notifications via multiple channels."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
email_sender: EmailSender,
|
||||
sms_sender: SmsSender | None = None,
|
||||
push_sender: PushSender | None = None,
|
||||
) -> None:
|
||||
self._email = email_sender
|
||||
self._sms = sms_sender
|
||||
self._push = push_sender
|
||||
|
||||
async def notify(
|
||||
self,
|
||||
user: User,
|
||||
message: str,
|
||||
channels: set[str] | None = None,
|
||||
) -> None:
|
||||
channels = channels or {"email"}
|
||||
|
||||
if "email" in channels:
|
||||
await self._email.send(user.email, message)
|
||||
|
||||
if "sms" in channels and self._sms and user.phone:
|
||||
await self._sms.send(user.phone, message)
|
||||
|
||||
if "push" in channels and self._push and user.device_token:
|
||||
await self._push.send(user.device_token, message)
|
||||
|
||||
# Easy to test with fakes
|
||||
service = NotificationService(
|
||||
email_sender=FakeEmailSender(),
|
||||
sms_sender=FakeSmsSender(),
|
||||
)
|
||||
```
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Rule of Three
|
||||
|
||||
Wait until you have three instances before abstracting.
|
||||
|
||||
```python
|
||||
# Two similar functions? Don't abstract yet
|
||||
def process_orders(orders: list[Order]) -> list[Result]:
|
||||
results = []
|
||||
for order in orders:
|
||||
validated = validate_order(order)
|
||||
result = process_validated_order(validated)
|
||||
results.append(result)
|
||||
return results
|
||||
|
||||
def process_returns(returns: list[Return]) -> list[Result]:
|
||||
results = []
|
||||
for ret in returns:
|
||||
validated = validate_return(ret)
|
||||
result = process_validated_return(validated)
|
||||
results.append(result)
|
||||
return results
|
||||
|
||||
# These look similar, but wait! Are they actually the same?
|
||||
# Different validation, different processing, different errors...
|
||||
# Duplication is often better than the wrong abstraction
|
||||
|
||||
# Only after a third case, consider if there's a real pattern
|
||||
# But even then, sometimes explicit is better than abstract
|
||||
```
|
||||
|
||||
### Pattern 6: Function Size Guidelines
|
||||
|
||||
Keep functions focused. Extract when a function:
|
||||
|
||||
- Exceeds 20-50 lines (varies by complexity)
|
||||
- Serves multiple distinct purposes
|
||||
- Has deeply nested logic (3+ levels)
|
||||
|
||||
```python
|
||||
# Too long, multiple concerns mixed
|
||||
def process_order(order: Order) -> Result:
|
||||
# 50 lines of validation...
|
||||
# 30 lines of inventory check...
|
||||
# 40 lines of payment processing...
|
||||
# 20 lines of notification...
|
||||
pass
|
||||
|
||||
# Better: Composed from focused functions
|
||||
def process_order(order: Order) -> Result:
|
||||
"""Process a customer order through the complete workflow."""
|
||||
validate_order(order)
|
||||
reserve_inventory(order)
|
||||
payment_result = charge_payment(order)
|
||||
send_confirmation(order, payment_result)
|
||||
return Result(success=True, order_id=order.id)
|
||||
```
|
||||
|
||||
### Pattern 7: Dependency Injection
|
||||
|
||||
Pass dependencies through constructors for testability.
|
||||
|
||||
```python
|
||||
from typing import Protocol
|
||||
|
||||
class Logger(Protocol):
|
||||
def info(self, msg: str, **kwargs) -> None: ...
|
||||
def error(self, msg: str, **kwargs) -> None: ...
|
||||
|
||||
class Cache(Protocol):
|
||||
async def get(self, key: str) -> str | None: ...
|
||||
async def set(self, key: str, value: str, ttl: int) -> None: ...
|
||||
|
||||
class UserService:
|
||||
"""Service with injected dependencies."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
repository: UserRepository,
|
||||
cache: Cache,
|
||||
logger: Logger,
|
||||
) -> None:
|
||||
self._repo = repository
|
||||
self._cache = cache
|
||||
self._logger = logger
|
||||
|
||||
async def get_user(self, user_id: str) -> User:
|
||||
# Check cache first
|
||||
cached = await self._cache.get(f"user:{user_id}")
|
||||
if cached:
|
||||
self._logger.info("Cache hit", user_id=user_id)
|
||||
return User.from_json(cached)
|
||||
|
||||
# Fetch from database
|
||||
user = await self._repo.get_by_id(user_id)
|
||||
if user:
|
||||
await self._cache.set(f"user:{user_id}", user.to_json(), ttl=300)
|
||||
|
||||
return user
|
||||
|
||||
# Production
|
||||
service = UserService(
|
||||
repository=PostgresUserRepository(db),
|
||||
cache=RedisCache(redis),
|
||||
logger=StructlogLogger(),
|
||||
)
|
||||
|
||||
# Testing
|
||||
service = UserService(
|
||||
repository=InMemoryUserRepository(),
|
||||
cache=FakeCache(),
|
||||
logger=NullLogger(),
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 8: Avoiding Common Anti-Patterns
|
||||
|
||||
**Don't expose internal types:**
|
||||
|
||||
```python
|
||||
# BAD: Leaking ORM model to API
|
||||
@app.get("/users/{id}")
|
||||
def get_user(id: str) -> UserModel: # SQLAlchemy model
|
||||
return db.query(UserModel).get(id)
|
||||
|
||||
# GOOD: Use response schemas
|
||||
@app.get("/users/{id}")
|
||||
def get_user(id: str) -> UserResponse:
|
||||
user = db.query(UserModel).get(id)
|
||||
return UserResponse.from_orm(user)
|
||||
```
|
||||
|
||||
**Don't mix I/O with business logic:**
|
||||
|
||||
```python
|
||||
# BAD: SQL embedded in business logic
|
||||
def calculate_discount(user_id: str) -> float:
|
||||
user = db.query("SELECT * FROM users WHERE id = ?", user_id)
|
||||
orders = db.query("SELECT * FROM orders WHERE user_id = ?", user_id)
|
||||
# Business logic mixed with data access
|
||||
|
||||
# GOOD: Repository pattern
|
||||
def calculate_discount(user: User, order_history: list[Order]) -> float:
|
||||
# Pure business logic, easily testable
|
||||
if len(order_history) > 10:
|
||||
return 0.15
|
||||
return 0.0
|
||||
```
|
||||
@@ -0,0 +1,193 @@
|
||||
---
|
||||
name: python-error-handling
|
||||
description: Python error handling patterns including input validation, exception hierarchies, and partial failure handling. Use when implementing validation logic, designing exception strategies, handling batch processing failures, or building robust APIs.
|
||||
---
|
||||
|
||||
# Python Error Handling
|
||||
|
||||
Build robust Python applications with proper input validation, meaningful exceptions, and graceful failure handling. Good error handling makes debugging easier and systems more reliable.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Validating user input and API parameters
|
||||
- Designing exception hierarchies for applications
|
||||
- Handling partial failures in batch operations
|
||||
- Converting external data to domain types
|
||||
- Building user-friendly error messages
|
||||
- Implementing fail-fast validation patterns
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Fail Fast
|
||||
|
||||
Validate inputs early, before expensive operations. Report all validation errors at once when possible.
|
||||
|
||||
### 2. Meaningful Exceptions
|
||||
|
||||
Use appropriate exception types with context. Messages should explain what failed, why, and how to fix it.
|
||||
|
||||
### 3. Partial Failures
|
||||
|
||||
In batch operations, don't let one failure abort everything. Track successes and failures separately.
|
||||
|
||||
### 4. Preserve Context
|
||||
|
||||
Chain exceptions to maintain the full error trail for debugging.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
def fetch_page(url: str, page_size: int) -> Page:
|
||||
if not url:
|
||||
raise ValueError("'url' is required")
|
||||
if not 1 <= page_size <= 100:
|
||||
raise ValueError(f"'page_size' must be 1-100, got {page_size}")
|
||||
# Now safe to proceed...
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Early Input Validation
|
||||
|
||||
Validate all inputs at API boundaries before any processing begins.
|
||||
|
||||
```python
|
||||
def process_order(
|
||||
order_id: str,
|
||||
quantity: int,
|
||||
discount_percent: float,
|
||||
) -> OrderResult:
|
||||
"""Process an order with validation."""
|
||||
# Validate required fields
|
||||
if not order_id:
|
||||
raise ValueError("'order_id' is required")
|
||||
|
||||
# Validate ranges
|
||||
if quantity <= 0:
|
||||
raise ValueError(f"'quantity' must be positive, got {quantity}")
|
||||
|
||||
if not 0 <= discount_percent <= 100:
|
||||
raise ValueError(
|
||||
f"'discount_percent' must be 0-100, got {discount_percent}"
|
||||
)
|
||||
|
||||
# Validation passed, proceed with processing
|
||||
return _process_validated_order(order_id, quantity, discount_percent)
|
||||
```
|
||||
|
||||
### Pattern 2: Convert to Domain Types Early
|
||||
|
||||
Parse strings and external data into typed domain objects at system boundaries.
|
||||
|
||||
```python
|
||||
from enum import Enum
|
||||
|
||||
class OutputFormat(Enum):
|
||||
JSON = "json"
|
||||
CSV = "csv"
|
||||
PARQUET = "parquet"
|
||||
|
||||
def parse_output_format(value: str) -> OutputFormat:
|
||||
"""Parse string to OutputFormat enum.
|
||||
|
||||
Args:
|
||||
value: Format string from user input.
|
||||
|
||||
Returns:
|
||||
Validated OutputFormat enum member.
|
||||
|
||||
Raises:
|
||||
ValueError: If format is not recognized.
|
||||
"""
|
||||
try:
|
||||
return OutputFormat(value.lower())
|
||||
except ValueError:
|
||||
valid_formats = [f.value for f in OutputFormat]
|
||||
raise ValueError(
|
||||
f"Invalid format '{value}'. "
|
||||
f"Valid options: {', '.join(valid_formats)}"
|
||||
)
|
||||
|
||||
# Usage at API boundary
|
||||
def export_data(data: list[dict], format_str: str) -> bytes:
|
||||
output_format = parse_output_format(format_str) # Fail fast
|
||||
# Rest of function uses typed OutputFormat
|
||||
...
|
||||
```
|
||||
|
||||
### Pattern 3: Pydantic for Complex Validation
|
||||
|
||||
Use Pydantic models for structured input validation with automatic error messages.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
class CreateUserInput(BaseModel):
|
||||
"""Input model for user creation."""
|
||||
|
||||
email: str = Field(..., min_length=5, max_length=255)
|
||||
name: str = Field(..., min_length=1, max_length=100)
|
||||
age: int = Field(ge=0, le=150)
|
||||
|
||||
@field_validator("email")
|
||||
@classmethod
|
||||
def validate_email_format(cls, v: str) -> str:
|
||||
if "@" not in v or "." not in v.split("@")[-1]:
|
||||
raise ValueError("Invalid email format")
|
||||
return v.lower()
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def normalize_name(cls, v: str) -> str:
|
||||
return v.strip().title()
|
||||
|
||||
# Usage
|
||||
try:
|
||||
user_input = CreateUserInput(
|
||||
email="user@example.com",
|
||||
name="john doe",
|
||||
age=25,
|
||||
)
|
||||
except ValidationError as e:
|
||||
# Pydantic provides detailed error information
|
||||
print(e.errors())
|
||||
```
|
||||
|
||||
### Pattern 4: Map Errors to Standard Exceptions
|
||||
|
||||
Use Python's built-in exception types appropriately, adding context as needed.
|
||||
|
||||
| Failure Type | Exception | Example |
|
||||
|--------------|-----------|---------|
|
||||
| Invalid input | `ValueError` | Bad parameter values |
|
||||
| Wrong type | `TypeError` | Expected string, got int |
|
||||
| Missing item | `KeyError` | Dict key not found |
|
||||
| Operational failure | `RuntimeError` | Service unavailable |
|
||||
| Timeout | `TimeoutError` | Operation took too long |
|
||||
| File not found | `FileNotFoundError` | Path doesn't exist |
|
||||
| Permission denied | `PermissionError` | Access forbidden |
|
||||
|
||||
```python
|
||||
# Good: Specific exception with context
|
||||
raise ValueError(f"'page_size' must be 1-100, got {page_size}")
|
||||
|
||||
# Avoid: Generic exception, no context
|
||||
raise Exception("Invalid parameter")
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Validate early** - Check inputs before expensive operations
|
||||
2. **Use specific exceptions** - `ValueError`, `TypeError`, not generic `Exception`
|
||||
3. **Include context** - Messages should explain what, why, and how to fix
|
||||
4. **Convert types at boundaries** - Parse strings to enums/domain types early
|
||||
5. **Chain exceptions** - Use `raise ... from e` to preserve debug info
|
||||
6. **Handle partial failures** - Don't abort batches on single item errors
|
||||
7. **Use Pydantic** - For complex input validation with structured errors
|
||||
8. **Document failure modes** - Docstrings should list possible exceptions
|
||||
9. **Log with context** - Include IDs, counts, and other debugging info
|
||||
10. **Test error paths** - Verify exceptions are raised correctly
|
||||
@@ -0,0 +1,171 @@
|
||||
# python-error-handling — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Custom Exceptions with Context
|
||||
|
||||
Create domain-specific exceptions that carry structured information.
|
||||
|
||||
```python
|
||||
class ApiError(Exception):
|
||||
"""Base exception for API errors."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
status_code: int,
|
||||
response_body: str | None = None,
|
||||
) -> None:
|
||||
self.status_code = status_code
|
||||
self.response_body = response_body
|
||||
super().__init__(message)
|
||||
|
||||
class RateLimitError(ApiError):
|
||||
"""Raised when rate limit is exceeded."""
|
||||
|
||||
def __init__(self, retry_after: int) -> None:
|
||||
self.retry_after = retry_after
|
||||
super().__init__(
|
||||
f"Rate limit exceeded. Retry after {retry_after}s",
|
||||
status_code=429,
|
||||
)
|
||||
|
||||
# Usage
|
||||
def handle_response(response: Response) -> dict:
|
||||
match response.status_code:
|
||||
case 200:
|
||||
return response.json()
|
||||
case 401:
|
||||
raise ApiError("Invalid credentials", 401)
|
||||
case 404:
|
||||
raise ApiError(f"Resource not found: {response.url}", 404)
|
||||
case 429:
|
||||
retry_after = int(response.headers.get("Retry-After", 60))
|
||||
raise RateLimitError(retry_after)
|
||||
case code if 400 <= code < 500:
|
||||
raise ApiError(f"Client error: {response.text}", code)
|
||||
case code if code >= 500:
|
||||
raise ApiError(f"Server error: {response.text}", code)
|
||||
```
|
||||
|
||||
### Pattern 6: Exception Chaining
|
||||
|
||||
Preserve the original exception when re-raising to maintain the debug trail.
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
class ServiceError(Exception):
|
||||
"""High-level service operation failed."""
|
||||
pass
|
||||
|
||||
def upload_file(path: str) -> str:
|
||||
"""Upload file and return URL."""
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
response = httpx.post("https://upload.example.com", files={"file": f})
|
||||
response.raise_for_status()
|
||||
return response.json()["url"]
|
||||
except FileNotFoundError as e:
|
||||
raise ServiceError(f"Upload failed: file not found at '{path}'") from e
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise ServiceError(
|
||||
f"Upload failed: server returned {e.response.status_code}"
|
||||
) from e
|
||||
except httpx.RequestError as e:
|
||||
raise ServiceError(f"Upload failed: network error") from e
|
||||
```
|
||||
|
||||
### Pattern 7: Batch Processing with Partial Failures
|
||||
|
||||
Never let one bad item abort an entire batch. Track results per item.
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class BatchResult[T]:
|
||||
"""Results from batch processing."""
|
||||
|
||||
succeeded: dict[int, T] # index -> result
|
||||
failed: dict[int, Exception] # index -> error
|
||||
|
||||
@property
|
||||
def success_count(self) -> int:
|
||||
return len(self.succeeded)
|
||||
|
||||
@property
|
||||
def failure_count(self) -> int:
|
||||
return len(self.failed)
|
||||
|
||||
@property
|
||||
def all_succeeded(self) -> bool:
|
||||
return len(self.failed) == 0
|
||||
|
||||
def process_batch(items: list[Item]) -> BatchResult[ProcessedItem]:
|
||||
"""Process items, capturing individual failures.
|
||||
|
||||
Args:
|
||||
items: Items to process.
|
||||
|
||||
Returns:
|
||||
BatchResult with succeeded and failed items by index.
|
||||
"""
|
||||
succeeded: dict[int, ProcessedItem] = {}
|
||||
failed: dict[int, Exception] = {}
|
||||
|
||||
for idx, item in enumerate(items):
|
||||
try:
|
||||
result = process_single_item(item)
|
||||
succeeded[idx] = result
|
||||
except Exception as e:
|
||||
failed[idx] = e
|
||||
|
||||
return BatchResult(succeeded=succeeded, failed=failed)
|
||||
|
||||
# Caller handles partial results
|
||||
result = process_batch(items)
|
||||
if not result.all_succeeded:
|
||||
logger.warning(
|
||||
f"Batch completed with {result.failure_count} failures",
|
||||
failed_indices=list(result.failed.keys()),
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 8: Progress Reporting for Long Operations
|
||||
|
||||
Provide visibility into batch progress without coupling business logic to UI.
|
||||
|
||||
```python
|
||||
from collections.abc import Callable
|
||||
|
||||
ProgressCallback = Callable[[int, int, str], None] # current, total, status
|
||||
|
||||
def process_large_batch(
|
||||
items: list[Item],
|
||||
on_progress: ProgressCallback | None = None,
|
||||
) -> BatchResult:
|
||||
"""Process batch with optional progress reporting.
|
||||
|
||||
Args:
|
||||
items: Items to process.
|
||||
on_progress: Optional callback receiving (current, total, status).
|
||||
"""
|
||||
total = len(items)
|
||||
succeeded = {}
|
||||
failed = {}
|
||||
|
||||
for idx, item in enumerate(items):
|
||||
if on_progress:
|
||||
on_progress(idx, total, f"Processing {item.id}")
|
||||
|
||||
try:
|
||||
succeeded[idx] = process_single_item(item)
|
||||
except Exception as e:
|
||||
failed[idx] = e
|
||||
|
||||
if on_progress:
|
||||
on_progress(total, total, "Complete")
|
||||
|
||||
return BatchResult(succeeded=succeeded, failed=failed)
|
||||
```
|
||||
@@ -0,0 +1,229 @@
|
||||
---
|
||||
name: python-observability
|
||||
description: Python observability patterns including structured logging, metrics, and distributed tracing. Use when adding logging, implementing metrics collection, setting up tracing, or debugging production systems.
|
||||
---
|
||||
|
||||
# Python Observability
|
||||
|
||||
Instrument Python applications with structured logs, metrics, and traces. When something breaks in production, you need to answer "what, where, and why" without deploying new code.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Adding structured logging to applications
|
||||
- Implementing metrics collection with Prometheus
|
||||
- Setting up distributed tracing across services
|
||||
- Propagating correlation IDs through request chains
|
||||
- Debugging production issues
|
||||
- Building observability dashboards
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Structured Logging
|
||||
|
||||
Emit logs as JSON with consistent fields for production environments. Machine-readable logs enable powerful queries and alerts. For local development, consider human-readable formats.
|
||||
|
||||
### 2. The Four Golden Signals
|
||||
|
||||
Track latency, traffic, errors, and saturation for every service boundary.
|
||||
|
||||
### 3. Correlation IDs
|
||||
|
||||
Thread a unique ID through all logs and spans for a single request, enabling end-to-end tracing.
|
||||
|
||||
### 4. Bounded Cardinality
|
||||
|
||||
Keep metric label values bounded. Unbounded labels (like user IDs) explode storage costs.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
import structlog
|
||||
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.JSONRenderer(),
|
||||
],
|
||||
)
|
||||
|
||||
logger = structlog.get_logger()
|
||||
logger.info("Request processed", user_id="123", duration_ms=45)
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Structured Logging with Structlog
|
||||
|
||||
Configure structlog for JSON output with consistent fields.
|
||||
|
||||
```python
|
||||
import logging
|
||||
import structlog
|
||||
|
||||
def configure_logging(log_level: str = "INFO") -> None:
|
||||
"""Configure structured logging for the application."""
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.contextvars.merge_contextvars,
|
||||
structlog.processors.add_log_level,
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.JSONRenderer(),
|
||||
],
|
||||
wrapper_class=structlog.make_filtering_bound_logger(
|
||||
getattr(logging, log_level.upper())
|
||||
),
|
||||
context_class=dict,
|
||||
logger_factory=structlog.PrintLoggerFactory(),
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
|
||||
# Initialize at application startup
|
||||
configure_logging("INFO")
|
||||
logger = structlog.get_logger()
|
||||
```
|
||||
|
||||
### Pattern 2: Consistent Log Fields
|
||||
|
||||
Every log entry should include standard fields for filtering and correlation.
|
||||
|
||||
```python
|
||||
import structlog
|
||||
from contextvars import ContextVar
|
||||
|
||||
# Store correlation ID in context
|
||||
correlation_id: ContextVar[str] = ContextVar("correlation_id", default="")
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
def process_request(request: Request) -> Response:
|
||||
"""Process request with structured logging."""
|
||||
logger.info(
|
||||
"Request received",
|
||||
correlation_id=correlation_id.get(),
|
||||
method=request.method,
|
||||
path=request.path,
|
||||
user_id=request.user_id,
|
||||
)
|
||||
|
||||
try:
|
||||
result = handle_request(request)
|
||||
logger.info(
|
||||
"Request completed",
|
||||
correlation_id=correlation_id.get(),
|
||||
status_code=200,
|
||||
duration_ms=elapsed,
|
||||
)
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
"Request failed",
|
||||
correlation_id=correlation_id.get(),
|
||||
error_type=type(e).__name__,
|
||||
error_message=str(e),
|
||||
)
|
||||
raise
|
||||
```
|
||||
|
||||
### Pattern 3: Semantic Log Levels
|
||||
|
||||
Use log levels consistently across the application.
|
||||
|
||||
| Level | Purpose | Examples |
|
||||
|-------|---------|----------|
|
||||
| `DEBUG` | Development diagnostics | Variable values, internal state |
|
||||
| `INFO` | Request lifecycle, operations | Request start/end, job completion |
|
||||
| `WARNING` | Recoverable anomalies | Retry attempts, fallback used |
|
||||
| `ERROR` | Failures needing attention | Exceptions, service unavailable |
|
||||
|
||||
```python
|
||||
# DEBUG: Detailed internal information
|
||||
logger.debug("Cache lookup", key=cache_key, hit=cache_hit)
|
||||
|
||||
# INFO: Normal operational events
|
||||
logger.info("Order created", order_id=order.id, total=order.total)
|
||||
|
||||
# WARNING: Abnormal but handled situations
|
||||
logger.warning(
|
||||
"Rate limit approaching",
|
||||
current_rate=950,
|
||||
limit=1000,
|
||||
reset_seconds=30,
|
||||
)
|
||||
|
||||
# ERROR: Failures requiring investigation
|
||||
logger.error(
|
||||
"Payment processing failed",
|
||||
order_id=order.id,
|
||||
error=str(e),
|
||||
payment_provider="stripe",
|
||||
)
|
||||
```
|
||||
|
||||
Never log expected behavior at `ERROR`. A user entering a wrong password is `INFO`, not `ERROR`.
|
||||
|
||||
### Pattern 4: Correlation ID Propagation
|
||||
|
||||
Generate a unique ID at ingress and thread it through all operations.
|
||||
|
||||
```python
|
||||
from contextvars import ContextVar
|
||||
import uuid
|
||||
import structlog
|
||||
|
||||
correlation_id: ContextVar[str] = ContextVar("correlation_id", default="")
|
||||
|
||||
def set_correlation_id(cid: str | None = None) -> str:
|
||||
"""Set correlation ID for current context."""
|
||||
cid = cid or str(uuid.uuid4())
|
||||
correlation_id.set(cid)
|
||||
structlog.contextvars.bind_contextvars(correlation_id=cid)
|
||||
return cid
|
||||
|
||||
# FastAPI middleware example
|
||||
from fastapi import Request
|
||||
|
||||
async def correlation_middleware(request: Request, call_next):
|
||||
"""Middleware to set and propagate correlation ID."""
|
||||
# Use incoming header or generate new
|
||||
cid = request.headers.get("X-Correlation-ID") or str(uuid.uuid4())
|
||||
set_correlation_id(cid)
|
||||
|
||||
response = await call_next(request)
|
||||
response.headers["X-Correlation-ID"] = cid
|
||||
return response
|
||||
```
|
||||
|
||||
Propagate to outbound requests:
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async def call_downstream_service(endpoint: str, data: dict) -> dict:
|
||||
"""Call downstream service with correlation ID."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
endpoint,
|
||||
json=data,
|
||||
headers={"X-Correlation-ID": correlation_id.get()},
|
||||
)
|
||||
return response.json()
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Use structured logging** - JSON logs with consistent fields
|
||||
2. **Propagate correlation IDs** - Thread through all requests and logs
|
||||
3. **Track the four golden signals** - Latency, traffic, errors, saturation
|
||||
4. **Bound label cardinality** - Never use unbounded values as metric labels
|
||||
5. **Log at appropriate levels** - Don't cry wolf with ERROR
|
||||
6. **Include context** - User ID, request ID, operation name in logs
|
||||
7. **Use context managers** - Consistent timing and error handling
|
||||
8. **Separate concerns** - Observability code shouldn't pollute business logic
|
||||
9. **Test your observability** - Verify logs and metrics in integration tests
|
||||
10. **Set up alerts** - Metrics are useless without alerting
|
||||
@@ -0,0 +1,176 @@
|
||||
# python-observability — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: The Four Golden Signals with Prometheus
|
||||
|
||||
Track these metrics for every service boundary:
|
||||
|
||||
```python
|
||||
from prometheus_client import Counter, Histogram, Gauge
|
||||
|
||||
# Latency: How long requests take
|
||||
REQUEST_LATENCY = Histogram(
|
||||
"http_request_duration_seconds",
|
||||
"Request latency in seconds",
|
||||
["method", "endpoint", "status"],
|
||||
buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
|
||||
)
|
||||
|
||||
# Traffic: Request rate
|
||||
REQUEST_COUNT = Counter(
|
||||
"http_requests_total",
|
||||
"Total HTTP requests",
|
||||
["method", "endpoint", "status"],
|
||||
)
|
||||
|
||||
# Errors: Error rate
|
||||
ERROR_COUNT = Counter(
|
||||
"http_errors_total",
|
||||
"Total HTTP errors",
|
||||
["method", "endpoint", "error_type"],
|
||||
)
|
||||
|
||||
# Saturation: Resource utilization
|
||||
DB_POOL_USAGE = Gauge(
|
||||
"db_connection_pool_used",
|
||||
"Number of database connections in use",
|
||||
)
|
||||
```
|
||||
|
||||
Instrument your endpoints:
|
||||
|
||||
```python
|
||||
import time
|
||||
from functools import wraps
|
||||
|
||||
def track_request(func):
|
||||
"""Decorator to track request metrics."""
|
||||
@wraps(func)
|
||||
async def wrapper(request: Request, *args, **kwargs):
|
||||
method = request.method
|
||||
endpoint = request.url.path
|
||||
start = time.perf_counter()
|
||||
|
||||
try:
|
||||
response = await func(request, *args, **kwargs)
|
||||
status = str(response.status_code)
|
||||
return response
|
||||
except Exception as e:
|
||||
status = "500"
|
||||
ERROR_COUNT.labels(
|
||||
method=method,
|
||||
endpoint=endpoint,
|
||||
error_type=type(e).__name__,
|
||||
).inc()
|
||||
raise
|
||||
finally:
|
||||
duration = time.perf_counter() - start
|
||||
REQUEST_COUNT.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
REQUEST_LATENCY.labels(method=method, endpoint=endpoint, status=status).observe(duration)
|
||||
|
||||
return wrapper
|
||||
```
|
||||
|
||||
### Pattern 6: Bounded Cardinality
|
||||
|
||||
Avoid labels with unbounded values to prevent metric explosion.
|
||||
|
||||
```python
|
||||
# BAD: User ID has potentially millions of values
|
||||
REQUEST_COUNT.labels(method="GET", user_id=user.id) # Don't do this!
|
||||
|
||||
# GOOD: Bounded values only
|
||||
REQUEST_COUNT.labels(method="GET", endpoint="/users", status="200")
|
||||
|
||||
# If you need per-user metrics, use a different approach:
|
||||
# - Log the user_id and query logs
|
||||
# - Use a separate analytics system
|
||||
# - Bucket users by type/tier
|
||||
REQUEST_COUNT.labels(
|
||||
method="GET",
|
||||
endpoint="/users",
|
||||
user_tier="premium", # Bounded set of values
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 7: Timed Operations with Context Manager
|
||||
|
||||
Create a reusable timing context manager for operations.
|
||||
|
||||
```python
|
||||
from contextlib import contextmanager
|
||||
import time
|
||||
import structlog
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
@contextmanager
|
||||
def timed_operation(name: str, **extra_fields):
|
||||
"""Context manager for timing and logging operations."""
|
||||
start = time.perf_counter()
|
||||
logger.debug("Operation started", operation=name, **extra_fields)
|
||||
|
||||
try:
|
||||
yield
|
||||
except Exception as e:
|
||||
elapsed_ms = (time.perf_counter() - start) * 1000
|
||||
logger.error(
|
||||
"Operation failed",
|
||||
operation=name,
|
||||
duration_ms=round(elapsed_ms, 2),
|
||||
error=str(e),
|
||||
**extra_fields,
|
||||
)
|
||||
raise
|
||||
else:
|
||||
elapsed_ms = (time.perf_counter() - start) * 1000
|
||||
logger.info(
|
||||
"Operation completed",
|
||||
operation=name,
|
||||
duration_ms=round(elapsed_ms, 2),
|
||||
**extra_fields,
|
||||
)
|
||||
|
||||
# Usage
|
||||
with timed_operation("fetch_user_orders", user_id=user.id):
|
||||
orders = await order_repository.get_by_user(user.id)
|
||||
```
|
||||
|
||||
### Pattern 8: OpenTelemetry Tracing
|
||||
|
||||
Set up distributed tracing with OpenTelemetry.
|
||||
|
||||
**Note:** OpenTelemetry is actively evolving. Check the [official Python documentation](https://opentelemetry.io/docs/languages/python/) for the latest API patterns and best practices.
|
||||
|
||||
```python
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
|
||||
|
||||
def configure_tracing(service_name: str, otlp_endpoint: str) -> None:
|
||||
"""Configure OpenTelemetry tracing."""
|
||||
provider = TracerProvider()
|
||||
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=otlp_endpoint))
|
||||
provider.add_span_processor(processor)
|
||||
trace.set_tracer_provider(provider)
|
||||
|
||||
tracer = trace.get_tracer(__name__)
|
||||
|
||||
async def process_order(order_id: str) -> Order:
|
||||
"""Process order with tracing."""
|
||||
with tracer.start_as_current_span("process_order") as span:
|
||||
span.set_attribute("order.id", order_id)
|
||||
|
||||
with tracer.start_as_current_span("validate_order"):
|
||||
validate_order(order_id)
|
||||
|
||||
with tracer.start_as_current_span("charge_payment"):
|
||||
charge_payment(order_id)
|
||||
|
||||
with tracer.start_as_current_span("send_confirmation"):
|
||||
send_confirmation(order_id)
|
||||
|
||||
return order
|
||||
```
|
||||
@@ -0,0 +1,166 @@
|
||||
---
|
||||
name: python-packaging
|
||||
description: Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code.
|
||||
---
|
||||
|
||||
# Python Packaging
|
||||
|
||||
Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Creating Python libraries for distribution
|
||||
- Building command-line tools with entry points
|
||||
- Publishing packages to PyPI or private repositories
|
||||
- Setting up Python project structure
|
||||
- Creating installable packages with dependencies
|
||||
- Building wheels and source distributions
|
||||
- Versioning and releasing Python packages
|
||||
- Creating namespace packages
|
||||
- Implementing package metadata and classifiers
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Package Structure
|
||||
|
||||
- **Source layout**: `src/package_name/` (recommended)
|
||||
- **Flat layout**: `package_name/` (simpler but less flexible)
|
||||
- **Package metadata**: pyproject.toml, setup.py, or setup.cfg
|
||||
- **Distribution formats**: wheel (.whl) and source distribution (.tar.gz)
|
||||
|
||||
### 2. Modern Packaging Standards
|
||||
|
||||
- **PEP 517/518**: Build system requirements
|
||||
- **PEP 621**: Metadata in pyproject.toml
|
||||
- **PEP 660**: Editable installs
|
||||
- **pyproject.toml**: Single source of configuration
|
||||
|
||||
### 3. Build Backends
|
||||
|
||||
- **setuptools**: Traditional, widely used
|
||||
- **hatchling**: Modern, opinionated
|
||||
- **flit**: Lightweight, for pure Python
|
||||
- **poetry**: Dependency management + packaging
|
||||
|
||||
### 4. Distribution
|
||||
|
||||
- **PyPI**: Python Package Index (public)
|
||||
- **TestPyPI**: Testing before production
|
||||
- **Private repositories**: JFrog, AWS CodeArtifact, etc.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Minimal Package Structure
|
||||
|
||||
```
|
||||
my-package/
|
||||
├── pyproject.toml
|
||||
├── README.md
|
||||
├── LICENSE
|
||||
├── src/
|
||||
│ └── my_package/
|
||||
│ ├── __init__.py
|
||||
│ └── module.py
|
||||
└── tests/
|
||||
└── test_module.py
|
||||
```
|
||||
|
||||
### Minimal pyproject.toml
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "my-package"
|
||||
version = "0.1.0"
|
||||
description = "A short description"
|
||||
authors = [{name = "Your Name", email = "you@example.com"}]
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.8"
|
||||
dependencies = [
|
||||
"requests>=2.28.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0",
|
||||
"black>=22.0",
|
||||
]
|
||||
```
|
||||
|
||||
## Package Structure Patterns
|
||||
|
||||
### Pattern 1: Source Layout (Recommended)
|
||||
|
||||
```
|
||||
my-package/
|
||||
├── pyproject.toml
|
||||
├── README.md
|
||||
├── LICENSE
|
||||
├── .gitignore
|
||||
├── src/
|
||||
│ └── my_package/
|
||||
│ ├── __init__.py
|
||||
│ ├── core.py
|
||||
│ ├── utils.py
|
||||
│ └── py.typed # For type hints
|
||||
├── tests/
|
||||
│ ├── __init__.py
|
||||
│ ├── test_core.py
|
||||
│ └── test_utils.py
|
||||
└── docs/
|
||||
└── index.md
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Prevents accidentally importing from source
|
||||
- Cleaner test imports
|
||||
- Better isolation
|
||||
|
||||
**pyproject.toml for source layout:**
|
||||
|
||||
```toml
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
```
|
||||
|
||||
### Pattern 2: Flat Layout
|
||||
|
||||
```
|
||||
my-package/
|
||||
├── pyproject.toml
|
||||
├── README.md
|
||||
├── my_package/
|
||||
│ ├── __init__.py
|
||||
│ └── module.py
|
||||
└── tests/
|
||||
└── test_module.py
|
||||
```
|
||||
|
||||
**Simpler but:**
|
||||
|
||||
- Can import package without installing
|
||||
- Less professional for libraries
|
||||
|
||||
### Pattern 3: Multi-Package Project
|
||||
|
||||
```
|
||||
project/
|
||||
├── pyproject.toml
|
||||
├── packages/
|
||||
│ ├── package-a/
|
||||
│ │ └── src/
|
||||
│ │ └── package_a/
|
||||
│ └── package-b/
|
||||
│ └── src/
|
||||
│ └── package_b/
|
||||
└── tests/
|
||||
```
|
||||
|
||||
## Detailed patterns and worked examples
|
||||
|
||||
Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
|
||||
|
||||
@@ -0,0 +1,357 @@
|
||||
# Python Packaging — Advanced Reference
|
||||
|
||||
Advanced packaging patterns including data files, namespace packages, C extensions, version management, testing installation, documentation templates, and distribution workflows.
|
||||
|
||||
## Pattern 11: Including Data Files
|
||||
|
||||
```toml
|
||||
[tool.setuptools.package-data]
|
||||
my_package = [
|
||||
"data/*.json",
|
||||
"templates/*.html",
|
||||
"static/css/*.css",
|
||||
"py.typed",
|
||||
]
|
||||
```
|
||||
|
||||
**Accessing data files:**
|
||||
|
||||
```python
|
||||
# src/my_package/loader.py
|
||||
from importlib.resources import files
|
||||
import json
|
||||
|
||||
def load_config():
|
||||
"""Load configuration from package data."""
|
||||
config_file = files("my_package").joinpath("data/config.json")
|
||||
with config_file.open() as f:
|
||||
return json.load(f)
|
||||
|
||||
# Python 3.9+
|
||||
from importlib.resources import files
|
||||
|
||||
data = files("my_package").joinpath("data/file.txt").read_text()
|
||||
```
|
||||
|
||||
## Pattern 12: Namespace Packages
|
||||
|
||||
**For large projects split across multiple repositories:**
|
||||
|
||||
```
|
||||
# Package 1: company-core
|
||||
company/
|
||||
└── core/
|
||||
├── __init__.py
|
||||
└── models.py
|
||||
|
||||
# Package 2: company-api
|
||||
company/
|
||||
└── api/
|
||||
├── __init__.py
|
||||
└── routes.py
|
||||
```
|
||||
|
||||
**Do NOT include __init__.py in the namespace directory (company/):**
|
||||
|
||||
```toml
|
||||
# company-core/pyproject.toml
|
||||
[project]
|
||||
name = "company-core"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["company.core*"]
|
||||
|
||||
# company-api/pyproject.toml
|
||||
[project]
|
||||
name = "company-api"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["company.api*"]
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
|
||||
```python
|
||||
# Both packages can be imported under same namespace
|
||||
from company.core import models
|
||||
from company.api import routes
|
||||
```
|
||||
|
||||
## Pattern 13: C Extensions
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools]
|
||||
ext-modules = [
|
||||
{name = "my_package.fast_module", sources = ["src/fast_module.c"]},
|
||||
]
|
||||
```
|
||||
|
||||
**Or with setup.py:**
|
||||
|
||||
```python
|
||||
# setup.py
|
||||
from setuptools import setup, Extension
|
||||
|
||||
setup(
|
||||
ext_modules=[
|
||||
Extension(
|
||||
"my_package.fast_module",
|
||||
sources=["src/fast_module.c"],
|
||||
include_dirs=["src/include"],
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
## Version Management
|
||||
|
||||
### Pattern 14: Semantic Versioning
|
||||
|
||||
```python
|
||||
# src/my_package/__init__.py
|
||||
__version__ = "1.2.3"
|
||||
|
||||
# Semantic versioning: MAJOR.MINOR.PATCH
|
||||
# MAJOR: Breaking changes
|
||||
# MINOR: New features (backward compatible)
|
||||
# PATCH: Bug fixes
|
||||
```
|
||||
|
||||
**Version constraints in dependencies:**
|
||||
|
||||
```toml
|
||||
dependencies = [
|
||||
"requests>=2.28.0,<3.0.0", # Compatible range
|
||||
"click~=8.1.0", # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0)
|
||||
"pydantic>=2.0", # Minimum version
|
||||
"numpy==1.24.3", # Exact version (avoid if possible)
|
||||
]
|
||||
```
|
||||
|
||||
### Pattern 15: Git-Based Versioning
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "my-package"
|
||||
dynamic = ["version"]
|
||||
|
||||
[tool.setuptools_scm]
|
||||
write_to = "src/my_package/_version.py"
|
||||
version_scheme = "post-release"
|
||||
local_scheme = "dirty-tag"
|
||||
```
|
||||
|
||||
**Creates versions like:**
|
||||
|
||||
- `1.0.0` (from git tag)
|
||||
- `1.0.1.dev3+g1234567` (3 commits after tag)
|
||||
|
||||
## Testing Installation
|
||||
|
||||
### Pattern 16: Editable Install
|
||||
|
||||
```bash
|
||||
# Install in development mode
|
||||
pip install -e .
|
||||
|
||||
# With optional dependencies
|
||||
pip install -e ".[dev]"
|
||||
pip install -e ".[dev,docs]"
|
||||
|
||||
# Now changes to source code are immediately reflected
|
||||
```
|
||||
|
||||
### Pattern 17: Testing in Isolated Environment
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python -m venv test-env
|
||||
source test-env/bin/activate # Linux/Mac
|
||||
# test-env\Scripts\activate # Windows
|
||||
|
||||
# Install package
|
||||
pip install dist/my_package-1.0.0-py3-none-any.whl
|
||||
|
||||
# Test it works
|
||||
python -c "import my_package; print(my_package.__version__)"
|
||||
|
||||
# Test CLI
|
||||
my-tool --help
|
||||
|
||||
# Cleanup
|
||||
deactivate
|
||||
rm -rf test-env
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
### Pattern 18: README.md Template
|
||||
|
||||
````markdown
|
||||
# My Package
|
||||
|
||||
[](https://pypi.org/project/my-package/)
|
||||
[](https://pypi.org/project/my-package/)
|
||||
[](https://github.com/username/my-package/actions)
|
||||
|
||||
Brief description of your package.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install my-package
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from my_package import something
|
||||
|
||||
result = something.do_stuff()
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Feature 1
|
||||
- Feature 2
|
||||
- Feature 3
|
||||
|
||||
## Documentation
|
||||
|
||||
Full documentation: https://my-package.readthedocs.io
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
git clone https://github.com/username/my-package.git
|
||||
cd my-package
|
||||
pip install -e ".[dev]"
|
||||
pytest
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
````
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 19: Multi-Architecture Wheels
|
||||
|
||||
```yaml
|
||||
# .github/workflows/wheels.yml
|
||||
name: Build wheels
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
build_wheels:
|
||||
name: Build wheels on ${{ matrix.os }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest, macos-latest]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Build wheels
|
||||
uses: pypa/cibuildwheel@v2.16.2
|
||||
|
||||
- uses: actions/upload-artifact@v3
|
||||
with:
|
||||
path: ./wheelhouse/*.whl
|
||||
```
|
||||
|
||||
### Pattern 20: Private Package Index
|
||||
|
||||
```bash
|
||||
# Install from private index
|
||||
pip install my-package --index-url https://private.pypi.org/simple/
|
||||
|
||||
# Or add to pip.conf
|
||||
[global]
|
||||
index-url = https://private.pypi.org/simple/
|
||||
extra-index-url = https://pypi.org/simple/
|
||||
|
||||
# Upload to private index
|
||||
twine upload --repository-url https://private.pypi.org/ dist/*
|
||||
```
|
||||
|
||||
## File Templates
|
||||
|
||||
### .gitignore for Python Packages
|
||||
|
||||
```gitignore
|
||||
# Build artifacts
|
||||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
*.egg
|
||||
.eggs/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
# Distribution
|
||||
*.whl
|
||||
*.tar.gz
|
||||
```
|
||||
|
||||
### MANIFEST.in
|
||||
|
||||
```
|
||||
# MANIFEST.in
|
||||
include README.md
|
||||
include LICENSE
|
||||
include pyproject.toml
|
||||
|
||||
recursive-include src/my_package/data *.json
|
||||
recursive-include src/my_package/templates *.html
|
||||
recursive-exclude * __pycache__
|
||||
recursive-exclude * *.py[co]
|
||||
```
|
||||
|
||||
## Checklist for Publishing
|
||||
|
||||
- [ ] Code is tested (pytest passing)
|
||||
- [ ] Documentation is complete (README, docstrings)
|
||||
- [ ] Version number updated
|
||||
- [ ] CHANGELOG.md updated
|
||||
- [ ] License file included
|
||||
- [ ] pyproject.toml is complete
|
||||
- [ ] Package builds without errors
|
||||
- [ ] Installation tested in clean environment
|
||||
- [ ] CLI tools work (if applicable)
|
||||
- [ ] PyPI metadata is correct (classifiers, keywords)
|
||||
- [ ] GitHub repository linked
|
||||
- [ ] Tested on TestPyPI first
|
||||
- [ ] Git tag created for release
|
||||
@@ -0,0 +1,350 @@
|
||||
# python-packaging — detailed patterns and worked examples
|
||||
|
||||
## Complete pyproject.toml Examples
|
||||
|
||||
### Pattern 4: Full-Featured pyproject.toml
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "my-awesome-package"
|
||||
version = "1.0.0"
|
||||
description = "An awesome Python package"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.8"
|
||||
license = {text = "MIT"}
|
||||
authors = [
|
||||
{name = "Your Name", email = "you@example.com"},
|
||||
]
|
||||
maintainers = [
|
||||
{name = "Maintainer Name", email = "maintainer@example.com"},
|
||||
]
|
||||
keywords = ["example", "package", "awesome"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
]
|
||||
|
||||
dependencies = [
|
||||
"requests>=2.28.0,<3.0.0",
|
||||
"click>=8.0.0",
|
||||
"pydantic>=2.0.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-cov>=4.0.0",
|
||||
"black>=23.0.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.0.0",
|
||||
]
|
||||
docs = [
|
||||
"sphinx>=5.0.0",
|
||||
"sphinx-rtd-theme>=1.0.0",
|
||||
]
|
||||
all = [
|
||||
"my-awesome-package[dev,docs]",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/username/my-awesome-package"
|
||||
Documentation = "https://my-awesome-package.readthedocs.io"
|
||||
Repository = "https://github.com/username/my-awesome-package"
|
||||
"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"
|
||||
Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
|
||||
|
||||
[project.scripts]
|
||||
my-cli = "my_package.cli:main"
|
||||
awesome-tool = "my_package.tools:run"
|
||||
|
||||
[project.entry-points."my_package.plugins"]
|
||||
plugin1 = "my_package.plugins:plugin1"
|
||||
|
||||
[tool.setuptools]
|
||||
package-dir = {"" = "src"}
|
||||
zip-safe = false
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
include = ["my_package*"]
|
||||
exclude = ["tests*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
my_package = ["py.typed", "*.pyi", "data/*.json"]
|
||||
|
||||
# Black configuration
|
||||
[tool.black]
|
||||
line-length = 100
|
||||
target-version = ["py38", "py39", "py310", "py311"]
|
||||
include = '\.pyi?$'
|
||||
|
||||
# Ruff configuration
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py38"
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "N", "W", "UP"]
|
||||
|
||||
# MyPy configuration
|
||||
[tool.mypy]
|
||||
python_version = "3.8"
|
||||
warn_return_any = true
|
||||
warn_unused_configs = true
|
||||
disallow_untyped_defs = true
|
||||
|
||||
# Pytest configuration
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
addopts = "-v --cov=my_package --cov-report=term-missing"
|
||||
|
||||
# Coverage configuration
|
||||
[tool.coverage.run]
|
||||
source = ["src"]
|
||||
omit = ["*/tests/*"]
|
||||
|
||||
[tool.coverage.report]
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"def __repr__",
|
||||
"raise AssertionError",
|
||||
"raise NotImplementedError",
|
||||
]
|
||||
```
|
||||
|
||||
### Pattern 5: Dynamic Versioning
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "my-package"
|
||||
dynamic = ["version"]
|
||||
description = "Package with dynamic version"
|
||||
|
||||
[tool.setuptools.dynamic]
|
||||
version = {attr = "my_package.__version__"}
|
||||
|
||||
# Or use setuptools-scm for git-based versioning
|
||||
[tool.setuptools_scm]
|
||||
write_to = "src/my_package/_version.py"
|
||||
```
|
||||
|
||||
**In **init**.py:**
|
||||
|
||||
```python
|
||||
# src/my_package/__init__.py
|
||||
__version__ = "1.0.0"
|
||||
|
||||
# Or with setuptools-scm
|
||||
from importlib.metadata import version
|
||||
__version__ = version("my-package")
|
||||
```
|
||||
|
||||
## Command-Line Interface (CLI) Patterns
|
||||
|
||||
### Pattern 6: CLI with Click
|
||||
|
||||
```python
|
||||
# src/my_package/cli.py
|
||||
import click
|
||||
|
||||
@click.group()
|
||||
@click.version_option()
|
||||
def cli():
|
||||
"""My awesome CLI tool."""
|
||||
pass
|
||||
|
||||
@cli.command()
|
||||
@click.argument("name")
|
||||
@click.option("--greeting", default="Hello", help="Greeting to use")
|
||||
def greet(name: str, greeting: str):
|
||||
"""Greet someone."""
|
||||
click.echo(f"{greeting}, {name}!")
|
||||
|
||||
@cli.command()
|
||||
@click.option("--count", default=1, help="Number of times to repeat")
|
||||
def repeat(count: int):
|
||||
"""Repeat a message."""
|
||||
for i in range(count):
|
||||
click.echo(f"Message {i + 1}")
|
||||
|
||||
def main():
|
||||
"""Entry point for CLI."""
|
||||
cli()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
**Register in pyproject.toml:**
|
||||
|
||||
```toml
|
||||
[project.scripts]
|
||||
my-tool = "my_package.cli:main"
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
my-tool greet World
|
||||
my-tool greet Alice --greeting="Hi"
|
||||
my-tool repeat --count=3
|
||||
```
|
||||
|
||||
### Pattern 7: CLI with argparse
|
||||
|
||||
```python
|
||||
# src/my_package/cli.py
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
def main():
|
||||
"""Main CLI entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="My awesome tool",
|
||||
prog="my-tool"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--version",
|
||||
action="version",
|
||||
version="%(prog)s 1.0.0"
|
||||
)
|
||||
|
||||
subparsers = parser.add_subparsers(dest="command", help="Commands")
|
||||
|
||||
# Add subcommand
|
||||
process_parser = subparsers.add_parser("process", help="Process data")
|
||||
process_parser.add_argument("input_file", help="Input file path")
|
||||
process_parser.add_argument(
|
||||
"--output", "-o",
|
||||
default="output.txt",
|
||||
help="Output file path"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "process":
|
||||
process_data(args.input_file, args.output)
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
def process_data(input_file: str, output_file: str):
|
||||
"""Process data from input to output."""
|
||||
print(f"Processing {input_file} -> {output_file}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
## Building and Publishing
|
||||
|
||||
### Pattern 8: Build Package Locally
|
||||
|
||||
```bash
|
||||
# Install build tools
|
||||
pip install build twine
|
||||
|
||||
# Build distribution
|
||||
python -m build
|
||||
|
||||
# This creates:
|
||||
# dist/
|
||||
# my-package-1.0.0.tar.gz (source distribution)
|
||||
# my_package-1.0.0-py3-none-any.whl (wheel)
|
||||
|
||||
# Check the distribution
|
||||
twine check dist/*
|
||||
```
|
||||
|
||||
### Pattern 9: Publishing to PyPI
|
||||
|
||||
```bash
|
||||
# Install publishing tools
|
||||
pip install twine
|
||||
|
||||
# Test on TestPyPI first
|
||||
twine upload --repository testpypi dist/*
|
||||
|
||||
# Install from TestPyPI to test
|
||||
pip install --index-url https://test.pypi.org/simple/ my-package
|
||||
|
||||
# If all good, publish to PyPI
|
||||
twine upload dist/*
|
||||
```
|
||||
|
||||
**Using API tokens (recommended):**
|
||||
|
||||
```bash
|
||||
# Create ~/.pypirc
|
||||
[distutils]
|
||||
index-servers =
|
||||
pypi
|
||||
testpypi
|
||||
|
||||
[pypi]
|
||||
username = __token__
|
||||
password = pypi-...your-token...
|
||||
|
||||
[testpypi]
|
||||
username = __token__
|
||||
password = pypi-...your-test-token...
|
||||
```
|
||||
|
||||
### Pattern 10: Automated Publishing with GitHub Actions
|
||||
|
||||
```yaml
|
||||
# .github/workflows/publish.yml
|
||||
name: Publish to PyPI
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [created]
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install build twine
|
||||
|
||||
- name: Build package
|
||||
run: python -m build
|
||||
|
||||
- name: Check package
|
||||
run: twine check dist/*
|
||||
|
||||
- name: Publish to PyPI
|
||||
env:
|
||||
TWINE_USERNAME: __token__
|
||||
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: twine upload dist/*
|
||||
```
|
||||
|
||||
For advanced patterns including data files, namespace packages, C extensions, version management, testing installation, documentation templates, and distribution workflows, see [references/advanced-patterns.md](references/advanced-patterns.md)
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
name: python-performance-optimization
|
||||
description: Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance.
|
||||
---
|
||||
|
||||
# Python Performance Optimization
|
||||
|
||||
Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Identifying performance bottlenecks in Python applications
|
||||
- Reducing application latency and response times
|
||||
- Optimizing CPU-intensive operations
|
||||
- Reducing memory consumption and memory leaks
|
||||
- Improving database query performance
|
||||
- Optimizing I/O operations
|
||||
- Speeding up data processing pipelines
|
||||
- Implementing high-performance algorithms
|
||||
- Profiling production applications
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Profiling Types
|
||||
|
||||
- **CPU Profiling**: Identify time-consuming functions
|
||||
- **Memory Profiling**: Track memory allocation and leaks
|
||||
- **Line Profiling**: Profile at line-by-line granularity
|
||||
- **Call Graph**: Visualize function call relationships
|
||||
|
||||
### 2. Performance Metrics
|
||||
|
||||
- **Execution Time**: How long operations take
|
||||
- **Memory Usage**: Peak and average memory consumption
|
||||
- **CPU Utilization**: Processor usage patterns
|
||||
- **I/O Wait**: Time spent on I/O operations
|
||||
|
||||
### 3. Optimization Strategies
|
||||
|
||||
- **Algorithmic**: Better algorithms and data structures
|
||||
- **Implementation**: More efficient code patterns
|
||||
- **Parallelization**: Multi-threading/processing
|
||||
- **Caching**: Avoid redundant computation
|
||||
- **Native Extensions**: C/Rust for critical paths
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Timing
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
def measure_time():
|
||||
"""Simple timing measurement."""
|
||||
start = time.time()
|
||||
|
||||
# Your code here
|
||||
result = sum(range(1000000))
|
||||
|
||||
elapsed = time.time() - start
|
||||
print(f"Execution time: {elapsed:.4f} seconds")
|
||||
return result
|
||||
|
||||
# Better: use timeit for accurate measurements
|
||||
import timeit
|
||||
|
||||
execution_time = timeit.timeit(
|
||||
"sum(range(1000000))",
|
||||
number=100
|
||||
)
|
||||
print(f"Average time: {execution_time/100:.6f} seconds")
|
||||
```
|
||||
|
||||
## Detailed patterns and worked examples
|
||||
|
||||
Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Profile before optimizing** - Measure to find real bottlenecks
|
||||
2. **Focus on hot paths** - Optimize code that runs most frequently
|
||||
3. **Use appropriate data structures** - Dict for lookups, set for membership
|
||||
4. **Avoid premature optimization** - Clarity first, then optimize
|
||||
5. **Use built-in functions** - They're implemented in C
|
||||
6. **Cache expensive computations** - Use lru_cache
|
||||
7. **Batch I/O operations** - Reduce system calls
|
||||
8. **Use generators** for large datasets
|
||||
9. **Consider NumPy** for numerical operations
|
||||
10. **Profile production code** - Use py-spy for live systems
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- Optimizing without profiling
|
||||
- Using global variables unnecessarily
|
||||
- Not using appropriate data structures
|
||||
- Creating unnecessary copies of data
|
||||
- Not using connection pooling for databases
|
||||
- Ignoring algorithmic complexity
|
||||
- Over-optimizing rare code paths
|
||||
- Not considering memory usage
|
||||
+419
@@ -0,0 +1,419 @@
|
||||
# Python Performance Optimization — Advanced Reference
|
||||
|
||||
Advanced optimization techniques including NumPy vectorization, caching, memory management, parallelization, async I/O, database optimization, and benchmarking tools.
|
||||
|
||||
## Advanced Optimization
|
||||
|
||||
### Pattern 11: NumPy for Numerical Operations
|
||||
|
||||
```python
|
||||
import timeit
|
||||
import numpy as np
|
||||
|
||||
def python_sum(n):
|
||||
"""Sum using pure Python."""
|
||||
return sum(range(n))
|
||||
|
||||
def numpy_sum(n):
|
||||
"""Sum using NumPy."""
|
||||
return np.arange(n).sum()
|
||||
|
||||
n = 1000000
|
||||
|
||||
python_time = timeit.timeit(lambda: python_sum(n), number=100)
|
||||
numpy_time = timeit.timeit(lambda: numpy_sum(n), number=100)
|
||||
|
||||
print(f"Python: {python_time:.4f}s")
|
||||
print(f"NumPy: {numpy_time:.4f}s")
|
||||
print(f"Speedup: {python_time/numpy_time:.2f}x")
|
||||
|
||||
# Vectorized operations
|
||||
def python_multiply():
|
||||
"""Element-wise multiplication in Python."""
|
||||
a = list(range(100000))
|
||||
b = list(range(100000))
|
||||
return [x * y for x, y in zip(a, b)]
|
||||
|
||||
def numpy_multiply():
|
||||
"""Vectorized multiplication in NumPy."""
|
||||
a = np.arange(100000)
|
||||
b = np.arange(100000)
|
||||
return a * b
|
||||
|
||||
py_time = timeit.timeit(python_multiply, number=100)
|
||||
np_time = timeit.timeit(numpy_multiply, number=100)
|
||||
|
||||
print(f"\nPython multiply: {py_time:.4f}s")
|
||||
print(f"NumPy multiply: {np_time:.4f}s")
|
||||
print(f"Speedup: {py_time/np_time:.2f}x")
|
||||
```
|
||||
|
||||
### Pattern 12: Caching with functools.lru_cache
|
||||
|
||||
```python
|
||||
from functools import lru_cache
|
||||
import timeit
|
||||
|
||||
def fibonacci_slow(n):
|
||||
"""Recursive fibonacci without caching."""
|
||||
if n < 2:
|
||||
return n
|
||||
return fibonacci_slow(n-1) + fibonacci_slow(n-2)
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def fibonacci_fast(n):
|
||||
"""Recursive fibonacci with caching."""
|
||||
if n < 2:
|
||||
return n
|
||||
return fibonacci_fast(n-1) + fibonacci_fast(n-2)
|
||||
|
||||
# Massive speedup for recursive algorithms
|
||||
n = 30
|
||||
|
||||
slow_time = timeit.timeit(lambda: fibonacci_slow(n), number=1)
|
||||
fast_time = timeit.timeit(lambda: fibonacci_fast(n), number=1000)
|
||||
|
||||
print(f"Without cache (1 run): {slow_time:.4f}s")
|
||||
print(f"With cache (1000 runs): {fast_time:.4f}s")
|
||||
|
||||
# Cache info
|
||||
print(f"Cache info: {fibonacci_fast.cache_info()}")
|
||||
```
|
||||
|
||||
### Pattern 13: Using __slots__ for Memory
|
||||
|
||||
```python
|
||||
import sys
|
||||
|
||||
class RegularClass:
|
||||
"""Regular class with __dict__."""
|
||||
def __init__(self, x, y, z):
|
||||
self.x = x
|
||||
self.y = y
|
||||
self.z = z
|
||||
|
||||
class SlottedClass:
|
||||
"""Class with __slots__ for memory efficiency."""
|
||||
__slots__ = ['x', 'y', 'z']
|
||||
|
||||
def __init__(self, x, y, z):
|
||||
self.x = x
|
||||
self.y = y
|
||||
self.z = z
|
||||
|
||||
# Memory comparison
|
||||
regular = RegularClass(1, 2, 3)
|
||||
slotted = SlottedClass(1, 2, 3)
|
||||
|
||||
print(f"Regular class size: {sys.getsizeof(regular)} bytes")
|
||||
print(f"Slotted class size: {sys.getsizeof(slotted)} bytes")
|
||||
|
||||
# Significant savings with many instances
|
||||
regular_objects = [RegularClass(i, i+1, i+2) for i in range(10000)]
|
||||
slotted_objects = [SlottedClass(i, i+1, i+2) for i in range(10000)]
|
||||
|
||||
print(f"\nMemory for 10000 regular objects: ~{sys.getsizeof(regular) * 10000} bytes")
|
||||
print(f"Memory for 10000 slotted objects: ~{sys.getsizeof(slotted) * 10000} bytes")
|
||||
```
|
||||
|
||||
### Pattern 14: Multiprocessing for CPU-Bound Tasks
|
||||
|
||||
```python
|
||||
import multiprocessing as mp
|
||||
import time
|
||||
|
||||
def cpu_intensive_task(n):
|
||||
"""CPU-intensive calculation."""
|
||||
return sum(i**2 for i in range(n))
|
||||
|
||||
def sequential_processing():
|
||||
"""Process tasks sequentially."""
|
||||
start = time.time()
|
||||
results = [cpu_intensive_task(1000000) for _ in range(4)]
|
||||
elapsed = time.time() - start
|
||||
return elapsed, results
|
||||
|
||||
def parallel_processing():
|
||||
"""Process tasks in parallel."""
|
||||
start = time.time()
|
||||
with mp.Pool(processes=4) as pool:
|
||||
results = pool.map(cpu_intensive_task, [1000000] * 4)
|
||||
elapsed = time.time() - start
|
||||
return elapsed, results
|
||||
|
||||
if __name__ == "__main__":
|
||||
seq_time, seq_results = sequential_processing()
|
||||
par_time, par_results = parallel_processing()
|
||||
|
||||
print(f"Sequential: {seq_time:.2f}s")
|
||||
print(f"Parallel: {par_time:.2f}s")
|
||||
print(f"Speedup: {seq_time/par_time:.2f}x")
|
||||
```
|
||||
|
||||
### Pattern 15: Async I/O for I/O-Bound Tasks
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import time
|
||||
import requests
|
||||
|
||||
urls = [
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
]
|
||||
|
||||
def synchronous_requests():
|
||||
"""Synchronous HTTP requests."""
|
||||
start = time.time()
|
||||
results = []
|
||||
for url in urls:
|
||||
response = requests.get(url)
|
||||
results.append(response.status_code)
|
||||
elapsed = time.time() - start
|
||||
return elapsed, results
|
||||
|
||||
async def async_fetch(session, url):
|
||||
"""Async HTTP request."""
|
||||
async with session.get(url) as response:
|
||||
return response.status
|
||||
|
||||
async def asynchronous_requests():
|
||||
"""Asynchronous HTTP requests."""
|
||||
start = time.time()
|
||||
async with aiohttp.ClientSession() as session:
|
||||
tasks = [async_fetch(session, url) for url in urls]
|
||||
results = await asyncio.gather(*tasks)
|
||||
elapsed = time.time() - start
|
||||
return elapsed, results
|
||||
|
||||
# Async is much faster for I/O-bound work
|
||||
sync_time, sync_results = synchronous_requests()
|
||||
async_time, async_results = asyncio.run(asynchronous_requests())
|
||||
|
||||
print(f"Synchronous: {sync_time:.2f}s")
|
||||
print(f"Asynchronous: {async_time:.2f}s")
|
||||
print(f"Speedup: {sync_time/async_time:.2f}x")
|
||||
```
|
||||
|
||||
## Database Optimization
|
||||
|
||||
### Pattern 16: Batch Database Operations
|
||||
|
||||
```python
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
def create_db():
|
||||
"""Create test database."""
|
||||
conn = sqlite3.connect(":memory:")
|
||||
conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)")
|
||||
return conn
|
||||
|
||||
def slow_inserts(conn, count):
|
||||
"""Insert records one at a time."""
|
||||
start = time.time()
|
||||
cursor = conn.cursor()
|
||||
for i in range(count):
|
||||
cursor.execute("INSERT INTO users (name) VALUES (?)", (f"User {i}",))
|
||||
conn.commit() # Commit each insert
|
||||
elapsed = time.time() - start
|
||||
return elapsed
|
||||
|
||||
def fast_inserts(conn, count):
|
||||
"""Batch insert with single commit."""
|
||||
start = time.time()
|
||||
cursor = conn.cursor()
|
||||
data = [(f"User {i}",) for i in range(count)]
|
||||
cursor.executemany("INSERT INTO users (name) VALUES (?)", data)
|
||||
conn.commit() # Single commit
|
||||
elapsed = time.time() - start
|
||||
return elapsed
|
||||
|
||||
# Benchmark
|
||||
conn1 = create_db()
|
||||
slow_time = slow_inserts(conn1, 1000)
|
||||
|
||||
conn2 = create_db()
|
||||
fast_time = fast_inserts(conn2, 1000)
|
||||
|
||||
print(f"Individual inserts: {slow_time:.4f}s")
|
||||
print(f"Batch insert: {fast_time:.4f}s")
|
||||
print(f"Speedup: {slow_time/fast_time:.2f}x")
|
||||
```
|
||||
|
||||
### Pattern 17: Query Optimization
|
||||
|
||||
```python
|
||||
# Use indexes for frequently queried columns
|
||||
"""
|
||||
-- Slow: No index
|
||||
SELECT * FROM users WHERE email = 'user@example.com';
|
||||
|
||||
-- Fast: With index
|
||||
CREATE INDEX idx_users_email ON users(email);
|
||||
SELECT * FROM users WHERE email = 'user@example.com';
|
||||
"""
|
||||
|
||||
# Use query planning
|
||||
import sqlite3
|
||||
|
||||
conn = sqlite3.connect("example.db")
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Analyze query performance
|
||||
cursor.execute("EXPLAIN QUERY PLAN SELECT * FROM users WHERE email = ?", ("test@example.com",))
|
||||
print(cursor.fetchall())
|
||||
|
||||
# Use SELECT only needed columns
|
||||
# Slow: SELECT *
|
||||
# Fast: SELECT id, name
|
||||
```
|
||||
|
||||
## Memory Optimization
|
||||
|
||||
### Pattern 18: Detecting Memory Leaks
|
||||
|
||||
```python
|
||||
import tracemalloc
|
||||
import gc
|
||||
|
||||
def memory_leak_example():
|
||||
"""Example that leaks memory."""
|
||||
leaked_objects = []
|
||||
|
||||
for i in range(100000):
|
||||
# Objects added but never removed
|
||||
leaked_objects.append([i] * 100)
|
||||
|
||||
# In real code, this would be an unintended reference
|
||||
|
||||
def track_memory_usage():
|
||||
"""Track memory allocations."""
|
||||
tracemalloc.start()
|
||||
|
||||
# Take snapshot before
|
||||
snapshot1 = tracemalloc.take_snapshot()
|
||||
|
||||
# Run code
|
||||
memory_leak_example()
|
||||
|
||||
# Take snapshot after
|
||||
snapshot2 = tracemalloc.take_snapshot()
|
||||
|
||||
# Compare
|
||||
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
|
||||
|
||||
print("Top 10 memory allocations:")
|
||||
for stat in top_stats[:10]:
|
||||
print(stat)
|
||||
|
||||
tracemalloc.stop()
|
||||
|
||||
# Monitor memory
|
||||
track_memory_usage()
|
||||
|
||||
# Force garbage collection
|
||||
gc.collect()
|
||||
```
|
||||
|
||||
### Pattern 19: Iterators vs Lists
|
||||
|
||||
```python
|
||||
import sys
|
||||
|
||||
def process_file_list(filename):
|
||||
"""Load entire file into memory."""
|
||||
with open(filename) as f:
|
||||
lines = f.readlines() # Loads all lines
|
||||
return sum(1 for line in lines if line.strip())
|
||||
|
||||
def process_file_iterator(filename):
|
||||
"""Process file line by line."""
|
||||
with open(filename) as f:
|
||||
return sum(1 for line in f if line.strip())
|
||||
|
||||
# Iterator uses constant memory
|
||||
# List loads entire file into memory
|
||||
```
|
||||
|
||||
### Pattern 20: Weakref for Caches
|
||||
|
||||
```python
|
||||
import weakref
|
||||
|
||||
class CachedResource:
|
||||
"""Resource that can be garbage collected."""
|
||||
def __init__(self, data):
|
||||
self.data = data
|
||||
|
||||
# Regular cache prevents garbage collection
|
||||
regular_cache = {}
|
||||
|
||||
def get_resource_regular(key):
|
||||
"""Get resource from regular cache."""
|
||||
if key not in regular_cache:
|
||||
regular_cache[key] = CachedResource(f"Data for {key}")
|
||||
return regular_cache[key]
|
||||
|
||||
# Weak reference cache allows garbage collection
|
||||
weak_cache = weakref.WeakValueDictionary()
|
||||
|
||||
def get_resource_weak(key):
|
||||
"""Get resource from weak cache."""
|
||||
resource = weak_cache.get(key)
|
||||
if resource is None:
|
||||
resource = CachedResource(f"Data for {key}")
|
||||
weak_cache[key] = resource
|
||||
return resource
|
||||
|
||||
# When no strong references exist, objects can be GC'd
|
||||
```
|
||||
|
||||
## Benchmarking Tools
|
||||
|
||||
### Custom Benchmark Decorator
|
||||
|
||||
```python
|
||||
import time
|
||||
from functools import wraps
|
||||
|
||||
def benchmark(func):
|
||||
"""Decorator to benchmark function execution."""
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
start = time.perf_counter()
|
||||
result = func(*args, **kwargs)
|
||||
elapsed = time.perf_counter() - start
|
||||
print(f"{func.__name__} took {elapsed:.6f} seconds")
|
||||
return result
|
||||
return wrapper
|
||||
|
||||
@benchmark
|
||||
def slow_function():
|
||||
"""Function to benchmark."""
|
||||
time.sleep(0.5)
|
||||
return sum(range(1000000))
|
||||
|
||||
result = slow_function()
|
||||
```
|
||||
|
||||
### Performance Testing with pytest-benchmark
|
||||
|
||||
```python
|
||||
# Install: pip install pytest-benchmark
|
||||
|
||||
def test_list_comprehension(benchmark):
|
||||
"""Benchmark list comprehension."""
|
||||
result = benchmark(lambda: [i**2 for i in range(10000)])
|
||||
assert len(result) == 10000
|
||||
|
||||
def test_map_function(benchmark):
|
||||
"""Benchmark map function."""
|
||||
result = benchmark(lambda: list(map(lambda x: x**2, range(10000))))
|
||||
assert len(result) == 10000
|
||||
|
||||
# Run with: pytest test_performance.py --benchmark-compare
|
||||
```
|
||||
+342
@@ -0,0 +1,342 @@
|
||||
# python-performance-optimization — detailed patterns and worked examples
|
||||
|
||||
## Profiling Tools
|
||||
|
||||
### Pattern 1: cProfile - CPU Profiling
|
||||
|
||||
```python
|
||||
import cProfile
|
||||
import pstats
|
||||
from pstats import SortKey
|
||||
|
||||
def slow_function():
|
||||
"""Function to profile."""
|
||||
total = 0
|
||||
for i in range(1000000):
|
||||
total += i
|
||||
return total
|
||||
|
||||
def another_function():
|
||||
"""Another function."""
|
||||
return [i**2 for i in range(100000)]
|
||||
|
||||
def main():
|
||||
"""Main function to profile."""
|
||||
result1 = slow_function()
|
||||
result2 = another_function()
|
||||
return result1, result2
|
||||
|
||||
# Profile the code
|
||||
if __name__ == "__main__":
|
||||
profiler = cProfile.Profile()
|
||||
profiler.enable()
|
||||
|
||||
main()
|
||||
|
||||
profiler.disable()
|
||||
|
||||
# Print stats
|
||||
stats = pstats.Stats(profiler)
|
||||
stats.sort_stats(SortKey.CUMULATIVE)
|
||||
stats.print_stats(10) # Top 10 functions
|
||||
|
||||
# Save to file for later analysis
|
||||
stats.dump_stats("profile_output.prof")
|
||||
```
|
||||
|
||||
**Command-line profiling:**
|
||||
|
||||
```bash
|
||||
# Profile a script
|
||||
python -m cProfile -o output.prof script.py
|
||||
|
||||
# View results
|
||||
python -m pstats output.prof
|
||||
# In pstats:
|
||||
# sort cumtime
|
||||
# stats 10
|
||||
```
|
||||
|
||||
### Pattern 2: line_profiler - Line-by-Line Profiling
|
||||
|
||||
```python
|
||||
# Install: pip install line-profiler
|
||||
|
||||
# Add @profile decorator (line_profiler provides this)
|
||||
@profile
|
||||
def process_data(data):
|
||||
"""Process data with line profiling."""
|
||||
result = []
|
||||
for item in data:
|
||||
processed = item * 2
|
||||
result.append(processed)
|
||||
return result
|
||||
|
||||
# Run with:
|
||||
# kernprof -l -v script.py
|
||||
```
|
||||
|
||||
**Manual line profiling:**
|
||||
|
||||
```python
|
||||
from line_profiler import LineProfiler
|
||||
|
||||
def process_data(data):
|
||||
"""Function to profile."""
|
||||
result = []
|
||||
for item in data:
|
||||
processed = item * 2
|
||||
result.append(processed)
|
||||
return result
|
||||
|
||||
if __name__ == "__main__":
|
||||
lp = LineProfiler()
|
||||
lp.add_function(process_data)
|
||||
|
||||
data = list(range(100000))
|
||||
|
||||
lp_wrapper = lp(process_data)
|
||||
lp_wrapper(data)
|
||||
|
||||
lp.print_stats()
|
||||
```
|
||||
|
||||
### Pattern 3: memory_profiler - Memory Usage
|
||||
|
||||
```python
|
||||
# Install: pip install memory-profiler
|
||||
|
||||
from memory_profiler import profile
|
||||
|
||||
@profile
|
||||
def memory_intensive():
|
||||
"""Function that uses lots of memory."""
|
||||
# Create large list
|
||||
big_list = [i for i in range(1000000)]
|
||||
|
||||
# Create large dict
|
||||
big_dict = {i: i**2 for i in range(100000)}
|
||||
|
||||
# Process data
|
||||
result = sum(big_list)
|
||||
|
||||
return result
|
||||
|
||||
if __name__ == "__main__":
|
||||
memory_intensive()
|
||||
|
||||
# Run with:
|
||||
# python -m memory_profiler script.py
|
||||
```
|
||||
|
||||
### Pattern 4: py-spy - Production Profiling
|
||||
|
||||
```bash
|
||||
# Install: pip install py-spy
|
||||
|
||||
# Profile a running Python process
|
||||
py-spy top --pid 12345
|
||||
|
||||
# Generate flamegraph
|
||||
py-spy record -o profile.svg --pid 12345
|
||||
|
||||
# Profile a script
|
||||
py-spy record -o profile.svg -- python script.py
|
||||
|
||||
# Dump current call stack
|
||||
py-spy dump --pid 12345
|
||||
```
|
||||
|
||||
## Optimization Patterns
|
||||
|
||||
### Pattern 5: List Comprehensions vs Loops
|
||||
|
||||
```python
|
||||
import timeit
|
||||
|
||||
# Slow: Traditional loop
|
||||
def slow_squares(n):
|
||||
"""Create list of squares using loop."""
|
||||
result = []
|
||||
for i in range(n):
|
||||
result.append(i**2)
|
||||
return result
|
||||
|
||||
# Fast: List comprehension
|
||||
def fast_squares(n):
|
||||
"""Create list of squares using comprehension."""
|
||||
return [i**2 for i in range(n)]
|
||||
|
||||
# Benchmark
|
||||
n = 100000
|
||||
|
||||
slow_time = timeit.timeit(lambda: slow_squares(n), number=100)
|
||||
fast_time = timeit.timeit(lambda: fast_squares(n), number=100)
|
||||
|
||||
print(f"Loop: {slow_time:.4f}s")
|
||||
print(f"Comprehension: {fast_time:.4f}s")
|
||||
print(f"Speedup: {slow_time/fast_time:.2f}x")
|
||||
|
||||
# Even faster for simple operations: map
|
||||
def faster_squares(n):
|
||||
"""Use map for even better performance."""
|
||||
return list(map(lambda x: x**2, range(n)))
|
||||
```
|
||||
|
||||
### Pattern 6: Generator Expressions for Memory
|
||||
|
||||
```python
|
||||
import sys
|
||||
|
||||
def list_approach():
|
||||
"""Memory-intensive list."""
|
||||
data = [i**2 for i in range(1000000)]
|
||||
return sum(data)
|
||||
|
||||
def generator_approach():
|
||||
"""Memory-efficient generator."""
|
||||
data = (i**2 for i in range(1000000))
|
||||
return sum(data)
|
||||
|
||||
# Memory comparison
|
||||
list_data = [i for i in range(1000000)]
|
||||
gen_data = (i for i in range(1000000))
|
||||
|
||||
print(f"List size: {sys.getsizeof(list_data)} bytes")
|
||||
print(f"Generator size: {sys.getsizeof(gen_data)} bytes")
|
||||
|
||||
# Generators use constant memory regardless of size
|
||||
```
|
||||
|
||||
### Pattern 7: String Concatenation
|
||||
|
||||
```python
|
||||
import timeit
|
||||
|
||||
def slow_concat(items):
|
||||
"""Slow string concatenation."""
|
||||
result = ""
|
||||
for item in items:
|
||||
result += str(item)
|
||||
return result
|
||||
|
||||
def fast_concat(items):
|
||||
"""Fast string concatenation with join."""
|
||||
return "".join(str(item) for item in items)
|
||||
|
||||
def faster_concat(items):
|
||||
"""Even faster with list."""
|
||||
parts = [str(item) for item in items]
|
||||
return "".join(parts)
|
||||
|
||||
items = list(range(10000))
|
||||
|
||||
# Benchmark
|
||||
slow = timeit.timeit(lambda: slow_concat(items), number=100)
|
||||
fast = timeit.timeit(lambda: fast_concat(items), number=100)
|
||||
faster = timeit.timeit(lambda: faster_concat(items), number=100)
|
||||
|
||||
print(f"Concatenation (+): {slow:.4f}s")
|
||||
print(f"Join (generator): {fast:.4f}s")
|
||||
print(f"Join (list): {faster:.4f}s")
|
||||
```
|
||||
|
||||
### Pattern 8: Dictionary Lookups vs List Searches
|
||||
|
||||
```python
|
||||
import timeit
|
||||
|
||||
# Create test data
|
||||
size = 10000
|
||||
items = list(range(size))
|
||||
lookup_dict = {i: i for i in range(size)}
|
||||
|
||||
def list_search(items, target):
|
||||
"""O(n) search in list."""
|
||||
return target in items
|
||||
|
||||
def dict_search(lookup_dict, target):
|
||||
"""O(1) search in dict."""
|
||||
return target in lookup_dict
|
||||
|
||||
target = size - 1 # Worst case for list
|
||||
|
||||
# Benchmark
|
||||
list_time = timeit.timeit(
|
||||
lambda: list_search(items, target),
|
||||
number=1000
|
||||
)
|
||||
dict_time = timeit.timeit(
|
||||
lambda: dict_search(lookup_dict, target),
|
||||
number=1000
|
||||
)
|
||||
|
||||
print(f"List search: {list_time:.6f}s")
|
||||
print(f"Dict search: {dict_time:.6f}s")
|
||||
print(f"Speedup: {list_time/dict_time:.0f}x")
|
||||
```
|
||||
|
||||
### Pattern 9: Local Variable Access
|
||||
|
||||
```python
|
||||
import timeit
|
||||
|
||||
# Global variable (slow)
|
||||
GLOBAL_VALUE = 100
|
||||
|
||||
def use_global():
|
||||
"""Access global variable."""
|
||||
total = 0
|
||||
for i in range(10000):
|
||||
total += GLOBAL_VALUE
|
||||
return total
|
||||
|
||||
def use_local():
|
||||
"""Use local variable."""
|
||||
local_value = 100
|
||||
total = 0
|
||||
for i in range(10000):
|
||||
total += local_value
|
||||
return total
|
||||
|
||||
# Local is faster
|
||||
global_time = timeit.timeit(use_global, number=1000)
|
||||
local_time = timeit.timeit(use_local, number=1000)
|
||||
|
||||
print(f"Global access: {global_time:.4f}s")
|
||||
print(f"Local access: {local_time:.4f}s")
|
||||
print(f"Speedup: {global_time/local_time:.2f}x")
|
||||
```
|
||||
|
||||
### Pattern 10: Function Call Overhead
|
||||
|
||||
```python
|
||||
import timeit
|
||||
|
||||
def calculate_inline():
|
||||
"""Inline calculation."""
|
||||
total = 0
|
||||
for i in range(10000):
|
||||
total += i * 2 + 1
|
||||
return total
|
||||
|
||||
def helper_function(x):
|
||||
"""Helper function."""
|
||||
return x * 2 + 1
|
||||
|
||||
def calculate_with_function():
|
||||
"""Calculation with function calls."""
|
||||
total = 0
|
||||
for i in range(10000):
|
||||
total += helper_function(i)
|
||||
return total
|
||||
|
||||
# Inline is faster due to no call overhead
|
||||
inline_time = timeit.timeit(calculate_inline, number=1000)
|
||||
function_time = timeit.timeit(calculate_with_function, number=1000)
|
||||
|
||||
print(f"Inline: {inline_time:.4f}s")
|
||||
print(f"Function calls: {function_time:.4f}s")
|
||||
```
|
||||
|
||||
For advanced optimization techniques including NumPy vectorization, caching, memory management, parallelization, async I/O, database optimization, and benchmarking tools, see [references/advanced-patterns.md](references/advanced-patterns.md)
|
||||
@@ -0,0 +1,252 @@
|
||||
---
|
||||
name: python-project-structure
|
||||
description: Python project organization, module architecture, and public API design. Use when setting up new projects, organizing modules, defining public interfaces with __all__, or planning directory layouts.
|
||||
---
|
||||
|
||||
# Python Project Structure & Module Architecture
|
||||
|
||||
Design well-organized Python projects with clear module boundaries, explicit public interfaces, and maintainable directory structures. Good organization makes code discoverable and changes predictable.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Starting a new Python project from scratch
|
||||
- Reorganizing an existing codebase for clarity
|
||||
- Defining module public APIs with `__all__`
|
||||
- Deciding between flat and nested directory structures
|
||||
- Determining test file placement strategies
|
||||
- Creating reusable library packages
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Module Cohesion
|
||||
|
||||
Group related code that changes together. A module should have a single, clear purpose.
|
||||
|
||||
### 2. Explicit Interfaces
|
||||
|
||||
Define what's public with `__all__`. Everything not listed is an internal implementation detail.
|
||||
|
||||
### 3. Flat Hierarchies
|
||||
|
||||
Prefer shallow directory structures. Add depth only for genuine sub-domains.
|
||||
|
||||
### 4. Consistent Conventions
|
||||
|
||||
Apply naming and organization patterns uniformly across the project.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```
|
||||
myproject/
|
||||
├── src/
|
||||
│ └── myproject/
|
||||
│ ├── __init__.py
|
||||
│ ├── services/
|
||||
│ ├── models/
|
||||
│ └── api/
|
||||
├── tests/
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: One Concept Per File
|
||||
|
||||
Each file should focus on a single concept or closely related set of functions. Consider splitting when a file:
|
||||
|
||||
- Handles multiple unrelated responsibilities
|
||||
- Grows beyond 300-500 lines (varies by complexity)
|
||||
- Contains classes that change for different reasons
|
||||
|
||||
```python
|
||||
# Good: Focused files
|
||||
# user_service.py - User business logic
|
||||
# user_repository.py - User data access
|
||||
# user_models.py - User data structures
|
||||
|
||||
# Avoid: Kitchen sink files
|
||||
# user.py - Contains service, repository, models, utilities...
|
||||
```
|
||||
|
||||
### Pattern 2: Explicit Public APIs with `__all__`
|
||||
|
||||
Define the public interface for every module. Unlisted members are internal implementation details.
|
||||
|
||||
```python
|
||||
# mypackage/services/__init__.py
|
||||
from .user_service import UserService
|
||||
from .order_service import OrderService
|
||||
from .exceptions import ServiceError, ValidationError
|
||||
|
||||
__all__ = [
|
||||
"UserService",
|
||||
"OrderService",
|
||||
"ServiceError",
|
||||
"ValidationError",
|
||||
]
|
||||
|
||||
# Internal helpers remain private by omission
|
||||
# from .internal_helpers import _validate_input # Not exported
|
||||
```
|
||||
|
||||
### Pattern 3: Flat Directory Structure
|
||||
|
||||
Prefer minimal nesting. Deep hierarchies make imports verbose and navigation difficult.
|
||||
|
||||
```
|
||||
# Preferred: Flat structure
|
||||
project/
|
||||
├── api/
|
||||
│ ├── routes.py
|
||||
│ └── middleware.py
|
||||
├── services/
|
||||
│ ├── user_service.py
|
||||
│ └── order_service.py
|
||||
├── models/
|
||||
│ ├── user.py
|
||||
│ └── order.py
|
||||
└── utils/
|
||||
└── validation.py
|
||||
|
||||
# Avoid: Deep nesting
|
||||
project/core/internal/services/impl/user/
|
||||
```
|
||||
|
||||
Add sub-packages only when there's a genuine sub-domain requiring isolation.
|
||||
|
||||
### Pattern 4: Test File Organization
|
||||
|
||||
Choose one approach and apply it consistently throughout the project.
|
||||
|
||||
**Option A: Colocated Tests**
|
||||
|
||||
```
|
||||
src/
|
||||
├── user_service.py
|
||||
├── test_user_service.py
|
||||
├── order_service.py
|
||||
└── test_order_service.py
|
||||
```
|
||||
|
||||
Benefits: Tests live next to the code they verify. Easy to see coverage gaps.
|
||||
|
||||
**Option B: Parallel Test Directory**
|
||||
|
||||
```
|
||||
src/
|
||||
├── services/
|
||||
│ ├── user_service.py
|
||||
│ └── order_service.py
|
||||
tests/
|
||||
├── services/
|
||||
│ ├── test_user_service.py
|
||||
│ └── test_order_service.py
|
||||
```
|
||||
|
||||
Benefits: Clean separation between production and test code. Standard for larger projects.
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Package Initialization
|
||||
|
||||
Use `__init__.py` to provide a clean public interface for package consumers.
|
||||
|
||||
```python
|
||||
# mypackage/__init__.py
|
||||
"""MyPackage - A library for doing useful things."""
|
||||
|
||||
from .core import MainClass, HelperClass
|
||||
from .exceptions import PackageError, ConfigError
|
||||
from .config import Settings
|
||||
|
||||
__all__ = [
|
||||
"MainClass",
|
||||
"HelperClass",
|
||||
"PackageError",
|
||||
"ConfigError",
|
||||
"Settings",
|
||||
]
|
||||
|
||||
__version__ = "1.0.0"
|
||||
```
|
||||
|
||||
Consumers can then import directly from the package:
|
||||
|
||||
```python
|
||||
from mypackage import MainClass, Settings
|
||||
```
|
||||
|
||||
### Pattern 6: Layered Architecture
|
||||
|
||||
Organize code by architectural layer for clear separation of concerns.
|
||||
|
||||
```
|
||||
myapp/
|
||||
├── api/ # HTTP handlers, request/response
|
||||
│ ├── routes/
|
||||
│ └── middleware/
|
||||
├── services/ # Business logic
|
||||
├── repositories/ # Data access
|
||||
├── models/ # Domain entities
|
||||
├── schemas/ # API schemas (Pydantic)
|
||||
└── config/ # Configuration
|
||||
```
|
||||
|
||||
Each layer should only depend on layers below it, never above.
|
||||
|
||||
### Pattern 7: Domain-Driven Structure
|
||||
|
||||
For complex applications, organize by business domain rather than technical layer.
|
||||
|
||||
```
|
||||
ecommerce/
|
||||
├── users/
|
||||
│ ├── models.py
|
||||
│ ├── services.py
|
||||
│ ├── repository.py
|
||||
│ └── api.py
|
||||
├── orders/
|
||||
│ ├── models.py
|
||||
│ ├── services.py
|
||||
│ ├── repository.py
|
||||
│ └── api.py
|
||||
└── shared/
|
||||
├── database.py
|
||||
└── exceptions.py
|
||||
```
|
||||
|
||||
## File and Module Naming
|
||||
|
||||
### Conventions
|
||||
|
||||
- Use `snake_case` for all file and module names: `user_repository.py`
|
||||
- Avoid abbreviations that obscure meaning: `user_repository.py` not `usr_repo.py`
|
||||
- Match class names to file names: `UserService` in `user_service.py`
|
||||
|
||||
### Import Style
|
||||
|
||||
Use absolute imports for clarity and reliability:
|
||||
|
||||
```python
|
||||
# Preferred: Absolute imports
|
||||
from myproject.services import UserService
|
||||
from myproject.models import User
|
||||
|
||||
# Avoid: Relative imports
|
||||
from ..services import UserService
|
||||
from . import models
|
||||
```
|
||||
|
||||
Relative imports can break when modules are moved or reorganized.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Keep files focused** - One concept per file, consider splitting at 300-500 lines (varies by complexity)
|
||||
2. **Define `__all__` explicitly** - Make public interfaces clear
|
||||
3. **Prefer flat structures** - Add depth only for genuine sub-domains
|
||||
4. **Use absolute imports** - More reliable and clearer
|
||||
5. **Be consistent** - Apply patterns uniformly across the project
|
||||
6. **Match names to content** - File names should describe their purpose
|
||||
7. **Separate concerns** - Keep layers distinct and dependencies flowing one direction
|
||||
8. **Document your structure** - Include a README explaining the organization
|
||||
@@ -0,0 +1,195 @@
|
||||
---
|
||||
name: python-resilience
|
||||
description: Python resilience patterns including automatic retries, exponential backoff, timeouts, and fault-tolerant decorators. Use when adding retry logic, implementing timeouts, building fault-tolerant services, or handling transient failures.
|
||||
---
|
||||
|
||||
# Python Resilience Patterns
|
||||
|
||||
Build fault-tolerant Python applications that gracefully handle transient failures, network issues, and service outages. Resilience patterns keep systems running when dependencies are unreliable.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Adding retry logic to external service calls
|
||||
- Implementing timeouts for network operations
|
||||
- Building fault-tolerant microservices
|
||||
- Handling rate limiting and backpressure
|
||||
- Creating infrastructure decorators
|
||||
- Designing circuit breakers
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Transient vs Permanent Failures
|
||||
|
||||
Retry transient errors (network timeouts, temporary service issues). Don't retry permanent errors (invalid credentials, bad requests).
|
||||
|
||||
### 2. Exponential Backoff
|
||||
|
||||
Increase wait time between retries to avoid overwhelming recovering services.
|
||||
|
||||
### 3. Jitter
|
||||
|
||||
Add randomness to backoff to prevent thundering herd when many clients retry simultaneously.
|
||||
|
||||
### 4. Bounded Retries
|
||||
|
||||
Cap both attempt count and total duration to prevent infinite retry loops.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from tenacity import retry, stop_after_attempt, wait_exponential_jitter
|
||||
|
||||
@retry(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential_jitter(initial=1, max=10),
|
||||
)
|
||||
def call_external_service(request: dict) -> dict:
|
||||
return httpx.post("https://api.example.com", json=request).json()
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Basic Retry with Tenacity
|
||||
|
||||
Use the `tenacity` library for production-grade retry logic. For simpler cases, consider built-in retry functionality or a lightweight custom implementation.
|
||||
|
||||
```python
|
||||
from tenacity import (
|
||||
retry,
|
||||
stop_after_attempt,
|
||||
stop_after_delay,
|
||||
wait_exponential_jitter,
|
||||
retry_if_exception_type,
|
||||
)
|
||||
|
||||
TRANSIENT_ERRORS = (ConnectionError, TimeoutError, OSError)
|
||||
|
||||
@retry(
|
||||
retry=retry_if_exception_type(TRANSIENT_ERRORS),
|
||||
stop=stop_after_attempt(5) | stop_after_delay(60),
|
||||
wait=wait_exponential_jitter(initial=1, max=30),
|
||||
)
|
||||
def fetch_data(url: str) -> dict:
|
||||
"""Fetch data with automatic retry on transient failures."""
|
||||
response = httpx.get(url, timeout=30)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
```
|
||||
|
||||
### Pattern 2: Retry Only Appropriate Errors
|
||||
|
||||
Whitelist specific transient exceptions. Never retry:
|
||||
|
||||
- `ValueError`, `TypeError` - These are bugs, not transient issues
|
||||
- `AuthenticationError` - Invalid credentials won't become valid
|
||||
- HTTP 4xx errors (except 429) - Client errors are permanent
|
||||
|
||||
```python
|
||||
from tenacity import retry, retry_if_exception_type
|
||||
import httpx
|
||||
|
||||
# Define what's retryable
|
||||
RETRYABLE_EXCEPTIONS = (
|
||||
ConnectionError,
|
||||
TimeoutError,
|
||||
httpx.ConnectTimeout,
|
||||
httpx.ReadTimeout,
|
||||
)
|
||||
|
||||
@retry(
|
||||
retry=retry_if_exception_type(RETRYABLE_EXCEPTIONS),
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential_jitter(initial=1, max=10),
|
||||
)
|
||||
def resilient_api_call(endpoint: str) -> dict:
|
||||
"""Make API call with retry on network issues."""
|
||||
return httpx.get(endpoint, timeout=10).json()
|
||||
```
|
||||
|
||||
### Pattern 3: HTTP Status Code Retries
|
||||
|
||||
Retry specific HTTP status codes that indicate transient issues.
|
||||
|
||||
```python
|
||||
from tenacity import retry, retry_if_result, stop_after_attempt
|
||||
import httpx
|
||||
|
||||
RETRY_STATUS_CODES = {429, 502, 503, 504}
|
||||
|
||||
def should_retry_response(response: httpx.Response) -> bool:
|
||||
"""Check if response indicates a retryable error."""
|
||||
return response.status_code in RETRY_STATUS_CODES
|
||||
|
||||
@retry(
|
||||
retry=retry_if_result(should_retry_response),
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential_jitter(initial=1, max=10),
|
||||
)
|
||||
def http_request(method: str, url: str, **kwargs) -> httpx.Response:
|
||||
"""Make HTTP request with retry on transient status codes."""
|
||||
return httpx.request(method, url, timeout=30, **kwargs)
|
||||
```
|
||||
|
||||
### Pattern 4: Combined Exception and Status Retry
|
||||
|
||||
Handle both network exceptions and HTTP status codes.
|
||||
|
||||
```python
|
||||
from tenacity import (
|
||||
retry,
|
||||
retry_if_exception_type,
|
||||
retry_if_result,
|
||||
stop_after_attempt,
|
||||
wait_exponential_jitter,
|
||||
before_sleep_log,
|
||||
)
|
||||
import logging
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TRANSIENT_EXCEPTIONS = (
|
||||
ConnectionError,
|
||||
TimeoutError,
|
||||
httpx.ConnectError,
|
||||
httpx.ReadTimeout,
|
||||
)
|
||||
RETRY_STATUS_CODES = {429, 500, 502, 503, 504}
|
||||
|
||||
def is_retryable_response(response: httpx.Response) -> bool:
|
||||
return response.status_code in RETRY_STATUS_CODES
|
||||
|
||||
@retry(
|
||||
retry=(
|
||||
retry_if_exception_type(TRANSIENT_EXCEPTIONS) |
|
||||
retry_if_result(is_retryable_response)
|
||||
),
|
||||
stop=stop_after_attempt(5),
|
||||
wait=wait_exponential_jitter(initial=1, max=30),
|
||||
before_sleep=before_sleep_log(logger, logging.WARNING),
|
||||
)
|
||||
def robust_http_call(
|
||||
method: str,
|
||||
url: str,
|
||||
**kwargs,
|
||||
) -> httpx.Response:
|
||||
"""HTTP call with comprehensive retry handling."""
|
||||
return httpx.request(method, url, timeout=30, **kwargs)
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Retry only transient errors** - Don't retry bugs or authentication failures
|
||||
2. **Use exponential backoff** - Give services time to recover
|
||||
3. **Add jitter** - Prevent thundering herd from synchronized retries
|
||||
4. **Cap total duration** - `stop_after_attempt(5) | stop_after_delay(60)`
|
||||
5. **Log every retry** - Silent retries hide systemic problems
|
||||
6. **Use decorators** - Keep retry logic separate from business logic
|
||||
7. **Inject dependencies** - Make infrastructure testable
|
||||
8. **Set timeouts everywhere** - Every network call needs a timeout
|
||||
9. **Fail gracefully** - Return cached/default values for non-critical paths
|
||||
10. **Monitor retry rates** - High retry rates indicate underlying issues
|
||||
@@ -0,0 +1,186 @@
|
||||
# python-resilience — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Logging Retry Attempts
|
||||
|
||||
Track retry behavior for debugging and alerting.
|
||||
|
||||
```python
|
||||
from tenacity import retry, stop_after_attempt, wait_exponential
|
||||
import structlog
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
def log_retry_attempt(retry_state):
|
||||
"""Log detailed retry information."""
|
||||
exception = retry_state.outcome.exception()
|
||||
logger.warning(
|
||||
"Retrying operation",
|
||||
attempt=retry_state.attempt_number,
|
||||
exception_type=type(exception).__name__,
|
||||
exception_message=str(exception),
|
||||
next_wait_seconds=retry_state.next_action.sleep if retry_state.next_action else None,
|
||||
)
|
||||
|
||||
@retry(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=1, max=10),
|
||||
before_sleep=log_retry_attempt,
|
||||
)
|
||||
def call_with_logging(request: dict) -> dict:
|
||||
"""External call with retry logging."""
|
||||
...
|
||||
```
|
||||
|
||||
### Pattern 6: Timeout Decorator
|
||||
|
||||
Create reusable timeout decorators for consistent timeout handling.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from functools import wraps
|
||||
from typing import TypeVar, Callable
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
def with_timeout(seconds: float):
|
||||
"""Decorator to add timeout to async functions."""
|
||||
def decorator(func: Callable[..., T]) -> Callable[..., T]:
|
||||
@wraps(func)
|
||||
async def wrapper(*args, **kwargs) -> T:
|
||||
return await asyncio.wait_for(
|
||||
func(*args, **kwargs),
|
||||
timeout=seconds,
|
||||
)
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
@with_timeout(30)
|
||||
async def fetch_with_timeout(url: str) -> dict:
|
||||
"""Fetch URL with 30 second timeout."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get(url)
|
||||
return response.json()
|
||||
```
|
||||
|
||||
### Pattern 7: Cross-Cutting Concerns via Decorators
|
||||
|
||||
Stack decorators to separate infrastructure from business logic.
|
||||
|
||||
```python
|
||||
from functools import wraps
|
||||
from typing import TypeVar, Callable
|
||||
import structlog
|
||||
|
||||
logger = structlog.get_logger()
|
||||
T = TypeVar("T")
|
||||
|
||||
def traced(name: str | None = None):
|
||||
"""Add tracing to function calls."""
|
||||
def decorator(func: Callable[..., T]) -> Callable[..., T]:
|
||||
span_name = name or func.__name__
|
||||
|
||||
@wraps(func)
|
||||
async def wrapper(*args, **kwargs) -> T:
|
||||
logger.info("Operation started", operation=span_name)
|
||||
try:
|
||||
result = await func(*args, **kwargs)
|
||||
logger.info("Operation completed", operation=span_name)
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error("Operation failed", operation=span_name, error=str(e))
|
||||
raise
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
# Stack multiple concerns
|
||||
@traced("fetch_user_data")
|
||||
@with_timeout(30)
|
||||
@retry(stop=stop_after_attempt(3), wait=wait_exponential_jitter())
|
||||
async def fetch_user_data(user_id: str) -> dict:
|
||||
"""Fetch user with tracing, timeout, and retry."""
|
||||
...
|
||||
```
|
||||
|
||||
### Pattern 8: Dependency Injection for Testability
|
||||
|
||||
Pass infrastructure components through constructors for easy testing.
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
class Logger(Protocol):
|
||||
def info(self, msg: str, **kwargs) -> None: ...
|
||||
def error(self, msg: str, **kwargs) -> None: ...
|
||||
|
||||
class MetricsClient(Protocol):
|
||||
def increment(self, metric: str, tags: dict | None = None) -> None: ...
|
||||
def timing(self, metric: str, value: float) -> None: ...
|
||||
|
||||
@dataclass
|
||||
class UserService:
|
||||
"""Service with injected infrastructure."""
|
||||
|
||||
repository: UserRepository
|
||||
logger: Logger
|
||||
metrics: MetricsClient
|
||||
|
||||
async def get_user(self, user_id: str) -> User:
|
||||
self.logger.info("Fetching user", user_id=user_id)
|
||||
start = time.perf_counter()
|
||||
|
||||
try:
|
||||
user = await self.repository.get(user_id)
|
||||
self.metrics.increment("user.fetch.success")
|
||||
return user
|
||||
except Exception as e:
|
||||
self.metrics.increment("user.fetch.error")
|
||||
self.logger.error("Failed to fetch user", user_id=user_id, error=str(e))
|
||||
raise
|
||||
finally:
|
||||
elapsed = time.perf_counter() - start
|
||||
self.metrics.timing("user.fetch.duration", elapsed)
|
||||
|
||||
# Easy to test with fakes
|
||||
service = UserService(
|
||||
repository=FakeRepository(),
|
||||
logger=FakeLogger(),
|
||||
metrics=FakeMetrics(),
|
||||
)
|
||||
```
|
||||
|
||||
### Pattern 9: Fail-Safe Defaults
|
||||
|
||||
Degrade gracefully when non-critical operations fail.
|
||||
|
||||
```python
|
||||
from typing import TypeVar
|
||||
from collections.abc import Callable
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
def fail_safe(default: T, log_failure: bool = True):
|
||||
"""Return default value on failure instead of raising."""
|
||||
def decorator(func: Callable[..., T]) -> Callable[..., T]:
|
||||
@wraps(func)
|
||||
async def wrapper(*args, **kwargs) -> T:
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except Exception as e:
|
||||
if log_failure:
|
||||
logger.warning(
|
||||
"Operation failed, using default",
|
||||
function=func.__name__,
|
||||
error=str(e),
|
||||
)
|
||||
return default
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
@fail_safe(default=[])
|
||||
async def get_recommendations(user_id: str) -> list[str]:
|
||||
"""Get recommendations, return empty list on failure."""
|
||||
...
|
||||
```
|
||||
@@ -0,0 +1,243 @@
|
||||
---
|
||||
name: python-resource-management
|
||||
description: Python resource management with context managers, cleanup patterns, and streaming. Use when managing connections, file handles, implementing cleanup logic, or building streaming responses with accumulated state.
|
||||
---
|
||||
|
||||
# Python Resource Management
|
||||
|
||||
Manage resources deterministically using context managers. Resources like database connections, file handles, and network sockets should be released reliably, even when exceptions occur.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Managing database connections and connection pools
|
||||
- Working with file handles and I/O
|
||||
- Implementing custom context managers
|
||||
- Building streaming responses with state
|
||||
- Handling nested resource cleanup
|
||||
- Creating async context managers
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Context Managers
|
||||
|
||||
The `with` statement ensures resources are released automatically, even on exceptions.
|
||||
|
||||
### 2. Protocol Methods
|
||||
|
||||
`__enter__`/`__exit__` for sync, `__aenter__`/`__aexit__` for async resource management.
|
||||
|
||||
### 3. Unconditional Cleanup
|
||||
|
||||
`__exit__` always runs, regardless of whether an exception occurred.
|
||||
|
||||
### 4. Exception Handling
|
||||
|
||||
Return `True` from `__exit__` to suppress exceptions, `False` to propagate them.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from contextlib import contextmanager
|
||||
|
||||
@contextmanager
|
||||
def managed_resource():
|
||||
resource = acquire_resource()
|
||||
try:
|
||||
yield resource
|
||||
finally:
|
||||
resource.cleanup()
|
||||
|
||||
with managed_resource() as r:
|
||||
r.do_work()
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Class-Based Context Manager
|
||||
|
||||
Implement the context manager protocol for complex resources.
|
||||
|
||||
```python
|
||||
class DatabaseConnection:
|
||||
"""Database connection with automatic cleanup."""
|
||||
|
||||
def __init__(self, dsn: str) -> None:
|
||||
self._dsn = dsn
|
||||
self._conn: Connection | None = None
|
||||
|
||||
def connect(self) -> None:
|
||||
"""Establish database connection."""
|
||||
self._conn = psycopg.connect(self._dsn)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close connection if open."""
|
||||
if self._conn is not None:
|
||||
self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
def __enter__(self) -> "DatabaseConnection":
|
||||
"""Enter context: connect and return self."""
|
||||
self.connect()
|
||||
return self
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: TracebackType | None,
|
||||
) -> None:
|
||||
"""Exit context: always close connection."""
|
||||
self.close()
|
||||
|
||||
# Usage with context manager (preferred)
|
||||
with DatabaseConnection(dsn) as db:
|
||||
result = db.execute(query)
|
||||
|
||||
# Manual management when needed
|
||||
db = DatabaseConnection(dsn)
|
||||
db.connect()
|
||||
try:
|
||||
result = db.execute(query)
|
||||
finally:
|
||||
db.close()
|
||||
```
|
||||
|
||||
### Pattern 2: Async Context Manager
|
||||
|
||||
For async resources, implement the async protocol.
|
||||
|
||||
```python
|
||||
class AsyncDatabasePool:
|
||||
"""Async database connection pool."""
|
||||
|
||||
def __init__(self, dsn: str, min_size: int = 1, max_size: int = 10) -> None:
|
||||
self._dsn = dsn
|
||||
self._min_size = min_size
|
||||
self._max_size = max_size
|
||||
self._pool: asyncpg.Pool | None = None
|
||||
|
||||
async def __aenter__(self) -> "AsyncDatabasePool":
|
||||
"""Create connection pool."""
|
||||
self._pool = await asyncpg.create_pool(
|
||||
self._dsn,
|
||||
min_size=self._min_size,
|
||||
max_size=self._max_size,
|
||||
)
|
||||
return self
|
||||
|
||||
async def __aexit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: TracebackType | None,
|
||||
) -> None:
|
||||
"""Close all connections in pool."""
|
||||
if self._pool is not None:
|
||||
await self._pool.close()
|
||||
|
||||
async def execute(self, query: str, *args) -> list[dict]:
|
||||
"""Execute query using pooled connection."""
|
||||
async with self._pool.acquire() as conn:
|
||||
return await conn.fetch(query, *args)
|
||||
|
||||
# Usage
|
||||
async with AsyncDatabasePool(dsn) as pool:
|
||||
users = await pool.execute("SELECT * FROM users WHERE active = $1", True)
|
||||
```
|
||||
|
||||
### Pattern 3: Using @contextmanager Decorator
|
||||
|
||||
Simplify context managers with the decorator for straightforward cases.
|
||||
|
||||
```python
|
||||
from contextlib import contextmanager, asynccontextmanager
|
||||
import time
|
||||
import structlog
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
@contextmanager
|
||||
def timed_block(name: str):
|
||||
"""Time a block of code."""
|
||||
start = time.perf_counter()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
elapsed = time.perf_counter() - start
|
||||
logger.info(f"{name} completed", duration_seconds=round(elapsed, 3))
|
||||
|
||||
# Usage
|
||||
with timed_block("data_processing"):
|
||||
process_large_dataset()
|
||||
|
||||
@asynccontextmanager
|
||||
async def database_transaction(conn: AsyncConnection):
|
||||
"""Manage database transaction."""
|
||||
await conn.execute("BEGIN")
|
||||
try:
|
||||
yield conn
|
||||
await conn.execute("COMMIT")
|
||||
except Exception:
|
||||
await conn.execute("ROLLBACK")
|
||||
raise
|
||||
|
||||
# Usage
|
||||
async with database_transaction(conn) as tx:
|
||||
await tx.execute("INSERT INTO users ...")
|
||||
await tx.execute("INSERT INTO audit_log ...")
|
||||
```
|
||||
|
||||
### Pattern 4: Unconditional Resource Release
|
||||
|
||||
Always clean up resources in `__exit__`, regardless of exceptions.
|
||||
|
||||
```python
|
||||
class FileProcessor:
|
||||
"""Process file with guaranteed cleanup."""
|
||||
|
||||
def __init__(self, path: str) -> None:
|
||||
self._path = path
|
||||
self._file: IO | None = None
|
||||
self._temp_files: list[Path] = []
|
||||
|
||||
def __enter__(self) -> "FileProcessor":
|
||||
self._file = open(self._path, "r")
|
||||
return self
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: TracebackType | None,
|
||||
) -> None:
|
||||
"""Clean up all resources unconditionally."""
|
||||
# Close main file
|
||||
if self._file is not None:
|
||||
self._file.close()
|
||||
|
||||
# Clean up any temporary files
|
||||
for temp_file in self._temp_files:
|
||||
try:
|
||||
temp_file.unlink()
|
||||
except OSError:
|
||||
pass # Best effort cleanup
|
||||
|
||||
# Return None/False to propagate any exception
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Always use context managers** - For any resource that needs cleanup
|
||||
2. **Clean up unconditionally** - `__exit__` runs even on exception
|
||||
3. **Don't suppress unexpectedly** - Return `False` unless suppression is intentional
|
||||
4. **Use @contextmanager** - For simple resource patterns
|
||||
5. **Implement both protocols** - Support `with` and manual management
|
||||
6. **Use ExitStack** - For dynamic numbers of resources
|
||||
7. **Accumulate efficiently** - List + join, not string concatenation
|
||||
8. **Track metrics** - Time-to-first-byte matters for streaming
|
||||
9. **Document behavior** - Especially exception suppression
|
||||
10. **Test cleanup paths** - Verify resources are released on errors
|
||||
@@ -0,0 +1,183 @@
|
||||
# python-resource-management — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Selective Exception Suppression
|
||||
|
||||
Only suppress specific, documented exceptions.
|
||||
|
||||
```python
|
||||
class StreamWriter:
|
||||
"""Writer that handles broken pipe gracefully."""
|
||||
|
||||
def __init__(self, stream) -> None:
|
||||
self._stream = stream
|
||||
|
||||
def __enter__(self) -> "StreamWriter":
|
||||
return self
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: TracebackType | None,
|
||||
) -> bool:
|
||||
"""Clean up, suppressing BrokenPipeError on shutdown."""
|
||||
self._stream.close()
|
||||
|
||||
# Suppress BrokenPipeError (client disconnected)
|
||||
# This is expected behavior, not an error
|
||||
if exc_type is BrokenPipeError:
|
||||
return True # Exception suppressed
|
||||
|
||||
return False # Propagate all other exceptions
|
||||
```
|
||||
|
||||
### Pattern 6: Streaming with Accumulated State
|
||||
|
||||
Maintain both incremental chunks and accumulated state during streaming.
|
||||
|
||||
```python
|
||||
from collections.abc import Generator
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
@dataclass
|
||||
class StreamingResult:
|
||||
"""Accumulated streaming result."""
|
||||
|
||||
chunks: list[str] = field(default_factory=list)
|
||||
_finalized: bool = False
|
||||
|
||||
@property
|
||||
def content(self) -> str:
|
||||
"""Get accumulated content."""
|
||||
return "".join(self.chunks)
|
||||
|
||||
def add_chunk(self, chunk: str) -> None:
|
||||
"""Add chunk to accumulator."""
|
||||
if self._finalized:
|
||||
raise RuntimeError("Cannot add to finalized result")
|
||||
self.chunks.append(chunk)
|
||||
|
||||
def finalize(self) -> str:
|
||||
"""Mark stream complete and return content."""
|
||||
self._finalized = True
|
||||
return self.content
|
||||
|
||||
def stream_with_accumulation(
|
||||
response: StreamingResponse,
|
||||
) -> Generator[tuple[str, str], None, str]:
|
||||
"""Stream response while accumulating content.
|
||||
|
||||
Yields:
|
||||
Tuple of (accumulated_content, new_chunk) for each chunk.
|
||||
|
||||
Returns:
|
||||
Final accumulated content.
|
||||
"""
|
||||
result = StreamingResult()
|
||||
|
||||
for chunk in response.iter_content():
|
||||
result.add_chunk(chunk)
|
||||
yield result.content, chunk
|
||||
|
||||
return result.finalize()
|
||||
```
|
||||
|
||||
### Pattern 7: Efficient String Accumulation
|
||||
|
||||
Avoid O(n²) string concatenation when accumulating.
|
||||
|
||||
```python
|
||||
def accumulate_stream(stream) -> str:
|
||||
"""Efficiently accumulate stream content."""
|
||||
# BAD: O(n²) due to string immutability
|
||||
# content = ""
|
||||
# for chunk in stream:
|
||||
# content += chunk # Creates new string each time
|
||||
|
||||
# GOOD: O(n) with list and join
|
||||
chunks: list[str] = []
|
||||
for chunk in stream:
|
||||
chunks.append(chunk)
|
||||
return "".join(chunks) # Single allocation
|
||||
```
|
||||
|
||||
### Pattern 8: Tracking Stream Metrics
|
||||
|
||||
Measure time-to-first-byte and total streaming time.
|
||||
|
||||
```python
|
||||
import time
|
||||
from collections.abc import Generator
|
||||
|
||||
def stream_with_metrics(
|
||||
response: StreamingResponse,
|
||||
) -> Generator[str, None, dict]:
|
||||
"""Stream response while collecting metrics.
|
||||
|
||||
Yields:
|
||||
Content chunks.
|
||||
|
||||
Returns:
|
||||
Metrics dictionary.
|
||||
"""
|
||||
start = time.perf_counter()
|
||||
first_chunk_time: float | None = None
|
||||
chunk_count = 0
|
||||
total_bytes = 0
|
||||
|
||||
for chunk in response.iter_content():
|
||||
if first_chunk_time is None:
|
||||
first_chunk_time = time.perf_counter() - start
|
||||
|
||||
chunk_count += 1
|
||||
total_bytes += len(chunk.encode())
|
||||
yield chunk
|
||||
|
||||
total_time = time.perf_counter() - start
|
||||
|
||||
return {
|
||||
"time_to_first_byte_ms": round((first_chunk_time or 0) * 1000, 2),
|
||||
"total_time_ms": round(total_time * 1000, 2),
|
||||
"chunk_count": chunk_count,
|
||||
"total_bytes": total_bytes,
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 9: Managing Multiple Resources with ExitStack
|
||||
|
||||
Handle a dynamic number of resources cleanly.
|
||||
|
||||
```python
|
||||
from contextlib import ExitStack, AsyncExitStack
|
||||
from pathlib import Path
|
||||
|
||||
def process_files(paths: list[Path]) -> list[str]:
|
||||
"""Process multiple files with automatic cleanup."""
|
||||
results = []
|
||||
|
||||
with ExitStack() as stack:
|
||||
# Open all files - they'll all be closed when block exits
|
||||
files = [stack.enter_context(open(p)) for p in paths]
|
||||
|
||||
for f in files:
|
||||
results.append(f.read())
|
||||
|
||||
return results
|
||||
|
||||
async def process_connections(hosts: list[str]) -> list[dict]:
|
||||
"""Process multiple async connections."""
|
||||
results = []
|
||||
|
||||
async with AsyncExitStack() as stack:
|
||||
connections = [
|
||||
await stack.enter_async_context(connect_to_host(host))
|
||||
for host in hosts
|
||||
]
|
||||
|
||||
for conn in connections:
|
||||
results.append(await conn.fetch_data())
|
||||
|
||||
return results
|
||||
```
|
||||
@@ -0,0 +1,278 @@
|
||||
---
|
||||
name: python-testing-patterns
|
||||
description: Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices.
|
||||
---
|
||||
|
||||
# Python Testing Patterns
|
||||
|
||||
Comprehensive guide to implementing robust testing strategies in Python using pytest, fixtures, mocking, parameterization, and test-driven development practices.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Writing unit tests for Python code
|
||||
- Setting up test suites and test infrastructure
|
||||
- Implementing test-driven development (TDD)
|
||||
- Creating integration tests for APIs and services
|
||||
- Mocking external dependencies and services
|
||||
- Testing async code and concurrent operations
|
||||
- Setting up continuous testing in CI/CD
|
||||
- Implementing property-based testing
|
||||
- Testing database operations
|
||||
- Debugging failing tests
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Test Types
|
||||
|
||||
- **Unit Tests**: Test individual functions/classes in isolation
|
||||
- **Integration Tests**: Test interaction between components
|
||||
- **Functional Tests**: Test complete features end-to-end
|
||||
- **Performance Tests**: Measure speed and resource usage
|
||||
|
||||
### 2. Test Structure (AAA Pattern)
|
||||
|
||||
- **Arrange**: Set up test data and preconditions
|
||||
- **Act**: Execute the code under test
|
||||
- **Assert**: Verify the results
|
||||
|
||||
### 3. Test Coverage
|
||||
|
||||
- Measure what code is exercised by tests
|
||||
- Identify untested code paths
|
||||
- Aim for meaningful coverage, not just high percentages
|
||||
|
||||
### 4. Test Isolation
|
||||
|
||||
- Tests should be independent
|
||||
- No shared state between tests
|
||||
- Each test should clean up after itself
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
# test_example.py
|
||||
def add(a, b):
|
||||
return a + b
|
||||
|
||||
def test_add():
|
||||
"""Basic test example."""
|
||||
result = add(2, 3)
|
||||
assert result == 5
|
||||
|
||||
def test_add_negative():
|
||||
"""Test with negative numbers."""
|
||||
assert add(-1, 1) == 0
|
||||
|
||||
# Run with: pytest test_example.py
|
||||
```
|
||||
|
||||
## Detailed patterns and worked examples
|
||||
|
||||
Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
|
||||
|
||||
## Testing Best Practices
|
||||
|
||||
### Test Organization
|
||||
|
||||
```python
|
||||
# tests/
|
||||
# __init__.py
|
||||
# conftest.py # Shared fixtures
|
||||
# test_unit/ # Unit tests
|
||||
# test_models.py
|
||||
# test_utils.py
|
||||
# test_integration/ # Integration tests
|
||||
# test_api.py
|
||||
# test_database.py
|
||||
# test_e2e/ # End-to-end tests
|
||||
# test_workflows.py
|
||||
```
|
||||
|
||||
### Test Naming Convention
|
||||
|
||||
A common pattern: `test_<unit>_<scenario>_<expected_outcome>`. Adapt to your team's preferences.
|
||||
|
||||
```python
|
||||
# Pattern: test_<unit>_<scenario>_<expected>
|
||||
def test_create_user_with_valid_data_returns_user():
|
||||
...
|
||||
|
||||
def test_create_user_with_duplicate_email_raises_conflict():
|
||||
...
|
||||
|
||||
def test_get_user_with_unknown_id_returns_none():
|
||||
...
|
||||
|
||||
# Good test names - clear and descriptive
|
||||
def test_user_creation_with_valid_data():
|
||||
"""Clear name describes what is being tested."""
|
||||
pass
|
||||
|
||||
def test_login_fails_with_invalid_password():
|
||||
"""Name describes expected behavior."""
|
||||
pass
|
||||
|
||||
def test_api_returns_404_for_missing_resource():
|
||||
"""Specific about inputs and expected outcomes."""
|
||||
pass
|
||||
|
||||
# Bad test names - avoid these
|
||||
def test_1(): # Not descriptive
|
||||
pass
|
||||
|
||||
def test_user(): # Too vague
|
||||
pass
|
||||
|
||||
def test_function(): # Doesn't explain what's tested
|
||||
pass
|
||||
```
|
||||
|
||||
### Testing Retry Behavior
|
||||
|
||||
Verify that retry logic works correctly using mock side effects.
|
||||
|
||||
```python
|
||||
from unittest.mock import Mock
|
||||
|
||||
def test_retries_on_transient_error():
|
||||
"""Test that service retries on transient failures."""
|
||||
client = Mock()
|
||||
# Fail twice, then succeed
|
||||
client.request.side_effect = [
|
||||
ConnectionError("Failed"),
|
||||
ConnectionError("Failed"),
|
||||
{"status": "ok"},
|
||||
]
|
||||
|
||||
service = ServiceWithRetry(client, max_retries=3)
|
||||
result = service.fetch()
|
||||
|
||||
assert result == {"status": "ok"}
|
||||
assert client.request.call_count == 3
|
||||
|
||||
def test_gives_up_after_max_retries():
|
||||
"""Test that service stops retrying after max attempts."""
|
||||
client = Mock()
|
||||
client.request.side_effect = ConnectionError("Failed")
|
||||
|
||||
service = ServiceWithRetry(client, max_retries=3)
|
||||
|
||||
with pytest.raises(ConnectionError):
|
||||
service.fetch()
|
||||
|
||||
assert client.request.call_count == 3
|
||||
|
||||
def test_does_not_retry_on_permanent_error():
|
||||
"""Test that permanent errors are not retried."""
|
||||
client = Mock()
|
||||
client.request.side_effect = ValueError("Invalid input")
|
||||
|
||||
service = ServiceWithRetry(client, max_retries=3)
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
service.fetch()
|
||||
|
||||
# Only called once - no retry for ValueError
|
||||
assert client.request.call_count == 1
|
||||
```
|
||||
|
||||
### Mocking Time with Freezegun
|
||||
|
||||
Use freezegun to control time in tests for predictable time-dependent behavior.
|
||||
|
||||
```python
|
||||
from freezegun import freeze_time
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
@freeze_time("2026-01-15 10:00:00")
|
||||
def test_token_expiry():
|
||||
"""Test token expires at correct time."""
|
||||
token = create_token(expires_in_seconds=3600)
|
||||
assert token.expires_at == datetime(2026, 1, 15, 11, 0, 0)
|
||||
|
||||
@freeze_time("2026-01-15 10:00:00")
|
||||
def test_is_expired_returns_false_before_expiry():
|
||||
"""Test token is not expired when within validity period."""
|
||||
token = create_token(expires_in_seconds=3600)
|
||||
assert not token.is_expired()
|
||||
|
||||
@freeze_time("2026-01-15 12:00:00")
|
||||
def test_is_expired_returns_true_after_expiry():
|
||||
"""Test token is expired after validity period."""
|
||||
token = Token(expires_at=datetime(2026, 1, 15, 11, 30, 0))
|
||||
assert token.is_expired()
|
||||
|
||||
def test_with_time_travel():
|
||||
"""Test behavior across time using freeze_time context."""
|
||||
with freeze_time("2026-01-01") as frozen_time:
|
||||
item = create_item()
|
||||
assert item.created_at == datetime(2026, 1, 1)
|
||||
|
||||
# Move forward in time
|
||||
frozen_time.move_to("2026-01-15")
|
||||
assert item.age_days == 14
|
||||
```
|
||||
|
||||
### Test Markers
|
||||
|
||||
```python
|
||||
# test_markers.py
|
||||
import pytest
|
||||
|
||||
@pytest.mark.slow
|
||||
def test_slow_operation():
|
||||
"""Mark slow tests."""
|
||||
import time
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_database_integration():
|
||||
"""Mark integration tests."""
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.skip(reason="Feature not implemented yet")
|
||||
def test_future_feature():
|
||||
"""Skip tests temporarily."""
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name == "nt", reason="Unix only test")
|
||||
def test_unix_specific():
|
||||
"""Conditional skip."""
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.xfail(reason="Known bug #123")
|
||||
def test_known_bug():
|
||||
"""Mark expected failures."""
|
||||
assert False
|
||||
|
||||
|
||||
# Run with:
|
||||
# pytest -m slow # Run only slow tests
|
||||
# pytest -m "not slow" # Skip slow tests
|
||||
# pytest -m integration # Run integration tests
|
||||
```
|
||||
|
||||
### Coverage Reporting
|
||||
|
||||
```bash
|
||||
# Install coverage
|
||||
pip install pytest-cov
|
||||
|
||||
# Run tests with coverage
|
||||
pytest --cov=myapp tests/
|
||||
|
||||
# Generate HTML report
|
||||
pytest --cov=myapp --cov-report=html tests/
|
||||
|
||||
# Fail if coverage below threshold
|
||||
pytest --cov=myapp --cov-fail-under=80 tests/
|
||||
|
||||
# Show missing lines
|
||||
pytest --cov=myapp --cov-report=term-missing tests/
|
||||
```
|
||||
|
||||
For advanced patterns (async testing, monkeypatching, property-based testing, database testing, CI/CD integration, and configuration), see [references/advanced-patterns.md](references/advanced-patterns.md)
|
||||
+411
@@ -0,0 +1,411 @@
|
||||
# Python Testing Patterns — Advanced Reference
|
||||
|
||||
Advanced testing patterns including async code, monkeypatching, temporary files, conftest setup, property-based testing, database testing, CI/CD integration, and configuration.
|
||||
|
||||
## Pattern 6: Testing Async Code
|
||||
|
||||
```python
|
||||
# test_async.py
|
||||
import pytest
|
||||
import asyncio
|
||||
|
||||
async def fetch_data(url: str) -> dict:
|
||||
"""Fetch data asynchronously."""
|
||||
await asyncio.sleep(0.1)
|
||||
return {"url": url, "data": "result"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_data():
|
||||
"""Test async function."""
|
||||
result = await fetch_data("https://api.example.com")
|
||||
assert result["url"] == "https://api.example.com"
|
||||
assert "data" in result
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_fetches():
|
||||
"""Test concurrent async operations."""
|
||||
urls = ["url1", "url2", "url3"]
|
||||
tasks = [fetch_data(url) for url in urls]
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
assert len(results) == 3
|
||||
assert all("data" in r for r in results)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def async_client():
|
||||
"""Async fixture."""
|
||||
client = {"connected": True}
|
||||
yield client
|
||||
client["connected"] = False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_with_async_fixture(async_client):
|
||||
"""Test using async fixture."""
|
||||
assert async_client["connected"] is True
|
||||
```
|
||||
|
||||
## Pattern 7: Monkeypatch for Testing
|
||||
|
||||
```python
|
||||
# test_environment.py
|
||||
import os
|
||||
import pytest
|
||||
|
||||
def get_database_url() -> str:
|
||||
"""Get database URL from environment."""
|
||||
return os.environ.get("DATABASE_URL", "sqlite:///:memory:")
|
||||
|
||||
|
||||
def test_database_url_default():
|
||||
"""Test default database URL."""
|
||||
# Will use actual environment variable if set
|
||||
url = get_database_url()
|
||||
assert url
|
||||
|
||||
|
||||
def test_database_url_custom(monkeypatch):
|
||||
"""Test custom database URL with monkeypatch."""
|
||||
monkeypatch.setenv("DATABASE_URL", "postgresql://localhost/test")
|
||||
assert get_database_url() == "postgresql://localhost/test"
|
||||
|
||||
|
||||
def test_database_url_not_set(monkeypatch):
|
||||
"""Test when env var is not set."""
|
||||
monkeypatch.delenv("DATABASE_URL", raising=False)
|
||||
assert get_database_url() == "sqlite:///:memory:"
|
||||
|
||||
|
||||
class Config:
|
||||
"""Configuration class."""
|
||||
|
||||
def __init__(self):
|
||||
self.api_key = "production-key"
|
||||
|
||||
def get_api_key(self):
|
||||
return self.api_key
|
||||
|
||||
|
||||
def test_monkeypatch_attribute(monkeypatch):
|
||||
"""Test monkeypatching object attributes."""
|
||||
config = Config()
|
||||
monkeypatch.setattr(config, "api_key", "test-key")
|
||||
assert config.get_api_key() == "test-key"
|
||||
```
|
||||
|
||||
## Pattern 8: Temporary Files and Directories
|
||||
|
||||
```python
|
||||
# test_file_operations.py
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
def save_data(filepath: Path, data: str):
|
||||
"""Save data to file."""
|
||||
filepath.write_text(data)
|
||||
|
||||
|
||||
def load_data(filepath: Path) -> str:
|
||||
"""Load data from file."""
|
||||
return filepath.read_text()
|
||||
|
||||
|
||||
def test_file_operations(tmp_path):
|
||||
"""Test file operations with temporary directory."""
|
||||
# tmp_path is a pathlib.Path object
|
||||
test_file = tmp_path / "test_data.txt"
|
||||
|
||||
# Save data
|
||||
save_data(test_file, "Hello, World!")
|
||||
|
||||
# Verify file exists
|
||||
assert test_file.exists()
|
||||
|
||||
# Load and verify data
|
||||
data = load_data(test_file)
|
||||
assert data == "Hello, World!"
|
||||
|
||||
|
||||
def test_multiple_files(tmp_path):
|
||||
"""Test with multiple temporary files."""
|
||||
files = {
|
||||
"file1.txt": "Content 1",
|
||||
"file2.txt": "Content 2",
|
||||
"file3.txt": "Content 3"
|
||||
}
|
||||
|
||||
for filename, content in files.items():
|
||||
filepath = tmp_path / filename
|
||||
save_data(filepath, content)
|
||||
|
||||
# Verify all files created
|
||||
assert len(list(tmp_path.iterdir())) == 3
|
||||
|
||||
# Verify contents
|
||||
for filename, expected_content in files.items():
|
||||
filepath = tmp_path / filename
|
||||
assert load_data(filepath) == expected_content
|
||||
```
|
||||
|
||||
## Pattern 9: Custom Fixtures and Conftest
|
||||
|
||||
```python
|
||||
# conftest.py
|
||||
"""Shared fixtures for all tests."""
|
||||
import pytest
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def database_url():
|
||||
"""Provide database URL for all tests."""
|
||||
return "postgresql://localhost/test_db"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_database(database_url):
|
||||
"""Auto-use fixture that runs before each test."""
|
||||
# Setup: Clear database
|
||||
print(f"Clearing database: {database_url}")
|
||||
yield
|
||||
# Teardown: Clean up
|
||||
print("Test completed")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_user():
|
||||
"""Provide sample user data."""
|
||||
return {
|
||||
"id": 1,
|
||||
"name": "Test User",
|
||||
"email": "test@example.com"
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_users():
|
||||
"""Provide list of sample users."""
|
||||
return [
|
||||
{"id": 1, "name": "User 1"},
|
||||
{"id": 2, "name": "User 2"},
|
||||
{"id": 3, "name": "User 3"},
|
||||
]
|
||||
|
||||
|
||||
# Parametrized fixture
|
||||
@pytest.fixture(params=["sqlite", "postgresql", "mysql"])
|
||||
def db_backend(request):
|
||||
"""Fixture that runs tests with different database backends."""
|
||||
return request.param
|
||||
|
||||
|
||||
def test_with_db_backend(db_backend):
|
||||
"""This test will run 3 times with different backends."""
|
||||
print(f"Testing with {db_backend}")
|
||||
assert db_backend in ["sqlite", "postgresql", "mysql"]
|
||||
```
|
||||
|
||||
## Pattern 10: Property-Based Testing
|
||||
|
||||
```python
|
||||
# test_properties.py
|
||||
from hypothesis import given, strategies as st
|
||||
import pytest
|
||||
|
||||
def reverse_string(s: str) -> str:
|
||||
"""Reverse a string."""
|
||||
return s[::-1]
|
||||
|
||||
|
||||
@given(st.text())
|
||||
def test_reverse_twice_is_original(s):
|
||||
"""Property: reversing twice returns original."""
|
||||
assert reverse_string(reverse_string(s)) == s
|
||||
|
||||
|
||||
@given(st.text())
|
||||
def test_reverse_length(s):
|
||||
"""Property: reversed string has same length."""
|
||||
assert len(reverse_string(s)) == len(s)
|
||||
|
||||
|
||||
@given(st.integers(), st.integers())
|
||||
def test_addition_commutative(a, b):
|
||||
"""Property: addition is commutative."""
|
||||
assert a + b == b + a
|
||||
|
||||
|
||||
@given(st.lists(st.integers()))
|
||||
def test_sorted_list_properties(lst):
|
||||
"""Property: sorted list is ordered."""
|
||||
sorted_lst = sorted(lst)
|
||||
|
||||
# Same length
|
||||
assert len(sorted_lst) == len(lst)
|
||||
|
||||
# All elements present
|
||||
assert set(sorted_lst) == set(lst)
|
||||
|
||||
# Is ordered
|
||||
for i in range(len(sorted_lst) - 1):
|
||||
assert sorted_lst[i] <= sorted_lst[i + 1]
|
||||
```
|
||||
|
||||
## Testing Database Code
|
||||
|
||||
```python
|
||||
# test_database_models.py
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, Column, Integer, String
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.orm import sessionmaker, Session
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
class User(Base):
|
||||
"""User model."""
|
||||
__tablename__ = "users"
|
||||
|
||||
id = Column(Integer, primary_key=True)
|
||||
name = Column(String(50))
|
||||
email = Column(String(100), unique=True)
|
||||
|
||||
|
||||
@pytest.fixture(scope="function")
|
||||
def db_session() -> Session:
|
||||
"""Create in-memory database for testing."""
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
|
||||
SessionLocal = sessionmaker(bind=engine)
|
||||
session = SessionLocal()
|
||||
|
||||
yield session
|
||||
|
||||
session.close()
|
||||
|
||||
|
||||
def test_create_user(db_session):
|
||||
"""Test creating a user."""
|
||||
user = User(name="Test User", email="test@example.com")
|
||||
db_session.add(user)
|
||||
db_session.commit()
|
||||
|
||||
assert user.id is not None
|
||||
assert user.name == "Test User"
|
||||
|
||||
|
||||
def test_query_user(db_session):
|
||||
"""Test querying users."""
|
||||
user1 = User(name="User 1", email="user1@example.com")
|
||||
user2 = User(name="User 2", email="user2@example.com")
|
||||
|
||||
db_session.add_all([user1, user2])
|
||||
db_session.commit()
|
||||
|
||||
users = db_session.query(User).all()
|
||||
assert len(users) == 2
|
||||
|
||||
|
||||
def test_unique_email_constraint(db_session):
|
||||
"""Test unique email constraint."""
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
user1 = User(name="User 1", email="same@example.com")
|
||||
user2 = User(name="User 2", email="same@example.com")
|
||||
|
||||
db_session.add(user1)
|
||||
db_session.commit()
|
||||
|
||||
db_session.add(user2)
|
||||
|
||||
with pytest.raises(IntegrityError):
|
||||
db_session.commit()
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
```yaml
|
||||
# .github/workflows/test.yml
|
||||
name: Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.9", "3.10", "3.11", "3.12"]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install -e ".[dev]"
|
||||
pip install pytest pytest-cov
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
pytest --cov=myapp --cov-report=xml
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
with:
|
||||
file: ./coverage.xml
|
||||
```
|
||||
|
||||
## Configuration Files
|
||||
|
||||
```ini
|
||||
# pytest.ini
|
||||
[pytest]
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
addopts =
|
||||
-v
|
||||
--strict-markers
|
||||
--tb=short
|
||||
--cov=myapp
|
||||
--cov-report=term-missing
|
||||
markers =
|
||||
slow: marks tests as slow
|
||||
integration: marks integration tests
|
||||
unit: marks unit tests
|
||||
e2e: marks end-to-end tests
|
||||
```
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--cov=myapp",
|
||||
"--cov-report=term-missing",
|
||||
]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["myapp"]
|
||||
omit = ["*/tests/*", "*/migrations/*"]
|
||||
|
||||
[tool.coverage.report]
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"def __repr__",
|
||||
"raise AssertionError",
|
||||
"raise NotImplementedError",
|
||||
]
|
||||
```
|
||||
@@ -0,0 +1,349 @@
|
||||
# python-testing-patterns — detailed patterns and worked examples
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Basic pytest Tests
|
||||
|
||||
```python
|
||||
# test_calculator.py
|
||||
import pytest
|
||||
|
||||
class Calculator:
|
||||
"""Simple calculator for testing."""
|
||||
|
||||
def add(self, a: float, b: float) -> float:
|
||||
return a + b
|
||||
|
||||
def subtract(self, a: float, b: float) -> float:
|
||||
return a - b
|
||||
|
||||
def multiply(self, a: float, b: float) -> float:
|
||||
return a * b
|
||||
|
||||
def divide(self, a: float, b: float) -> float:
|
||||
if b == 0:
|
||||
raise ValueError("Cannot divide by zero")
|
||||
return a / b
|
||||
|
||||
|
||||
def test_addition():
|
||||
"""Test addition."""
|
||||
calc = Calculator()
|
||||
assert calc.add(2, 3) == 5
|
||||
assert calc.add(-1, 1) == 0
|
||||
assert calc.add(0, 0) == 0
|
||||
|
||||
|
||||
def test_subtraction():
|
||||
"""Test subtraction."""
|
||||
calc = Calculator()
|
||||
assert calc.subtract(5, 3) == 2
|
||||
assert calc.subtract(0, 5) == -5
|
||||
|
||||
|
||||
def test_multiplication():
|
||||
"""Test multiplication."""
|
||||
calc = Calculator()
|
||||
assert calc.multiply(3, 4) == 12
|
||||
assert calc.multiply(0, 5) == 0
|
||||
|
||||
|
||||
def test_division():
|
||||
"""Test division."""
|
||||
calc = Calculator()
|
||||
assert calc.divide(6, 3) == 2
|
||||
assert calc.divide(5, 2) == 2.5
|
||||
|
||||
|
||||
def test_division_by_zero():
|
||||
"""Test division by zero raises error."""
|
||||
calc = Calculator()
|
||||
with pytest.raises(ValueError, match="Cannot divide by zero"):
|
||||
calc.divide(5, 0)
|
||||
```
|
||||
|
||||
### Pattern 2: Fixtures for Setup and Teardown
|
||||
|
||||
```python
|
||||
# test_database.py
|
||||
import pytest
|
||||
from typing import Generator
|
||||
|
||||
class Database:
|
||||
"""Simple database class."""
|
||||
|
||||
def __init__(self, connection_string: str):
|
||||
self.connection_string = connection_string
|
||||
self.connected = False
|
||||
|
||||
def connect(self):
|
||||
"""Connect to database."""
|
||||
self.connected = True
|
||||
|
||||
def disconnect(self):
|
||||
"""Disconnect from database."""
|
||||
self.connected = False
|
||||
|
||||
def query(self, sql: str) -> list:
|
||||
"""Execute query."""
|
||||
if not self.connected:
|
||||
raise RuntimeError("Not connected")
|
||||
return [{"id": 1, "name": "Test"}]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db() -> Generator[Database, None, None]:
|
||||
"""Fixture that provides connected database."""
|
||||
# Setup
|
||||
database = Database("sqlite:///:memory:")
|
||||
database.connect()
|
||||
|
||||
# Provide to test
|
||||
yield database
|
||||
|
||||
# Teardown
|
||||
database.disconnect()
|
||||
|
||||
|
||||
def test_database_query(db):
|
||||
"""Test database query with fixture."""
|
||||
results = db.query("SELECT * FROM users")
|
||||
assert len(results) == 1
|
||||
assert results[0]["name"] == "Test"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def app_config():
|
||||
"""Session-scoped fixture - created once per test session."""
|
||||
return {
|
||||
"database_url": "postgresql://localhost/test",
|
||||
"api_key": "test-key",
|
||||
"debug": True
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def api_client(app_config):
|
||||
"""Module-scoped fixture - created once per test module."""
|
||||
# Setup expensive resource
|
||||
client = {"config": app_config, "session": "active"}
|
||||
yield client
|
||||
# Cleanup
|
||||
client["session"] = "closed"
|
||||
|
||||
|
||||
def test_api_client(api_client):
|
||||
"""Test using api client fixture."""
|
||||
assert api_client["session"] == "active"
|
||||
assert api_client["config"]["debug"] is True
|
||||
```
|
||||
|
||||
### Pattern 3: Parameterized Tests
|
||||
|
||||
```python
|
||||
# test_validation.py
|
||||
import pytest
|
||||
|
||||
def is_valid_email(email: str) -> bool:
|
||||
"""Check if email is valid."""
|
||||
return "@" in email and "." in email.split("@")[1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("email,expected", [
|
||||
("user@example.com", True),
|
||||
("test.user@domain.co.uk", True),
|
||||
("invalid.email", False),
|
||||
("@example.com", False),
|
||||
("user@domain", False),
|
||||
("", False),
|
||||
])
|
||||
def test_email_validation(email, expected):
|
||||
"""Test email validation with various inputs."""
|
||||
assert is_valid_email(email) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("a,b,expected", [
|
||||
(2, 3, 5),
|
||||
(0, 0, 0),
|
||||
(-1, 1, 0),
|
||||
(100, 200, 300),
|
||||
(-5, -5, -10),
|
||||
])
|
||||
def test_addition_parameterized(a, b, expected):
|
||||
"""Test addition with multiple parameter sets."""
|
||||
from test_calculator import Calculator
|
||||
calc = Calculator()
|
||||
assert calc.add(a, b) == expected
|
||||
|
||||
|
||||
# Using pytest.param for special cases
|
||||
@pytest.mark.parametrize("value,expected", [
|
||||
pytest.param(1, True, id="positive"),
|
||||
pytest.param(0, False, id="zero"),
|
||||
pytest.param(-1, False, id="negative"),
|
||||
])
|
||||
def test_is_positive(value, expected):
|
||||
"""Test with custom test IDs."""
|
||||
assert (value > 0) == expected
|
||||
```
|
||||
|
||||
### Pattern 4: Mocking with unittest.mock
|
||||
|
||||
```python
|
||||
# test_api_client.py
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
import requests
|
||||
|
||||
class APIClient:
|
||||
"""Simple API client."""
|
||||
|
||||
def __init__(self, base_url: str):
|
||||
self.base_url = base_url
|
||||
|
||||
def get_user(self, user_id: int) -> dict:
|
||||
"""Fetch user from API."""
|
||||
response = requests.get(f"{self.base_url}/users/{user_id}")
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
def create_user(self, data: dict) -> dict:
|
||||
"""Create new user."""
|
||||
response = requests.post(f"{self.base_url}/users", json=data)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
|
||||
def test_get_user_success():
|
||||
"""Test successful API call with mock."""
|
||||
client = APIClient("https://api.example.com")
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.json.return_value = {"id": 1, "name": "John Doe"}
|
||||
mock_response.raise_for_status.return_value = None
|
||||
|
||||
with patch("requests.get", return_value=mock_response) as mock_get:
|
||||
user = client.get_user(1)
|
||||
|
||||
assert user["id"] == 1
|
||||
assert user["name"] == "John Doe"
|
||||
mock_get.assert_called_once_with("https://api.example.com/users/1")
|
||||
|
||||
|
||||
def test_get_user_not_found():
|
||||
"""Test API call with 404 error."""
|
||||
client = APIClient("https://api.example.com")
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.raise_for_status.side_effect = requests.HTTPError("404 Not Found")
|
||||
|
||||
with patch("requests.get", return_value=mock_response):
|
||||
with pytest.raises(requests.HTTPError):
|
||||
client.get_user(999)
|
||||
|
||||
|
||||
@patch("requests.post")
|
||||
def test_create_user(mock_post):
|
||||
"""Test user creation with decorator syntax."""
|
||||
client = APIClient("https://api.example.com")
|
||||
|
||||
mock_post.return_value.json.return_value = {"id": 2, "name": "Jane Doe"}
|
||||
mock_post.return_value.raise_for_status.return_value = None
|
||||
|
||||
user_data = {"name": "Jane Doe", "email": "jane@example.com"}
|
||||
result = client.create_user(user_data)
|
||||
|
||||
assert result["id"] == 2
|
||||
mock_post.assert_called_once()
|
||||
call_args = mock_post.call_args
|
||||
assert call_args.kwargs["json"] == user_data
|
||||
```
|
||||
|
||||
### Pattern 5: Testing Exceptions
|
||||
|
||||
```python
|
||||
# test_exceptions.py
|
||||
import pytest
|
||||
|
||||
def divide(a: float, b: float) -> float:
|
||||
"""Divide a by b."""
|
||||
if b == 0:
|
||||
raise ZeroDivisionError("Division by zero")
|
||||
if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
|
||||
raise TypeError("Arguments must be numbers")
|
||||
return a / b
|
||||
|
||||
|
||||
def test_zero_division():
|
||||
"""Test exception is raised for division by zero."""
|
||||
with pytest.raises(ZeroDivisionError):
|
||||
divide(10, 0)
|
||||
|
||||
|
||||
def test_zero_division_with_message():
|
||||
"""Test exception message."""
|
||||
with pytest.raises(ZeroDivisionError, match="Division by zero"):
|
||||
divide(5, 0)
|
||||
|
||||
|
||||
def test_type_error():
|
||||
"""Test type error exception."""
|
||||
with pytest.raises(TypeError, match="must be numbers"):
|
||||
divide("10", 5)
|
||||
|
||||
|
||||
def test_exception_info():
|
||||
"""Test accessing exception info."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
int("not a number")
|
||||
|
||||
assert "invalid literal" in str(exc_info.value)
|
||||
```
|
||||
|
||||
For advanced patterns including async testing, monkeypatching, temporary files, conftest setup, property-based testing, database testing, CI/CD integration, and configuration files, see [references/advanced-patterns.md](references/advanced-patterns.md)
|
||||
|
||||
## Test Design Principles
|
||||
|
||||
### One Behavior Per Test
|
||||
|
||||
Each test should verify exactly one behavior. This makes failures easy to diagnose and tests easy to maintain.
|
||||
|
||||
```python
|
||||
# BAD - testing multiple behaviors
|
||||
def test_user_service():
|
||||
user = service.create_user(data)
|
||||
assert user.id is not None
|
||||
assert user.email == data["email"]
|
||||
updated = service.update_user(user.id, {"name": "New"})
|
||||
assert updated.name == "New"
|
||||
|
||||
# GOOD - focused tests
|
||||
def test_create_user_assigns_id():
|
||||
user = service.create_user(data)
|
||||
assert user.id is not None
|
||||
|
||||
def test_create_user_stores_email():
|
||||
user = service.create_user(data)
|
||||
assert user.email == data["email"]
|
||||
|
||||
def test_update_user_changes_name():
|
||||
user = service.create_user(data)
|
||||
updated = service.update_user(user.id, {"name": "New"})
|
||||
assert updated.name == "New"
|
||||
```
|
||||
|
||||
### Test Error Paths
|
||||
|
||||
Always test failure cases, not just happy paths.
|
||||
|
||||
```python
|
||||
def test_get_user_raises_not_found():
|
||||
with pytest.raises(UserNotFoundError) as exc_info:
|
||||
service.get_user("nonexistent-id")
|
||||
|
||||
assert "nonexistent-id" in str(exc_info.value)
|
||||
|
||||
def test_create_user_rejects_invalid_email():
|
||||
with pytest.raises(ValueError, match="Invalid email format"):
|
||||
service.create_user({"email": "not-an-email"})
|
||||
```
|
||||
@@ -0,0 +1,200 @@
|
||||
---
|
||||
name: python-type-safety
|
||||
description: Python type safety with type hints, generics, protocols, and strict type checking. Use when adding type annotations, implementing generic classes, defining structural interfaces, or configuring mypy/pyright.
|
||||
---
|
||||
|
||||
# Python Type Safety
|
||||
|
||||
Leverage Python's type system to catch errors at static analysis time. Type annotations serve as enforced documentation that tooling validates automatically.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Adding type hints to existing code
|
||||
- Creating generic, reusable classes
|
||||
- Defining structural interfaces with protocols
|
||||
- Configuring mypy or pyright for strict checking
|
||||
- Understanding type narrowing and guards
|
||||
- Building type-safe APIs and libraries
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Type Annotations
|
||||
|
||||
Declare expected types for function parameters, return values, and variables.
|
||||
|
||||
### 2. Generics
|
||||
|
||||
Write reusable code that preserves type information across different types.
|
||||
|
||||
### 3. Protocols
|
||||
|
||||
Define structural interfaces without inheritance (duck typing with type safety).
|
||||
|
||||
### 4. Type Narrowing
|
||||
|
||||
Use guards and conditionals to narrow types within code blocks.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
def get_user(user_id: str) -> User | None:
|
||||
"""Return type makes 'might not exist' explicit."""
|
||||
...
|
||||
|
||||
# Type checker enforces handling None case
|
||||
user = get_user("123")
|
||||
if user is None:
|
||||
raise UserNotFoundError("123")
|
||||
print(user.name) # Type checker knows user is User here
|
||||
```
|
||||
|
||||
## Fundamental Patterns
|
||||
|
||||
### Pattern 1: Annotate All Public Signatures
|
||||
|
||||
Every public function, method, and class should have type annotations.
|
||||
|
||||
```python
|
||||
def get_user(user_id: str) -> User:
|
||||
"""Retrieve user by ID."""
|
||||
...
|
||||
|
||||
def process_batch(
|
||||
items: list[Item],
|
||||
max_workers: int = 4,
|
||||
) -> BatchResult[ProcessedItem]:
|
||||
"""Process items concurrently."""
|
||||
...
|
||||
|
||||
class UserRepository:
|
||||
def __init__(self, db: Database) -> None:
|
||||
self._db = db
|
||||
|
||||
async def find_by_id(self, user_id: str) -> User | None:
|
||||
"""Return User if found, None otherwise."""
|
||||
...
|
||||
|
||||
async def find_by_email(self, email: str) -> User | None:
|
||||
...
|
||||
|
||||
async def save(self, user: User) -> User:
|
||||
"""Save and return user with generated ID."""
|
||||
...
|
||||
```
|
||||
|
||||
Use `mypy --strict` or `pyright` in CI to catch type errors early. For existing projects, enable strict mode incrementally using per-module overrides.
|
||||
|
||||
### Pattern 2: Use Modern Union Syntax
|
||||
|
||||
Python 3.10+ provides cleaner union syntax.
|
||||
|
||||
```python
|
||||
# Preferred (3.10+)
|
||||
def find_user(user_id: str) -> User | None:
|
||||
...
|
||||
|
||||
def parse_value(v: str) -> int | float | str:
|
||||
...
|
||||
|
||||
# Older style (still valid, needed for 3.9)
|
||||
from typing import Optional, Union
|
||||
|
||||
def find_user(user_id: str) -> Optional[User]:
|
||||
...
|
||||
```
|
||||
|
||||
### Pattern 3: Type Narrowing with Guards
|
||||
|
||||
Use conditionals to narrow types for the type checker.
|
||||
|
||||
```python
|
||||
def process_user(user_id: str) -> UserData:
|
||||
user = find_user(user_id)
|
||||
|
||||
if user is None:
|
||||
raise UserNotFoundError(f"User {user_id} not found")
|
||||
|
||||
# Type checker knows user is User here, not User | None
|
||||
return UserData(
|
||||
name=user.name,
|
||||
email=user.email,
|
||||
)
|
||||
|
||||
def process_items(items: list[Item | None]) -> list[ProcessedItem]:
|
||||
# Filter and narrow types
|
||||
valid_items = [item for item in items if item is not None]
|
||||
# valid_items is now list[Item]
|
||||
return [process(item) for item in valid_items]
|
||||
```
|
||||
|
||||
### Pattern 4: Generic Classes
|
||||
|
||||
Create type-safe reusable containers.
|
||||
|
||||
```python
|
||||
from typing import TypeVar, Generic
|
||||
|
||||
T = TypeVar("T")
|
||||
E = TypeVar("E", bound=Exception)
|
||||
|
||||
class Result(Generic[T, E]):
|
||||
"""Represents either a success value or an error."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
value: T | None = None,
|
||||
error: E | None = None,
|
||||
) -> None:
|
||||
if (value is None) == (error is None):
|
||||
raise ValueError("Exactly one of value or error must be set")
|
||||
self._value = value
|
||||
self._error = error
|
||||
|
||||
@property
|
||||
def is_success(self) -> bool:
|
||||
return self._error is None
|
||||
|
||||
@property
|
||||
def is_failure(self) -> bool:
|
||||
return self._error is not None
|
||||
|
||||
def unwrap(self) -> T:
|
||||
"""Get value or raise the error."""
|
||||
if self._error is not None:
|
||||
raise self._error
|
||||
return self._value # type: ignore[return-value]
|
||||
|
||||
def unwrap_or(self, default: T) -> T:
|
||||
"""Get value or return default."""
|
||||
if self._error is not None:
|
||||
return default
|
||||
return self._value # type: ignore[return-value]
|
||||
|
||||
# Usage preserves types
|
||||
def parse_config(path: str) -> Result[Config, ConfigError]:
|
||||
try:
|
||||
return Result(value=Config.from_file(path))
|
||||
except ConfigError as e:
|
||||
return Result(error=e)
|
||||
|
||||
result = parse_config("config.yaml")
|
||||
if result.is_success:
|
||||
config = result.unwrap() # Type: Config
|
||||
```
|
||||
|
||||
## Detailed worked examples and patterns
|
||||
|
||||
Detailed sections (starting with `## Advanced Patterns`) live in `references/details.md`. Read that file when the navigation summary above is insufficient.
|
||||
|
||||
## Best Practices Summary
|
||||
|
||||
1. **Annotate all public APIs** - Functions, methods, class attributes
|
||||
2. **Use `T | None`** - Modern union syntax over `Optional[T]`
|
||||
3. **Run strict type checking** - `mypy --strict` in CI
|
||||
4. **Use generics** - Preserve type info in reusable code
|
||||
5. **Define protocols** - Structural typing for interfaces
|
||||
6. **Narrow types** - Use guards to help the type checker
|
||||
7. **Bound type vars** - Restrict generics to meaningful types
|
||||
8. **Create type aliases** - Meaningful names for complex types
|
||||
9. **Minimize `Any`** - Use specific types or generics. `Any` is acceptable for truly dynamic data or when interfacing with untyped third-party code
|
||||
10. **Document with types** - Types are enforceable documentation
|
||||
@@ -0,0 +1,237 @@
|
||||
# python-type-safety — detailed worked examples
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Pattern 5: Generic Repository
|
||||
|
||||
Create type-safe data access patterns.
|
||||
|
||||
```python
|
||||
from typing import TypeVar, Generic
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
T = TypeVar("T")
|
||||
ID = TypeVar("ID")
|
||||
|
||||
class Repository(ABC, Generic[T, ID]):
|
||||
"""Generic repository interface."""
|
||||
|
||||
@abstractmethod
|
||||
async def get(self, id: ID) -> T | None:
|
||||
"""Get entity by ID."""
|
||||
...
|
||||
|
||||
@abstractmethod
|
||||
async def save(self, entity: T) -> T:
|
||||
"""Save and return entity."""
|
||||
...
|
||||
|
||||
@abstractmethod
|
||||
async def delete(self, id: ID) -> bool:
|
||||
"""Delete entity, return True if existed."""
|
||||
...
|
||||
|
||||
class UserRepository(Repository[User, str]):
|
||||
"""Concrete repository for Users with string IDs."""
|
||||
|
||||
async def get(self, id: str) -> User | None:
|
||||
row = await self._db.fetchrow(
|
||||
"SELECT * FROM users WHERE id = $1", id
|
||||
)
|
||||
return User(**row) if row else None
|
||||
|
||||
async def save(self, entity: User) -> User:
|
||||
...
|
||||
|
||||
async def delete(self, id: str) -> bool:
|
||||
...
|
||||
```
|
||||
|
||||
### Pattern 6: TypeVar with Bounds
|
||||
|
||||
Restrict generic parameters to specific types.
|
||||
|
||||
```python
|
||||
from typing import TypeVar
|
||||
from pydantic import BaseModel
|
||||
|
||||
ModelT = TypeVar("ModelT", bound=BaseModel)
|
||||
|
||||
def validate_and_create(model_cls: type[ModelT], data: dict) -> ModelT:
|
||||
"""Create a validated Pydantic model from dict."""
|
||||
return model_cls.model_validate(data)
|
||||
|
||||
# Works with any BaseModel subclass
|
||||
class User(BaseModel):
|
||||
name: str
|
||||
email: str
|
||||
|
||||
user = validate_and_create(User, {"name": "Alice", "email": "a@b.com"})
|
||||
# user is typed as User
|
||||
|
||||
# Type error: str is not a BaseModel subclass
|
||||
result = validate_and_create(str, {"name": "Alice"}) # Error!
|
||||
```
|
||||
|
||||
### Pattern 7: Protocols for Structural Typing
|
||||
|
||||
Define interfaces without requiring inheritance.
|
||||
|
||||
```python
|
||||
from typing import Protocol, runtime_checkable
|
||||
|
||||
@runtime_checkable
|
||||
class Serializable(Protocol):
|
||||
"""Any class that can be serialized to/from dict."""
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
...
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "Serializable":
|
||||
...
|
||||
|
||||
# User satisfies Serializable without inheriting from it
|
||||
class User:
|
||||
def __init__(self, id: str, name: str) -> None:
|
||||
self.id = id
|
||||
self.name = name
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"id": self.id, "name": self.name}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "User":
|
||||
return cls(id=data["id"], name=data["name"])
|
||||
|
||||
def serialize(obj: Serializable) -> str:
|
||||
"""Works with any Serializable object."""
|
||||
return json.dumps(obj.to_dict())
|
||||
|
||||
# Works - User matches the protocol
|
||||
serialize(User("1", "Alice"))
|
||||
|
||||
# Runtime checking with @runtime_checkable
|
||||
isinstance(User("1", "Alice"), Serializable) # True
|
||||
```
|
||||
|
||||
### Pattern 8: Common Protocol Patterns
|
||||
|
||||
Define reusable structural interfaces.
|
||||
|
||||
```python
|
||||
from typing import Protocol
|
||||
|
||||
class Closeable(Protocol):
|
||||
"""Resource that can be closed."""
|
||||
def close(self) -> None: ...
|
||||
|
||||
class AsyncCloseable(Protocol):
|
||||
"""Async resource that can be closed."""
|
||||
async def close(self) -> None: ...
|
||||
|
||||
class Readable(Protocol):
|
||||
"""Object that can be read from."""
|
||||
def read(self, n: int = -1) -> bytes: ...
|
||||
|
||||
class HasId(Protocol):
|
||||
"""Object with an ID property."""
|
||||
@property
|
||||
def id(self) -> str: ...
|
||||
|
||||
class Comparable(Protocol):
|
||||
"""Object that supports comparison."""
|
||||
def __lt__(self, other: "Comparable") -> bool: ...
|
||||
def __le__(self, other: "Comparable") -> bool: ...
|
||||
```
|
||||
|
||||
### Pattern 9: Type Aliases
|
||||
|
||||
Create meaningful type names.
|
||||
|
||||
**Note:** The `type Alias = ...` statement syntax (PEP 695) was introduced in **Python 3.12**, not 3.10. For projects targeting earlier versions (including 3.10/3.11), use the `TypeAlias` annotation (PEP 613, available since Python 3.10).
|
||||
|
||||
```python
|
||||
# Python 3.12+ type statement (PEP 695)
|
||||
type UserId = str
|
||||
type UserDict = dict[str, Any]
|
||||
|
||||
# Python 3.12+ type statement with generics (PEP 695)
|
||||
type Handler[T] = Callable[[Request], T]
|
||||
type AsyncHandler[T] = Callable[[Request], Awaitable[T]]
|
||||
```
|
||||
|
||||
```python
|
||||
# Python 3.10-3.11 style (needed for broader compatibility)
|
||||
from typing import TypeAlias
|
||||
from collections.abc import Callable, Awaitable
|
||||
|
||||
UserId: TypeAlias = str
|
||||
Handler: TypeAlias = Callable[[Request], Response]
|
||||
```
|
||||
|
||||
```python
|
||||
# Usage
|
||||
def register_handler(path: str, handler: Handler[Response]) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
### Pattern 10: Callable Types
|
||||
|
||||
Type function parameters and callbacks.
|
||||
|
||||
```python
|
||||
from collections.abc import Callable, Awaitable
|
||||
|
||||
# Sync callback
|
||||
ProgressCallback = Callable[[int, int], None] # (current, total)
|
||||
|
||||
# Async callback
|
||||
AsyncHandler = Callable[[Request], Awaitable[Response]]
|
||||
|
||||
# With named parameters (using Protocol)
|
||||
class OnProgress(Protocol):
|
||||
def __call__(
|
||||
self,
|
||||
current: int,
|
||||
total: int,
|
||||
*,
|
||||
message: str = "",
|
||||
) -> None: ...
|
||||
|
||||
def process_items(
|
||||
items: list[Item],
|
||||
on_progress: ProgressCallback | None = None,
|
||||
) -> list[Result]:
|
||||
for i, item in enumerate(items):
|
||||
if on_progress:
|
||||
on_progress(i, len(items))
|
||||
...
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Strict Mode Checklist
|
||||
|
||||
For `mypy --strict` compliance:
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[tool.mypy]
|
||||
python_version = "3.12"
|
||||
strict = true
|
||||
warn_return_any = true
|
||||
warn_unused_ignores = true
|
||||
disallow_untyped_defs = true
|
||||
disallow_incomplete_defs = true
|
||||
no_implicit_optional = true
|
||||
```
|
||||
|
||||
Incremental adoption goals:
|
||||
- All function parameters annotated
|
||||
- All return types annotated
|
||||
- Class attributes annotated
|
||||
- Minimize `Any` usage (acceptable for truly dynamic data)
|
||||
- Generic collections use type parameters (`list[str]` not `list`)
|
||||
|
||||
For existing codebases, enable strict mode per-module using `# mypy: strict` or configure per-module overrides in `pyproject.toml`.
|
||||
@@ -0,0 +1,345 @@
|
||||
---
|
||||
name: uv-package-manager
|
||||
description: Master the uv package manager for fast Python dependency management, virtual environments, and modern Python project workflows. Use when setting up Python projects, managing dependencies, or optimizing Python development workflows with uv.
|
||||
---
|
||||
|
||||
# UV Package Manager
|
||||
|
||||
Comprehensive guide to using uv, an extremely fast Python package installer and resolver written in Rust, for modern Python project management and dependency workflows.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Setting up new Python projects quickly
|
||||
- Managing Python dependencies faster than pip
|
||||
- Creating and managing virtual environments
|
||||
- Installing Python interpreters
|
||||
- Resolving dependency conflicts efficiently
|
||||
- Migrating from pip/pip-tools/poetry
|
||||
- Speeding up CI/CD pipelines
|
||||
- Managing monorepo Python projects
|
||||
- Working with lockfiles for reproducible builds
|
||||
- Optimizing Docker builds with Python dependencies
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. What is uv?
|
||||
|
||||
- **Ultra-fast package installer**: 10-100x faster than pip
|
||||
- **Written in Rust**: Leverages Rust's performance
|
||||
- **Drop-in pip replacement**: Compatible with pip workflows
|
||||
- **Virtual environment manager**: Create and manage venvs
|
||||
- **Python installer**: Download and manage Python versions
|
||||
- **Resolver**: Advanced dependency resolution
|
||||
- **Lockfile support**: Reproducible installations
|
||||
|
||||
### 2. Key Features
|
||||
|
||||
- Blazing fast installation speeds
|
||||
- Disk space efficient with global cache
|
||||
- Compatible with pip, pip-tools, poetry
|
||||
- Comprehensive dependency resolution
|
||||
- Cross-platform support (Linux, macOS, Windows)
|
||||
- No Python required for installation
|
||||
- Built-in virtual environment support
|
||||
|
||||
### 3. UV vs Traditional Tools
|
||||
|
||||
- **vs pip**: 10-100x faster, better resolver
|
||||
- **vs pip-tools**: Faster, simpler, better UX
|
||||
- **vs poetry**: Faster, less opinionated, lighter
|
||||
- **vs conda**: Faster, Python-focused
|
||||
|
||||
## Installation
|
||||
|
||||
### Quick Install
|
||||
|
||||
```bash
|
||||
# macOS/Linux
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
|
||||
# Windows (PowerShell)
|
||||
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
|
||||
# Using pip (if you already have Python)
|
||||
pip install uv
|
||||
|
||||
# Using Homebrew (macOS)
|
||||
brew install uv
|
||||
|
||||
# Using cargo (if you have Rust)
|
||||
cargo install --git https://github.com/astral-sh/uv uv
|
||||
```
|
||||
|
||||
### Verify Installation
|
||||
|
||||
```bash
|
||||
uv --version
|
||||
# uv 0.x.x
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Create a New Project
|
||||
|
||||
```bash
|
||||
# Create new project with virtual environment
|
||||
uv init my-project
|
||||
cd my-project
|
||||
|
||||
# Or create in current directory
|
||||
uv init .
|
||||
|
||||
# Initialize creates:
|
||||
# - .python-version (Python version)
|
||||
# - pyproject.toml (project config)
|
||||
# - README.md
|
||||
# - .gitignore
|
||||
```
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
# Install packages (creates venv if needed)
|
||||
uv add requests pandas
|
||||
|
||||
# Install dev dependencies
|
||||
uv add --dev pytest black ruff
|
||||
|
||||
# Install from requirements.txt
|
||||
uv pip install -r requirements.txt
|
||||
|
||||
# Install from pyproject.toml
|
||||
uv sync
|
||||
```
|
||||
|
||||
## Virtual Environment Management
|
||||
|
||||
### Pattern 1: Creating Virtual Environments
|
||||
|
||||
```bash
|
||||
# Create virtual environment with uv
|
||||
uv venv
|
||||
|
||||
# Create with specific Python version
|
||||
uv venv --python 3.12
|
||||
|
||||
# Create with custom name
|
||||
uv venv my-env
|
||||
|
||||
# Create with system site packages
|
||||
uv venv --system-site-packages
|
||||
|
||||
# Specify location
|
||||
uv venv /path/to/venv
|
||||
```
|
||||
|
||||
### Pattern 2: Activating Virtual Environments
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
source .venv/bin/activate
|
||||
|
||||
# Windows (Command Prompt)
|
||||
.venv\Scripts\activate.bat
|
||||
|
||||
# Windows (PowerShell)
|
||||
.venv\Scripts\Activate.ps1
|
||||
|
||||
# Or use uv run (no activation needed)
|
||||
uv run python script.py
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
### Pattern 3: Using uv run
|
||||
|
||||
```bash
|
||||
# Run Python script (auto-activates venv)
|
||||
uv run python app.py
|
||||
|
||||
# Run installed CLI tool
|
||||
uv run black .
|
||||
uv run pytest
|
||||
|
||||
# Run with specific Python version
|
||||
uv run --python 3.11 python script.py
|
||||
|
||||
# Pass arguments
|
||||
uv run python script.py --arg value
|
||||
```
|
||||
|
||||
## Package Management
|
||||
|
||||
### Pattern 4: Adding Dependencies
|
||||
|
||||
```bash
|
||||
# Add package (adds to pyproject.toml)
|
||||
uv add requests
|
||||
|
||||
# Add with version constraint
|
||||
uv add "django>=4.0,<5.0"
|
||||
|
||||
# Add multiple packages
|
||||
uv add numpy pandas matplotlib
|
||||
|
||||
# Add dev dependency
|
||||
uv add --dev pytest pytest-cov
|
||||
|
||||
# Add optional dependency group
|
||||
uv add --optional docs sphinx
|
||||
|
||||
# Add from git
|
||||
uv add git+https://github.com/user/repo.git
|
||||
|
||||
# Add from git with specific ref
|
||||
uv add git+https://github.com/user/repo.git@v1.0.0
|
||||
|
||||
# Add from local path
|
||||
uv add ./local-package
|
||||
|
||||
# Add editable local package
|
||||
uv add -e ./local-package
|
||||
```
|
||||
|
||||
### Pattern 5: Removing Dependencies
|
||||
|
||||
```bash
|
||||
# Remove package
|
||||
uv remove requests
|
||||
|
||||
# Remove dev dependency
|
||||
uv remove --dev pytest
|
||||
|
||||
# Remove multiple packages
|
||||
uv remove numpy pandas matplotlib
|
||||
```
|
||||
|
||||
### Pattern 6: Upgrading Dependencies
|
||||
|
||||
```bash
|
||||
# Upgrade specific package
|
||||
uv add --upgrade requests
|
||||
|
||||
# Upgrade all packages
|
||||
uv sync --upgrade
|
||||
|
||||
# Upgrade package to latest
|
||||
uv add --upgrade requests
|
||||
|
||||
# Show what would be upgraded
|
||||
uv tree --outdated
|
||||
```
|
||||
|
||||
### Pattern 7: Locking Dependencies
|
||||
|
||||
```bash
|
||||
# Generate uv.lock file
|
||||
uv lock
|
||||
|
||||
# Update lock file
|
||||
uv lock --upgrade
|
||||
|
||||
# Lock without installing
|
||||
uv lock --no-install
|
||||
|
||||
# Lock specific package
|
||||
uv lock --upgrade-package requests
|
||||
```
|
||||
|
||||
## Python Version Management
|
||||
|
||||
### Pattern 8: Installing Python Versions
|
||||
|
||||
```bash
|
||||
# Install Python version
|
||||
uv python install 3.12
|
||||
|
||||
# Install multiple versions
|
||||
uv python install 3.11 3.12 3.13
|
||||
|
||||
# Install latest version
|
||||
uv python install
|
||||
|
||||
# List installed versions
|
||||
uv python list
|
||||
|
||||
# Find available versions
|
||||
uv python list --all-versions
|
||||
```
|
||||
|
||||
### Pattern 9: Setting Python Version
|
||||
|
||||
```bash
|
||||
# Set Python version for project
|
||||
uv python pin 3.12
|
||||
|
||||
# This creates/updates .python-version file
|
||||
|
||||
# Use specific Python version for command
|
||||
uv --python 3.11 run python script.py
|
||||
|
||||
# Create venv with specific version
|
||||
uv venv --python 3.12
|
||||
```
|
||||
|
||||
## Project Configuration
|
||||
|
||||
### Pattern 10: pyproject.toml with uv
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "my-project"
|
||||
version = "0.1.0"
|
||||
description = "My awesome project"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.8"
|
||||
dependencies = [
|
||||
"requests>=2.31.0",
|
||||
"pydantic>=2.0.0",
|
||||
"click>=8.1.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.4.0",
|
||||
"pytest-cov>=4.1.0",
|
||||
"black>=23.0.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.5.0",
|
||||
]
|
||||
docs = [
|
||||
"sphinx>=7.0.0",
|
||||
"sphinx-rtd-theme>=1.3.0",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.uv]
|
||||
dev-dependencies = [
|
||||
# Additional dev dependencies managed by uv
|
||||
]
|
||||
|
||||
[tool.uv.sources]
|
||||
# Custom package sources
|
||||
my-package = { git = "https://github.com/user/repo.git" }
|
||||
```
|
||||
|
||||
### Pattern 11: Using uv with Existing Projects
|
||||
|
||||
```bash
|
||||
# Migrate from requirements.txt
|
||||
uv add -r requirements.txt
|
||||
|
||||
# Migrate from poetry
|
||||
# Already have pyproject.toml, just use:
|
||||
uv sync
|
||||
|
||||
# Export to requirements.txt
|
||||
uv pip freeze > requirements.txt
|
||||
|
||||
# Export with hashes
|
||||
uv pip freeze --require-hashes > requirements.txt
|
||||
```
|
||||
|
||||
For advanced workflows including Docker integration, lockfile management, performance optimization, tool comparison, common workflows, tool integration, troubleshooting, best practices, migration guides, and command reference, see [references/advanced-patterns.md](references/advanced-patterns.md)
|
||||
@@ -0,0 +1,473 @@
|
||||
# UV Package Manager — Advanced Reference
|
||||
|
||||
Advanced workflows including Docker integration, lockfile management, performance optimization, tool comparison, common workflows, tool integration, troubleshooting, best practices, migration guides, and command reference.
|
||||
|
||||
## Advanced Workflows
|
||||
|
||||
### Pattern 12: Monorepo Support
|
||||
|
||||
```bash
|
||||
# Project structure
|
||||
# monorepo/
|
||||
# packages/
|
||||
# package-a/
|
||||
# pyproject.toml
|
||||
# package-b/
|
||||
# pyproject.toml
|
||||
# pyproject.toml (root)
|
||||
|
||||
# Root pyproject.toml
|
||||
[tool.uv.workspace]
|
||||
members = ["packages/*"]
|
||||
|
||||
# Install all workspace packages
|
||||
uv sync
|
||||
|
||||
# Add workspace dependency
|
||||
uv add --path ./packages/package-a
|
||||
```
|
||||
|
||||
### Pattern 13: CI/CD Integration
|
||||
|
||||
```yaml
|
||||
# .github/workflows/test.yml
|
||||
name: Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v2
|
||||
with:
|
||||
enable-cache: true
|
||||
|
||||
- name: Set up Python
|
||||
run: uv python install 3.12
|
||||
|
||||
- name: Install dependencies
|
||||
run: uv sync --all-extras --dev
|
||||
|
||||
- name: Run tests
|
||||
run: uv run pytest
|
||||
|
||||
- name: Run linting
|
||||
run: |
|
||||
uv run ruff check .
|
||||
uv run black --check .
|
||||
```
|
||||
|
||||
### Pattern 14: Docker Integration
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Install uv
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.6 /uv /usr/local/bin/uv
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy dependency files
|
||||
COPY pyproject.toml uv.lock ./
|
||||
|
||||
# Install dependencies
|
||||
RUN uv sync --frozen --no-dev
|
||||
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
# Run application
|
||||
CMD ["uv", "run", "python", "app.py"]
|
||||
```
|
||||
|
||||
**Optimized multi-stage build:**
|
||||
|
||||
```dockerfile
|
||||
# Multi-stage Dockerfile
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
# Install uv
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.6 /uv /usr/local/bin/uv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies to venv
|
||||
COPY pyproject.toml uv.lock ./
|
||||
RUN uv sync --frozen --no-dev --no-editable
|
||||
|
||||
# Runtime stage
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy venv from builder
|
||||
COPY --from=builder /app/.venv .venv
|
||||
COPY . .
|
||||
|
||||
# Use venv
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
|
||||
CMD ["python", "app.py"]
|
||||
```
|
||||
|
||||
### Pattern 15: Lockfile Workflows
|
||||
|
||||
```bash
|
||||
# Create lockfile (uv.lock)
|
||||
uv lock
|
||||
|
||||
# Install from lockfile (exact versions)
|
||||
uv sync --frozen
|
||||
|
||||
# Update lockfile without installing
|
||||
uv lock --no-install
|
||||
|
||||
# Upgrade specific package in lock
|
||||
uv lock --upgrade-package requests
|
||||
|
||||
# Check if lockfile is up to date
|
||||
uv lock --check
|
||||
|
||||
# Export lockfile to requirements.txt
|
||||
uv export --format requirements-txt > requirements.txt
|
||||
|
||||
# Export with hashes for security
|
||||
uv export --format requirements-txt --hash > requirements.txt
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Pattern 16: Using Global Cache
|
||||
|
||||
```bash
|
||||
# UV automatically uses global cache at:
|
||||
# Linux: ~/.cache/uv
|
||||
# macOS: ~/Library/Caches/uv
|
||||
# Windows: %LOCALAPPDATA%\uv\cache
|
||||
|
||||
# Clear cache
|
||||
uv cache clean
|
||||
|
||||
# Check cache size
|
||||
uv cache dir
|
||||
```
|
||||
|
||||
### Pattern 17: Parallel Installation
|
||||
|
||||
```bash
|
||||
# UV installs packages in parallel by default
|
||||
|
||||
# Control parallelism
|
||||
uv pip install --jobs 4 package1 package2
|
||||
|
||||
# No parallel (sequential)
|
||||
uv pip install --jobs 1 package
|
||||
```
|
||||
|
||||
### Pattern 18: Offline Mode
|
||||
|
||||
```bash
|
||||
# Install from cache only (no network)
|
||||
uv pip install --offline package
|
||||
|
||||
# Sync from lockfile offline
|
||||
uv sync --frozen --offline
|
||||
```
|
||||
|
||||
## Comparison with Other Tools
|
||||
|
||||
### uv vs pip
|
||||
|
||||
```bash
|
||||
# pip
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install requests pandas numpy
|
||||
# ~30 seconds
|
||||
|
||||
# uv
|
||||
uv venv
|
||||
uv add requests pandas numpy
|
||||
# ~2 seconds (10-15x faster)
|
||||
```
|
||||
|
||||
### uv vs poetry
|
||||
|
||||
```bash
|
||||
# poetry
|
||||
poetry init
|
||||
poetry add requests pandas
|
||||
poetry install
|
||||
# ~20 seconds
|
||||
|
||||
# uv
|
||||
uv init
|
||||
uv add requests pandas
|
||||
uv sync
|
||||
# ~3 seconds (6-7x faster)
|
||||
```
|
||||
|
||||
### uv vs pip-tools
|
||||
|
||||
```bash
|
||||
# pip-tools
|
||||
pip-compile requirements.in
|
||||
pip-sync requirements.txt
|
||||
# ~15 seconds
|
||||
|
||||
# uv
|
||||
uv lock
|
||||
uv sync --frozen
|
||||
# ~2 seconds (7-8x faster)
|
||||
```
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Pattern 19: Starting a New Project
|
||||
|
||||
```bash
|
||||
# Complete workflow
|
||||
uv init my-project
|
||||
cd my-project
|
||||
|
||||
# Set Python version
|
||||
uv python pin 3.12
|
||||
|
||||
# Add dependencies
|
||||
uv add fastapi uvicorn pydantic
|
||||
|
||||
# Add dev dependencies
|
||||
uv add --dev pytest black ruff mypy
|
||||
|
||||
# Create structure
|
||||
mkdir -p src/my_project tests
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Format code
|
||||
uv run black .
|
||||
uv run ruff check .
|
||||
```
|
||||
|
||||
### Pattern 20: Maintaining Existing Project
|
||||
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://github.com/user/project.git
|
||||
cd project
|
||||
|
||||
# Install dependencies (creates venv automatically)
|
||||
uv sync
|
||||
|
||||
# Install with dev dependencies
|
||||
uv sync --all-extras
|
||||
|
||||
# Update dependencies
|
||||
uv lock --upgrade
|
||||
|
||||
# Run application
|
||||
uv run python app.py
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Add new dependency
|
||||
uv add new-package
|
||||
|
||||
# Commit updated files
|
||||
git add pyproject.toml uv.lock
|
||||
git commit -m "Add new-package dependency"
|
||||
```
|
||||
|
||||
## Tool Integration
|
||||
|
||||
### Pattern 21: Pre-commit Hooks
|
||||
|
||||
```yaml
|
||||
# .pre-commit-config.yaml
|
||||
repos:
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: uv-lock
|
||||
name: uv lock
|
||||
entry: uv lock
|
||||
language: system
|
||||
pass_filenames: false
|
||||
|
||||
- id: ruff
|
||||
name: ruff
|
||||
entry: uv run ruff check --fix
|
||||
language: system
|
||||
types: [python]
|
||||
|
||||
- id: black
|
||||
name: black
|
||||
entry: uv run black
|
||||
language: system
|
||||
types: [python]
|
||||
```
|
||||
|
||||
### Pattern 22: VS Code Integration
|
||||
|
||||
```json
|
||||
// .vscode/settings.json
|
||||
{
|
||||
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
|
||||
"python.terminal.activateEnvironment": true,
|
||||
"python.testing.pytestEnabled": true,
|
||||
"python.testing.pytestArgs": ["-v"],
|
||||
"python.linting.enabled": true,
|
||||
"python.formatting.provider": "black",
|
||||
"[python]": {
|
||||
"editor.defaultFormatter": "ms-python.black-formatter",
|
||||
"editor.formatOnSave": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
```bash
|
||||
# Issue: uv not found
|
||||
# Solution: Add to PATH or reinstall
|
||||
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc
|
||||
|
||||
# Issue: Wrong Python version
|
||||
# Solution: Pin version explicitly
|
||||
uv python pin 3.12
|
||||
uv venv --python 3.12
|
||||
|
||||
# Issue: Dependency conflict
|
||||
# Solution: Check resolution
|
||||
uv lock --verbose
|
||||
|
||||
# Issue: Cache issues
|
||||
# Solution: Clear cache
|
||||
uv cache clean
|
||||
|
||||
# Issue: Lockfile out of sync
|
||||
# Solution: Regenerate
|
||||
uv lock --upgrade
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Project Setup
|
||||
|
||||
1. **Always use lockfiles** for reproducibility
|
||||
2. **Pin Python version** with .python-version
|
||||
3. **Separate dev dependencies** from production
|
||||
4. **Use uv run** instead of activating venv
|
||||
5. **Commit uv.lock** to version control
|
||||
6. **Use --frozen in CI** for consistent builds
|
||||
7. **Leverage global cache** for speed
|
||||
8. **Use workspace** for monorepos
|
||||
9. **Export requirements.txt** for compatibility
|
||||
10. **Keep uv updated** for latest features
|
||||
|
||||
### Performance Tips
|
||||
|
||||
```bash
|
||||
# Use frozen installs in CI
|
||||
uv sync --frozen
|
||||
|
||||
# Use offline mode when possible
|
||||
uv sync --offline
|
||||
|
||||
# Parallel operations (automatic)
|
||||
# uv does this by default
|
||||
|
||||
# Reuse cache across environments
|
||||
# uv shares cache globally
|
||||
|
||||
# Use lockfiles to skip resolution
|
||||
uv sync --frozen # skips resolution
|
||||
```
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From pip + requirements.txt
|
||||
|
||||
```bash
|
||||
# Before
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
# After
|
||||
uv venv
|
||||
uv pip install -r requirements.txt
|
||||
# Or better:
|
||||
uv init
|
||||
uv add -r requirements.txt
|
||||
```
|
||||
|
||||
### From Poetry
|
||||
|
||||
```bash
|
||||
# Before
|
||||
poetry install
|
||||
poetry add requests
|
||||
|
||||
# After
|
||||
uv sync
|
||||
uv add requests
|
||||
|
||||
# Keep existing pyproject.toml
|
||||
# uv reads [project] and [tool.poetry] sections
|
||||
```
|
||||
|
||||
### From pip-tools
|
||||
|
||||
```bash
|
||||
# Before
|
||||
pip-compile requirements.in
|
||||
pip-sync requirements.txt
|
||||
|
||||
# After
|
||||
uv lock
|
||||
uv sync --frozen
|
||||
```
|
||||
|
||||
## Command Reference
|
||||
|
||||
### Essential Commands
|
||||
|
||||
```bash
|
||||
# Project management
|
||||
uv init [PATH] # Initialize project
|
||||
uv add PACKAGE # Add dependency
|
||||
uv remove PACKAGE # Remove dependency
|
||||
uv sync # Install dependencies
|
||||
uv lock # Create/update lockfile
|
||||
|
||||
# Virtual environments
|
||||
uv venv [PATH] # Create venv
|
||||
uv run COMMAND # Run in venv
|
||||
|
||||
# Python management
|
||||
uv python install VERSION # Install Python
|
||||
uv python list # List installed Pythons
|
||||
uv python pin VERSION # Pin Python version
|
||||
|
||||
# Package installation (pip-compatible)
|
||||
uv pip install PACKAGE # Install package
|
||||
uv pip uninstall PACKAGE # Uninstall package
|
||||
uv pip freeze # List installed
|
||||
uv pip list # List packages
|
||||
|
||||
# Utility
|
||||
uv cache clean # Clear cache
|
||||
uv cache dir # Show cache location
|
||||
uv --version # Show version
|
||||
```
|
||||
Reference in New Issue
Block a user