wehub-resource-sync
133 lines
4.8 KiB
Markdown
133 lines
4.8 KiB
Markdown
# Docker Environments
|
|
|
|
Pre-built images are available on Docker Hub (`docker.io/ml4t/`). Most readers need only the main `ml4t` image.
|
|
|
|
## Images
|
|
|
|
| Image | Docker Hub | Python | Platforms | Size |
|
|
|-------|-----------|--------|-----------|------|
|
|
| **ml4t** | `ml4t/ml4t:latest` | 3.14 | amd64 + arm64 | ~12 GB / ~3 GB |
|
|
| **py312** | `ml4t/ml4t-py312:latest` | 3.12 | amd64 only | ~9.6 GB |
|
|
| **benchmark** | `ml4t/ml4t-benchmark:latest` | 3.14 | amd64 + arm64 | ~1.7 GB |
|
|
| **rapids** | (build locally) | 3.12 | amd64 + NVIDIA GPU | ~15 GB |
|
|
|
|
### ml4t (Main)
|
|
|
|
Covers all 27 chapters and 9 case studies. Includes PyTorch with CUDA 12.8 support, LightGBM, scikit-learn, Polars, Plotly, and all ML4T libraries.
|
|
|
|
```bash
|
|
docker compose pull ml4t
|
|
docker compose up ml4t # Jupyter Lab at http://localhost:8888
|
|
docker compose run --rm ml4t python nb.py # Run a notebook directly
|
|
```
|
|
|
|
GPU passthrough (same image, NVIDIA runtime required):
|
|
|
|
```bash
|
|
docker compose --profile gpu run --rm ml4t-gpu python notebook.py
|
|
```
|
|
|
|
### py312 (Python 3.12 Dependencies)
|
|
|
|
For notebooks requiring libraries without Python 3.14 wheels:
|
|
|
|
| Notebook | Library |
|
|
|----------|---------|
|
|
| Ch05 `03_sigcwgan_signatures` | signatory |
|
|
| Ch09 `06_path_signatures`, `12_wasserstein_regimes` | signatory, esig |
|
|
| Ch10 `01_word2vec`, `02_asset_embeddings`, `03_sentiment_evolution` | gensim |
|
|
| Ch15 `06_fed_announcement_bsts` | tfcausalimpact (TFP BSTS) |
|
|
| Ch21 `05_deep_hedging_pfhedge` | pfhedge |
|
|
|
|
```bash
|
|
docker compose --profile py312 pull py312
|
|
docker compose --profile py312 run --rm py312 python 05_synthetic_data/03_sigcwgan_signatures.py
|
|
```
|
|
|
|
Not available on Apple Silicon — view pre-executed `.ipynb` files instead.
|
|
|
|
### benchmark (Storage Benchmarks)
|
|
|
|
For Chapter 2 storage benchmarks comparing file formats and databases (TimescaleDB, ClickHouse, QuestDB, InfluxDB).
|
|
|
|
```bash
|
|
docker compose pull benchmark
|
|
|
|
# Start database services
|
|
docker compose --profile benchmark up -d timescaledb clickhouse questdb influxdb
|
|
|
|
# Run benchmark
|
|
docker compose --profile benchmark run --rm benchmark \
|
|
python 02_financial_data_universe/21_storage_benchmark_database.py
|
|
|
|
# Stop databases
|
|
docker compose --profile benchmark down
|
|
```
|
|
|
|
### rapids (GPU Benchmarks)
|
|
|
|
For Chapter 12 GBM GPU benchmark with RAPIDS cuML and LightGBM CUDA. Requires NVIDIA GPU. Must be built locally:
|
|
|
|
```bash
|
|
docker compose --profile rapids build rapids
|
|
docker compose --profile rapids run --rm rapids python 12_gradient_boosting/02_gbm_comparison.py
|
|
```
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
envs/
|
|
├── README.md # This file
|
|
├── ml4t/Dockerfile # Main image (Python 3.14)
|
|
├── py312/
|
|
│ ├── Dockerfile # Python 3.12 for signatory/esig/gensim/pfhedge/tfcausalimpact
|
|
│ └── pyproject.toml # py312-specific dependencies
|
|
├── benchmark/
|
|
│ ├── Dockerfile # Benchmark image with DB clients
|
|
│ └── pyproject.toml # Benchmark-specific dependencies
|
|
├── rapids/
|
|
│ └── Dockerfile # RAPIDS cuML + LightGBM CUDA
|
|
└── test_all_imports.py # Import verification script (63 packages)
|
|
```
|
|
|
|
## Import Verification
|
|
|
|
Each Docker image has a self-test that verifies all packages needed by its notebooks are importable. Run it after pulling or building to confirm the environment is healthy:
|
|
|
|
```bash
|
|
# ml4t image (baked-in command)
|
|
docker compose run --rm ml4t ml4t-test-imports
|
|
|
|
# Or explicitly
|
|
docker compose run --rm ml4t python envs/test_all_imports.py
|
|
|
|
# py312 image
|
|
docker compose --profile py312 run --rm py312 python envs/test_all_imports.py --image py312
|
|
|
|
# Benchmark image
|
|
docker compose --profile benchmark run --rm benchmark python envs/test_all_imports.py --image benchmark
|
|
|
|
# Test a specific chapter
|
|
docker compose run --rm ml4t python envs/test_all_imports.py --chapter 15 --verbose
|
|
```
|
|
|
|
**What's tested per image:**
|
|
|
|
| Image | Packages | ML4T Libs | Utils Modules | Chapters |
|
|
|-------|----------|-----------|---------------|----------|
|
|
| **ml4t** | 50 third-party | 6 (data, engineer, models, diagnostic, backtest, live) | 22 (4 repo + 18 case study) | Ch01-Ch26 |
|
|
| **py312** | 5 (signatory, esig, gensim, pfhedge, tfcausalimpact) | 1 (diagnostic) | — | Ch05, Ch09, Ch10, Ch12, Ch14, Ch15, Ch21 |
|
|
| **benchmark** | 5 (duckdb, tables, DB clients) | 0 | — | Ch02 |
|
|
|
|
The test groups packages by chapter, so failures map directly to which notebooks are affected. Exit code is 0 (all pass) or 1 (failures). py312-only packages are shown as informational when running the ml4t test.
|
|
|
|
## Building Locally
|
|
|
|
If you prefer to build from source instead of pulling from Docker Hub:
|
|
|
|
```bash
|
|
docker compose build ml4t # ~45 min on x86, ~15 min on ARM64
|
|
docker compose --profile py312 build py312 # ~30 min
|
|
docker compose --profile benchmark build benchmark # ~10 min
|
|
```
|