stefan-jansen--machine-learning-for-trading/docs/installation.md

Installation Guide

This guide walks you through setting up Docker to run the ML4T notebooks. Pre-built images on Docker Hub mean you can be running notebooks in minutes.

---

Before You Begin

Every command in this guide is typed into a terminal on your own computer, not into the GitHub website. GitHub stores the code; your terminal is where you tell your machine to fetch and run it.

Platform	How to open a terminal
Windows	Start menu → type PowerShell → open Windows PowerShell. Some steps below need Run as administrator (right-click → Run as administrator).
macOS	Applications → Utilities → Terminal
Linux	Ctrl+Alt+T, or search for Terminal

Commands shown in a block like this are typed at the terminal prompt, one line at a time, then Enter:

git clone https://github.com/stefan-jansen/machine-learning-for-trading.git

Type flags exactly as written. --install is two dashes attached to the word with no space before it. wsl -- install is a different command and will not do what you want.

If a command is not found, the tool it belongs to is not installed yet. git ships with Git for Windows and with the Xcode command-line tools on macOS (xcode-select --install).

---

Platform Support

Platform	ml4t	py312	Benchmark	GPU
Linux x86_64	✅	✅	✅	✅*
Windows 11 (WSL2)	✅	✅	✅	✅*
macOS Intel	✅	✅	✅	-
macOS Apple Silicon	✅	-	✅	-

* Requires NVIDIA GPU + nvidia-container-toolkit

Which image do I need?

Image	What it covers	Platforms
ml4t	All chapters (Ch01-Ch27) + all 9 case studies	amd64 + arm64
ml4t-py312	Ch05 NB01/03/07, Ch09 NB06/12, Ch10 NB01-03, Ch12 NB10, Ch14 NB06, Ch15 NB06, Ch21 deep_hedging (signatory, esig, gensim, pfhedge, tfcausalimpact)	amd64 only
benchmark	Ch02 storage benchmarks (DuckDB, HDF5, database clients)	amd64 + arm64
rapids	Ch12 GBM GPU benchmark (RAPIDS cuML, LightGBM CUDA)	amd64 + NVIDIA GPU

Most readers need only ml4t. The other images are for specific notebooks.

Apple Silicon users: The notebooks requiring ml4t-py312 are not runnable on ARM64 because the underlying libraries (signatory, esig) have no ARM64 builds. View the pre-executed .ipynb files on GitHub or in Jupyter instead.

---

Quick Start (All Platforms)

# 1. Clone the repository
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading

# 2. Copy environment template
cp .env.example .env

# 3. Pull the pre-built image from Docker Hub
docker compose pull ml4t

# 4. Start Jupyter Lab
docker compose up ml4t
# Open http://localhost:8888

# 5. Or run a notebook directly
docker compose run --rm ml4t python 01_process_is_edge/factor_regimes.py

That's it. No build step needed — Docker pulls the pre-built image (~12 GB on x86, ~3 GB on ARM64).

To build locally instead (if you prefer or need to modify the environment):

docker compose build ml4t    # ~45 min on x86, ~15 min on ARM64

---

Platform-Specific Setup

Ubuntu / Linux

# Install Docker
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
# Log out and back in for group membership

# Verify
docker run --rm hello-world
docker compose version

If Docker Compose is missing: sudo apt install docker-compose-plugin

Windows 11 (WSL2)

Docker Desktop on Windows runs its engine inside WSL2. WSL2 must be working before Docker Desktop can start, so complete steps 1-3 in order and do not skip the restart.

0. Check that hardware virtualization is on. WSL2 cannot run without it, and it is disabled by default on some machines. Press Ctrl+Shift+Esc → Performance tab → CPU, and look for Virtualization.

   • Enabled: continue to step 1.
   • Disabled: turn it on in your BIOS/UEFI setup screen, where it is called Intel VT-x, AMD-V, or Virtualization Technology. The key to enter setup varies by manufacturer (commonly F2, F10, or Del during boot). Nothing below will work until this reads Enabled.

1. Install WSL2 and a Linux distribution. Open PowerShell as Administrator:

   wsl --install -d Ubuntu
   

   Two dashes, no space: --install, not -- install. The -d Ubuntu is required. Without it, some Windows builds install the WSL runtime but no Linux distribution, and later steps fail with Windows Subsystem for Linux has no installed distributions.

2. Restart your computer. This is a required step, not a conditional one. wsl --install enables a Windows feature that does not take effect until you reboot, and Windows does not always prompt you. If the command printed The operation completed successfully, restart now.

   After the restart, Ubuntu opens and asks you to create a username and password. The password is not echoed as you type, which is expected.

3. Verify WSL2 before installing Docker. In PowerShell:

   wsl --list --verbose
   

   You should see Ubuntu with STATE Running (or Stopped) and VERSION 2. If the list is empty, repeat step 1 and confirm you restarted. If VERSION reads 1, run wsl --set-version Ubuntu 2.

4. Increase WSL2 memory limit: WSL2 defaults to 50% of host RAM, which may not be enough for data-heavy notebooks. Create or edit %USERPROFILE%\.wslconfig:

   [wsl2]
   memory=12GB
   swap=4GB
   

   Then restart WSL: wsl --shutdown from PowerShell, then reopen your terminal.

5. Install Docker Desktop from docker.com/products/docker-desktop

   Install it only after wsl --list --verbose shows a VERSION 2 distribution. Docker Desktop started against a non-working WSL2 backend hangs on "Starting the Docker Engine…" indefinitely.

   • Ensure "Use WSL 2 based engine" is checked in Settings → General
   • In Settings → Resources → WSL Integration, enable your Ubuntu distribution
   • In Settings → Resources, allocate at least 8 GB memory and 60 GB disk

6. Verify Docker Desktop integration: Open your WSL Ubuntu terminal and run:

   docker version
   

   If this fails with "Cannot connect to the Docker daemon", Docker Desktop's WSL integration is not enabled for your distribution. Check step 5 above.

7. Clone in WSL (not on Windows drives — much faster):

   cd ~
   git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
   cd machine-learning-for-trading
   cp .env.example .env
   docker compose pull ml4t
   

Important: Always run docker commands from inside a WSL terminal (Ubuntu), not from Windows PowerShell or Command Prompt. Docker Desktop exposes the Docker socket to WSL distributions, but the Docker CLI in Windows may behave differently.

Tip: Keep the repo at ~/machine-learning-for-trading in WSL, not /mnt/c/.... The Windows filesystem (/mnt/c/) is dramatically slower due to the 9P protocol bridge. Access WSL files from Windows Explorer via \\wsl$\Ubuntu\home\<username>\machine-learning-for-trading.

macOS (Intel and Apple Silicon)

1. Install Docker Desktop from docker.com/products/docker-desktop

   • Choose the correct chip: Intel or Apple chip
   • Recommended resources: 4+ CPUs, 8+ GB memory, 64+ GB disk

2. Apple Silicon only: In Docker Desktop Settings → General, enable Use Rosetta for x86_64/amd64 emulation (needed only for the ml4t-py312 image, which most readers won't use).

3. Clone and pull:

   git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
   cd machine-learning-for-trading
   cp .env.example .env
   docker compose pull ml4t
   

---

GPU Support (NVIDIA)

GPU acceleration benefits deep learning chapters (Ch05, Ch10, Ch13, Ch14, Ch21). Requires NVIDIA GPU with CUDA support.

Requirements

• NVIDIA GPU (GTX 1060 or better)
• NVIDIA Driver 525+ (for CUDA 12.x)
• Linux (native) or Windows 11 (WSL2)
• Not available on macOS

Ubuntu: Install nvidia-container-toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Windows WSL2

GPU passthrough works automatically with NVIDIA Driver 525+ installed on Windows and Docker Desktop with WSL2 backend.

Verify and Run

# Verify GPU is visible
docker compose --profile gpu run --rm ml4t-gpu python -c \
  "import torch; print(f'CUDA: {torch.cuda.is_available()}')"
# Should print: CUDA: True

# Run with GPU
docker compose --profile gpu run --rm ml4t-gpu python 13_dl_time_series/01_core_architectures.py

---

Storage Benchmarks (Chapter 2)

Chapter 2 includes storage benchmarks comparing file formats and databases.

# Pull benchmark image
docker compose pull benchmark

# Start database services
docker compose --profile benchmark up -d timescaledb clickhouse questdb influxdb

# Wait for databases to be healthy
docker compose --profile benchmark ps

# Run benchmark
docker compose --profile benchmark run --rm benchmark \
  python 02_financial_data_universe/21_storage_benchmark_database.py

# Stop databases when done
docker compose --profile benchmark down

---

Py312 Image (Specific Notebooks)

A small number of notebooks require Python 3.12 libraries not available on Python 3.14:

Notebook	Library	Chapter
01_timegan, 03_sigcwgan_signatures, 07_dp_gan	signatory, torch CUDA bug on 3.14	Ch05
06_path_signatures, 12_wasserstein_regimes	signatory, esig	Ch09
01_word2vec, 02_asset_embeddings, 03_sentiment_evolution	gensim	Ch10
10_shap_nlp_sentiment	torch CUDA bug + shap	Ch12
06_conditional_autoencoder	torch CUDA bug + shap	Ch14
06_fed_announcement_bsts	tfcausalimpact (TFP BSTS)	Ch15
05_deep_hedging_pfhedge	pfhedge (unmaintained, numpy<2)	Ch21

# x86 systems only (Linux, Windows WSL2, macOS Intel)
docker compose --profile py312 pull py312
docker compose --profile py312 run --rm py312 python 05_synthetic_data/03_sigcwgan_signatures.py

Apple Silicon: These notebooks cannot run natively. View the pre-executed .ipynb files in Jupyter or on GitHub.

---

Troubleshooting

Docker Desktop hangs on "Starting the Docker Engine…" (Windows)

Check the status bar at the bottom of the Docker Desktop window. If it reads RAM 0.00 GB and CPU 0.00%, the engine's virtual machine never started, and the cause is the WSL2 backend rather than Docker itself. Work through it in this order:

1. Virtualization off in firmware. Task Manager → Performance → CPU → Virtualization. If it says Disabled, enable Intel VT-x / AMD-V in BIOS/UEFI. See step 0 of the Windows setup above.
2. Pending reboot. If you ran wsl --install and did not restart, restart now.
3. No Linux distribution. Run wsl --list --verbose in PowerShell. If it prints has no installed distributions, run wsl --install -d Ubuntu and restart.
4. WSL2 backend not selected. Docker Desktop → Settings → General → "Use the WSL 2 based engine".

Then quit Docker Desktop fully (right-click the tray icon → Quit) and start it again.

"Cannot connect to Docker daemon"

• Linux: sudo systemctl start docker && sudo systemctl enable docker
• Windows/macOS: Ensure Docker Desktop is running (system tray / menu bar)
• Windows WSL2: Make sure you are running from a WSL terminal, not PowerShell. Verify integration: Docker Desktop → Settings → Resources → WSL Integration → enable your distribution

Out of memory or container killed (WSL2)

WSL2 defaults to 50% of host RAM. Large notebooks (Ch13 deep learning, case study pipelines) may exceed this. Edit %USERPROFILE%\.wslconfig:

[wsl2]
memory=12GB
swap=4GB

Then restart: wsl --shutdown from PowerShell and reopen your terminal.

"Permission denied" on Linux

sudo usermod -aG docker $USER
# Log out and back in

Slow on Apple Silicon

If a container is slow, check if it's running under x86 emulation:

docker compose run --rm ml4t uname -m
# Should print: aarch64 (native) not x86_64 (emulated)

If you see x86_64, the image may not have an arm64 variant. The ml4t and benchmark images both have native arm64 builds.

"No space left on device"

docker system prune -a    # Remove unused images/containers
docker system df           # Check space usage

Build fails with network errors

# Behind a proxy:
export HTTP_PROXY=http://proxy:port
export HTTPS_PROXY=http://proxy:port
docker compose pull ml4t

---

Local Setup with uv (Alternative to Docker)

Docker is recommended because it guarantees a consistent environment. But if you prefer a local Python setup — for faster iteration, IDE integration, or GPU access without container overhead — uv handles everything from Python installation through dependency resolution.

What uv Does

uv is a fast Python package manager written in Rust. It replaces pip, venv, pip-tools, and pyenv in a single tool. When you run uv sync, it:

1. Reads pyproject.toml for dependency specifications
2. Reads uv.lock for exact pinned versions (reproducible across machines)
3. Creates a virtual environment in .venv/
4. Installs all packages including PyTorch with CUDA support

Setup

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone and enter the repository
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading

# Install all dependencies (creates .venv/, installs ~300 packages)
uv sync

# Copy environment template and add API keys
cp .env.example .env
# Edit .env — see data/README.md for API key instructions

# Verify
uv run python -c "import polars, torch, lightgbm; print('Ready')"

How pyproject.toml Works

The pyproject.toml at the repository root defines all Python dependencies:

• Core data science: NumPy, SciPy, Pandas, Polars, PyArrow
• Visualization: Plotly, Matplotlib, Seaborn
• Machine learning: scikit-learn, XGBoost, LightGBM, CatBoost, Optuna, SHAP
• Deep learning: PyTorch 2.x (with CUDA 12.8 on Linux/Windows)
• NLP: Hugging Face Transformers, sentence-transformers, FinBERT
• ML4T libraries: ml4t-data, ml4t-engineer, ml4t-models, ml4t-diagnostic, ml4t-backtest, ml4t-live (installed from PyPI)

The lockfile uv.lock pins every transitive dependency to exact versions, so uv sync produces the same environment regardless of when you install.

What Local Setup Cannot Run

A few notebooks require Docker because their dependencies have no Python 3.14 wheel or need external services:

Notebook	Reason	Docker Image
Ch05 03_sigcwgan_signatures	signatory requires Python 3.12	py312
Ch09 06_path_signatures	esig requires Python 3.12	py312
Ch10 01-03 (word2vec, embeddings, sentiment)	gensim requires Python 3.12	py312
Ch12 10_shap_nlp_sentiment	torch CUDA bug on 3.14 + shap	py312
Ch14 06_conditional_autoencoder	torch CUDA bug on 3.14 + shap	py312
Ch15 06_fed_announcement_bsts	tfcausalimpact requires Python 3.12	py312
Ch21 05_deep_hedging_pfhedge	pfhedge requires numpy<2	py312
Ch02 21_storage_benchmark_database	requires database services	benchmark

For these, use docker compose with the appropriate profile even if your main workflow is local.

GPU with Local Setup

PyTorch auto-detects NVIDIA GPUs when CUDA drivers are installed. No special configuration needed:

uv run python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"N/A\"}')"

GPU-intensive notebooks (Ch05 GANs, Ch13 deep learning, Ch14 autoencoders, Ch21 RL) benefit from GPU but all include CPU fallback with reduced parameters.

---

Next Steps

• Running Notebooks — How to execute notebooks, Papermill test mode, case study pipelines
• Data Guide — Download required datasets