wehub/stefan-jansen--machine-learning-for-trading
1
0
Fork 0
Code Issues Pull Requests Actions 3 Deployments Packages Projects Releases Wiki Activity
Files
main
stefan-jansen--machine-lear…/docs/running-notebooks.md
T
wehub-resource-sync 0249a267e1 chore: import upstream snapshot with attribution
2026-07-13 13:26:28 +08:00

15 KiB
Raw Permalink Blame History

Running Notebooks

This guide explains how to execute notebooks, work with case studies, and experiment with your own strategies.

Two Ways to Run

Option A: Docker (Recommended)

Docker provides a consistent environment across all platforms with pre-built images on Docker Hub. After installation:

# Pull the image (one time, ~12 GB on x86, ~3 GB on ARM64)
docker compose pull ml4t

# Start Jupyter Lab
docker compose up ml4t
# Open http://localhost:8888

# Run a notebook directly
docker compose run --rm ml4t python 11_ml_pipeline/01_ols_inference.py

# Run with GPU (deep learning chapters)
docker compose --profile gpu run --rm ml4t-gpu python 13_dl_time_series/01_core_architectures.py

Docker covers all notebooks across all 27 chapters and 9 case studies, though a small subset requires a non-default profile such as py312, benchmark, or rapids.

Option B: Local with uv (Advanced)

uv is a fast Python package manager that handles virtual environments automatically. A local setup covers ~90% of notebooks; a few require Docker:

Docker-Only Notebooks Reason Image
Ch05 03_sigcwgan_signatures signatory (no Python 3.14 wheel) py312
Ch09 06_path_signatures, 12_wasserstein_regimes signatory, esig (no Python 3.14 wheel) py312
Ch10 01_word2vec, 02_asset_embeddings, 03_sentiment_evolution gensim (no Python 3.14 wheel) py312
Ch12 10_shap_nlp_sentiment torch CUDA bug on 3.14 + shap py312
Ch14 06_conditional_autoencoder torch CUDA bug on 3.14 + shap py312
Ch15 06_fed_announcement_bsts tfcausalimpact (TFP BSTS, py<3.13) py312
Ch21 05_deep_hedging_pfhedge pfhedge (unmaintained, numpy<2) py312
Ch02 21_storage_benchmark_database requires benchmark image + database services benchmark
Ch12 02_gbm_comparison (GPU section) RAPIDS cuML, LightGBM CUDA rapids 
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/stefan-jansen/machine-learning-for-trading.git
cd machine-learning-for-trading

# Set up environment
cp .env.example .env
# Edit .env to add API keys (see data/README.md)

# Install all dependencies
uv sync

# Run a notebook
uv run python 11_ml_pipeline/01_ols_inference.py

Platform notes for local setup:

  • Python 3.14+ required
  • TA-Lib must be installed separately (instructions)
  • GPU: PyTorch auto-detects CUDA if NVIDIA drivers are installed
  • Apple Silicon: Most packages have native ARM64 wheels; the py312 notebooks above cannot run on ARM64 — view their pre-executed .ipynb files instead

Notebook Format

Notebooks use Jupytext percent format: the source of truth is the .py file, and .ipynb is generated from it.

11_ml_pipeline/
  01_ols_inference.py      # Source (edit this)
  01_ols_inference.ipynb   # Generated (view in Jupyter)

Viewing: Open .ipynb files in Jupyter Lab, VS Code, or on GitHub (rendered with outputs).

Running: Execute the .py file from the repo root:

uv run python 11_ml_pipeline/01_ols_inference.py
# or
docker compose run --rm ml4t python 11_ml_pipeline/01_ols_inference.py

Important: Always run from the repository root. Running from a subdirectory will fail with ImportError: No module named 'utils'.

Chapter Notebooks

Each chapter directory contains teaching notebooks that demonstrate concepts from the book:

07_defining_the_learning_task/
  01_data_preprocessing.py       # Notebook (Jupytext source)
  01_data_preprocessing.ipynb    # Notebook (Jupyter, with outputs)
  02_label_methods.py
  ...
  README.md                      # Chapter overview and notebook guide

These notebooks are self-contained. Run them in order within a chapter, or jump to any notebook that interests you — most chapter notebooks only depend on downloaded data, not on other notebooks.

Case Study Notebooks

Each case study applies the same end-to-end research workflow to a different market — ETFs, crypto perpetuals, intraday equities, options, FX, futures, and equity factor panels. Each is a pipeline: every stage writes artifacts (labels, features, predictions, backtests) that later stages consume. The stages are already programmed and extensible, but they read like a research process — run one straight through, or open any stage, change it, and re-run from there.

The Workflow

Every case study follows the same sequence of phases, and each phase maps to a book chapter. The stage numbers differ from one case study to the next — each market gets a different set of model-family stages (more or fewer deep-learning architectures, latent-factor models where they apply) — but the phase order is identical everywhere:

Phase Chapter What it does
Feasibility Ch6 Universe breadth, point-in-time eligibility, horizon-cost feasibility, walk-forward setup
Labels Ch7 Forward returns and classification labels with walk-forward splits
Financial features Ch8 Momentum, volatility, carry, and cross-sectional ranking features
Model-based features Ch9 ARIMA, GARCH, HMM, and spectral features from walk-forward fits
Evaluation Ch7–9 Feature–label IC diagnostics across all engineered features
Linear Ch11 Ridge / LASSO / ElasticNet baseline every later model must beat
Gradient boosting Ch12 LightGBM with Optuna tuning
Tabular DL Ch12 TabM / TabPFN tabular deep learning
Sequence DL Ch13 LSTM, TCN, TSMixer, PatchTST (architectures vary by market)
Latent factors Ch14 PCA, IPCA, autoencoders, SDF (where applicable)
Causal DML Ch15 Double ML — does the signal cause returns or reflect confounders?
Model analysis Ch11–15 Cross-family IC comparison, fold stability, checkpoint sensitivity
Backtest Ch16 Strategy simulation, falsified against an equal-weight benchmark
Portfolio Ch17 Score-weighted, risk-parity, inverse-vol, MVO, HRP, conformal allocation
Costs Ch18 Transaction-cost impact on the edge
Risk Ch19 Position-level stops, trailing stops, time exits
Strategy analysis Ch20 End-to-end assessment — IC, Sharpe, cost survival, holdout

Each case study's own README.md lists its exact stage files with this mapping. To see a given case study's stages, list them:

ls case_studies/etfs/        # 01_feasibility_analysis.py … 18_strategy_analysis.py

Running a Case Study End to End

Run the stages in order from the repo root. Using the ETF case study (stages 01–18):

# Define, label, engineer, evaluate (Ch6–9)
uv run python case_studies/etfs/01_feasibility_analysis.py
uv run python case_studies/etfs/02_labels.py
uv run python case_studies/etfs/03_financial_features.py
uv run python case_studies/etfs/04_model_based_features.py
uv run python case_studies/etfs/05_evaluation.py

# Train model families — run any or all (Ch11–15)
uv run python case_studies/etfs/06_linear.py
uv run python case_studies/etfs/07_gbm.py
# … 08_tabular_dl, 09_dl_lstm, 10_dl_tsmixer, 11_latent_factors, 12_causal_dml
uv run python case_studies/etfs/13_model_analysis.py

# Build the strategy — backtest, portfolio, costs, risk, synthesis (Ch16–20)
uv run python case_studies/etfs/14_backtest.py
uv run python case_studies/etfs/15_portfolio_management.py
uv run python case_studies/etfs/16_costs.py
uv run python case_studies/etfs/17_risk_management.py
uv run python case_studies/etfs/18_strategy_analysis.py

Each stage checks for the artifacts it needs and tells you which earlier stage to run if anything is missing, so you can always pick up partway through.

The Run Log

Every model training run, prediction set, causal-effect estimate, and backtest is recorded in a per-case-study run log (run_log/). The SQLite catalog run_log/registry.db is the single source of truth for all metrics discussed in the book — IC scores, Sharpe ratios, drawdowns, etc.

See case_studies/RUN_LOG.md for the schema and querying API.

Pre-Computed Results (Download Artifacts)

Running all nine case study pipelines end-to-end (training ~50 model configurations, running ~1,000 backtests per case study) takes days of compute. To let you explore results immediately, we provide a curated subset of artifacts as a GitHub release:

# Download all case study artifacts (~1.6 GB total)
uv run python scripts/download_artifacts.py

# Download a single case study
uv run python scripts/download_artifacts.py --cs etfs

# Check what's installed
uv run python scripts/download_artifacts.py --list

This populates case_studies/{cs}/run_log/ with:

  • registry.db — full metrics database (all training runs, predictions, backtests)
  • Best predictions per model family — validation predictions for cross-model comparison (Ch11-15 insight notebooks)
  • Top-10 predictions by IC — for backtest analysis (Ch16)
  • Top backtests by stage — signal, allocation, cost sensitivity (Ch17-19 strategy notebooks)
  • Holdout predictions — for out-of-sample synthesis (Ch20)

With these artifacts, you can:

  1. Browse results immediately — the model-analysis and strategy-analysis stages load predictions and metrics from the registry
  2. Reproduce selectively — run any model notebook to verify or extend results
  3. Experiment — new runs register automatically alongside the shipped baselines
  4. Compare — analytical notebooks query whatever is in the registry, so your experiments appear next to the book's results

What's not included: The full set of ~1,000 backtest variations per case study (these total ~97 GB). The download provides the ~20 best-performing configurations that the book discusses. You can generate the rest by running the backtest stages yourself.

Experimenting

The case study pipeline is designed for experimentation. Here are common workflows:

Try Different Model Hyperparameters

Open a model notebook (e.g., 07_gbm.py), modify the configuration, and run it. The new run registers with a unique hash — your results coexist with the originals.

# In 07_gbm.py, change the parameter grid:
PARAM_GRID = {
    "num_leaves": [31, 63, 127],      # Try more complex trees
    "learning_rate": [0.01, 0.05],     # Different learning rates
    "min_child_samples": [20, 50],
}

Try a Different Backtest Configuration

Modify the signal-to-position mapping, change cost assumptions, or adjust position sizing:

# In 14_backtest.py, change the strategy:
TOP_N = 10              # Hold top 10 instead of top 20
COST_BPS = 15           # Higher transaction costs
REBALANCE_FREQ = "W"    # Weekly instead of monthly

Compare Your Experiments

Open the analysis notebook — it automatically picks up all registry entries:

uv run python case_studies/etfs/18_strategy_analysis.py
# Shows your new runs alongside the book's baselines

Data Requirements

Notebooks require downloaded datasets. See data/README.md for the complete data guide.

Quick start with free data:

# Download ETF data (Yahoo Finance, no API key needed)
uv run python data/etfs/market/download.py

# Download all free datasets
uv run python data/download_all.py --free-only

Some datasets require API keys (set in .env):

  • OANDA (FX pairs): Free API key from oanda.com
  • NASDAQ Data Link (US equities): Free API key from data.nasdaq.com
  • Databento (CME futures): $125 free signup credit from databento.com
  • AlgoSeek (microstructure, options): Requires commercial license

Accelerated Execution with Papermill

Every notebook has a parameters cell (# %% tags=["parameters"]) with production defaults — the values readers see in the book. Papermill can inject override values that reduce data scope, training epochs, or universe size so notebooks complete in minutes instead of hours.

How It Works

  1. The parameters cell defines production values:

    # %% tags=["parameters"]
MAX_SYMBOLS = 0      # 0 = all symbols (production)
N_EPOCHS = 500
START_DATE = "2006-01-01"

  2. Papermill creates an injected cell after the tagged cell that overrides selected values:

    # Injected by Papermill
MAX_SYMBOLS = 15     # Reduced for fast execution
N_EPOCHS = 2

  3. The notebook code sees only the final (overridden) values. Same code path always runs — there are no if TEST: branches.

Running a Single Notebook with Overrides

# Run with reduced parameters (output goes to /dev/null)
uv run papermill notebook.ipynb /dev/null \
    --cwd . -k python3 \
    -p MAX_SYMBOLS 15 \
    -p N_EPOCHS 2

# Or save the executed notebook
uv run papermill notebook.ipynb output.ipynb \
    --cwd . -k python3 \
    -p MAX_SYMBOLS 15

Running via pytest (Recommended)

The test suite reads per-notebook overrides from tests/overrides.yaml and runs each notebook through Papermill with appropriate parameter reductions:

# Run all notebooks in a chapter
uv run pytest tests/test_chapter_notebooks.py -v -k "11_ml_pipeline"

# Run a specific notebook
uv run pytest tests/test_chapter_notebooks.py -v -k "01_ols_inference"

# Run all case study notebooks for ETFs
uv run pytest tests/test_chapter_notebooks.py -v -k "etfs"

# Run everything (takes ~2 hours with reduced parameters)
uv run pytest tests/test_chapter_notebooks.py -v

Override Configuration

Test parameter overrides are defined in tests/overrides.yaml, keyed by notebook path:

# Example entries
11_ml_pipeline/01_ols_inference:
  timeout: 180
  parameters:
    MAX_SYMBOLS: 15

case_studies/etfs/07_gbm:
  timeout: 300
  parameters:
    MAX_SYMBOLS: 15
    START_DATE: "2020-01-01"

To customize for your machine: copy tests/overrides.yaml to tests/overrides.local.yaml (gitignored) and adjust timeouts or parameter values. The test runner checks for the local file first.

Output Isolation

When the environment variable ML4T_OUTPUT_DIR is set (which pytest does automatically), all notebook outputs are redirected to a temporary directory. This prevents test runs from overwriting production artifacts like trained models or backtest results.

# Manual output isolation
ML4T_OUTPUT_DIR=/tmp/ml4t-test uv run python case_studies/etfs/07_gbm.py

Headless Execution

For running notebooks without a display (e.g., on a server or in CI):

MPLBACKEND=Agg PLOTLY_RENDERER=json uv run python notebook.py

Troubleshooting

"No module named 'utils'"

You're running from a subdirectory. Always run from the repository root:

# Wrong
cd 11_ml_pipeline && python 01_ols_inference.py

# Right
uv run python 11_ml_pipeline/01_ols_inference.py

Missing prerequisite files

Case study notebooks check for upstream artifacts. If a file is missing, the notebook tells you which notebook to run first.

Slow notebooks

Some model training notebooks take several minutes. Notebooks with long runtimes print progress during execution. For faster iteration, reduce the data scope in the parameters cell at the top of each notebook.

Reference in New Issue View Git Blame Copy Permalink