wehub/stefan-jansen--machine-learning-for-trading
1
0
Fork 0
Code Issues Pull Requests Actions 3 Deployments Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:26:28 +08:00
commit 0249a267e1
495 changed files with 1277094 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+93
View File
@@ -0,0 +1,93 @@
# ML4T Configuration Template
# Copy this to .env and configure for your environment:
# cp .env.example .env
# # Then edit .env with your actual paths
# ============================================================================
# Path Configuration
# ============================================================================
# Repository path (auto-detected, rarely needs to be set)
# ML4T_PATH=/path/to/third-edition
# Data directory path
# - Default: ./data (repo's data folder)
# - Override for external data: /path/to/your/data
# ML4T_DATA_PATH=/path/to/data
# ============================================================================
# Docker Configuration (Only needed when using Docker)
# ============================================================================
# User mapping to prevent permission issues in Docker
# Set these to your host user ID and group ID:
# UID=$(id -u) # Usually 1000
# GID=$(id -g) # Usually 1000
UID=1000
GID=1000
# ============================================================================
# Test / CI
# ============================================================================
# Output redirection (used by the pytest harness - not needed for manual runs).
# When set, notebook outputs go here instead of chapter/output/. The harness
# also reduces epochs/folds/rows by injecting papermill `parameters`-tagged
# cells at execution time; there is no global fast-mode flag for manual runs.
# ML4T_OUTPUT_DIR=/tmp/ml4t-test-output
# ============================================================================
# API Keys (Optional - only needed for downloading data)
# ============================================================================
# SEC EDGAR - required for Ch04 NB02 (SEC filing explorer) and NB14 (text
# extraction). The SEC mandates a real User-Agent (your name + email) on every
# EDGAR request and blocks placeholder addresses; the notebook raises if unset.
# Format: "Jane Doe jane@example.org"
EDGAR_IDENTITY=
# FRED - Federal Reserve Economic Data (free API key)
# Get key: https://fred.stlouisfed.org/docs/api/api_key.html
FRED_API_KEY=
# DataBento - Futures and market data (paid)
# Get key: https://databento.com
DATABENTO_API_KEY=
# Oanda - Forex data (free tier available)
# Get key: https://www.oanda.com/
OANDA_API_KEY=
# Alpaca - Equity and crypto trading (free tier available)
# Get key: https://alpaca.markets/
ALPACA_API_KEY=
ALPACA_SECRET_KEY=
# Polygon - Multi-asset market data (free tier available)
# Get key: https://polygon.io/
POLYGON_API_KEY=
# LLM providers for Ch. 24 Autonomous Agent demo (any one is enough; the
# client auto-selects Anthropic -> OpenAI -> Google -> OpenRouter -> local
# Ollama, then falls back to a deterministic mock if none are set).
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
GOOGLE_API_KEY=
# OpenRouter: one key, any model. Optionally pin OPENROUTER_MODEL.
OPENROUTER_API_KEY=
# OPENROUTER_MODEL=anthropic/claude-sonnet-4
# Tavily - Agentic web search for Ch. 24 Autonomous Agent demo
TAVILY_API_KEY=
# TabPFN - Ch. 12 (12_gradient_boosting/03_dl_vs_gbm). The tabpfn package
# requires a one-time license acceptance to download model weights for local
# inference. Register at https://ux.priorlabs.ai, accept the license, and copy
# your API key from https://ux.priorlabs.ai/account. The other models in the
# notebook (LightGBM, MLP, TabM) run without it.
TABPFN_TOKEN=
# Ch. 24 research operator (nb11) — companion skill library (github.com/ml4t/skills).
# Default: cloned next to the code repo (../skills). Set this to override, or
# when running in Docker (where ../skills is not present).
# RESEARCH_OPERATOR_SKILLS_ROOT=/path/to/skills
.github/dependabot.yml
+24
View File
@@ -0,0 +1,24 @@
# Dependabot configuration for the ML4T companion repository.
#
# This repo is an educational companion to the book — readers clone it and run
# notebooks locally; it is not a deployed service. We therefore keep Dependabot
# to SECURITY updates only, and batch them:
# - open-pull-requests-limit: 0 disables routine version-bump PRs. Security
# updates are unaffected (they use a separate internal limit), so genuine
# vulnerability fixes still arrive.
# - the `groups` block collapses all security updates into a single PR instead
# of one-per-package, on a monthly cadence, to avoid churn.
# The dependency lock is canonical in the private code repo and mirrored here, so
# updates are reviewed and batched rather than merged piecemeal on this mirror.
version: 2
updates:
- package-ecosystem: "uv"
directory: "/"
schedule:
interval: "monthly"
open-pull-requests-limit: 0
groups:
security:
applies-to: "security-updates"
patterns:
- "*"
.github/workflows/container-smoke.yml
+100
View File
@@ -0,0 +1,100 @@
# ML4T 3rd Edition — Container first-contact smoke (Phase 2b)
#
# Runs the reader's ACTUAL first command through docker compose, in the reader's
# real image, and proves the free-data download can write to /data. This is the
# only layer that catches compose/mount/env regressions like #361 (the `/data`
# read-only mount) — a unit test can't, because it never composes the container.
#
# Cost control: the ml4t image is ~12 GB, so per-PR we run this ONLY when the
# compose/env surface changes (docker-compose.yml, envs/**). The download-script
# *logic* is covered deterministically by the native `test-unit` job on every PR
# (tests/test_download_helpers.py et al.) and the static mount contract by
# tests/test_compose_mounts.py — neither needs the image. The weekly run exercises
# the full path regardless and files an issue if the reader's first contact breaks.
name: Container Smoke
on:
schedule:
- cron: '0 6 * * 1' # Monday 06:00 UTC
pull_request:
branches: [main]
paths:
- 'docker-compose.yml'
- 'envs/**'
- '.github/workflows/container-smoke.yml'
workflow_dispatch:
jobs:
container-smoke:
name: container-smoke
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@v6
- name: First-contact download in the composed container
id: smoke
env:
# Point the compose data mount at a workspace dir we own, so we can
# assert the downloaded files actually land on the host side of the
# bind mount (proves the mount is writable AND correctly wired).
ML4T_DATA_PATH: ${{ github.workspace }}/_smoke_data
run: |
set -o pipefail
mkdir -p "$ML4T_DATA_PATH"
# The reader's real first command, scoped to a single symbol so the
# pull is tiny. --user root mirrors the documented in-container
# download invocation and still fails hard on a read-only mount (even
# root cannot write to a `:ro` bind), so this catches the #361 class.
docker compose run --rm --user root ml4t \
python data/etfs/market/download.py --symbol SPY 2>&1 | tee smoke.log
echo "----- assertions -----"
# 1. No read-only-mount failure (the #361 signature).
if grep -qi "Read-only file system" smoke.log; then
echo "FAIL: container hit a read-only /data mount (#361 regression)"
exit 1
fi
# 2. The download actually populated the host side of the /data mount.
out="$ML4T_DATA_PATH/etfs/market/etf_universe.parquet"
if [ ! -f "$out" ]; then
echo "FAIL: expected $out on the host mount, but it is missing"
echo "Contents of $ML4T_DATA_PATH:"
find "$ML4T_DATA_PATH" -maxdepth 3 -print || true
exit 1
fi
echo "OK: $out present ($(du -h "$out" | cut -f1)) — /data mount is writable and wired"
- name: Open issue on weekly failure
if: failure() && github.event_name == 'schedule'
uses: actions/github-script@v7
with:
script: |
const owner = context.repo.owner;
const repo = context.repo.repo;
const runUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`;
const title = 'container-smoke: first-contact download failed';
const q = `repo:${owner}/${repo} is:issue is:open in:title "${title}"`;
const existing = await github.rest.search.issuesAndPullRequests({ q });
if (existing.data.total_count > 0) {
const num = existing.data.items[0].number;
await github.rest.issues.createComment({
owner, repo, issue_number: num,
body: `Failed again in the weekly container smoke: ${runUrl}`,
});
} else {
await github.rest.issues.create({
owner, repo, title,
body: `The weekly container first-contact smoke failed.\n\n` +
`The reader's \`docker compose run … data/etfs/market/download.py --symbol SPY\` ` +
`did not populate \`/data\` in the composed \`ml4t\` container.\n\n` +
`This is the #361 class (compose mount / env wiring). Run: ${runUrl}`,
labels: ['weekly-flake'],
});
}
.github/workflows/full-test.yml
+198
View File
@@ -0,0 +1,198 @@
# ML4T Full Test Suite — Release Gate
#
# Runs ALL 443 notebooks inside Docker. Manual dispatch only.
# Expected runtime: 60-90 minutes.
#
# Use this before releases to verify everything works end-to-end.
# For incremental testing on push/PR, use test.yml instead.
name: Full Test Suite
on:
workflow_dispatch:
inputs:
scope:
description: "Test scope"
type: choice
options:
- all
- chapters-only
- case-studies-only
default: all
env:
ML4T_DATA_PATH: ${{ github.workspace }}/test-data/data
ML4T_OUTPUT_DIR: /tmp/ml4t-test-output
MPLBACKEND: Agg
PLOTLY_RENDERER: json
jobs:
# ---------------------------------------------------------------------------
# All chapter notebooks (Ch01-Ch26)
# ---------------------------------------------------------------------------
chapters:
if: >-
github.event.inputs.scope == 'all' ||
github.event.inputs.scope == 'chapters-only'
timeout-minutes: 120
runs-on: ubuntu-latest
container:
image: ml4t/ml4t:latest
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Seed intermediates
run: |
mkdir -p $ML4T_OUTPUT_DIR
if [ -d test-data/intermediates ]; then
cp -r test-data/intermediates/* $ML4T_OUTPUT_DIR/
fi
- name: Run ALL chapter notebooks
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pytest tests/test_chapter_notebooks.py \
-v --tb=short \
--timeout=600
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: chapter-results
path: /tmp/ml4t-test-results.jsonl
if-no-files-found: ignore
# ---------------------------------------------------------------------------
# All case study pipelines (9 case studies)
# ---------------------------------------------------------------------------
case-studies:
if: >-
github.event.inputs.scope == 'all' ||
github.event.inputs.scope == 'case-studies-only'
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
case-study:
- etfs
- crypto_perps_funding
- nasdaq100_microstructure
- sp500_equity_option_analytics
- us_firm_characteristics
- fx_pairs
- cme_futures
- sp500_options
- us_equities_panel
runs-on: ubuntu-latest
container:
image: ml4t/ml4t:latest
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Run pipeline (${{ matrix.case-study }})
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pytest tests/test_case_studies.py \
-v --tb=short \
-k "${{ matrix.case-study }}" \
--timeout=600
# ---------------------------------------------------------------------------
# Py312 notebooks (signatory, gensim, esig, pfhedge, tfcausalimpact, torch CUDA bug)
# ---------------------------------------------------------------------------
py312:
if: github.event.inputs.scope == 'all'
timeout-minutes: 45
runs-on: ubuntu-latest
container:
image: ml4t/ml4t-py312:latest
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
- name: Run py312 notebooks
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pytest tests/test_docker_notebooks.py \
-v --tb=short \
-k "03_sigcwgan or 06_path_signatures or 12_wasserstein \
or 01_word2vec or 02_asset_embeddings or 03_sentiment_evolution \
or 01_timegan or 07_dp_gan \
or 10_shap_nlp_sentiment or 06_conditional_autoencoder \
or 06_fed_announcement_bsts \
or deep_hedging_pfhedge"
# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
summary:
runs-on: ubuntu-latest
needs: [chapters, case-studies, py312]
if: always()
steps:
- name: Download results
uses: actions/download-artifact@v4
with:
pattern: "*-results"
merge-multiple: true
path: results/
continue-on-error: true
- name: Report
run: |
echo "============================================"
echo "ML4T Full Test Suite — Summary"
echo "============================================"
echo "Chapters: ${{ needs.chapters.result }}"
echo "Case Studies: ${{ needs.case-studies.result }}"
echo "Py312: ${{ needs.py312.result }}"
echo "============================================"
if [[ "${{ needs.chapters.result }}" == "failure" ]] || \
[[ "${{ needs.case-studies.result }}" == "failure" ]] || \
[[ "${{ needs.py312.result }}" == "failure" ]]; then
echo "RESULT: SOME TESTS FAILED"
exit 1
else
echo "RESULT: ALL TESTS PASSED"
fi
.github/workflows/test.yml
+676
View File
@@ -0,0 +1,676 @@
# ML4T 3rd Edition — CI Test Workflow
#
# Tests notebooks inside pre-built Docker images (ml4t/ml4t:latest).
# This ensures CI tests exactly what readers will use.
#
# Architecture:
# - All notebook tests run inside Docker containers (not native Python)
# - Incremental: dorny/paths-filter only tests changed chapters
# - Pre-built images from Docker Hub (built by docker-publish.yml)
# - Special images for py312 (signatory/gensim/tfcausalimpact), Neo4j, benchmarks
#
# Trigger policy:
# - PR: incremental (path-filtered, only changed chapters)
# - Weekly: full suite on schedule (Monday 6 AM UTC)
# - After Docker Publish: full suite (workflow_run trigger)
# - Manual: workflow_dispatch with optional run_all flag
# - Push to main: NOT triggered (conserves minutes)
name: Tests
on:
pull_request:
branches: [main]
schedule:
- cron: '0 6 * * 1' # Weekly Monday 6 AM UTC
workflow_run:
workflows: ["Docker Publish"]
types: [completed]
workflow_dispatch:
inputs:
run_all:
description: "Run all tests (ignore path filters)"
type: boolean
default: false
test_filter:
description: "pytest -k filter (e.g. 'us_equities_panel or nasdaq100' to run only those)"
type: string
default: ""
env:
ML4T_DATA_PATH: ${{ github.workspace }}/test-data/data
ML4T_OUTPUT_DIR: /tmp/ml4t-test-output
MPLBACKEND: Agg
PLOTLY_RENDERER: json
jobs:
# ---------------------------------------------------------------------------
# Detect which paths changed and build dynamic matrices
# ---------------------------------------------------------------------------
changes:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
outputs:
chapter_matrix: ${{ steps.build.outputs.chapter_matrix }}
case_matrix: ${{ steps.build.outputs.case_matrix }}
run_py312: ${{ steps.build.outputs.run_py312 }}
run_neo4j: ${{ steps.build.outputs.run_neo4j }}
run_benchmark: ${{ steps.build.outputs.run_benchmark }}
steps:
- uses: actions/checkout@v6
- uses: dorny/paths-filter@v4
id: filter
with:
filters: |
shared:
- 'utils/**'
- 'data/**'
- 'tests/**'
- 'pyproject.toml'
- '.github/workflows/test.yml'
- 'matplotlibrc'
ch01:
- '01_process_is_edge/**'
ch02-03:
- '02_financial_data_universe/**'
- '03_market_microstructure/**'
- 'data/**'
ch04:
- '04_fundamental_alternative_data/**'
ch05:
- '05_synthetic_data/**'
ch06-07:
- '06_strategy_definition/**'
- '07_defining_the_learning_task/**'
ch08-09:
- '08_financial_features/**'
- '09_model_based_features/**'
ch10:
- '10_text_feature_engineering/**'
ch11-12:
- '11_ml_pipeline/**'
- '12_gradient_boosting/**'
ch13:
- '13_dl_time_series/**'
ch14-15:
- '14_latent_factors/**'
- '15_causal_estimation/**'
ch16-17:
- '16_strategy_simulation/**'
- '17_portfolio_construction/**'
ch18-20:
- '18_transaction_costs/**'
- '19_risk_management/**'
- '20_strategy_synthesis/**'
ch21:
- '21_rl_execution_hedging/**'
ch22:
- '22_rag_financial_research/**'
ch23:
- '23_knowledge_graphs/**'
ch24:
- '24_autonomous_agents/**'
ch25:
- '25_live_trading/**'
ch26:
- '26_mlops_governance/**'
cs-etfs:
- 'case_studies/etfs/**'
cs-crypto:
- 'case_studies/crypto_perps_funding/**'
cs-nasdaq:
- 'case_studies/nasdaq100_microstructure/**'
cs-sp500eoa:
- 'case_studies/sp500_equity_option_analytics/**'
cs-firmchar:
- 'case_studies/us_firm_characteristics/**'
cs-fx:
- 'case_studies/fx_pairs/**'
cs-futures:
- 'case_studies/cme_futures/**'
cs-sp500opt:
- 'case_studies/sp500_options/**'
cs-usequity:
- 'case_studies/us_equities_panel/**'
- name: Build dynamic matrices
id: build
run: |
shared="${{ steps.filter.outputs.shared }}"
schedule="${{ github.event_name == 'schedule' }}"
after_docker="${{ github.event_name == 'workflow_run' }}"
run_all="${{ github.event.inputs.run_all }}"
all="false"
if [[ "$shared" == "true" ]] || [[ "$schedule" == "true" ]] || [[ "$run_all" == "true" ]] || [[ "$after_docker" == "true" ]]; then
all="true"
fi
# --- Chapter matrix (all chapters including Ch01 and Ch05) ---
chapters="[]"
# Public-subset scoping: include a chapter only if at least one of its
# NN_* directories is actually present. The public repo ships chapters
# incrementally (beat by beat), so this auto-scopes the matrix to what
# is shipped — no per-beat workflow edits, and unshipped chapters can't
# fail with "no tests collected".
add_ch() {
local n="$1" f="$2" p
for p in $(echo "$f" | grep -oE '[0-9]{2}_' || true); do
if compgen -G "${p}*" >/dev/null 2>&1; then
chapters=$(echo "$chapters" | jq -c --arg n "$n" --arg f "$f" '. + [{"name":$n,"filter":$f}]')
return
fi
done
}
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch01 }}" == "true" ]]; then
add_ch "ch01" "01_process"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch02-03 }}" == "true" ]]; then
add_ch "ch02-03" "02_financial or 03_market"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch04 }}" == "true" ]]; then
add_ch "ch04" "04_fundamental"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch05 }}" == "true" ]]; then
add_ch "ch05" "05_synthetic"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch06-07 }}" == "true" ]]; then
add_ch "ch06-07" "06_strategy or 07_defining"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch08-09 }}" == "true" ]]; then
add_ch "ch08-09" "08_financial or 09_model"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch10 }}" == "true" ]]; then
add_ch "ch10" "10_text"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch11-12 }}" == "true" ]]; then
add_ch "ch11-12" "11_ml_pipeline or 12_gradient"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch13 }}" == "true" ]]; then
add_ch "ch13" "13_dl_time"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch14-15 }}" == "true" ]]; then
add_ch "ch14-15" "14_latent or 15_causal"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch16-17 }}" == "true" ]]; then
add_ch "ch16-17" "16_strategy or 17_portfolio"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch18-20 }}" == "true" ]]; then
add_ch "ch18-20" "18_transaction or 19_risk or 20_strategy"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch21 }}" == "true" ]]; then
add_ch "ch21" "21_rl"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch22 }}" == "true" ]]; then
add_ch "ch22" "22_rag"
fi
# Ch23 skipped — requires Neo4j service (tested in test-neo4j when ready)
# if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch23 }}" == "true" ]]; then
# add_ch "ch23" "23_knowledge"
# fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch24 }}" == "true" ]]; then
add_ch "ch24" "24_autonomous"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch25 }}" == "true" ]]; then
add_ch "ch25" "25_live"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.ch26 }}" == "true" ]]; then
add_ch "ch26" "26_mlops"
fi
echo "chapter_matrix=$chapters" >> "$GITHUB_OUTPUT"
# --- Py312 flag ---
run_py312="false"
if [[ "$all" == "true" ]] \
|| [[ "${{ steps.filter.outputs.ch05 }}" == "true" ]] \
|| [[ "${{ steps.filter.outputs.ch08-09 }}" == "true" ]] \
|| [[ "${{ steps.filter.outputs.ch10 }}" == "true" ]] \
|| [[ "${{ steps.filter.outputs.ch14-15 }}" == "true" ]] \
|| [[ "${{ steps.filter.outputs.ch21 }}" == "true" ]]; then
run_py312="true"
fi
echo "run_py312=$run_py312" >> "$GITHUB_OUTPUT"
# --- Neo4j flag (needs populated Neo4j — skip until infra is ready) ---
run_neo4j="false"
echo "run_neo4j=$run_neo4j" >> "$GITHUB_OUTPUT"
# --- Benchmark flag (disabled — QuestDB container fails health check in CI) ---
run_benchmark="false"
echo "run_benchmark=$run_benchmark" >> "$GITHUB_OUTPUT"
# --- Case study matrix ---
cases="[]"
# Public repo ships case-study CONFIG but gates the pipeline code.
# test_case_studies.py auto-discovers stages by globbing
# case_studies/<cs>/[0-9][0-9]*.py, and the results registry is seeded
# from the test-data repo at run time (NOT shipped to this repo). So
# gate on the same signal the test collects: enroll a case study only
# once its pipeline notebooks are shipped here — currently only
# config/setup.yaml ships, so case-study jobs stay skipped. Future
# beats that ship the notebooks auto-enroll, no workflow edits.
add_cs() {
if compgen -G "case_studies/$1/[0-9][0-9]*.py" >/dev/null 2>&1; then
cases=$(echo "$cases" | jq -c --arg cs "$1" '. + [{"case-study":$cs}]')
fi
}
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-etfs }}" == "true" ]]; then
add_cs "etfs"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-crypto }}" == "true" ]]; then
add_cs "crypto_perps_funding"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-nasdaq }}" == "true" ]]; then
add_cs "nasdaq100_microstructure"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-sp500eoa }}" == "true" ]]; then
add_cs "sp500_equity_option_analytics"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-firmchar }}" == "true" ]]; then
add_cs "us_firm_characteristics"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-fx }}" == "true" ]]; then
add_cs "fx_pairs"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-futures }}" == "true" ]]; then
add_cs "cme_futures"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-sp500opt }}" == "true" ]]; then
add_cs "sp500_options"
fi
if [[ "$all" == "true" ]] || [[ "${{ steps.filter.outputs.cs-usequity }}" == "true" ]]; then
add_cs "us_equities_panel"
fi
echo "case_matrix=$cases" >> "$GITHUB_OUTPUT"
echo "Chapters: $chapters"
echo "Cases: $cases"
echo "Py312: $run_py312"
echo "Neo4j: $run_neo4j"
echo "Benchmark: $run_benchmark"
# ---------------------------------------------------------------------------
# Lint (always runs, fast)
# ---------------------------------------------------------------------------
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: astral-sh/setup-uv@v6
with:
version: "latest"
- name: Ruff check
run: uvx ruff@0.15.8 check utils/ data/ tests/
- name: Ruff format
run: uvx ruff@0.15.8 format --check utils/ data/ tests/
# ---------------------------------------------------------------------------
# Fast native unit tests — download-script logic (no Docker, no network,
# no test-data, no secrets). Seconds per PR.
#
# Notebook tests run against pre-seeded data, so the data/**/download.py code
# path is invisible to them — this is how the in-container free-data download
# bug (#361) reached readers. These tests exercise that logic directly:
# path resolution / output placement, download_all dispatch modes, the
# firm-characteristics archive flatten, download-script registry drift, the
# compose /data-mount contract (static regression-lock for #361), and the
# drift-coverage ratchet (every free dataset keeps a live external smoke).
# The networked container smoke (Phase 2b) and live drift pulls (Phase 3) run
# in container-smoke.yml / weekly-external.yml, not here.
# ---------------------------------------------------------------------------
test-unit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: astral-sh/setup-uv@v6
with:
version: "latest"
- name: Install minimal test deps
# Just the import surface of the download-logic tests (utils/downloading
# + data/download_all + firm-char extract_zip). Intentionally NOT the
# full project env (no torch/ml4t-*) so this stays a fast per-commit gate.
run: |
uv venv --python 3.14
uv pip install pytest polars pyyaml python-dotenv plotly
- name: Run download-logic unit tests
env:
PYTHONPATH: ${{ github.workspace }}
# Override the workflow-level test-data path: this job checks out no
# test-data. utils.config validates ML4T_DATA_PATH exists at import and
# reads the env var ahead of .env, so point both at the repo's own dirs.
ML4T_PATH: ${{ github.workspace }}
ML4T_DATA_PATH: ${{ github.workspace }}/data
run: |
# utils.config also requires a .env file to exist; create a minimal one.
printf 'ML4T_PATH=%s\nML4T_DATA_PATH=%s/data\n' \
"$GITHUB_WORKSPACE" "$GITHUB_WORKSPACE" > .env
.venv/bin/python -m pytest \
tests/test_download_helpers.py \
tests/test_firm_characteristics_extract.py \
tests/test_download_all_dispatch.py \
tests/test_download_scripts_registry.py \
tests/test_compose_mounts.py \
tests/test_download_coverage.py \
-v --tb=short
# ---------------------------------------------------------------------------
# Chapter notebook tests — inside Docker (ml4t/ml4t:latest)
#
# Runs the same Python 3.14 environment that readers use.
# Notebooks needing signatory/gensim/esig/tfcausalimpact are skipped here,
# tested in test-py312 instead.
# ---------------------------------------------------------------------------
test-chapters:
name: ${{ matrix.name }}
timeout-minutes: 60
needs: [lint, changes]
if: needs.changes.outputs.chapter_matrix != '[]'
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.changes.outputs.chapter_matrix) }}
runs-on: ubuntu-latest
container:
image: ml4t/ml4t:latest
steps:
- name: Install git (for checkout inside container)
run: apt-get update -qq && apt-get install -y -qq git git-lfs openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Seed case study intermediates
run: |
mkdir -p $ML4T_OUTPUT_DIR
if [ -d test-data/intermediates ]; then
cp -r test-data/intermediates/* $ML4T_OUTPUT_DIR/
echo "Seeded $(du -sh $ML4T_OUTPUT_DIR | cut -f1) of intermediates"
fi
# Ch20 outputs go to repo dir (get_chapter_dir doesn't redirect)
if [ -d test-data/intermediates/ch20_synthesis/output ]; then
mkdir -p 20_strategy_synthesis/output
cp test-data/intermediates/ch20_synthesis/output/* 20_strategy_synthesis/output/
echo "Seeded Ch20 synthesis outputs"
fi
# Ch24/11 research_operator: artifacts dir is gitignored in this repo;
# source them from test-data into the notebook dir.
if [ -d test-data/data/autonomous_agents/operator_artifacts ]; then
mkdir -p 24_autonomous_agents/operator_artifacts
cp test-data/data/autonomous_agents/operator_artifacts/*.json \
24_autonomous_agents/operator_artifacts/
echo "Seeded ch24/11 operator artifacts"
fi
- name: Run chapter notebooks (${{ matrix.name }})
env:
PYTHONPATH: ${{ github.workspace }}
TEST_FILTER_INPUT: ${{ github.event.inputs.test_filter }}
MATRIX_FILTER: ${{ matrix.filter }}
FRED_API_KEY: ${{ secrets.FRED_API_KEY }}
run: |
FILTER="${TEST_FILTER_INPUT:-$MATRIX_FILTER}"
python -m pytest tests/test_chapter_notebooks.py \
-v --tb=short \
-k "$FILTER"
# ---------------------------------------------------------------------------
# Case study pipeline tests — inside Docker (ml4t/ml4t:latest)
# ---------------------------------------------------------------------------
test-case-studies:
name: cs-${{ matrix.case-study }}
timeout-minutes: 90
needs: [lint, changes]
if: needs.changes.outputs.case_matrix != '[]'
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.changes.outputs.case_matrix) }}
runs-on: ubuntu-latest
container:
image: ml4t/ml4t:latest
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git git-lfs openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Seed case study intermediates
run: |
mkdir -p $ML4T_OUTPUT_DIR
if [ -d test-data/intermediates ]; then
cp -r test-data/intermediates/* $ML4T_OUTPUT_DIR/
echo "Seeded $(du -sh $ML4T_OUTPUT_DIR | cut -f1) of intermediates"
fi
- name: Run pipeline (${{ matrix.case-study }})
env:
PYTHONPATH: ${{ github.workspace }}
TEST_FILTER_INPUT: ${{ github.event.inputs.test_filter }}
MATRIX_CASE_STUDY: ${{ matrix.case-study }}
FRED_API_KEY: ${{ secrets.FRED_API_KEY }}
run: |
FILTER="${TEST_FILTER_INPUT:-$MATRIX_CASE_STUDY}"
python -m pytest tests/test_case_studies.py \
-v --tb=short \
-k "$FILTER"
# ---------------------------------------------------------------------------
# Py312 Docker image (Python 3.12: gensim, signatory, esig, pfhedge,
# tfcausalimpact, plus notebooks affected by the Py3.14 torch CUDA bug)
# ---------------------------------------------------------------------------
test-py312:
timeout-minutes: 45
needs: [lint, changes]
if: needs.changes.outputs.run_py312 == 'true'
runs-on: ubuntu-latest
container:
image: ml4t/ml4t-py312:latest
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git git-lfs openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Run py312 notebooks
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pytest tests/test_docker_notebooks.py \
-v --tb=short \
-k "03_sigcwgan or 06_path_signatures or 12_wasserstein \
or 01_word2vec or 02_asset_embeddings or 03_sentiment_evolution \
or 01_timegan or 07_dp_gan \
or 10_shap_nlp_sentiment or 06_conditional_autoencoder \
or 06_fed_announcement_bsts \
or deep_hedging_pfhedge"
# ---------------------------------------------------------------------------
# Neo4j Docker service (Ch23 Knowledge Graphs)
# ---------------------------------------------------------------------------
test-neo4j:
timeout-minutes: 30
needs: [lint, changes]
if: needs.changes.outputs.run_neo4j == 'true'
runs-on: ubuntu-latest
container:
image: ml4t/ml4t:latest
env:
NEO4J_URI: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: password
services:
neo4j:
image: neo4j:2026.02.2
env:
NEO4J_AUTH: neo4j/password
ports:
- 7474:7474
- 7687:7687
options: >-
--health-cmd "cypher-shell -u neo4j -p password 'RETURN 1;'"
--health-interval 10s
--health-timeout 10s
--health-retries 10
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git git-lfs openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Run Ch23 Knowledge Graph notebooks (pipeline order)
env:
PYTHONPATH: ${{ github.workspace }}
run: |
# Run in dependency order: upstream notebooks generate data for downstream
python -m pytest tests/test_docker_notebooks.py -v --tb=short \
-k "08_8k_event_extraction" && \
python -m pytest tests/test_docker_notebooks.py -v --tb=short \
-k "02_supply_chain_kg_construction" && \
python -m pytest tests/test_docker_notebooks.py -v --tb=short \
-k "07_dynamic_kg_temporal or 03_graph_rag_qa or 05_institutional_holdings_kg \
or 09_knowledge_graph_features"
# ---------------------------------------------------------------------------
# Benchmark Docker image (Ch02 storage benchmarks + database services)
# ---------------------------------------------------------------------------
test-benchmark:
timeout-minutes: 30
needs: [lint, changes]
if: needs.changes.outputs.run_benchmark == 'true'
runs-on: ubuntu-latest
container:
image: ml4t/ml4t-benchmark:latest
env:
CLICKHOUSE_HOST: clickhouse
CLICKHOUSE_PORT: "8123"
QUESTDB_HOST: questdb
QUESTDB_HTTP_PORT: "9000"
TIMESCALE_HOST: timescaledb
TIMESCALE_PORT: "5432"
TIMESCALE_PASSWORD: benchmark
INFLUXDB_HOST: influxdb
INFLUXDB_PORT: "8086"
INFLUXDB_ORG: ml4t
INFLUXDB_TOKEN: benchmark-token
INFLUXDB_BUCKET: market_data
services:
timescaledb:
image: timescale/timescaledb:latest-pg16
env:
POSTGRES_PASSWORD: benchmark
POSTGRES_DB: ml4t
POSTGRES_USER: postgres
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 10s
--health-timeout 5s
--health-retries 5
clickhouse:
image: clickhouse/clickhouse-server:latest
env:
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1"
CLICKHOUSE_USER: default
CLICKHOUSE_PASSWORD: ""
options: >-
--health-cmd "wget --spider -q http://localhost:8123/ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
questdb:
image: questdb/questdb:latest
env:
QDB_METRICS_ENABLED: "true"
options: >-
--health-cmd "curl -sf http://localhost:9003/status || curl -sf http://localhost:9000/exec?query=SELECT%201"
--health-interval 10s
--health-timeout 10s
--health-retries 10
influxdb:
image: influxdb:2.7
env:
DOCKER_INFLUXDB_INIT_MODE: setup
DOCKER_INFLUXDB_INIT_USERNAME: admin
DOCKER_INFLUXDB_INIT_PASSWORD: benchmark123
DOCKER_INFLUXDB_INIT_ORG: ml4t
DOCKER_INFLUXDB_INIT_BUCKET: market_data
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: benchmark-token
options: >-
--health-cmd "curl -f http://localhost:8086/health"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Install git
run: |
apt-get update -qq && apt-get install -y -qq git git-lfs openssh-client >/dev/null 2>&1 || true
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Run Ch02 database benchmark notebooks
env:
PYTHONPATH: ${{ github.workspace }}
run: |
python -m pytest tests/test_docker_notebooks.py \
-v --tb=short \
-k "18_storage_benchmark_database or 20_storage_benchmark_database"
.github/workflows/weekly-external.yml
+296
View File
@@ -0,0 +1,296 @@
# ML4T 3rd Edition — Weekly External Workflow (Tier 2)
#
# Runs notebooks tagged `tier: weekly` in tests/overrides.yaml — these have
# external dependencies (IB Gateway, OKX, chromadb, real LLM APIs, real HTTP)
# that aren't available in the per-commit Tier 1 workflow.
#
# Routing: ML4T_TEST_TIER=weekly env var → conftest tier-skip allows weekly NBs
# to run (per-commit run skips them automatically with the inverse rule).
#
# Today most still skip via existing `skip: true` because the underlying
# infrastructure (IB Gateway, OKX, etc.) isn't provisioned yet. The job's
# purpose is to exercise the routing/flake-retry/issue-opening plumbing so
# that when fixtures land (Steps 4-9 of the test-infra coverage plan) the
# weekly job auto-picks them up with no workflow change.
#
# Failure handling: any failing NB opens a `weekly-flake` issue (idempotent
# by NB id in title); pytest-rerunfailures retries flaky NBs per-NB based on
# `reruns: N` in overrides.yaml.
name: Weekly External
on:
schedule:
- cron: '0 6 * * 1' # Monday 06:00 UTC
workflow_dispatch:
inputs:
test_filter:
description: "Optional pytest -k filter to narrow scope"
type: string
default: ""
env:
ML4T_TEST_TIER: weekly
ML4T_DATA_PATH: ${{ github.workspace }}/test-data/data
ML4T_OUTPUT_DIR: /tmp/ml4t-test-output
MPLBACKEND: Agg
PLOTLY_RENDERER: json
jobs:
weekly-chapters:
name: weekly-chapters
runs-on: ubuntu-latest
timeout-minutes: 60
container:
image: ml4t/ml4t:latest
permissions:
contents: read
issues: write
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git git-lfs openssh-client >/dev/null
- uses: actions/checkout@v6
- name: Checkout test data
uses: actions/checkout@v6
with:
repository: ml4t/third-edition-test-data
path: test-data
ssh-key: ${{ secrets.TEST_DATA_DEPLOY_KEY }}
lfs: false
- name: Seed case study intermediates
run: |
mkdir -p $ML4T_OUTPUT_DIR
if [ -d test-data/intermediates ]; then
cp -r test-data/intermediates/* $ML4T_OUTPUT_DIR/
fi
- name: Run weekly chapter notebooks
id: pytest_chapters
env:
PYTHONPATH: ${{ github.workspace }}
TEST_FILTER_INPUT: ${{ github.event.inputs.test_filter }}
# NO-SPEND POLICY: only FREE providers are wired here. Billed APIs
# (OpenAI, OKX authenticated, Databento) are intentionally NOT passed,
# so any notebook needing them skips rather than incurs a charge.
FRED_API_KEY: ${{ secrets.FRED_API_KEY }}
EDGAR_IDENTITY: ${{ secrets.EDGAR_IDENTITY }}
run: |
set +e
# Build -k as an array so a multi-word filter (e.g. "foo or bar")
# stays a single pytest argument instead of being word-split.
FILTER_ARGS=()
if [ -n "${TEST_FILTER_INPUT:-}" ]; then
FILTER_ARGS=(-k "${TEST_FILTER_INPUT}")
fi
# Public repo: chapter notebooks only. Case-study pipeline tests need
# gated results/registries that aren't shipped, so they're excluded.
python -m pytest tests/test_chapter_notebooks.py \
-v --tb=short -m weekly "${FILTER_ARGS[@]}" \
--junitxml=junit-weekly.xml
echo "exit_code=$?" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Upload junit report
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-weekly
path: junit-weekly.xml
retention-days: 30
- name: Open issues for failed notebooks
if: always() && steps.pytest_chapters.outputs.exit_code != '0'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const { execSync } = require('child_process');
let xml;
try {
xml = fs.readFileSync('junit-weekly.xml', 'utf8');
} catch (e) {
core.info('No junit-weekly.xml found; nothing to file.');
return;
}
// Crude but dependency-free junit parse: pull <testcase ...> blocks
// and check each for a child <failure> or <error>.
const testcaseRe = /<testcase\b[^>]*>([\s\S]*?)<\/testcase>|<testcase\b[^>]*\/>/g;
const attrRe = (name) => new RegExp(`${name}="([^"]*)"`);
const failures = [];
let m;
while ((m = testcaseRe.exec(xml)) !== null) {
const block = m[0];
const body = m[1] || '';
if (!/<failure\b|<error\b/.test(body)) continue;
const nameMatch = block.match(attrRe('name'));
const classMatch = block.match(attrRe('classname'));
if (nameMatch) {
failures.push({
name: nameMatch[1],
classname: classMatch ? classMatch[1] : '',
});
}
}
core.info(`Found ${failures.length} failing testcases.`);
const owner = context.repo.owner;
const repo = context.repo.repo;
const runUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`;
for (const f of failures) {
const nbId = f.name.replace(/^test_[a-z_]+\[/, '').replace(/\]$/, '');
const title = `weekly-flake: ${nbId}`;
// Search for existing open issue with same title to dedupe.
const q = `repo:${owner}/${repo} is:issue is:open in:title "weekly-flake: ${nbId}"`;
const existing = await github.rest.search.issuesAndPullRequests({ q });
if (existing.data.total_count > 0) {
const num = existing.data.items[0].number;
core.info(`Existing issue #${num} for ${nbId}; commenting.`);
await github.rest.issues.createComment({
owner, repo, issue_number: num,
body: `Failed again in weekly run: ${runUrl}`,
});
} else {
core.info(`Opening new issue for ${nbId}.`);
await github.rest.issues.create({
owner, repo, title,
body: `Weekly external workflow failed for \`${nbId}\`.\n\n` +
`Run: ${runUrl}\n\n` +
`Test class: \`${f.classname}\`\n\n` +
`Dedupes by NB id — comments will append on subsequent failures.`,
labels: ['weekly-flake'],
});
}
}
- name: Fail job if pytest failed
if: steps.pytest_chapters.outputs.exit_code != '0'
run: |
echo "Weekly pytest exited with code ${{ steps.pytest_chapters.outputs.exit_code }}"
exit 1
# ---------------------------------------------------------------------------
# Phase 3 — live external drift smoke.
#
# One tiny REAL pull per free data source (tests/test_external_drift.py, marked
# `drift`). Catches upstream provider/schema drift: when a source moves,
# renames a file, or changes its payload shape, the matching test fails and we
# file a deduped issue. No billed APIs — key-gated sources (FRED, OANDA) skip
# when their free key is absent; geo-restricted sources (Kalshi/Polymarket)
# skip when blocked. Flake-tolerant via pytest-rerunfailures when available.
# ---------------------------------------------------------------------------
weekly-drift:
name: weekly-drift
runs-on: ubuntu-latest
timeout-minutes: 20
container:
image: ml4t/ml4t:latest
permissions:
contents: read
issues: write
steps:
- name: Install git
run: apt-get update -qq && apt-get install -y -qq git >/dev/null
- uses: actions/checkout@v6
- name: Run external drift smoke
id: pytest_drift
env:
PYTHONPATH: ${{ github.workspace }}
# NO-SPEND POLICY: only FREE keys. Billed APIs stay unset → their
# drift tests skip rather than charge.
FRED_API_KEY: ${{ secrets.FRED_API_KEY }}
EDGAR_IDENTITY: ${{ secrets.EDGAR_IDENTITY }}
run: |
set +e
# utils.config requires a .env file to exist; create a minimal one and
# point the data path at the repo's own data/ dir (no test-data here).
printf 'ML4T_PATH=%s\nML4T_DATA_PATH=%s/data\n' \
"$GITHUB_WORKSPACE" "$GITHUB_WORKSPACE" > .env
# Flake tolerance only if pytest-rerunfailures is installed in the image.
RERUN_ARGS=""
if python -c "import pytest_rerunfailures" 2>/dev/null; then
RERUN_ARGS="--reruns 2 --reruns-delay 30"
fi
python -m pytest tests/test_external_drift.py -m drift \
-v --tb=short $RERUN_ARGS \
--junitxml=junit-drift.xml
echo "exit_code=$?" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Upload junit report
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-drift
path: junit-drift.xml
retention-days: 30
- name: Open issues for drifted sources
if: always() && steps.pytest_drift.outputs.exit_code != '0'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
let xml;
try {
xml = fs.readFileSync('junit-drift.xml', 'utf8');
} catch (e) {
core.info('No junit-drift.xml found; nothing to file.');
return;
}
// Dependency-free junit parse: <testcase> blocks with a child
// <failure>/<error> are real drift (skips are not failures).
const testcaseRe = /<testcase\b[^>]*>([\s\S]*?)<\/testcase>|<testcase\b[^>]*\/>/g;
const attrRe = (name) => new RegExp(`${name}="([^"]*)"`);
const failures = [];
let m;
while ((m = testcaseRe.exec(xml)) !== null) {
const block = m[0];
const body = m[1] || '';
if (!/<failure\b|<error\b/.test(body)) continue;
const nameMatch = block.match(attrRe('name'));
if (nameMatch) failures.push(nameMatch[1]);
}
core.info(`Found ${failures.length} drifted source(s).`);
const owner = context.repo.owner;
const repo = context.repo.repo;
const runUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`;
for (const name of failures) {
// test_etf_yahoo_reachable -> etf_yahoo
const source = name.replace(/^test_/, '').replace(/_reachable$/, '');
const title = `external-drift: ${source}`;
const q = `repo:${owner}/${repo} is:issue is:open in:title "external-drift: ${source}"`;
const existing = await github.rest.search.issuesAndPullRequests({ q });
if (existing.data.total_count > 0) {
const num = existing.data.items[0].number;
await github.rest.issues.createComment({
owner, repo, issue_number: num,
body: `Drifted again in the weekly run: ${runUrl}`,
});
} else {
await github.rest.issues.create({
owner, repo, title,
body: `The weekly external drift smoke failed for \`${name}\`.\n\n` +
`An upstream free data source appears to have moved, renamed a ` +
`file, or changed its payload schema.\n\n` +
`Run: ${runUrl}\n\n` +
`Dedupes by source — comments append on subsequent failures.`,
labels: ['external-drift', 'weekly-flake'],
});
}
}
- name: Fail job if drift smoke failed
if: steps.pytest_drift.outputs.exit_code != '0'
run: |
echo "Drift smoke exited with code ${{ steps.pytest_drift.outputs.exit_code }}"
exit 1
.gitignore
+203
View File
@@ -0,0 +1,203 @@
# ML4T Technical Review - .gitignore
# Environment and secrets
.env
.env.local
.env.*.local
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
*.egg
*.egg-info/
dist/
build/
eggs/
.eggs/
*.manifest
*.spec
# Virtual environments
.venv/
venv/
ENV/
# Package manager
pip-log.txt
# Claude Code
.claude/
.mcp.json
# IDE and editors
.idea/
.vscode/
*.swp
*.swo
*~
# Jupyter
.ipynb_checkpoints/
*.ipynb_checkpoints/
# NOTE: .ipynb files ARE committed (for reviewer convenience)
# Both .py (Jupytext source) and .ipynb (generated) are tracked
# Testing
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
tests/_internal/notebook_catalog.db
# Linter caches
.ruff_cache/
# OS files
.DS_Store
Thumbs.db
# Data files (downloaded, not tracked)
# Download using: uv run python data/download_all.py
data/**/*.parquet
data/**/*.csv
data/**/*.h5
data/**/*.hdf5
data/**/*.gz
data/**/*.pkl
data/**/*.json
data/**/*.bin
data/**/*.pcap
data/**/*.npy
data/**/*.txt
data/**/*.xlsx
data/**/*.pdf
data/**/*.parquet.gzip
data/**/*.npz
data/**/*.meta
data/**/*.index
data/**/*.data-*
data/**/checkpoint
data/**/__MACOSX/
data/**/*.xml
data/**/*.zip
# Caches
data/**/__pycache__/
data/**/.cache/
# Chapter outputs (generated by notebooks)
*/output/
!*/output/.gitkeep
# Root-level output/ — notebooks run from repo root write chapter-relative
# output/ paths here; regenerable, never committed.
/output/
# Executed-state builder logs (scripts/build_executed_state.py)
scripts/executed_state_logs/
# Ch23 supply chain extraction cache (11KB, saves 25-min GPU re-run)
# Whitelist the parent path, then re-exclude all siblings of the cache pair below.
!23_knowledge_graphs/output/
23_knowledge_graphs/output/*
!23_knowledge_graphs/output/supply_chain_cache/
23_knowledge_graphs/output/supply_chain_cache/*
!23_knowledge_graphs/output/supply_chain_cache/extracted_triples.parquet
!23_knowledge_graphs/output/supply_chain_cache/extracted_triples.meta.json
# Figure outputs (regenerated by notebooks; publication figures live in book repo)
*/figures/
!*/figures/.gitkeep
case_studies/*/figures/
# Local notebook/model run artifacts
.deploy_source.json
.ml4t_risk_state.json
catalog.sqlite
catboost_info/
mlruns/
mlflow.db
lib/
tests/notebook_catalog.db
11_ml_pipeline/models/
# Test-status DB is the SSOT in the agents repo, not here — a stray run
# from the wrong cwd can drop an empty one at this repo root. Never track it.
test_status.db
# Pipeline artifacts (case study outputs — generated, not tracked)
case_studies/*/features/
case_studies/*/labels/
case_studies/*/models/
case_studies/*/synthesis/
case_studies/*/backtest/
case_studies/*/strategy/
case_studies/*/evaluation/
case_studies/*/exploration/
case_studies/*/config/exploration/
case_studies/*/config/cv_config.json
case_studies/*/eligibility.csv
case_studies/*/registry.db
case_studies/*/results/
case_studies/*/*_v1_contaminated/
# Unified registry and run artifacts — all gitignored
# registry.db is SSOT but too large for git (43-180 MB per case study)
# Regenerated from pipeline runs; not needed for readers
case_studies/*/run_log/
# Model checkpoints (from training notebooks)
**/trainer_great/
**/trainer_output/
**/checkpoint-*/
# Temporary files
*.tmp
*.temp
*.bak
*.log
/tmp/
# Executed notebooks (test artifacts)
_executed_*.ipynb
# Misc
FinFutYY.txt
f_year.txt
.test-output/
test_results/
# Notebook catalogs live in book repo
*/catalog/*.json
case_studies/*/catalog/*.json
# Chapter 25 deployment loop artefacts (regenerated by 02_etfs_deployment_loop)
25_live_trading/live_artifacts/
# Chapter 24 research-operator captured traces. The directory is ignored (live
# re-runs write throwaway traces here), EXCEPT the two canonical pinned traces
# 11_research_operator replays by default — those are committed (host home path
# normalized to ~) so the chapter reproduces deterministically on any clone.
24_autonomous_agents/operator_artifacts/*
!24_autonomous_agents/operator_artifacts/run_etfs_20260504T223150.json
!24_autonomous_agents/operator_artifacts/run_us_firm_characteristics_20260504T225521.json
# Chapter 24 forecasting run traces. The directory is ignored (live re-runs and
# CI smoke-tests write throwaway traces here), EXCEPT the canonical pinned traces
# the notebooks replay by default — those are committed (sanitized) so the
# chapter reproduces deterministically on any clone. See each notebook's
# PINNED_TRACE / PINNED_TRACES parameter.
24_autonomous_agents/forecast_traces/*
!24_autonomous_agents/forecast_traces/04_research_agent_20260609T141730Z_b694ab4d0453.json
!24_autonomous_agents/forecast_traces/06_multi_agent_research_20260609T141413Z_9fc4a2655471.json
!24_autonomous_agents/forecast_traces/07_adversarial_debate_20260609T141631Z_cf1ad76cf379.json
!24_autonomous_agents/forecast_traces/08_forecasting_pipeline_20260609T141954Z_ef5bca7b95bd.json
!24_autonomous_agents/forecast_traces/08_forecasting_pipeline_20260609T142158Z_24e083e7fe54.json
!24_autonomous_agents/forecast_traces/10_framework_comparison_20260609T150909Z_ba1855847f4c.json
!24_autonomous_agents/forecast_traces/10_framework_comparison_20260609T151202Z_c8cba3007562.json
!24_autonomous_agents/forecast_traces/01_react_reasoning_20260615T191047Z_04eb6e7c603d.json
!24_autonomous_agents/forecast_traces/09_evaluation_and_governance_20260615T191431Z_d5030378899c.json
# Review tooling config (dev-only)
.roborev.toml
.pre-commit-config.yaml
+50
View File
@@ -0,0 +1,50 @@
# Pre-commit hooks for ML4T Technical Review
# Install: pre-commit install
# Run manually: pre-commit run --all-files
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.8
hooks:
- id: ruff
args: [--fix]
types: [python]
exclude: \.ipynb$
- id: ruff-format
types: [python]
exclude: \.ipynb$
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
exclude: \.md$
- id: end-of-file-fixer
exclude: \.md$
- id: check-yaml
- id: check-toml
- repo: local
hooks:
# Compile-time syntax check. ast.parse misses errors raised only at
# compile time (duplicate kwargs, await outside async, return at module
# level, etc.) — `python -m py_compile` invokes the full compile()
# path and catches them. Cheap; runs on every staged Python file.
- id: py-compile
name: python -m py_compile (catches duplicate kwargs etc.)
language: system
entry: python3 -m py_compile
types: [python]
exclude: \.ipynb$
# Sync gate: a stamped notebook must be its CURRENT .py executed in a
# production run. Fails if any provenance-stamped .ipynb is stale (its
# paired .py changed since execution) or was committed from a TEST-mode
# run. Unstamped notebooks are advisory only (gradual adoption). See
# scripts/notebook_provenance.py.
- id: notebook-sync
name: notebook provenance sync gate
language: system
entry: python3 scripts/notebook_provenance.py check
pass_filenames: false
always_run: true
01_process_is_edge/README.md
+71
View File
@@ -0,0 +1,71 @@
# Chapter 1: The Process Is Your Edge
The chapter establishes the chapter's central claim: in trading, durable performance depends less on picking a sophisticated model than on maintaining a disciplined research process that can survive changing markets, noisy signals, and real-world frictions. It gives readers a usable vocabulary for market change, shows why recent shocks exposed fragile assumptions, and reframes ML for trading as an adaptation problem rather than a model-selection contest.
## Learning Objectives
* Distinguish structural breaks, regimes, data drift, concept drift, and online detection, and explain why static trading models degrade in changing markets
* Explain the ML4T Workflow as a research-to-production system, including its data infrastructure foundation, scoping invariants, iterative research modules, and feedback loops from live trading back to research
* Define the evidence boundary between exploration and confirmation, and explain how trial logging, sealed holdouts, and selection-aware evaluation preserve research integrity
* Describe how causal inference and generative AI fit within a disciplined trading workflow, including the main benefits they provide and the new failure modes they introduce
* Apply regime thinking, implementability checks, and monitoring logic to diagnose strategy vulnerabilities and to adapt workflow discipline across independent and institutional settings
## Sections
### 1.1 Why Process Discipline Matters
This section establishes the chapter's central claim: in trading, durable performance depends less on picking a sophisticated model than on maintaining a disciplined research process that can survive changing markets, noisy signals, and real-world frictions. It gives readers a usable vocabulary for market change, shows why recent shocks exposed fragile assumptions, and reframes ML for trading as an adaptation problem rather than a model-selection contest.
### 1.2 Introducing the ML4T Workflow
This section presents the book's core framework: a research-to-production workflow built on point-in-time-correct data infrastructure, explicit scoping rules, iterative feature and model development, realistic strategy design, deployment discipline, and ongoing monitoring. The key value for readers is that it turns trading research into a managed lifecycle with auditable artifacts, clear handoffs, and an explicit boundary between exploration and confirmation.
### 1.3 Causal Inference and Generative AI in the Workflow
This section places two modern method families inside the workflow rather than treating them as standalone trends. Causal inference is framed as a way to sharpen mechanisms, assumptions, and diagnosis; generative AI is framed as a way to expand research and unstructured-data processing while also creating new risks such as leakage, hallucination, and workflow bloat. Readers should care because the section makes clear that new tools increase the value of discipline rather than replacing it.
### 1.4 Market Regimes: Change Is the Constant
This section turns non-stationarity into something operational. It shows how regime concepts can support explanation, robustness checks, and live monitoring, while insisting that regimes are primarily a risk lens rather than a reliable timing signal. The factor and macro examples make the idea concrete: regime methods are useful when they help identify adverse environments and connect them to predefined risk actions.
- [`factor_regimes`](factor_regimes.ipynb) — Demonstrates unsupervised learning for market regime detection using Gaussian Mixture Models (GMM) on factor returns from the AQR Century of Factor Premia dataset.
- [`macro_regimes`](macro_regimes.ipynb) — Demonstrates unsupervised learning for market regime detection using macroeconomic indicators from FRED, validated against S&P 500 volatility and drawdowns.
### 1.5 In the Real World: Independent vs. Institutional
This section translates the workflow into real operating contexts. It explains how institutions benefit from built-in friction and review, while independent researchers must create their own governance through documentation, checkpoints, and explicit stop criteria. The practical payoff is strong: it helps readers see where solo practitioners are vulnerable, where they can still compete, and how reusable infrastructure compounds research quality over time.
## Running the Notebooks
```bash
# From the repository root
uv run python 01_process_is_edge/<notebook>.py
# Test mode (reduced data via Papermill)
uv run pytest tests/test_notebooks.py -v -k "01_process_is_edge"
```
## References
- **Andrew Ang and Geert Bekaert** (2002). [International Asset Allocation With Regime Shifts](https://doi.org/10.1093/rfs/15.4.1137). *Review of Financial Studies*.
- **Robert D. Arnott et al.** (2018). [A Backtesting Protocol in the Era of Machine Learning](https://doi.org/10.2139/ssrn.3275654).
- **Darrell Duffie** (2020). [Still the World's Safe Haven? Redesigning the U.S. Treasury Market After the COVID-19 Crisis](https://www.brookings.edu/wp-content/uploads/2020/05/WP62_Duffie_v2.pdf).
- **David Easley et al.** (2012). [The Volume Clock: Insights into the High Frequency Paradigm](https://doi.org/10.2139/ssrn.2034858).
- **Frank J. Fabozzi et al.** (2024). [Paradigm Shift: Embracing Holism in Causal Modeling for Investment Applications](https://doi.org/10.3905/jpm.2024.51.1.159). *The Journal of Portfolio Management*.
- **Frank J. Fabozzi and Caleb C. Stenholm** (2025). [Strategic Discipline: How Asset Management Mirrors Military Operations](https://doi.org/10.3905/jpm.2025.1.769). *The Journal of Portfolio Management*.
- **Ziang Fang and Jason Moore** (2025). What AI Can (and Can't Yet) Do for Alpha.
- **Stefano Giglio et al.** (2022). [Factor Models, Machine Learning, and Asset Pricing](https://doi.org/10.1146/annurev-financial-101521-104735). *Annual Review of Financial Economics*.
- **Campbell R. Harvey et al.** (2016). [...and the Cross-Section of Expected Returns](https://doi.org/10.1093/rfs/hhv059). *Review of Financial Studies*.
- **Blanka Horvath et al.** (2021). [Clustering Market Regimes Using the Wasserstein Distance](https://doi.org/10.2139/ssrn.3947905).
- **Antti Ilmanen et al.** (2021). [How Do Factor Premia Vary Over Time? A Century of Evidence](https://doi.org/10.2139/ssrn.3400998).
- **Justina Lee** (2025). [Man Group Says Agentic AI Is Now Devising Quant Trading Signals](https://www.bloomberg.com/news/articles/2025-07-10/man-group-says-agentic-ai-is-now-devising-quant-trading-signals). *Bloomberg.com*.
- **Andrew W. Lo** (2004). [The Adaptive Markets Hypothesis: Market Efficiency from an Evolutionary Perspective](https://papers.ssrn.com/abstract=602222).
- **Martin Luk** (2023). [Generative AI: Overview, Economic Impact, and Applications in Asset Management](https://doi.org/10.2139/ssrn.4574814).
- **Judea Pearl** (2019). [The seven tools of causal inference, with reflections on machine learning](https://doi.org/10.1145/3241036). *Communications of the ACM*.
- **Marcos López de Prado** (2018). The 10 Reasons Most Machine Learning Funds Fail. *The Journal of Portfolio Management*.
- **Marcos Lopez de Prado et al.** (2024). [The Case for Causal Factor Investing](https://doi.org/10.2139/ssrn.4774522).
- **Marcos López de Prado and Vincent Zoonekynd** (2025). [Correcting the Factor Mirage: A Research Protocol for Causal Factor Investing](https://doi.org/10.3905/jpm.2025.1.794). *The Journal of Portfolio Management*.
- **James Ryseff et al.** (2024). [The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Avoiding the Anti-Patterns of AI](https://www.rand.org/pubs/research_reports/RRA2680-1.html).
- **Bernhard Schölkopf et al.** (2021). [Towards Causal Representation Learning](https://doi.org/10.48550/arXiv.2102.11107).
- **Stefan Studer et al.** (2021). [Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology](https://doi.org/10.3390/make3020020). *Machine Learning and Knowledge Extraction*.
- **A. Sinem Uysal and John M. Mulvey** (2021). [A Machine Learning Approach in Regime-Switching Risk Parity Portfolios](https://doi.org/10.3905/jfds.2021.1.057). *The Journal of Financial Data Science*.
01_process_is_edge/factor_regimes.ipynb
+2386
View File
File diff suppressed because one or more lines are too long
01_process_is_edge/factor_regimes.py
+1037
View File
File diff suppressed because it is too large Load Diff
01_process_is_edge/macro_regimes.ipynb
+2647
View File
File diff suppressed because one or more lines are too long
01_process_is_edge/macro_regimes.py
+1012
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/01_us_equities_eda.ipynb
+789
View File
@@ -0,0 +1,789 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "5ad079dc",
"metadata": {
"papermill": {
"duration": 0.005414,
"end_time": "2026-06-13T02:32:24.544112+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:24.538698+00:00",
"status": "completed"
}
},
"source": [
"# US Equities — Exploratory Data Analysis\n",
"\n",
"**Docker image**: `ml4t`\n",
"\n",
"**Purpose**: Profile the Wiki Prices dataset of US equity OHLCV history and confirm\n",
"the inactive-symbol coverage that makes it usable for survivorship-bias-free\n",
"backtests.\n",
"\n",
"**Learning objectives**:\n",
"\n",
"- Load the equity panel via `data.load_us_equities` and inspect its canonical schema.\n",
"- Distinguish raw and split/dividend-adjusted price columns.\n",
"- Quantify the share of symbols that stop trading before the dataset end date.\n",
"- Check OHLC invariants and null rates across the full panel.\n",
"\n",
"**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — Equities).\n",
"\n",
"**Prerequisites**: `data` package on `PYTHONPATH`; Wiki Prices parquet present at\n",
"`ML4T_DATA_PATH/equities/market/us_equities/`. Run\n",
"`python data/equities/market/us_equities/download.py` if missing."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "fcba0234",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:24.551754Z",
"iopub.status.busy": "2026-06-13T02:32:24.551580Z",
"iopub.status.idle": "2026-06-13T02:32:25.784522Z",
"shell.execute_reply": "2026-06-13T02:32:25.784148Z"
},
"papermill": {
"duration": 1.237163,
"end_time": "2026-06-13T02:32:25.785024+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:24.547861+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"\"\"\"US Equities — Exploratory data analysis of the Wiki Prices dataset.\"\"\"\n",
"\n",
"import polars as pl\n",
"\n",
"from data import load_us_equities\n",
"from utils.data_quality import check_ohlc_invariants"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e2e6b56b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:25.788403Z",
"iopub.status.busy": "2026-06-13T02:32:25.788307Z",
"iopub.status.idle": "2026-06-13T02:32:25.789984Z",
"shell.execute_reply": "2026-06-13T02:32:25.789599Z"
},
"papermill": {
"duration": 0.004284,
"end_time": "2026-06-13T02:32:25.790350+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:25.786066+00:00",
"status": "completed"
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"# Production defaults — Papermill injects overrides for CI\n",
"MAX_SYMBOLS = 0 # 0 = all"
]
},
{
"cell_type": "markdown",
"id": "a05e8378",
"metadata": {
"papermill": {
"duration": 0.000762,
"end_time": "2026-06-13T02:32:25.791966+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:25.791204+00:00",
"status": "completed"
}
},
"source": [
"## 1. Load and Inspect"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "18d4c304",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:25.793964Z",
"iopub.status.busy": "2026-06-13T02:32:25.793895Z",
"iopub.status.idle": "2026-06-13T02:32:26.253321Z",
"shell.execute_reply": "2026-06-13T02:32:26.252871Z"
},
"papermill": {
"duration": 0.460993,
"end_time": "2026-06-13T02:32:26.253751+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:25.792758+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Wiki Prices Dataset ===\n",
"Shape: (15389314, 14)\n",
"Columns: ['symbol', 'timestamp', 'open', 'high', 'low', 'close', 'volume', 'ex-dividend', 'split_ratio', 'adj_open', 'adj_high', 'adj_low', 'adj_close', 'adj_volume']\n"
]
}
],
"source": [
"wiki = load_us_equities()\n",
"\n",
"print(\"=== Wiki Prices Dataset ===\")\n",
"print(f\"Shape: {wiki.shape}\")\n",
"print(f\"Columns: {wiki.columns}\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "f3ce7b2d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.256226Z",
"iopub.status.busy": "2026-06-13T02:32:26.256110Z",
"iopub.status.idle": "2026-06-13T02:32:26.257967Z",
"shell.execute_reply": "2026-06-13T02:32:26.257731Z"
},
"papermill": {
"duration": 0.003422,
"end_time": "2026-06-13T02:32:26.258212+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.254790+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Schema:\n",
" symbol: String\n",
" timestamp: Date\n",
" open: Float64\n",
" high: Float64\n",
" low: Float64\n",
" close: Float64\n",
" volume: Float64\n",
" ex-dividend: Float64\n",
" split_ratio: Float64\n",
" adj_open: Float64\n",
" adj_high: Float64\n",
" adj_low: Float64\n",
" adj_close: Float64\n",
" adj_volume: Float64\n"
]
}
],
"source": [
"# Schema overview\n",
"print(\"\\nSchema:\")\n",
"for col, dtype in wiki.schema.items():\n",
" print(f\" {col}: {dtype}\")"
]
},
{
"cell_type": "markdown",
"id": "7cf8fb2a",
"metadata": {
"papermill": {
"duration": 0.000835,
"end_time": "2026-06-13T02:32:26.259980+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.259145+00:00",
"status": "completed"
}
},
"source": [
"### Adjusted vs Raw Prices\n",
"\n",
"**Important**: This dataset contains both raw and adjusted prices.\n",
"\n",
"| Column Type | Examples | Use Case |\n",
"|-------------|----------|----------|\n",
"| Raw | `open`, `high`, `low`, `close`, `volume` | Historical analysis at actual prices |\n",
"| Adjusted | `adj_open`, `adj_high`, `adj_low`, `adj_close`, `adj_volume` | **Backtesting** (handles splits/dividends) |\n",
"\n",
"Always use `adj_*` columns for return calculations and strategy backtesting."
]
},
{
"cell_type": "markdown",
"id": "6d550e3b",
"metadata": {
"papermill": {
"duration": 0.000832,
"end_time": "2026-06-13T02:32:26.261676+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.260844+00:00",
"status": "completed"
}
},
"source": [
"## 2. Coverage Summary"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "3007d3af",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.264176Z",
"iopub.status.busy": "2026-06-13T02:32:26.264095Z",
"iopub.status.idle": "2026-06-13T02:32:26.348271Z",
"shell.execute_reply": "2026-06-13T02:32:26.347931Z"
},
"papermill": {
"duration": 0.086007,
"end_time": "2026-06-13T02:32:26.348675+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.262668+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Coverage ===\n",
"Unique tickers: 3,199\n",
"Date range: 1962-01-02 to 2018-03-27\n",
"Total rows: 15,389,314\n"
]
}
],
"source": [
"# Overall coverage\n",
"print(\"=== Coverage ===\")\n",
"print(f\"Unique tickers: {wiki['symbol'].n_unique():,}\")\n",
"print(f\"Date range: {wiki['timestamp'].min()} to {wiki['timestamp'].max()}\")\n",
"print(f\"Total rows: {len(wiki):,}\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "07f7febc",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.351317Z",
"iopub.status.busy": "2026-06-13T02:32:26.351234Z",
"iopub.status.idle": "2026-06-13T02:32:26.723550Z",
"shell.execute_reply": "2026-06-13T02:32:26.723172Z"
},
"papermill": {
"duration": 0.374552,
"end_time": "2026-06-13T02:32:26.724359+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.349807+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"# Per-stock statistics\n",
"stock_stats = wiki.group_by(\"symbol\").agg(\n",
" [\n",
" pl.len().alias(\"days\"),\n",
" pl.col(\"timestamp\").min().alias(\"start\"),\n",
" pl.col(\"timestamp\").max().alias(\"end\"),\n",
" pl.col(\"adj_close\").mean().alias(\"avg_price\"),\n",
" ]\n",
")"
]
},
{
"cell_type": "markdown",
"id": "bc6a8d8e",
"metadata": {
"papermill": {
"duration": 0.001661,
"end_time": "2026-06-13T02:32:26.728002+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.726341+00:00",
"status": "completed"
}
},
"source": [
"Per-symbol coverage distribution — number of trading days and mean adjusted price."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "eca30494",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.731023Z",
"iopub.status.busy": "2026-06-13T02:32:26.730937Z",
"iopub.status.idle": "2026-06-13T02:32:26.735753Z",
"shell.execute_reply": "2026-06-13T02:32:26.735507Z"
},
"papermill": {
"duration": 0.006674,
"end_time": "2026-06-13T02:32:26.736156+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.729482+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (9, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>statistic</th><th>days</th><th>avg_price</th></tr><tr><td>str</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>&quot;count&quot;</td><td>3199.0</td><td>3199.0</td></tr><tr><td>&quot;null_count&quot;</td><td>0.0</td><td>0.0</td></tr><tr><td>&quot;mean&quot;</td><td>4810.663957</td><td>118.245432</td></tr><tr><td>&quot;std&quot;</td><td>2621.170846</td><td>2893.682465</td></tr><tr><td>&quot;min&quot;</td><td>4.0</td><td>0.669129</td></tr><tr><td>&quot;25%&quot;</td><td>2611.0</td><td>11.539385</td></tr><tr><td>&quot;50%&quot;</td><td>4974.0</td><td>18.519854</td></tr><tr><td>&quot;75%&quot;</td><td>6666.0</td><td>29.777315</td></tr><tr><td>&quot;max&quot;</td><td>14155.0</td><td>132897.234865</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (9, 3)\n",
"┌────────────┬─────────────┬───────────────┐\n",
"│ statistic ┆ days ┆ avg_price │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ str ┆ f64 ┆ f64 │\n",
"╞════════════╪═════════════╪═══════════════╡\n",
"│ count ┆ 3199.0 ┆ 3199.0 │\n",
"│ null_count ┆ 0.0 ┆ 0.0 │\n",
"│ mean ┆ 4810.663957 ┆ 118.245432 │\n",
"│ std ┆ 2621.170846 ┆ 2893.682465 │\n",
"│ min ┆ 4.0 ┆ 0.669129 │\n",
"│ 25% ┆ 2611.0 ┆ 11.539385 │\n",
"│ 50% ┆ 4974.0 ┆ 18.519854 │\n",
"│ 75% ┆ 6666.0 ┆ 29.777315 │\n",
"│ max ┆ 14155.0 ┆ 132897.234865 │\n",
"└────────────┴─────────────┴───────────────┘"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"stock_stats.select([\"days\", \"avg_price\"]).describe()"
]
},
{
"cell_type": "markdown",
"id": "e3726365",
"metadata": {
"papermill": {
"duration": 0.000927,
"end_time": "2026-06-13T02:32:26.738073+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.737146+00:00",
"status": "completed"
}
},
"source": [
"## 3. Survivorship Analysis\n",
"\n",
"A key feature of this dataset is that it includes stocks that ceased trading\n",
"before the dataset end date. This is critical for avoiding survivorship bias.\n",
"\n",
"**Note**: Stocks marked as \"inactive before end\" include both:\n",
"- Actually delisted companies (bankruptcy, acquisition, etc.)\n",
"- Stocks with incomplete coverage in this dataset\n",
"\n",
"The important point: these stocks are included, preventing survivorship bias."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "9dfb3c79",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.740278Z",
"iopub.status.busy": "2026-06-13T02:32:26.740210Z",
"iopub.status.idle": "2026-06-13T02:32:26.747184Z",
"shell.execute_reply": "2026-06-13T02:32:26.746804Z"
},
"papermill": {
"duration": 0.008672,
"end_time": "2026-06-13T02:32:26.747650+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.738978+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Survivorship Analysis ===\n",
"Dataset end: 2018-03-27\n",
"Active at dataset end: 2,422 (75.7%)\n",
"Inactive before end: 777 (24.3%)\n",
"\n",
"This 24% inactive rate helps mitigate survivorship bias in backtests.\n"
]
}
],
"source": [
"dataset_end = wiki.select(pl.col(\"timestamp\").max()).item()\n",
"\n",
"# Identify stocks that stopped trading before dataset end\n",
"stock_stats = stock_stats.with_columns((pl.col(\"end\") < dataset_end).alias(\"inactive_before_end\"))\n",
"\n",
"n_active = stock_stats.filter(~pl.col(\"inactive_before_end\")).height\n",
"n_inactive = stock_stats.filter(pl.col(\"inactive_before_end\")).height\n",
"total = n_active + n_inactive\n",
"inactive_pct = n_inactive / total * 100\n",
"\n",
"print(\"=== Survivorship Analysis ===\")\n",
"print(f\"Dataset end: {dataset_end}\")\n",
"print(f\"Active at dataset end: {n_active:,} ({n_active / total * 100:.1f}%)\")\n",
"print(f\"Inactive before end: {n_inactive:,} ({inactive_pct:.1f}%)\")\n",
"print(f\"\\nThis {inactive_pct:.0f}% inactive rate helps mitigate survivorship bias in backtests.\")"
]
},
{
"cell_type": "markdown",
"id": "333e64d2",
"metadata": {
"papermill": {
"duration": 0.000978,
"end_time": "2026-06-13T02:32:26.749790+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.748812+00:00",
"status": "completed"
}
},
"source": [
"## 4. Data Quality"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "8041dec8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.752249Z",
"iopub.status.busy": "2026-06-13T02:32:26.752172Z",
"iopub.status.idle": "2026-06-13T02:32:26.755615Z",
"shell.execute_reply": "2026-06-13T02:32:26.755147Z"
},
"papermill": {
"duration": 0.00514,
"end_time": "2026-06-13T02:32:26.755903+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.750763+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Data Quality ===\n",
"Total null values: 1,299\n",
" open: 538 (0.0035%)\n",
" high: 55 (0.0004%)\n",
" low: 55 (0.0004%)\n",
" close: 1 (0.0000%)\n",
" split_ratio: 1 (0.0000%)\n",
" adj_open: 538 (0.0035%)\n",
" adj_high: 55 (0.0004%)\n",
" adj_low: 55 (0.0004%)\n",
" adj_close: 1 (0.0000%)\n",
"\n",
"Null rate: 0.0006%\n"
]
}
],
"source": [
"# Check for nulls across columns\n",
"null_counts = wiki.null_count()\n",
"total_nulls = null_counts.sum_horizontal()[0]\n",
"print(\"=== Data Quality ===\")\n",
"print(f\"Total null values: {total_nulls:,}\")\n",
"\n",
"# Show per-column breakdown (only columns with nulls)\n",
"for col in null_counts.columns:\n",
" val = null_counts[col][0]\n",
" if val > 0:\n",
" print(f\" {col}: {val:,} ({val / len(wiki) * 100:.4f}%)\")\n",
"\n",
"print(f\"\\nNull rate: {total_nulls / (len(wiki) * len(wiki.columns)) * 100:.4f}%\")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "ac521b11",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.759001Z",
"iopub.status.busy": "2026-06-13T02:32:26.758834Z",
"iopub.status.idle": "2026-06-13T02:32:26.986021Z",
"shell.execute_reply": "2026-06-13T02:32:26.985683Z"
},
"papermill": {
"duration": 0.229428,
"end_time": "2026-06-13T02:32:26.986481+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.757053+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"OHLC Invariants (adjusted prices):\n",
" [OK] high_gte_low: 100.00%\n",
" [OK] high_gte_open: 100.00%\n",
" [OK] high_gte_close: 100.00%\n",
" [OK] low_lte_open: 100.00%\n",
" [OK] low_lte_close: 100.00%\n",
" [OK] volume_non_negative: 100.00%\n"
]
}
],
"source": [
"# OHLC invariants on adjusted prices\n",
"invariants = check_ohlc_invariants(\n",
" wiki,\n",
" open_col=\"adj_open\",\n",
" high_col=\"adj_high\",\n",
" low_col=\"adj_low\",\n",
" close_col=\"adj_close\",\n",
" volume_col=\"adj_volume\",\n",
")\n",
"\n",
"print(\"\\nOHLC Invariants (adjusted prices):\")\n",
"for row in invariants.iter_rows(named=True):\n",
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")"
]
},
{
"cell_type": "markdown",
"id": "f11b91fe",
"metadata": {
"papermill": {
"duration": 0.001525,
"end_time": "2026-06-13T02:32:26.989389+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.987864+00:00",
"status": "completed"
}
},
"source": [
"## 5. Example: Single Stock"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "911af454",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:26.992105Z",
"iopub.status.busy": "2026-06-13T02:32:26.992013Z",
"iopub.status.idle": "2026-06-13T02:32:27.006055Z",
"shell.execute_reply": "2026-06-13T02:32:27.005651Z"
},
"papermill": {
"duration": 0.015918,
"end_time": "2026-06-13T02:32:27.006360+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:26.990442+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== AAPL Example ===\n",
"Trading days: 9,400\n",
"Date range: 1980-12-12 to 2018-03-27\n"
]
}
],
"source": [
"# AAPL as example\n",
"aapl = wiki.filter(pl.col(\"symbol\") == \"AAPL\").sort(\"timestamp\")\n",
"\n",
"print(\"=== AAPL Example ===\")\n",
"print(f\"Trading days: {len(aapl):,}\")\n",
"print(f\"Date range: {aapl['timestamp'].min()} to {aapl['timestamp'].max()}\")"
]
},
{
"cell_type": "markdown",
"id": "84290f70",
"metadata": {
"papermill": {
"duration": 0.001127,
"end_time": "2026-06-13T02:32:27.008827+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:27.007700+00:00",
"status": "completed"
}
},
"source": [
"Five most recent trading days for AAPL on adjusted prices."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "83c34cd1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:27.011649Z",
"iopub.status.busy": "2026-06-13T02:32:27.011526Z",
"iopub.status.idle": "2026-06-13T02:32:27.014436Z",
"shell.execute_reply": "2026-06-13T02:32:27.014207Z"
},
"papermill": {
"duration": 0.00492,
"end_time": "2026-06-13T02:32:27.014850+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:27.009930+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>adj_open</th><th>adj_high</th><th>adj_low</th><th>adj_close</th><th>adj_volume</th></tr><tr><td>date</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>2018-03-21</td><td>175.04</td><td>175.09</td><td>171.26</td><td>171.27</td><td>3.5247358e7</td></tr><tr><td>2018-03-22</td><td>170.0</td><td>172.68</td><td>168.6</td><td>168.845</td><td>4.1051076e7</td></tr><tr><td>2018-03-23</td><td>168.39</td><td>169.92</td><td>164.94</td><td>164.94</td><td>4.0248954e7</td></tr><tr><td>2018-03-26</td><td>168.07</td><td>173.1</td><td>166.44</td><td>172.77</td><td>3.6272617e7</td></tr><tr><td>2018-03-27</td><td>173.68</td><td>175.15</td><td>166.92</td><td>168.34</td><td>3.8962839e7</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5, 6)\n",
"┌────────────┬──────────┬──────────┬─────────┬───────────┬─────────────┐\n",
"│ timestamp ┆ adj_open ┆ adj_high ┆ adj_low ┆ adj_close ┆ adj_volume │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ date ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │\n",
"╞════════════╪══════════╪══════════╪═════════╪═══════════╪═════════════╡\n",
"│ 2018-03-21 ┆ 175.04 ┆ 175.09 ┆ 171.26 ┆ 171.27 ┆ 3.5247358e7 │\n",
"│ 2018-03-22 ┆ 170.0 ┆ 172.68 ┆ 168.6 ┆ 168.845 ┆ 4.1051076e7 │\n",
"│ 2018-03-23 ┆ 168.39 ┆ 169.92 ┆ 164.94 ┆ 164.94 ┆ 4.0248954e7 │\n",
"│ 2018-03-26 ┆ 168.07 ┆ 173.1 ┆ 166.44 ┆ 172.77 ┆ 3.6272617e7 │\n",
"│ 2018-03-27 ┆ 173.68 ┆ 175.15 ┆ 166.92 ┆ 168.34 ┆ 3.8962839e7 │\n",
"└────────────┴──────────┴──────────┴─────────┴───────────┴─────────────┘"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"aapl.select([\"timestamp\", \"adj_open\", \"adj_high\", \"adj_low\", \"adj_close\", \"adj_volume\"]).tail(5)"
]
},
{
"cell_type": "markdown",
"id": "cc275e7f",
"metadata": {
"papermill": {
"duration": 0.001054,
"end_time": "2026-06-13T02:32:27.017022+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:27.015968+00:00",
"status": "completed"
}
},
"source": [
"## Key Takeaways\n",
"\n",
"1. **Mitigates survivorship bias**: 24% of symbols stop trading before the\n",
" dataset end — including these inactive tickers is what makes the panel\n",
" usable for unbiased backtests.\n",
"2. **Always use adjusted prices for returns**: the `adj_*` columns absorb\n",
" splits and dividends; the raw `open/high/low/close/volume` columns remain\n",
" available for analyses that need actual traded levels.\n",
"3. **Long history, broad cross-section**: 3,199 symbols with a max single-symbol\n",
" span of ~56 years (14,155 trading days), covering multiple market regimes.\n",
"4. **Clean panel**: null rate is 0.0006% of values and the six adjusted-price\n",
" OHLC invariants hold on 100% of rows.\n",
"\n",
"**Next**: `02_corporate_actions` validates the adjustment factors that\n",
"produce the `adj_*` columns. **Book reference**: §2.2 (Equities)."
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "tags,-all"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.3"
},
"papermill": {
"default_parameters": {},
"duration": 3.93752,
"end_time": "2026-06-13T02:32:27.835362+00:00",
"environment_variables": {},
"exception": null,
"input_path": "02_financial_data_universe/01_us_equities_eda.ipynb",
"output_path": "02_financial_data_universe/01_us_equities_eda.ipynb",
"parameters": {},
"start_time": "2026-06-13T02:32:23.897842+00:00",
"version": "2.7.0"
},
"ml4t_provenance": {
"source_py_blob": "f34a592b987b8854cd8e3015baf91ad8f580fc4e",
"executed_at": "2026-06-13T02:32:28.047302+00:00",
"executor": "cpu-local",
"production": true,
"parameters": {},
"notes": "executed-state finalization batch (2026-06-13 run)"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02_financial_data_universe/01_us_equities_eda.py
+200
View File
@@ -0,0 +1,200 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # US Equities — Exploratory Data Analysis
#
# **Docker image**: `ml4t`
#
# **Purpose**: Profile the Wiki Prices dataset of US equity OHLCV history and confirm
# the inactive-symbol coverage that makes it usable for survivorship-bias-free
# backtests.
#
# **Learning objectives**:
#
# - Load the equity panel via `data.load_us_equities` and inspect its canonical schema.
# - Distinguish raw and split/dividend-adjusted price columns.
# - Quantify the share of symbols that stop trading before the dataset end date.
# - Check OHLC invariants and null rates across the full panel.
#
# **Book reference**: §2.2 ("The Asset-Class Market Data Landscape" — Equities).
#
# **Prerequisites**: `data` package on `PYTHONPATH`; Wiki Prices parquet present at
# `ML4T_DATA_PATH/equities/market/us_equities/`. Run
# `python data/equities/market/us_equities/download.py` if missing.
# %%
"""US Equities — Exploratory data analysis of the Wiki Prices dataset."""
import polars as pl
from data import load_us_equities
from utils.data_quality import check_ohlc_invariants
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
MAX_SYMBOLS = 0 # 0 = all
# %% [markdown]
# ## 1. Load and Inspect
# %%
wiki = load_us_equities()
print("=== Wiki Prices Dataset ===")
print(f"Shape: {wiki.shape}")
print(f"Columns: {wiki.columns}")
# %%
# Schema overview
print("\nSchema:")
for col, dtype in wiki.schema.items():
print(f" {col}: {dtype}")
# %% [markdown]
# ### Adjusted vs Raw Prices
#
# **Important**: This dataset contains both raw and adjusted prices.
#
# | Column Type | Examples | Use Case |
# |-------------|----------|----------|
# | Raw | `open`, `high`, `low`, `close`, `volume` | Historical analysis at actual prices |
# | Adjusted | `adj_open`, `adj_high`, `adj_low`, `adj_close`, `adj_volume` | **Backtesting** (handles splits/dividends) |
#
# Always use `adj_*` columns for return calculations and strategy backtesting.
# %% [markdown]
# ## 2. Coverage Summary
# %%
# Overall coverage
print("=== Coverage ===")
print(f"Unique tickers: {wiki['symbol'].n_unique():,}")
print(f"Date range: {wiki['timestamp'].min()} to {wiki['timestamp'].max()}")
print(f"Total rows: {len(wiki):,}")
# %%
# Per-stock statistics
stock_stats = wiki.group_by("symbol").agg(
[
pl.len().alias("days"),
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.col("adj_close").mean().alias("avg_price"),
]
)
# %% [markdown]
# Per-symbol coverage distribution — number of trading days and mean adjusted price.
# %%
stock_stats.select(["days", "avg_price"]).describe()
# %% [markdown]
# ## 3. Survivorship Analysis
#
# A key feature of this dataset is that it includes stocks that ceased trading
# before the dataset end date. This is critical for avoiding survivorship bias.
#
# **Note**: Stocks marked as "inactive before end" include both:
# - Actually delisted companies (bankruptcy, acquisition, etc.)
# - Stocks with incomplete coverage in this dataset
#
# The important point: these stocks are included, preventing survivorship bias.
# %%
dataset_end = wiki.select(pl.col("timestamp").max()).item()
# Identify stocks that stopped trading before dataset end
stock_stats = stock_stats.with_columns((pl.col("end") < dataset_end).alias("inactive_before_end"))
n_active = stock_stats.filter(~pl.col("inactive_before_end")).height
n_inactive = stock_stats.filter(pl.col("inactive_before_end")).height
total = n_active + n_inactive
inactive_pct = n_inactive / total * 100
print("=== Survivorship Analysis ===")
print(f"Dataset end: {dataset_end}")
print(f"Active at dataset end: {n_active:,} ({n_active / total * 100:.1f}%)")
print(f"Inactive before end: {n_inactive:,} ({inactive_pct:.1f}%)")
print(f"\nThis {inactive_pct:.0f}% inactive rate helps mitigate survivorship bias in backtests.")
# %% [markdown]
# ## 4. Data Quality
# %%
# Check for nulls across columns
null_counts = wiki.null_count()
total_nulls = null_counts.sum_horizontal()[0]
print("=== Data Quality ===")
print(f"Total null values: {total_nulls:,}")
# Show per-column breakdown (only columns with nulls)
for col in null_counts.columns:
val = null_counts[col][0]
if val > 0:
print(f" {col}: {val:,} ({val / len(wiki) * 100:.4f}%)")
print(f"\nNull rate: {total_nulls / (len(wiki) * len(wiki.columns)) * 100:.4f}%")
# %%
# OHLC invariants on adjusted prices
invariants = check_ohlc_invariants(
wiki,
open_col="adj_open",
high_col="adj_high",
low_col="adj_low",
close_col="adj_close",
volume_col="adj_volume",
)
print("\nOHLC Invariants (adjusted prices):")
for row in invariants.iter_rows(named=True):
status = "[OK]" if row["valid_pct"] >= 99.99 else "[WARN]"
print(f" {status} {row['check']}: {row['valid_pct']:.2f}%")
# %% [markdown]
# ## 5. Example: Single Stock
# %%
# AAPL as example
aapl = wiki.filter(pl.col("symbol") == "AAPL").sort("timestamp")
print("=== AAPL Example ===")
print(f"Trading days: {len(aapl):,}")
print(f"Date range: {aapl['timestamp'].min()} to {aapl['timestamp'].max()}")
# %% [markdown]
# Five most recent trading days for AAPL on adjusted prices.
# %%
aapl.select(["timestamp", "adj_open", "adj_high", "adj_low", "adj_close", "adj_volume"]).tail(5)
# %% [markdown]
# ## Key Takeaways
#
# 1. **Mitigates survivorship bias**: 24% of symbols stop trading before the
# dataset end — including these inactive tickers is what makes the panel
# usable for unbiased backtests.
# 2. **Always use adjusted prices for returns**: the `adj_*` columns absorb
# splits and dividends; the raw `open/high/low/close/volume` columns remain
# available for analyses that need actual traded levels.
# 3. **Long history, broad cross-section**: 3,199 symbols with a max single-symbol
# span of ~56 years (14,155 trading days), covering multiple market regimes.
# 4. **Clean panel**: null rate is 0.0006% of values and the six adjusted-price
# OHLC invariants hold on 100% of rows.
#
# **Next**: `02_corporate_actions` validates the adjustment factors that
# produce the `adj_*` columns. **Book reference**: §2.2 (Equities).
02_financial_data_universe/02_corporate_actions.ipynb
+1637
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/02_corporate_actions.py
+541
View File
@@ -0,0 +1,541 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Corporate Actions: Adjusting for Splits and Dividends
#
# **Docker image**: `ml4t`
#
# **Purpose**: Demonstrate why unadjusted price series mis-represent returns
# across stock splits and cash dividends, derive the industry-standard backward
# adjustment, and validate the `ml4t.data.adjustments.apply_corporate_actions`
# implementation against the pre-adjusted Quandl WIKI series.
#
# **Learning objectives**:
#
# - Identify how splits and dividends break raw price continuity.
# - Apply the backward-adjustment formula for splits and dividends.
# - Use `apply_corporate_actions` to produce an adjusted OHLCV panel.
# - Validate adjusted prices against a trusted reference series.
# - Pick the right price representation for a given strategy horizon.
#
# **Book reference**: §2.3 ("A Due Diligence Framework for Data Sourcing" —
# corporate-action handling) and §2.2 (Equities).
#
# **Prerequisites**: Wiki Prices US-equities parquet on disk
# (`load_us_equities` resolves it via `ML4T_DATA_PATH`); `ml4t-data` library
# installed.
# %%
"""Corporate Actions — Adjusting for splits and dividends in historical price series."""
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
from ml4t.data.adjustments import apply_corporate_actions
from data import load_us_equities
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ## 1. Why Corporate Actions Matter
#
# Corporate actions break the continuity of price series. Without proper
# adjustments:
#
# - **Stock splits** appear as massive price drops (e.g., a 2:1 split looks like
# a $-50\%$ return).
# - **Dividends** cause ex-date price drops that distort return calculations.
# - **ML features** computed on unadjusted prices are therefore systematically
# wrong.
#
# ### Types of corporate actions
#
# | Type | Description | Effect on price | Effect on shares |
# |------|-------------|-----------------|------------------|
# | Stock split | e.g., 2:1 split | Halved | Doubled |
# | Reverse split | e.g., 1:4 split | Quadrupled | Quartered |
# | Cash dividend | Payment to shareholders | Drops by amount | Unchanged |
# | Stock dividend | Additional shares issued | Drops pro rata | Increased |
# %% [markdown]
# ## 2. Load the Wiki Prices Panel
#
# The Quandl WIKI dataset is well-suited for studying corporate actions because
# it ships raw prices alongside the original split and dividend events plus a
# pre-calculated adjusted series — so the adjusted column doubles as a
# reference to validate against.
# %%
raw_wiki = load_us_equities()
print(f"Wiki Prices loaded: {len(raw_wiki):,} rows")
# %% [markdown]
# Apple is a clean illustrative example: four splits and 54 cash dividends
# across 1980–2018.
# %%
dividend_col = "ex_dividend" if "ex_dividend" in raw_wiki.columns else "ex-dividend"
aapl = raw_wiki.filter(pl.col("symbol") == "AAPL").sort("timestamp")
print(f"AAPL records: {len(aapl):,} ({aapl['timestamp'].min()} to {aapl['timestamp'].max()})")
# %% [markdown]
# Stock splits in the AAPL history.
# %%
splits = aapl.filter(pl.col("split_ratio") != 1.0)
splits.select(["timestamp", "close", "split_ratio", "adj_close"])
# %% [markdown]
# First ten of 54 cash dividends.
# %%
dividends = aapl.filter(pl.col(dividend_col) > 0)
dividends.select(["timestamp", "close", dividend_col, "adj_close"]).head(10)
# %% [markdown]
# ## 3. Three Ways to Represent Prices
#
# 1. **Raw (unadjusted)** — exactly as traded on the exchange. Best for
# order-execution simulation; returns are distorted at action dates.
# 2. **Split-adjusted** — adjusts for stock splits only. Reasonable for
# short-horizon trading where dividends are negligible.
# 3. **Total-return (split + dividend) adjusted** — the standard for ML
# features, factor research, and long-horizon backtests.
#
# The choice of representation determines whether returns are economically
# meaningful or dominated by accounting artifacts.
# %% [markdown]
# ## 4. Raw vs Adjusted Cumulative Return
#
# Compute cumulative returns from both raw and adjusted close prices and plot
# them on a log scale; each split shows up as a discontinuity in the raw
# series.
# %%
dates_arr = aapl["timestamp"].to_numpy()
raw_close = aapl["close"].to_numpy()
adj_close = aapl["adj_close"].to_numpy()
split_dates = splits["timestamp"].to_numpy()
split_ratios = splits["split_ratio"].to_numpy()
raw_returns = np.diff(raw_close) / raw_close[:-1]
adj_returns = np.diff(adj_close) / adj_close[:-1]
raw_cumret = np.cumprod(1 + raw_returns)
adj_cumret = np.cumprod(1 + adj_returns)
# %%
fig, ax = plt.subplots(figsize=(12, 6))
ax.plot(dates_arr[1:], adj_cumret, label="Adjusted prices (correct)", color="#1E3A5F", linewidth=2)
ax.plot(dates_arr[1:], raw_cumret, label="Raw prices (wrong)", color="#8B0000", linewidth=2)
for d, ratio in zip(split_dates, split_ratios, strict=False):
ax.axvline(d, color="gray", linestyle="--", alpha=0.4, linewidth=1)
idx = np.searchsorted(dates_arr[1:], d)
if idx < len(adj_cumret):
ax.annotate(
f"{int(ratio)}:1 split",
xy=(d, adj_cumret[idx]),
xytext=(10, 20),
textcoords="offset points",
fontsize=9,
color="gray",
arrowprops=dict(arrowstyle="-", color="gray", alpha=0.5),
)
ax.annotate(
f"Raw: {raw_cumret[-1]:.0f}x\nAdjusted: {adj_cumret[-1]:.0f}x\n({adj_cumret[-1] / raw_cumret[-1]:.0f}x difference)",
xy=(dates_arr[-1], (raw_cumret[-1] + adj_cumret[-1]) / 2),
xytext=(-120, 0),
textcoords="offset points",
fontsize=11,
fontweight="bold",
bbox=dict(boxstyle="round,pad=0.3", facecolor="wheat", alpha=0.8),
ha="right",
)
ax.annotate(
"Each split looks like a crash\nin raw prices, causing\ncumulative returns to diverge",
xy=(split_dates[2], raw_cumret[np.searchsorted(dates_arr[1:], split_dates[2])]),
xytext=(50, -30),
textcoords="offset points",
fontsize=9,
style="italic",
bbox=dict(boxstyle="round,pad=0.2", facecolor="white", alpha=0.8, edgecolor="gray"),
arrowprops=dict(arrowstyle="->", color="gray"),
)
ax.set_ylabel("Cumulative return (start = 1)")
ax.set_xlabel("Date")
ax.set_title("Raw vs adjusted cumulative return — AAPL 1980–2018")
ax.legend(loc="upper left", fontsize=10)
ax.set_yscale("log")
ax.set_ylim(0.5, adj_cumret[-1] * 1.5)
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda x, _: f"{x:.0f}x" if x >= 1 else f"{x:.1f}x"))
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# %%
# Persist Figure 2.3 artifact for book script (Hard Rule 15).
import polars as _pl
from utils.paths import get_chapter_dir as _get_chapter_dir
_ARTIFACTS_DIR = _get_chapter_dir(2) / "output" / "book_figure_artifacts"
_ARTIFACTS_DIR.mkdir(parents=True, exist_ok=True)
_pl.DataFrame(
{
"timestamp": dates_arr[1:],
"adj_cumret": adj_cumret,
"raw_cumret": raw_cumret,
}
).write_parquet(_ARTIFACTS_DIR / "figure_2_3_corporate_actions_curves.parquet")
_pl.DataFrame(
{
"split_date": split_dates,
"split_ratio": split_ratios,
}
).write_parquet(_ARTIFACTS_DIR / "figure_2_3_corporate_actions_splits.parquet")
print(
f"Persisted figure_2_3_corporate_actions: {len(adj_cumret)} dates, "
f"{len(split_dates)} splits, final raw={raw_cumret[-1]:.1f}x adj={adj_cumret[-1]:.1f}x"
)
# %% [markdown]
# Quantify the divergence: a buy-and-hold of AAPL since 1980 returned roughly
# $400\times$ on a total-return basis but only $\sim 6\times$ on raw prices —
# the entire dividend stream and the share-count effect of the four splits are
# missing from the raw series.
# %%
print(f"Final cumulative return — raw prices: {raw_cumret[-1]:.1f}x")
print(f"Final cumulative return — adjusted prices: {adj_cumret[-1]:.1f}x")
print(f"Ratio: {adj_cumret[-1] / raw_cumret[-1]:.0f}x difference")
print(
f"Raw prices understate the cumulative return by {(adj_cumret[-1] / raw_cumret[-1] - 1) * 100:.0f}%"
)
# %% [markdown]
# ## 5. The `apply_corporate_actions` API
#
# The `apply_corporate_actions` function in `ml4t.data.adjustments` implements
# the backward-adjustment methodology shared by Quandl and most major data
# vendors. The function signature documents the convention it expects.
# %%
help(apply_corporate_actions)
# %% [markdown]
# Apply the adjustment to AAPL. The function expects a `date` column and
# returns the input frame plus `adj_*` columns; we round-trip through the
# canonical `timestamp` schema.
# %%
adjusted = apply_corporate_actions(
aapl.rename({"timestamp": "date"}),
split_col="split_ratio",
dividend_col=dividend_col,
price_cols=["open", "high", "low", "close"],
volume_col="volume",
).rename({"date": "timestamp"})
print("Adjusted columns:", [c for c in adjusted.columns if c.startswith("adj_")])
# %% [markdown]
# Side-by-side comparison at the IPO, three split dates, and the dataset end.
# `our_adj_close` comes from this notebook's invocation of
# `apply_corporate_actions`; `quandl_adj_close` is the pre-computed adjusted
# series shipped with the dataset.
# %%
from datetime import date as _date
example_dates = [
_date(1980, 12, 12),
_date(1987, 6, 16),
_date(2000, 6, 21),
_date(2014, 6, 9),
_date(2018, 3, 27),
]
comparison = (
adjusted.filter(pl.col("timestamp").is_in(example_dates))
.select(
pl.col("timestamp"),
pl.col("close").alias("raw_close"),
pl.col("adj_close").alias("our_adj_close"),
pl.col("split_ratio"),
)
.join(
aapl.select(
pl.col("timestamp"),
pl.col("adj_close").alias("quandl_adj_close"),
),
on="timestamp",
)
.sort("timestamp")
)
comparison
# %% [markdown]
# ## 6. Validation Against Quandl's Reference
#
# Compare every row of the locally-computed adjusted close to Quandl's
# pre-calculated value. With ~9,400 iterative backward updates, small
# floating-point error accumulates; the published convention tolerates a
# relative difference of about $5 \times 10^{-4}$.
# %%
our_adj = adjusted["adj_close"].to_numpy()
quandl_adj = aapl["adj_close"].to_numpy()
dates = adjusted["timestamp"].to_numpy()
absolute_diff = np.abs(our_adj - quandl_adj)
relative_diff = absolute_diff / quandl_adj * 100
tolerance_pct = 0.05 # 0.05 %
print(f"Total comparisons: {len(our_adj):,}")
print(f"Max absolute difference: ${absolute_diff.max():.6f}")
print(f"Max relative difference: {relative_diff.max():.4f}%")
print(f"Mean relative difference: {relative_diff.mean():.6f}%")
print(f"Median relative difference: {np.median(relative_diff):.6f}%")
print()
if np.allclose(our_adj, quandl_adj, rtol=tolerance_pct / 100):
print(f"VALIDATION PASSED — within {tolerance_pct:.4f}% relative tolerance.")
else:
n_out = int((relative_diff > tolerance_pct).sum())
print(f"VALIDATION FAILED — {n_out} rows outside {tolerance_pct:.4f}% tolerance.")
# %% [markdown]
# Visual check — the two series overlay on the log-scale price chart, and the
# rolling relative difference stays below the tolerance line.
# %%
fig, axes = plt.subplots(2, 1, figsize=(14, 8))
ax1 = axes[0]
ax1.plot(dates, our_adj, label="Our adjustment", alpha=0.85)
ax1.plot(dates, quandl_adj, label="Quandl adjustment", alpha=0.85, linestyle="--")
ax1.set_ylabel("Adjusted price ($)")
ax1.set_title("Local vs Quandl pre-calculated adjusted close")
ax1.set_yscale("log")
ax1.legend()
ax2 = axes[1]
ax2.plot(dates, relative_diff, color="#2E86AB", linewidth=1.2, alpha=0.9)
ax2.axhline(
tolerance_pct,
color="red",
linestyle="--",
linewidth=2,
label=f"Tolerance ({tolerance_pct:.4f}%)",
)
ax2.set_ylabel("Relative difference (%)")
ax2.set_xlabel("Date")
ax2.set_title("Relative difference between local and Quandl series")
ax2.set_ylim(0, max(relative_diff.max() * 1.1, tolerance_pct * 2))
ax2.legend()
plt.tight_layout()
plt.show()
# %% [markdown]
# ## 7. The Adjustment Formulas
#
# The industry-standard method is **backward adjustment**: start at the most
# recent date with `factor = 1` and walk backwards in time, multiplying the
# factor at each corporate action so future prices remain unchanged and
# historical prices are scaled down.
#
# **Split adjustment.** On the day before a stock split with ratio $R$, all
# prior prices must be divided by $R$:
#
# $$\text{factor}_{t-1} = \text{factor}_{t} \times \frac{1}{R_t}.$$
#
# **Dividend adjustment.** On ex-dividend date $t$, with closing price
# $P_{t-1}$ on the day before and dividend amount $D_t$, the multiplier for
# pre-ex-date prices is
#
# $$m_t = \frac{P_{t-1} - D_t}{P_{t-1}} = 1 - \frac{D_t}{P_{t-1}}, \qquad
# \text{factor}_{t-1} = \text{factor}_{t} \times m_t.$$
#
# **Combined.** For any date $d$ the adjusted price is
#
# $$P^{\text{adj}}_d = P^{\text{raw}}_d \cdot \text{factor}_d.$$
#
# This backward-looking rule guarantees that returns computed on the adjusted
# series equal the total-return of holding the stock with dividends reinvested.
# %% [markdown]
# ### Numeric demonstration — split
# %%
def demonstrate_split_adjustment(
price_before: float = 270.0, shares_before: int = 100, ratio: float = 3.0
) -> None:
"""Print a worked example of backward split adjustment."""
value_before = price_before * shares_before
price_after = price_before / ratio
shares_after = shares_before * ratio
value_after = price_after * shares_after
print(
f"Before {int(ratio)}:1 split — price ${price_before:.2f}, shares {shares_before:d}, value ${value_before:,.2f}"
)
print(
f"After {int(ratio)}:1 split — price ${price_after:.2f}, shares {int(shares_after):d}, value ${value_after:,.2f}"
)
print(
f"Backward-adjusted historical price: ${price_before:.2f} / {ratio:.0f} = ${price_before / ratio:.2f}"
)
print(f"Adjusted series is now continuous: ${price_before / ratio:.2f} -> ${price_after:.2f}")
# %%
demonstrate_split_adjustment()
# %% [markdown]
# ### Numeric demonstration — dividend
# %%
def demonstrate_dividend_adjustment(price_before_ex: float = 100.0, dividend: float = 5.0) -> None:
"""Print a worked example of backward dividend adjustment."""
price_on_ex = price_before_ex - dividend
raw_return = (price_on_ex - price_before_ex) / price_before_ex
multiplier = price_on_ex / price_before_ex
adjusted_before = price_before_ex * multiplier
adj_return = (price_on_ex - adjusted_before) / adjusted_before
print(
f"Day before ex: ${price_before_ex:.2f} Dividend: ${dividend:.2f} Ex-date: ${price_on_ex:.2f}"
)
print(f"Raw return on ex-date: {raw_return:.1%}")
print(f"Backward multiplier: {price_on_ex:.2f} / {price_before_ex:.2f} = {multiplier:.4f}")
print(
f"Adjusted pre-ex price: ${price_before_ex:.2f} × {multiplier:.4f} = ${adjusted_before:.2f}"
)
print(f"Adjusted return on ex-date: {adj_return:.1%}")
# %%
demonstrate_dividend_adjustment()
# %% [markdown]
# The convention treats the dividend as if it were reinvested in the stock at
# the close on the day before the ex-date. In real markets the ex-date price
# move is not exactly equal to the dividend; the adjustment is a *convention*
# for building continuous total-return series, not a description of price
# behavior.
# %% [markdown]
# ## 8. Practical Implications for ML Trading
#
# | Use case | Price type | Reason |
# |-------------------------------|---------------------------|--------|
# | Feature engineering | Total-return adjusted | Returns must be economically meaningful |
# | Long-horizon backtest | Total-return adjusted | Captures dividend stream and split discontinuities |
# | Order-execution simulation | Raw | Matches actual exchange prices |
# | Intraday trading (< 1 day) | Split-adjusted | Dividends negligible at high frequency |
# | Factor research | Total-return adjusted | Standard in academic literature |
# %% [markdown]
# ### Worked example — dividend impact on momentum
#
# A five-ETF rotational momentum sleeve illustrates how price-only momentum
# can mis-rank the cross-section when dividend yields differ materially. The
# numbers below are a stylized one-period example, not measured returns.
# %%
universe = pl.DataFrame(
{
"symbol": ["SPY", "QQQ", "TLT", "GLD", "EFA"],
"price_return": [0.10, 0.15, -0.02, 0.08, 0.07],
"dividend_yield": [0.015, 0.005, 0.025, 0.00, 0.025],
}
).with_columns(total_return=pl.col("price_return") + pl.col("dividend_yield"))
ranked = universe.with_columns(
rank_price=pl.col("price_return").rank(descending=True, method="ordinal"),
rank_total=pl.col("total_return").rank(descending=True, method="ordinal"),
).with_columns(rank_change=pl.col("rank_price") - pl.col("rank_total"))
ranked.select(
[
"symbol",
"price_return",
"dividend_yield",
"total_return",
"rank_price",
"rank_total",
"rank_change",
]
)
# %% [markdown]
# Across the five ETFs the dividend stream changes the ranking: EFA moves up
# one place when dividends are included and GLD moves down one place. TLT
# flips from a negative price return to a small positive total return but
# stays last. Price-only momentum signals would miss those re-orderings,
# which compound across many decisions over a multi-year horizon.
# %% [markdown]
# ## 9. Data-Source Conventions
#
# Different vendors use different adjustment conventions. The conventions the
# reader is most likely to encounter are summarised below; always validate
# against a known split or dividend date before assuming what a vendor's
# `close` column actually contains.
#
# | Provider | "Close" column | "Adj Close" column |
# |--------------------|-----------------------|------------------------|
# | Quandl WIKI | Truly unadjusted | Split + dividend adjusted |
# | Yahoo Finance | May be split-adjusted | Split + dividend adjusted |
# | Bloomberg | Unadjusted | Configurable |
# | Binance (crypto) | Unadjusted | n/a (no splits) |
#
# Yahoo Finance behaviour in particular varies by interface (web UI,
# `yfinance` library, REST API). The safe procedure when integrating a new
# source is to pull a known split date — for AAPL, 2014-06-09 (7:1) is the
# canonical test case — and check whether the pre-split price has been
# divided by the split ratio.
# %% [markdown]
# ## Key Takeaways
#
# 1. **AAPL has 4 stock splits and 54 cash dividends** in the Wiki Prices
# sample (1980-12-12 to 2018-03-27); ignoring them turns a 398.2× total
# return into a 5.9× raw-price return — a 68× difference, or a 6,701%
# underestimation.
# 2. **Backward adjustment is the standard convention.** The local
# `apply_corporate_actions` implementation matches Quandl's pre-computed
# series within a 0.05% relative tolerance; the largest observed
# discrepancy is 0.0249%, attributable to floating-point accumulation
# across the 9,400-row recursion.
# 3. **Pick the price representation that matches the use case** — total-return
# adjusted for ML features and long-horizon backtests, raw for execution
# simulation, split-adjusted for short-horizon trading.
# 4. **Vendor conventions differ**, so always verify against a known split or
# dividend date before integrating a new data source.
#
# **Next**: `03_etfs_eda` profiles the ETF universe used in the rotational
# strategy. **Book reference**: §2.3 (corporate-action handling) and §2.2
# (Equities).
02_financial_data_universe/03_etfs_eda.ipynb
+1193
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/03_etfs_eda.py
+311
View File
@@ -0,0 +1,311 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # ETFs — Exploratory Data Analysis
#
# **Docker image**: `ml4t`
#
# **Purpose**: Profile the 100-ETF candidate universe sourced from Yahoo Finance
# and confirm the category coverage, history, and data-quality characteristics
# that drive the ETF rotation case study.
#
# **Learning objectives**:
#
# - Load the ETF panel via `data.load_etfs` and inspect its canonical schema.
# - Quantify per-symbol coverage and identify ETFs with shorter history.
# - Check OHLC invariants and null rates across the full panel.
# - Compare liquidity and price ranges across asset-class categories.
#
# **Book reference**: §2.2 ("The Asset-Class Market Data Landscape" — ETPs).
#
# **Prerequisites**: `data` package on `PYTHONPATH`; ETF parquet present at
# `ML4T_DATA_PATH/etfs/market/`. Run `python data/etfs/market/download.py` if
# missing.
# %%
"""ETFs — Exploratory data analysis of the multi-asset ETF universe."""
import polars as pl
from ml4t.data.etfs import ETFDataManager
from data import load_etfs
from utils.data_quality import check_ohlc_invariants, per_asset_stats
from utils.paths import REPO_ROOT
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
MAX_SYMBOLS = 0 # 0 = all
# %% [markdown]
# ## 1. Load and Inspect
#
# The ETF universe is stored as a single Parquet file containing daily OHLCV data
# for 50 ETFs spanning multiple asset classes.
# %%
etfs = load_etfs()
print("=== ETF Dataset ===")
print(f"Shape: {etfs.shape}")
print(f"Columns: {etfs.columns}")
# %%
# Schema overview
print("\nSchema:")
for col, dtype in etfs.schema.items():
print(f" {col}: {dtype}")
# %% [markdown]
# ### Adjusted Prices
#
# Yahoo Finance returns split- and dividend-adjusted OHLC. The `close` column is
# the adjusted close, so returns can be computed directly without a separate
# `adj_close` column.
# %% [markdown]
# ## 2. Coverage Summary
# %%
# Unique symbols
symbols = etfs["symbol"].unique().sort().to_list()
print("=== Coverage ===")
print(f"Number of ETFs: {len(symbols)}")
print(f"\nSymbols: {', '.join(symbols)}")
# %%
# Overall date range
date_range = etfs.select(
[
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.col("timestamp").n_unique().alias("unique_dates"),
]
)
print(f"\nDate range: {date_range['start'][0]} to {date_range['end'][0]}")
print(f"Trading days: {date_range['unique_dates'][0]}")
# %%
symbol_stats = per_asset_stats(
etfs,
time_col="timestamp",
asset_col="symbol",
price_col="close",
volume_col="volume",
)
full_start = date_range["start"][0]
full_end = date_range["end"][0]
partial = symbol_stats.filter(
(pl.col("start").cast(pl.Date) != pl.lit(full_start).cast(pl.Date))
| (pl.col("end").cast(pl.Date) != pl.lit(full_end).cast(pl.Date))
)
print(f"Symbols with full coverage: {len(symbol_stats) - len(partial)}")
print(f"Symbols with partial coverage: {len(partial)}")
# %% [markdown]
# Most ETFs predate the 2006 start of the dataset, but a sizeable minority were
# launched later — visible below as ETFs whose first observation is after
# 2006-01-03.
# %%
partial.select(["symbol", "start", "end", "rows"]).sort("start")
# %% [markdown]
# ## 3. ETF Categories
#
# Six categories cover 50 ETFs that span the major asset-class buckets used by
# the rotation case study. The full universe contains 100 ETFs; the remaining
# 50 are kept as candidates for the universe-construction work in Chapter 6.
# %%
ETF_CATEGORIES = {
"US Equity Broad": ["SPY", "QQQ", "IWM", "DIA", "VTI", "MDY", "IJR"],
"US Sectors": ["XLB", "XLE", "XLF", "XLI", "XLK", "XLP", "XLU", "XLV", "XLY", "VNQ", "IYR"],
"International": [
"EFA",
"EEM",
"VEA",
"VWO",
"EWJ",
"EWG",
"FXI",
"EWZ",
"EWY",
"EWC",
"EWA",
"ACWI",
],
"Fixed Income": ["AGG", "BND", "TLT", "IEF", "SHY", "LQD", "HYG", "TIP", "EMB", "MUB"],
"Commodities": ["GLD", "SLV", "USO", "UNG", "DBC", "GSG"],
"Specialty": ["IBB", "SMH", "KRE", "OIH"],
}
print("=== ETF Categories ===")
for category, tickers in ETF_CATEGORIES.items():
available = [t for t in tickers if t in symbols]
print(f" {category}: {len(available)}/{len(tickers)} ETFs")
# %% [markdown]
# ## 4. Data Quality
# %%
# Check for nulls
null_counts = etfs.null_count()
total_nulls = null_counts.sum_horizontal()[0]
print("=== Data Quality ===")
print(f"Total null values: {total_nulls}")
# %%
# Check for zero volume days
zero_volume = etfs.filter(pl.col("volume") == 0)
print(f"Zero volume rows: {len(zero_volume)} ({100 * len(zero_volume) / len(etfs):.2f}%)")
# %%
# OHLC invariants
invariants = check_ohlc_invariants(etfs)
print("\nOHLC Invariants:")
for row in invariants.iter_rows(named=True):
status = "[OK]" if row["valid_pct"] >= 99.99 else "[WARN]"
print(f" {status} {row['check']}: {row['valid_pct']:.2f}%")
# Count violations
violations = etfs.filter(
(pl.col("high") < pl.col("low"))
| (pl.col("high") < pl.col("open"))
| (pl.col("high") < pl.col("close"))
)
print(f"\nTotal OHLC violations: {len(violations)}")
# %% [markdown]
# Violations occur where `high < close` or `low > close` after price adjustment.
# These arise from floating-point/rounding precision in the stored adjusted OHLC
# values (the same cumulative ratio is applied across all four fields, but small
# per-field rounding can break the strict invariants the raw quotes satisfied).
# At a tenth of a percent of rows they are immaterial for return and feature
# calculations but worth being aware of when computing intraday range statistics.
# %% [markdown]
# ## 5. Volume and Liquidity Distribution
#
# Understanding volume distributions helps identify liquidity constraints for trading.
# %%
# Average daily volume by ETF
volume_stats = (
etfs.group_by("symbol")
.agg(
[
pl.col("volume").mean().alias("avg_volume"),
pl.col("volume").median().alias("median_volume"),
pl.col("volume").std().alias("volume_std"),
]
)
.sort("avg_volume", descending=True)
)
print("=== Volume Distribution (Top 10 Most Liquid) ===")
volume_stats.head(10)
# %%
# Volume by category
print("\n=== Average Daily Volume by Category ===")
for category, tickers in ETF_CATEGORIES.items():
category_vol = (
etfs.filter(pl.col("symbol").is_in(tickers)).select(pl.col("volume").mean()).item()
)
if category_vol is not None:
print(f" {category}: {category_vol:,.0f} shares/day")
# %% [markdown]
# ## 6. Price Distribution
#
# Price levels across the ETF universe span a wide range.
# %%
# Current price levels (most recent date)
latest = etfs.filter(pl.col("timestamp") == etfs["timestamp"].max())
price_dist = latest.select(
[
pl.col("close").min().alias("min_price"),
pl.col("close").max().alias("max_price"),
pl.col("close").median().alias("median_price"),
pl.col("close").mean().alias("mean_price"),
]
)
print("=== Price Distribution (Latest Date) ===")
price_dist
# %%
# Price range by category
print("=== Price Range by Category ===")
for category, tickers in ETF_CATEGORIES.items():
category_prices = latest.filter(pl.col("symbol").is_in(tickers))
min_p = category_prices["close"].min()
max_p = category_prices["close"].max()
print(f" {category}: ${min_p:.2f} - ${max_p:.2f}")
# %% [markdown]
# ## 7. Loading by Symbol or via the ml4t-data Library
#
# `load_etfs(symbols=[...])` filters the panel to a subset; `ETFDataManager` is
# the config-driven entry point used by the production download/refresh workflow
# in `data/etfs/market/`.
# %%
spy = load_etfs(symbols=["SPY"])
print(f"SPY via loader: {spy.shape}")
# %%
config_path = REPO_ROOT / "data" / "etfs" / "market" / "config.yaml"
etf_mgr = ETFDataManager.from_config(str(config_path))
configured = sum(len(group["symbols"]) for group in etf_mgr.config.tickers.values())
print(f"ETFDataManager loaded from {config_path}")
print(f" Provider: {etf_mgr.config.provider}")
print(f" Date range: {etf_mgr.config.start} to {etf_mgr.config.end}")
print(f" Configured symbols: {configured} across {len(etf_mgr.config.tickers)} categories")
# %% [markdown]
# ## Key Takeaways
#
# 1. **Pre-adjusted prices**: Yahoo Finance returns split- and dividend-adjusted
# OHLC. The `close` column is the adjusted close — return calculations need
# no further adjustment.
# 2. **Coverage**: 100 ETFs across daily history from 2006-01-03 to 2025-12-31
# (5,031 trading days), with 59 ETFs spanning the full window and 41 starting
# later as new products were launched.
# 3. **Categorization**: Six categories (US Equity Broad, US Sectors,
# International, Fixed Income, Commodities, Specialty) cover 50 of the 100
# ETFs and form the candidate set for the rotation case study; the other 50
# are reserved for the universe-construction work in Chapter 6.
# 4. **Mostly clean data**: Zero nulls and 473 OHLC violations
# (`high < close` or `low > close`) — about 0.1% of rows, immaterial for
# return and feature calculations.
# 5. **Liquidity variation**: SPY averages 126M shares/day; the Commodities
# bucket averages 5.5M — a 23× difference that matters for transaction-cost
# modeling in later chapters.
#
# ### Next Steps
#
# - **§2.6 / `13_data_quality_framework`**: Systematic data-quality checks across
# the seven datasets.
# - **§2.7 / `15_survivorship_bias_detection`**: Survivorship and selection bias
# in equity panels — a contrast to the ETF universe shown here.
# - **Chapter 6**: Universe construction filters this 100-ETF candidate pool down
# to the trading universe used by the ETF rotation case study.
02_financial_data_universe/04_cme_futures_eda.ipynb
+865
View File
@@ -0,0 +1,865 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "fb07336e",
"metadata": {
"papermill": {
"duration": 0.006471,
"end_time": "2026-06-13T02:32:41.536804+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:41.530333+00:00",
"status": "completed"
}
},
"source": [
"# CME Futures — Exploratory Data Analysis\n",
"\n",
"**Docker image**: `ml4t`\n",
"\n",
"**Purpose**: Profile the 30-product CME futures dataset (Databento, hourly,\n",
"2011–2025) and surface the contract / continuous structure that downstream\n",
"notebooks rely on.\n",
"\n",
"**Learning objectives**:\n",
"\n",
"- Understand the futures data hierarchy: product → contract → continuous series.\n",
"- Load individual contracts and continuous (rolled) series via\n",
" `load_cme_futures` and inspect the canonical `timestamp` / `product`\n",
" schema.\n",
"- Summarize per-product coverage and group products by asset class.\n",
"- Verify OHLC invariants on a representative continuous series.\n",
"\n",
"**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — Futures).\n",
"\n",
"**Prerequisites**: `data` package on `PYTHONPATH`; CME parquet present at\n",
"`ML4T_DATA_PATH/futures/`. Run `python data/futures/market/download.py` if\n",
"missing (Databento API key required)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "1830247e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:41.541085Z",
"iopub.status.busy": "2026-06-13T02:32:41.540762Z",
"iopub.status.idle": "2026-06-13T02:32:42.760904Z",
"shell.execute_reply": "2026-06-13T02:32:42.760453Z"
},
"papermill": {
"duration": 1.222807,
"end_time": "2026-06-13T02:32:42.761451+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:41.538644+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"\"\"\"CME Futures — Exploratory data analysis of the futures universe.\"\"\"\n",
"\n",
"import polars as pl\n",
"\n",
"from data import list_cme_products, load_cme_futures\n",
"from utils.data_quality import check_ohlc_invariants"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "bdf60381",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.765255Z",
"iopub.status.busy": "2026-06-13T02:32:42.765168Z",
"iopub.status.idle": "2026-06-13T02:32:42.767341Z",
"shell.execute_reply": "2026-06-13T02:32:42.766636Z"
},
"papermill": {
"duration": 0.005182,
"end_time": "2026-06-13T02:32:42.767745+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.762563+00:00",
"status": "completed"
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"# Production defaults — Papermill injects overrides for CI\n",
"MAX_SYMBOLS = 0 # 0 = all"
]
},
{
"cell_type": "markdown",
"id": "70f4bc2e",
"metadata": {
"papermill": {
"duration": 0.000844,
"end_time": "2026-06-13T02:32:42.769552+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.768708+00:00",
"status": "completed"
}
},
"source": [
"## 1. Configuration and Data Discovery\n",
"\n",
"The futures data uses a Hive-partitioned structure for efficient queries:\n",
"- `futures/continuous/product={PRODUCT}/`: Volume-rolled continuous contracts (hourly)\n",
"- `futures/individual/{PRODUCT}/data.parquet`: Individual contract price data\n",
"\n",
"We use `load_cme_futures()` for proper data loading with partition pruning."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "246bdafd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.771820Z",
"iopub.status.busy": "2026-06-13T02:32:42.771760Z",
"iopub.status.idle": "2026-06-13T02:32:42.774699Z",
"shell.execute_reply": "2026-06-13T02:32:42.774121Z"
},
"papermill": {
"duration": 0.004585,
"end_time": "2026-06-13T02:32:42.775020+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.770435+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Futures Universe ===\n",
"Available products: 30\n",
"\n",
"Products: 6A, 6B, 6C, 6E, 6J, 6S, CL, ES, GC, GF, HE, HG, HO, LE, NG, NQ, PL, RB, RTY, SI, YM, ZB, ZC, ZF, ZL, ZM, ZN, ZS, ZT, ZW\n"
]
}
],
"source": [
"# Discover available products via the CME loader\n",
"products = list_cme_products()\n",
"\n",
"print(\"=== Futures Universe ===\")\n",
"print(f\"Available products: {len(products)}\")\n",
"print(f\"\\nProducts: {', '.join(products)}\")"
]
},
{
"cell_type": "markdown",
"id": "893ba4a8",
"metadata": {
"papermill": {
"duration": 0.000878,
"end_time": "2026-06-13T02:32:42.776865+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.775987+00:00",
"status": "completed"
}
},
"source": [
"Map each product to a coarse asset-class bucket. The mapping covers every\n",
"product in the dataset; downstream chapters use the same bucket labels for\n",
"universe-construction and risk reporting."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "19ed20cd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.779264Z",
"iopub.status.busy": "2026-06-13T02:32:42.779153Z",
"iopub.status.idle": "2026-06-13T02:32:42.781984Z",
"shell.execute_reply": "2026-06-13T02:32:42.781419Z"
},
"papermill": {
"duration": 0.004493,
"end_time": "2026-06-13T02:32:42.782271+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.777778+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"ASSET_CLASS_MAP = {\n",
" \"ES\": \"Equity Index\",\n",
" \"NQ\": \"Equity Index\",\n",
" \"YM\": \"Equity Index\",\n",
" \"RTY\": \"Equity Index\",\n",
" \"ZN\": \"Rates\",\n",
" \"ZB\": \"Rates\",\n",
" \"ZF\": \"Rates\",\n",
" \"ZT\": \"Rates\",\n",
" \"CL\": \"Energy\",\n",
" \"NG\": \"Energy\",\n",
" \"HO\": \"Energy\",\n",
" \"RB\": \"Energy\",\n",
" \"GC\": \"Metals\",\n",
" \"SI\": \"Metals\",\n",
" \"HG\": \"Metals\",\n",
" \"PL\": \"Metals\",\n",
" \"6E\": \"FX\",\n",
" \"6J\": \"FX\",\n",
" \"6B\": \"FX\",\n",
" \"6A\": \"FX\",\n",
" \"6C\": \"FX\",\n",
" \"6S\": \"FX\",\n",
" \"ZC\": \"Grains\",\n",
" \"ZS\": \"Grains\",\n",
" \"ZW\": \"Grains\",\n",
" \"ZM\": \"Grains\",\n",
" \"ZL\": \"Grains\",\n",
" \"LE\": \"Livestock\",\n",
" \"HE\": \"Livestock\",\n",
" \"GF\": \"Livestock\",\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "7dcc7f2b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.784842Z",
"iopub.status.busy": "2026-06-13T02:32:42.784729Z",
"iopub.status.idle": "2026-06-13T02:32:42.815497Z",
"shell.execute_reply": "2026-06-13T02:32:42.814607Z"
},
"papermill": {
"duration": 0.032845,
"end_time": "2026-06-13T02:32:42.816135+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.783290+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Products by Asset Class:\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (7, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>asset_class</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>&quot;FX&quot;</td><td>6</td></tr><tr><td>&quot;Grains&quot;</td><td>5</td></tr><tr><td>&quot;Energy&quot;</td><td>4</td></tr><tr><td>&quot;Equity Index&quot;</td><td>4</td></tr><tr><td>&quot;Metals&quot;</td><td>4</td></tr><tr><td>&quot;Rates&quot;</td><td>4</td></tr><tr><td>&quot;Livestock&quot;</td><td>3</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (7, 2)\n",
"┌──────────────┬─────┐\n",
"│ asset_class ┆ len │\n",
"│ --- ┆ --- │\n",
"│ str ┆ u32 │\n",
"╞══════════════╪═════╡\n",
"│ FX ┆ 6 │\n",
"│ Grains ┆ 5 │\n",
"│ Energy ┆ 4 │\n",
"│ Equity Index ┆ 4 │\n",
"│ Metals ┆ 4 │\n",
"│ Rates ┆ 4 │\n",
"│ Livestock ┆ 3 │\n",
"└──────────────┴─────┘"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Count products by asset class\n",
"class_counts = (\n",
" pl.DataFrame({\"product\": products})\n",
" .with_columns(asset_class=pl.col(\"product\").replace(ASSET_CLASS_MAP))\n",
" .group_by(\"asset_class\")\n",
" .len()\n",
" .sort([\"len\", \"asset_class\"], descending=[True, False])\n",
")\n",
"\n",
"print(\"Products by Asset Class:\")\n",
"class_counts"
]
},
{
"cell_type": "markdown",
"id": "07175acd",
"metadata": {
"papermill": {
"duration": 0.001659,
"end_time": "2026-06-13T02:32:42.820076+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.818417+00:00",
"status": "completed"
}
},
"source": [
"## 2. Data Structure Example: E-mini S&P 500 (ES)\n",
"\n",
"### Futures Key Hierarchy\n",
"\n",
"| Level | Example | Description |\n",
"|-------|---------|-------------|\n",
"| **Product** | ES | The underlying (E-mini S&P 500) |\n",
"| **Contract** | ESH4 | Specific expiration (March 2024) |\n",
"| **Continuous** | c0, c1 | Front month, first deferred |"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d61969a4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.823125Z",
"iopub.status.busy": "2026-06-13T02:32:42.823038Z",
"iopub.status.idle": "2026-06-13T02:32:42.831229Z",
"shell.execute_reply": "2026-06-13T02:32:42.830764Z"
},
"papermill": {
"duration": 0.010089,
"end_time": "2026-06-13T02:32:42.831567+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.821478+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== ES Individual Contracts ===\n",
"Shape: (19361, 10)\n",
"Columns: ['timestamp', 'rtype', 'publisher_id', 'instrument_id', 'open', 'high', 'low', 'close', 'volume', 'product']\n",
"Date range: 2016-01-03 00:00:00+00:00 to 2025-12-30 00:00:00+00:00\n",
"Unique contracts: 194\n"
]
}
],
"source": [
"es_individual = load_cme_futures(products=[\"ES\"], continuous=False, frequency=\"hourly\")\n",
"\n",
"print(\"=== ES Individual Contracts ===\")\n",
"print(f\"Shape: {es_individual.shape}\")\n",
"print(f\"Columns: {es_individual.columns}\")\n",
"print(f\"Date range: {es_individual['timestamp'].min()} to {es_individual['timestamp'].max()}\")\n",
"print(f\"Unique contracts: {es_individual['instrument_id'].n_unique()}\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a409f932",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.834329Z",
"iopub.status.busy": "2026-06-13T02:32:42.834216Z",
"iopub.status.idle": "2026-06-13T02:32:42.843754Z",
"shell.execute_reply": "2026-06-13T02:32:42.843399Z"
},
"papermill": {
"duration": 0.011434,
"end_time": "2026-06-13T02:32:42.844170+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.832736+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== ES Continuous Series (front month) ===\n",
"Shape: (89031, 13)\n",
"Date range: 2011-01-02 23:00:00+00:00 to 2025-12-30 23:00:00+00:00\n"
]
}
],
"source": [
"es_continuous = load_cme_futures(products=[\"ES\"], tenors=[0], continuous=True, frequency=\"hourly\")\n",
"\n",
"print(\"=== ES Continuous Series (front month) ===\")\n",
"print(f\"Shape: {es_continuous.shape}\")\n",
"print(f\"Date range: {es_continuous['timestamp'].min()} to {es_continuous['timestamp'].max()}\")"
]
},
{
"cell_type": "markdown",
"id": "1c8de1ff",
"metadata": {
"papermill": {
"duration": 0.001035,
"end_time": "2026-06-13T02:32:42.846393+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.845358+00:00",
"status": "completed"
}
},
"source": [
"Each individual contract trades for a finite window before expiry. Aggregating\n",
"by `instrument_id` shows the rollover pattern — quarterly contracts overlap\n",
"during the roll period."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "3be70a7d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.848957Z",
"iopub.status.busy": "2026-06-13T02:32:42.848871Z",
"iopub.status.idle": "2026-06-13T02:32:42.853358Z",
"shell.execute_reply": "2026-06-13T02:32:42.852983Z"
},
"papermill": {
"duration": 0.006217,
"end_time": "2026-06-13T02:32:42.853630+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.847413+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total ES contracts: 194\n",
"Most recent 5 contracts:\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>instrument_id</th><th>first_trade</th><th>last_trade</th><th>total_volume</th><th>observations</th></tr><tr><td>u32</td><td>datetime[ns, UTC]</td><td>datetime[ns, UTC]</td><td>u64</td><td>u32</td></tr></thead><tbody><tr><td>42004904</td><td>2025-10-02 00:00:00 UTC</td><td>2025-12-29 00:00:00 UTC</td><td>104</td><td>7</td></tr><tr><td>42140860</td><td>2025-10-03 00:00:00 UTC</td><td>2025-10-03 00:00:00 UTC</td><td>1</td><td>1</td></tr><tr><td>42018017</td><td>2025-10-10 00:00:00 UTC</td><td>2025-12-30 00:00:00 UTC</td><td>3831</td><td>52</td></tr><tr><td>42008091</td><td>2025-10-23 00:00:00 UTC</td><td>2025-12-16 00:00:00 UTC</td><td>7</td><td>5</td></tr><tr><td>42016422</td><td>2025-10-23 00:00:00 UTC</td><td>2025-12-17 00:00:00 UTC</td><td>10</td><td>4</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5, 5)\n",
"┌───────────────┬─────────────────────────┬─────────────────────────┬──────────────┬──────────────┐\n",
"│ instrument_id ┆ first_trade ┆ last_trade ┆ total_volume ┆ observations │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ u32 ┆ datetime[ns, UTC] ┆ datetime[ns, UTC] ┆ u64 ┆ u32 │\n",
"╞═══════════════╪═════════════════════════╪═════════════════════════╪══════════════╪══════════════╡\n",
"│ 42004904 ┆ 2025-10-02 00:00:00 UTC ┆ 2025-12-29 00:00:00 UTC ┆ 104 ┆ 7 │\n",
"│ 42140860 ┆ 2025-10-03 00:00:00 UTC ┆ 2025-10-03 00:00:00 UTC ┆ 1 ┆ 1 │\n",
"│ 42018017 ┆ 2025-10-10 00:00:00 UTC ┆ 2025-12-30 00:00:00 UTC ┆ 3831 ┆ 52 │\n",
"│ 42008091 ┆ 2025-10-23 00:00:00 UTC ┆ 2025-12-16 00:00:00 UTC ┆ 7 ┆ 5 │\n",
"│ 42016422 ┆ 2025-10-23 00:00:00 UTC ┆ 2025-12-17 00:00:00 UTC ┆ 10 ┆ 4 │\n",
"└───────────────┴─────────────────────────┴─────────────────────────┴──────────────┴──────────────┘"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"contract_stats = (\n",
" es_individual.group_by(\"instrument_id\")\n",
" .agg(\n",
" pl.col(\"timestamp\").min().alias(\"first_trade\"),\n",
" pl.col(\"timestamp\").max().alias(\"last_trade\"),\n",
" pl.col(\"volume\").sum().alias(\"total_volume\"),\n",
" pl.len().alias(\"observations\"),\n",
" )\n",
" .sort(\"first_trade\")\n",
")\n",
"print(f\"Total ES contracts: {len(contract_stats)}\")\n",
"print(\"Most recent 5 contracts:\")\n",
"contract_stats.tail(5)"
]
},
{
"cell_type": "markdown",
"id": "030744ec",
"metadata": {
"lines_to_next_cell": 2,
"papermill": {
"duration": 0.001054,
"end_time": "2026-06-13T02:32:42.855801+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.854747+00:00",
"status": "completed"
}
},
"source": [
"## 3. Coverage Summary\n",
"\n",
"Check data availability across all products."
]
},
{
"cell_type": "markdown",
"id": "0be32d87",
"metadata": {
"lines_to_next_cell": 2,
"papermill": {
"duration": 0.001041,
"end_time": "2026-06-13T02:32:42.857925+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.856884+00:00",
"status": "completed"
}
},
"source": [
"Summarize per-product coverage by loading the front-month continuous series\n",
"(`tenor=0`) for every product and recording its row count and date range."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "5fdcffb8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.860611Z",
"iopub.status.busy": "2026-06-13T02:32:42.860534Z",
"iopub.status.idle": "2026-06-13T02:32:42.863095Z",
"shell.execute_reply": "2026-06-13T02:32:42.862662Z"
},
"papermill": {
"duration": 0.004347,
"end_time": "2026-06-13T02:32:42.863352+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.859005+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"def get_product_coverage(product_list: list[str]) -> pl.DataFrame:\n",
" \"\"\"Summarize continuous series coverage for each product (front month).\"\"\"\n",
" summaries = []\n",
" for product in product_list:\n",
" df = load_cme_futures(products=[product], tenors=[0], continuous=True, frequency=\"hourly\")\n",
" summaries.append(\n",
" {\n",
" \"product\": product,\n",
" \"asset_class\": ASSET_CLASS_MAP[product],\n",
" \"rows\": len(df),\n",
" \"start_date\": str(df[\"timestamp\"].min())[:10],\n",
" \"end_date\": str(df[\"timestamp\"].max())[:10],\n",
" }\n",
" )\n",
" return pl.DataFrame(summaries)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "f8fa8a21",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:42.866298Z",
"iopub.status.busy": "2026-06-13T02:32:42.866227Z",
"iopub.status.idle": "2026-06-13T02:32:43.007188Z",
"shell.execute_reply": "2026-06-13T02:32:43.006836Z"
},
"papermill": {
"duration": 0.142974,
"end_time": "2026-06-13T02:32:43.007534+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:42.864560+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Products with data: 30 / 30\n",
"Coverage by asset class:\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (7, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>asset_class</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>&quot;FX&quot;</td><td>6</td></tr><tr><td>&quot;Grains&quot;</td><td>5</td></tr><tr><td>&quot;Energy&quot;</td><td>4</td></tr><tr><td>&quot;Equity Index&quot;</td><td>4</td></tr><tr><td>&quot;Metals&quot;</td><td>4</td></tr><tr><td>&quot;Rates&quot;</td><td>4</td></tr><tr><td>&quot;Livestock&quot;</td><td>3</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (7, 2)\n",
"┌──────────────┬─────┐\n",
"│ asset_class ┆ len │\n",
"│ --- ┆ --- │\n",
"│ str ┆ u32 │\n",
"╞══════════════╪═════╡\n",
"│ FX ┆ 6 │\n",
"│ Grains ┆ 5 │\n",
"│ Energy ┆ 4 │\n",
"│ Equity Index ┆ 4 │\n",
"│ Metals ┆ 4 │\n",
"│ Rates ┆ 4 │\n",
"│ Livestock ┆ 3 │\n",
"└──────────────┴─────┘"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coverage = get_product_coverage(products)\n",
"print(f\"Products with data: {len(coverage)} / {len(products)}\")\n",
"print(\"Coverage by asset class:\")\n",
"coverage.group_by(\"asset_class\").len().sort([\"len\", \"asset_class\"], descending=[True, False])"
]
},
{
"cell_type": "markdown",
"id": "54663965",
"metadata": {
"papermill": {
"duration": 0.001516,
"end_time": "2026-06-13T02:32:43.010480+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:43.008964+00:00",
"status": "completed"
}
},
"source": [
"A handful of representative products from each asset-class bucket:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "c4bc6406",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:43.013592Z",
"iopub.status.busy": "2026-06-13T02:32:43.013498Z",
"iopub.status.idle": "2026-06-13T02:32:43.016213Z",
"shell.execute_reply": "2026-06-13T02:32:43.015939Z"
},
"papermill": {
"duration": 0.004893,
"end_time": "2026-06-13T02:32:43.016650+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:43.011757+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (7, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>product</th><th>asset_class</th><th>rows</th><th>start_date</th><th>end_date</th></tr><tr><td>str</td><td>str</td><td>i64</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>&quot;6E&quot;</td><td>&quot;FX&quot;</td><td>88326</td><td>&quot;2011-01-02&quot;</td><td>&quot;2025-12-30&quot;</td></tr><tr><td>&quot;CL&quot;</td><td>&quot;Energy&quot;</td><td>89383</td><td>&quot;2011-01-02&quot;</td><td>&quot;2025-12-30&quot;</td></tr><tr><td>&quot;ES&quot;</td><td>&quot;Equity Index&quot;</td><td>89031</td><td>&quot;2011-01-02&quot;</td><td>&quot;2025-12-30&quot;</td></tr><tr><td>&quot;GC&quot;</td><td>&quot;Metals&quot;</td><td>89424</td><td>&quot;2011-01-02&quot;</td><td>&quot;2025-12-30&quot;</td></tr><tr><td>&quot;NQ&quot;</td><td>&quot;Equity Index&quot;</td><td>89049</td><td>&quot;2011-01-02&quot;</td><td>&quot;2025-12-30&quot;</td></tr><tr><td>&quot;ZC&quot;</td><td>&quot;Grains&quot;</td><td>71426</td><td>&quot;2011-01-03&quot;</td><td>&quot;2025-12-30&quot;</td></tr><tr><td>&quot;ZN&quot;</td><td>&quot;Rates&quot;</td><td>88235</td><td>&quot;2011-01-02&quot;</td><td>&quot;2025-12-30&quot;</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (7, 5)\n",
"┌─────────┬──────────────┬───────┬────────────┬────────────┐\n",
"│ product ┆ asset_class ┆ rows ┆ start_date ┆ end_date │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ i64 ┆ str ┆ str │\n",
"╞═════════╪══════════════╪═══════╪════════════╪════════════╡\n",
"│ 6E ┆ FX ┆ 88326 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
"│ CL ┆ Energy ┆ 89383 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
"│ ES ┆ Equity Index ┆ 89031 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
"│ GC ┆ Metals ┆ 89424 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
"│ NQ ┆ Equity Index ┆ 89049 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
"│ ZC ┆ Grains ┆ 71426 ┆ 2011-01-03 ┆ 2025-12-30 │\n",
"│ ZN ┆ Rates ┆ 88235 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
"└─────────┴──────────────┴───────┴────────────┴────────────┘"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"key_products = [\"ES\", \"NQ\", \"CL\", \"GC\", \"ZN\", \"6E\", \"ZC\"]\n",
"coverage.filter(pl.col(\"product\").is_in(key_products))"
]
},
{
"cell_type": "markdown",
"id": "880232f1",
"metadata": {
"papermill": {
"duration": 0.001243,
"end_time": "2026-06-13T02:32:43.019253+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:43.018010+00:00",
"status": "completed"
}
},
"source": [
"## 4. Data Quality"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "9c22a3a9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:32:43.022480Z",
"iopub.status.busy": "2026-06-13T02:32:43.022386Z",
"iopub.status.idle": "2026-06-13T02:32:43.028338Z",
"shell.execute_reply": "2026-06-13T02:32:43.028002Z"
},
"papermill": {
"duration": 0.008384,
"end_time": "2026-06-13T02:32:43.028857+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:43.020473+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== OHLC Invariants (ES Continuous) ===\n",
" [OK] high_gte_low: 100.00%\n",
" [OK] high_gte_open: 100.00%\n",
" [OK] high_gte_close: 100.00%\n",
" [OK] low_lte_open: 100.00%\n",
" [OK] low_lte_close: 100.00%\n",
" [OK] volume_non_negative: 100.00%\n"
]
}
],
"source": [
"invariants = check_ohlc_invariants(es_continuous)\n",
"print(\"=== OHLC Invariants (ES Continuous) ===\")\n",
"for row in invariants.iter_rows(named=True):\n",
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")"
]
},
{
"cell_type": "markdown",
"id": "baee6bf4",
"metadata": {
"papermill": {
"duration": 0.001275,
"end_time": "2026-06-13T02:32:43.031631+00:00",
"exception": false,
"start_time": "2026-06-13T02:32:43.030356+00:00",
"status": "completed"
}
},
"source": [
"## Key Takeaways\n",
"\n",
"1. **30 products across 7 asset-class buckets**: FX (6), Grains (5),\n",
" Energy / Equity Index / Metals / Rates (4 each), Livestock (3).\n",
"2. **Hierarchy**: each product has 100+ individual contracts (194 for ES) and\n",
" one or more continuous series; downstream notebooks operate on the\n",
" continuous front month unless they specifically need contract-level data.\n",
"3. **Hourly granularity**, full coverage 2011-01-02 through 2025-12-30 for\n",
" products with the longest history. ES individual contract data starts\n",
" later (2016) because earlier contracts have already rolled off.\n",
"4. **Canonical schema**: `timestamp` for time and `product` for entity (CME's\n",
" contract identity is non-trivial — see also `instrument_id` for individual\n",
" contracts).\n",
"5. **OHLC invariants hold at 100% for ES continuous** — all six checks pass\n",
" on every observation.\n",
"\n",
"### Next Steps\n",
"\n",
"- **`05_futures_session_aggregation`**: Aligning hourly bars to CME trading\n",
" sessions.\n",
"- **`06_futures_continuous`**: Roll detection and the three adjustment\n",
" methods (ratio, difference, calendar).\n",
"- **Chapter 8**: Feature engineering on term structure and roll yield."
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "tags,-all"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.3"
},
"papermill": {
"default_parameters": {},
"duration": 5.126337,
"end_time": "2026-06-13T02:32:46.012766+00:00",
"environment_variables": {},
"exception": null,
"input_path": "02_financial_data_universe/04_cme_futures_eda.ipynb",
"output_path": "02_financial_data_universe/04_cme_futures_eda.ipynb",
"parameters": {},
"start_time": "2026-06-13T02:32:40.886429+00:00",
"version": "2.7.0"
},
"ml4t_provenance": {
"source_py_blob": "526df0bb2e3ca5cf445b7e70dafafc50f7895c6b",
"executed_at": "2026-06-13T02:32:46.227406+00:00",
"executor": "cpu-local",
"production": true,
"parameters": {},
"notes": "executed-state finalization batch (2026-06-13 run)"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02_financial_data_universe/04_cme_futures_eda.py
+243
View File
@@ -0,0 +1,243 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # CME Futures — Exploratory Data Analysis
#
# **Docker image**: `ml4t`
#
# **Purpose**: Profile the 30-product CME futures dataset (Databento, hourly,
# 2011–2025) and surface the contract / continuous structure that downstream
# notebooks rely on.
#
# **Learning objectives**:
#
# - Understand the futures data hierarchy: product → contract → continuous series.
# - Load individual contracts and continuous (rolled) series via
# `load_cme_futures` and inspect the canonical `timestamp` / `product`
# schema.
# - Summarize per-product coverage and group products by asset class.
# - Verify OHLC invariants on a representative continuous series.
#
# **Book reference**: §2.2 ("The Asset-Class Market Data Landscape" — Futures).
#
# **Prerequisites**: `data` package on `PYTHONPATH`; CME parquet present at
# `ML4T_DATA_PATH/futures/`. Run `python data/futures/market/download.py` if
# missing (Databento API key required).
# %%
"""CME Futures — Exploratory data analysis of the futures universe."""
import polars as pl
from data import list_cme_products, load_cme_futures
from utils.data_quality import check_ohlc_invariants
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
MAX_SYMBOLS = 0 # 0 = all
# %% [markdown]
# ## 1. Configuration and Data Discovery
#
# The futures data uses a Hive-partitioned structure for efficient queries:
# - `futures/continuous/product={PRODUCT}/`: Volume-rolled continuous contracts (hourly)
# - `futures/individual/{PRODUCT}/data.parquet`: Individual contract price data
#
# We use `load_cme_futures()` for proper data loading with partition pruning.
# %%
# Discover available products via the CME loader
products = list_cme_products()
print("=== Futures Universe ===")
print(f"Available products: {len(products)}")
print(f"\nProducts: {', '.join(products)}")
# %% [markdown]
# Map each product to a coarse asset-class bucket. The mapping covers every
# product in the dataset; downstream chapters use the same bucket labels for
# universe-construction and risk reporting.
# %%
ASSET_CLASS_MAP = {
"ES": "Equity Index",
"NQ": "Equity Index",
"YM": "Equity Index",
"RTY": "Equity Index",
"ZN": "Rates",
"ZB": "Rates",
"ZF": "Rates",
"ZT": "Rates",
"CL": "Energy",
"NG": "Energy",
"HO": "Energy",
"RB": "Energy",
"GC": "Metals",
"SI": "Metals",
"HG": "Metals",
"PL": "Metals",
"6E": "FX",
"6J": "FX",
"6B": "FX",
"6A": "FX",
"6C": "FX",
"6S": "FX",
"ZC": "Grains",
"ZS": "Grains",
"ZW": "Grains",
"ZM": "Grains",
"ZL": "Grains",
"LE": "Livestock",
"HE": "Livestock",
"GF": "Livestock",
}
# %%
# Count products by asset class
class_counts = (
pl.DataFrame({"product": products})
.with_columns(asset_class=pl.col("product").replace(ASSET_CLASS_MAP))
.group_by("asset_class")
.len()
.sort(["len", "asset_class"], descending=[True, False])
)
print("Products by Asset Class:")
class_counts
# %% [markdown]
# ## 2. Data Structure Example: E-mini S&P 500 (ES)
#
# ### Futures Key Hierarchy
#
# | Level | Example | Description |
# |-------|---------|-------------|
# | **Product** | ES | The underlying (E-mini S&P 500) |
# | **Contract** | ESH4 | Specific expiration (March 2024) |
# | **Continuous** | c0, c1 | Front month, first deferred |
# %%
es_individual = load_cme_futures(products=["ES"], continuous=False, frequency="hourly")
print("=== ES Individual Contracts ===")
print(f"Shape: {es_individual.shape}")
print(f"Columns: {es_individual.columns}")
print(f"Date range: {es_individual['timestamp'].min()} to {es_individual['timestamp'].max()}")
print(f"Unique contracts: {es_individual['instrument_id'].n_unique()}")
# %%
es_continuous = load_cme_futures(products=["ES"], tenors=[0], continuous=True, frequency="hourly")
print("=== ES Continuous Series (front month) ===")
print(f"Shape: {es_continuous.shape}")
print(f"Date range: {es_continuous['timestamp'].min()} to {es_continuous['timestamp'].max()}")
# %% [markdown]
# Each individual contract trades for a finite window before expiry. Aggregating
# by `instrument_id` shows the rollover pattern — quarterly contracts overlap
# during the roll period.
# %%
contract_stats = (
es_individual.group_by("instrument_id")
.agg(
pl.col("timestamp").min().alias("first_trade"),
pl.col("timestamp").max().alias("last_trade"),
pl.col("volume").sum().alias("total_volume"),
pl.len().alias("observations"),
)
.sort("first_trade")
)
print(f"Total ES contracts: {len(contract_stats)}")
print("Most recent 5 contracts:")
contract_stats.tail(5)
# %% [markdown]
# ## 3. Coverage Summary
#
# Check data availability across all products.
# %% [markdown]
# Summarize per-product coverage by loading the front-month continuous series
# (`tenor=0`) for every product and recording its row count and date range.
# %%
def get_product_coverage(product_list: list[str]) -> pl.DataFrame:
"""Summarize continuous series coverage for each product (front month)."""
summaries = []
for product in product_list:
df = load_cme_futures(products=[product], tenors=[0], continuous=True, frequency="hourly")
summaries.append(
{
"product": product,
"asset_class": ASSET_CLASS_MAP[product],
"rows": len(df),
"start_date": str(df["timestamp"].min())[:10],
"end_date": str(df["timestamp"].max())[:10],
}
)
return pl.DataFrame(summaries)
# %%
coverage = get_product_coverage(products)
print(f"Products with data: {len(coverage)} / {len(products)}")
print("Coverage by asset class:")
coverage.group_by("asset_class").len().sort(["len", "asset_class"], descending=[True, False])
# %% [markdown]
# A handful of representative products from each asset-class bucket:
# %%
key_products = ["ES", "NQ", "CL", "GC", "ZN", "6E", "ZC"]
coverage.filter(pl.col("product").is_in(key_products))
# %% [markdown]
# ## 4. Data Quality
# %%
invariants = check_ohlc_invariants(es_continuous)
print("=== OHLC Invariants (ES Continuous) ===")
for row in invariants.iter_rows(named=True):
status = "[OK]" if row["valid_pct"] >= 99.99 else "[WARN]"
print(f" {status} {row['check']}: {row['valid_pct']:.2f}%")
# %% [markdown]
# ## Key Takeaways
#
# 1. **30 products across 7 asset-class buckets**: FX (6), Grains (5),
# Energy / Equity Index / Metals / Rates (4 each), Livestock (3).
# 2. **Hierarchy**: each product has 100+ individual contracts (194 for ES) and
# one or more continuous series; downstream notebooks operate on the
# continuous front month unless they specifically need contract-level data.
# 3. **Hourly granularity**, full coverage 2011-01-02 through 2025-12-30 for
# products with the longest history. ES individual contract data starts
# later (2016) because earlier contracts have already rolled off.
# 4. **Canonical schema**: `timestamp` for time and `product` for entity (CME's
# contract identity is non-trivial — see also `instrument_id` for individual
# contracts).
# 5. **OHLC invariants hold at 100% for ES continuous** — all six checks pass
# on every observation.
#
# ### Next Steps
#
# - **`05_futures_session_aggregation`**: Aligning hourly bars to CME trading
# sessions.
# - **`06_futures_continuous`**: Roll detection and the three adjustment
# methods (ratio, difference, calendar).
# - **Chapter 8**: Feature engineering on term structure and roll yield.
02_financial_data_universe/05_futures_session_aggregation.ipynb
+1681
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/05_futures_session_aggregation.py
+536
View File
@@ -0,0 +1,536 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Futures Session Aggregation: Hourly to Daily
#
# **Docker image**: `ml4t`
#
# **Purpose**: Convert hourly continuous futures bars (Databento, UTC) to
# session-aware daily bars that respect the 4:00 PM Central Time CME session
# boundary, applying ratio back-adjustment to eliminate roll-induced price
# gaps.
#
# **Learning objectives**:
#
# - Understand why CME session dates differ from UTC calendar dates and how
# bars on Sunday evening belong to Monday's session.
# - Apply ratio (multiplicative) back-adjustment to a continuous series so
# percentage returns are preserved across rolls.
# - Aggregate hourly bars to session-correct daily OHLCV across all 30 products
# and three tenors (front month, first deferred, second deferred).
#
# **Book reference**: §2.2 ("The Asset-Class Market Data Landscape" — Futures);
# adjustment methodology compared in `06_futures_continuous`.
#
# **Prerequisites**: `data` package on `PYTHONPATH`; hourly continuous parquet
# present at `ML4T_DATA_PATH/futures/market/continuous/`. See
# [`06_futures_continuous`](06_futures_continuous.ipynb) for the teaching
# explanation of ratio vs Panama adjustment.
# %%
"""Session-aware aggregation of hourly futures to daily bars."""
import os
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import numpy as np
import polars as pl
from data import load_cme_futures
from utils import ML4T_DATA_PATH
from utils.paths import get_chapter_dir
# Output path for session-aggregated daily data. Default writes to a
# chapter-local directory; set WRITE_TO_DATA=1 to materialize the canonical
# daily parquet under ML4T_DATA_PATH for downstream notebooks.
WRITE_TO_DATA = os.environ.get("WRITE_TO_DATA", "0") == "1"
OUTPUT_DIR = (
ML4T_DATA_PATH / "futures" / "market" / "continuous" / "daily"
if WRITE_TO_DATA
else get_chapter_dir(2) / "output" / "futures_daily"
)
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ## 1. CME Session Boundaries
#
# ### Session Definition
#
# CME Globex sessions follow this schedule:
# - **Session Start**: Sunday 5:00 PM CT (for Monday session)
# - **Session End**: 4:00 PM CT (defines the session date)
# - **Daily Maintenance**: 4:00-5:00 PM CT (1-hour break)
#
# ### Why This Matters
#
# If we aggregate by calendar day (midnight UTC), we split a single trading
# session across two days, creating incorrect daily bars:
#
# | Approach | Sunday 11 PM UTC | Monday 3 PM UTC |
# |----------|------------------|-----------------|
# | **Calendar Day (Wrong)** | Sunday | Monday |
# | **CME Session (Correct)** | Monday | Monday |
#
# Both bars belong to Monday's session (which ends Monday 4 PM CT).
# %%
# Timezone constants
CT = ZoneInfo("America/Chicago")
UTC = ZoneInfo("UTC")
# CME session ends at 4 PM CT
SESSION_END_HOUR_CT = 16 # 4:00 PM
def assign_cme_session_date(ts: datetime) -> datetime:
"""
Assign CME session date to a UTC timestamp.
The session date is the date when the session ENDS (4 PM CT).
A bar at Sunday 11 PM UTC belongs to Monday's session.
CME closes Friday at 4 PM CT and reopens Sunday 5 PM CT.
Bars after Friday 4 PM CT still belong to Friday's session —
they must NOT roll to Saturday.
Args:
ts: UTC timestamp
Returns:
Session date (as date, no time component)
"""
# Convert to Central Time
ts_ct = ts.astimezone(CT)
# If we're past 4 PM CT, this belongs to tomorrow's session
if ts_ct.hour >= SESSION_END_HOUR_CT:
candidate = ts_ct.date() + timedelta(days=1)
# Friday after 4 PM CT → keep as Friday (no Saturday session)
# isoweekday: Mon=1, Fri=5, Sat=6
if candidate.isoweekday() == 6: # Saturday
candidate = ts_ct.date() # Keep as Friday
session_date = candidate
else:
session_date = ts_ct.date()
return session_date
# %%
# Quick test
test_times = [
datetime(2024, 1, 7, 23, 0, tzinfo=UTC), # Sunday 11 PM UTC = Sunday 5 PM CT -> Monday
datetime(2024, 1, 8, 15, 0, tzinfo=UTC), # Monday 3 PM UTC = Monday 9 AM CT -> Monday
datetime(2024, 1, 8, 22, 0, tzinfo=UTC), # Monday 10 PM UTC = Monday 4 PM CT -> Tuesday
datetime(
2024, 1, 12, 22, 0, tzinfo=UTC
), # Friday 10 PM UTC = Friday 4 PM CT -> Friday (NOT Saturday)
]
print("Session Assignment Examples:")
for ts in test_times:
ts_ct = ts.astimezone(CT)
session = assign_cme_session_date(ts)
print(f" {ts} ({ts_ct.strftime('%a %I:%M %p CT')}) -> Session: {session}")
# %% [markdown]
# ## 2. Load Hourly Continuous Data
#
# We load all products and tenors from the DataBento hourly data.
# %%
hourly = load_cme_futures(continuous=True, frequency="hourly")
products = sorted(hourly["product"].unique().to_list())
print(f"Loaded {len(hourly):,} hourly bars")
print(f"Products: {hourly['product'].n_unique()}")
print(f"Tenors: {sorted(hourly['tenor'].unique().to_list())}")
print(f"Date range: {hourly['timestamp'].min()} to {hourly['timestamp'].max()}")
print(f"Available products: {', '.join(products)}")
# %%
hourly.filter(pl.col("product") == "ES").select(
"timestamp", "product", "tenor", "open", "high", "low", "close", "volume"
).head(10)
# %% [markdown]
# ## 3. Assign Session Dates
#
# We add a `session_date` column using Polars expressions for efficiency.
# %%
# Vectorized session date assignment using Polars
# Convert to Central Time, then check if hour >= 16 (4 PM)
def add_session_date(df: pl.DataFrame) -> pl.DataFrame:
"""Add session_date column based on CME session boundaries.
Friday after 4 PM CT stays as Friday — CME has no Saturday session.
"""
return (
df.with_columns(pl.col("timestamp").dt.convert_time_zone("America/Chicago").alias("ts_ct"))
.with_columns(
pl.col("ts_ct").dt.date().alias("_ct_date"),
(pl.col("ts_ct").dt.hour() >= SESSION_END_HOUR_CT).alias("_after_close"),
# isoweekday: Mon=1 ... Fri=5, Sat=6, Sun=7
(pl.col("ts_ct").dt.weekday() == 5).alias("_is_friday"),
)
.with_columns(
# After 4 PM CT → next day, UNLESS it's Friday (no Saturday session)
pl.when(pl.col("_after_close") & ~pl.col("_is_friday"))
.then(pl.col("_ct_date") + pl.duration(days=1))
.otherwise(pl.col("_ct_date"))
.alias("session_date")
)
.drop("ts_ct", "_ct_date", "_after_close", "_is_friday")
)
# %%
hourly_with_sessions = add_session_date(hourly)
print("Session dates assigned (ES sample):")
hourly_with_sessions.filter(pl.col("product") == "ES").select(
"timestamp", "session_date", "product", "tenor", "close", "volume"
).head(15)
# %% [markdown]
# Walk a single calendar day for the ES front month: bars with `timestamp <
# 2024-01-08 22:00 UTC` carry session_date 2024-01-08; bars at or after 22:00
# UTC (= 16:00 CT, the close) carry session_date 2024-01-09.
# %%
(
hourly_with_sessions.filter(
(pl.col("product") == "ES")
& (pl.col("tenor") == 0)
& (pl.col("timestamp").dt.date() == pl.lit("2024-01-08").str.to_date())
)
.sort("timestamp")
.select("timestamp", "session_date", "close", "volume")
)
# %% [markdown]
# ## 3b. Ratio Back-Adjustment
#
# Databento's continuous contracts are **unadjusted** — price gaps at roll transitions
# produce spurious returns (e.g., ES Mar 2020: -11.08% artificial gap). We apply
# **ratio (multiplicative)** back-adjustment using `instrument_id` to detect roll points:
#
# 1. Detect where `instrument_id` changes between adjacent hourly bars
# 2. Compute ratio = new contract open / old contract close at each roll
# 3. Accumulate ratios backward (most recent prices stay unadjusted)
# 4. Multiply all OHLC prices by cumulative ratio
#
# Ratio adjustment preserves **percentage returns** (critical for IC, momentum features,
# and backtesting) unlike Panama (additive) which distorts returns for old data and can
# push prices negative for commodities with large cumulative adjustments.
#
# See [`06_futures_continuous`](06_futures_continuous.ipynb) for a teaching explanation of adjustment methods.
# %%
# Sort and detect roll transitions per (product, tenor)
hourly_sorted = hourly_with_sessions.sort(["product", "tenor", "timestamp"])
# Detect instrument_id changes within each (product, tenor) group
hourly_sorted = hourly_sorted.with_columns(
pl.col("instrument_id").shift(1).over("product", "tenor").alias("_prev_instrument_id"),
pl.col("close").shift(1).over("product", "tenor").alias("_prev_close"),
)
# Roll points: where instrument_id changes (excluding first row of each group)
rolls = hourly_sorted.filter(
pl.col("_prev_instrument_id").is_not_null()
& (pl.col("instrument_id") != pl.col("_prev_instrument_id"))
)
# Ratio = new contract's open / old contract's close (adjacent hourly bars)
roll_ratios = rolls.select(
"product",
"tenor",
"timestamp",
(pl.col("open") / pl.col("_prev_close")).alias("ratio"),
)
print(f"Roll transitions detected: {len(roll_ratios)}")
print(f"Products with rolls: {roll_ratios['product'].n_unique()}")
es_rolls = roll_ratios.filter(pl.col("product") == "ES").sort("timestamp")
print(f"ES roll ratios ({len(es_rolls)} rolls):")
es_rolls.select("timestamp", "ratio").head(10)
# %% [markdown]
# ### Ratio Back-Adjustment Function
#
# Walk backward through each (product, tenor) group, accumulating roll ratios to
# build a cumulative multiplier for all OHLC prices.
# %%
def ratio_adjust(group: pl.DataFrame) -> pl.DataFrame:
"""Apply ratio back-adjustment to a single (product, tenor) group."""
group = group.sort("timestamp")
# Get roll ratios for this group
group_rolls = roll_ratios.filter(
(pl.col("product") == group["product"][0]) & (pl.col("tenor") == group["tenor"][0])
).select("timestamp", "ratio")
if len(group_rolls) == 0:
return group.with_columns(pl.lit(1.0).alias("_cumulative_ratio"))
# Join roll ratios
group = group.join(group_rolls, on="timestamp", how="left").with_columns(
pl.col("ratio").fill_null(1.0)
)
# Cumulative ratio: product of all FUTURE ratios (reverse cumprod)
# Bars BEFORE a roll get multiplied; bars ON and AFTER the roll do not
n = len(group)
ratios = group["ratio"].to_numpy()
adj = np.ones(n)
cumulative = 1.0
for i in range(n - 1, -1, -1):
adj[i] = cumulative
if ratios[i] != 1.0:
cumulative *= ratios[i]
return group.with_columns(pl.Series("_cumulative_ratio", adj)).drop("ratio")
# %%
# Apply per group
adjusted_groups = []
products_tenors = hourly_sorted.select("product", "tenor").unique().sort("product", "tenor")
n_groups = len(products_tenors)
for i, row in enumerate(products_tenors.iter_rows(named=True)):
group = hourly_sorted.filter(
(pl.col("product") == row["product"]) & (pl.col("tenor") == row["tenor"])
)
adjusted = ratio_adjust(group)
adjusted_groups.append(adjusted)
if (i + 1) % 30 == 0 or i == n_groups - 1:
print(f" Adjusted {i + 1}/{n_groups} groups")
hourly_adjusted = pl.concat(adjusted_groups)
# Apply ratio adjustment to OHLC (multiply, not add)
hourly_adjusted = hourly_adjusted.with_columns(
(pl.col("open") * pl.col("_cumulative_ratio")).alias("open"),
(pl.col("high") * pl.col("_cumulative_ratio")).alias("high"),
(pl.col("low") * pl.col("_cumulative_ratio")).alias("low"),
(pl.col("close") * pl.col("_cumulative_ratio")).alias("close"),
)
print(f"\nRatio adjustment applied to {len(hourly_adjusted):,} hourly bars")
# Show adjustment magnitude for ES front month
es_adj = hourly_adjusted.filter((pl.col("product") == "ES") & (pl.col("tenor") == 0)).sort(
"timestamp"
)
print(
f"ES front month cumulative ratio range: "
f"{es_adj['_cumulative_ratio'].min():.4f} to {es_adj['_cumulative_ratio'].max():.4f}"
)
# Replace hourly_with_sessions with adjusted data for downstream aggregation
hourly_with_sessions = hourly_adjusted.drop(
"_prev_instrument_id", "_prev_close", "_cumulative_ratio"
)
# %% [markdown]
# ## 4. Aggregate to Daily OHLCV
#
# Aggregate hourly bars to daily using session boundaries:
# - **Open**: First bar's open (ratio-adjusted)
# - **High**: Maximum high (ratio-adjusted)
# - **Low**: Minimum low (ratio-adjusted)
# - **Close**: Last bar's close (ratio-adjusted)
# - **Volume**: Sum of all volumes
# %%
# Aggregate to daily by session_date, product, tenor
daily = (
hourly_with_sessions.sort(["product", "tenor", "timestamp"])
.group_by(["session_date", "product", "tenor"])
.agg(
[
pl.col("open").first(),
pl.col("high").max(),
pl.col("low").min(),
pl.col("close").last(),
pl.col("volume").sum(),
pl.len().alias("bar_count"),
pl.col("timestamp").min().alias("session_start"),
pl.col("timestamp").max().alias("session_end"),
]
)
.sort(["product", "tenor", "session_date"])
)
print(f"Daily bars: {len(daily):,}")
print(f"Products: {daily['product'].n_unique()}")
print(f"Session date range: {daily['session_date'].min()} to {daily['session_date'].max()}")
# %%
es_daily = daily.filter((pl.col("product") == "ES") & (pl.col("tenor") == 0))
print("ES front month daily bars (first 20 sessions):")
es_daily.select("session_date", "open", "high", "low", "close", "volume", "bar_count").head(20)
# %% [markdown]
# ## 5. Validate Aggregation
#
# Check that daily aggregation is correct:
# - Bar counts should be ~23 per session (23-hour trading day)
# - OHLC relationships should hold (Low ≤ Open/Close ≤ High)
# %%
bar_counts = daily.group_by("bar_count").len().sort("bar_count")
typical_sessions = daily.filter(pl.col("bar_count").is_between(20, 24))
print(f"Typical sessions (20-24 bars): {len(typical_sessions):,} / {len(daily):,}")
print("Bar counts per session:")
bar_counts
# %%
# OHLC invariant check
ohlc_check = daily.with_columns(
[
(pl.col("low") <= pl.col("open")).alias("low_le_open"),
(pl.col("low") <= pl.col("close")).alias("low_le_close"),
(pl.col("high") >= pl.col("open")).alias("high_ge_open"),
(pl.col("high") >= pl.col("close")).alias("high_ge_close"),
]
)
print("OHLC Invariant Check:")
for col in ["low_le_open", "low_le_close", "high_ge_open", "high_ge_close"]:
pct = ohlc_check[col].mean() * 100
status = "[OK]" if pct > 99.9 else "[FAIL]"
print(f" {status} {col}: {pct:.2f}%")
# %% [markdown]
# ## 6. Coverage Summary
#
# Summary of daily data coverage by product.
# %%
# Coverage by product
coverage = (
daily.group_by("product")
.agg(
[
pl.col("session_date").min().alias("start_date"),
pl.col("session_date").max().alias("end_date"),
pl.len().alias("total_bars"),
pl.col("tenor").n_unique().alias("tenors"),
]
)
.sort("product")
)
print("Daily data coverage by product:")
coverage
# %%
tenor_coverage = (
daily.group_by("tenor")
.agg(
pl.col("product").n_unique().alias("products"),
pl.len().alias("total_bars"),
)
.sort("tenor")
)
print("Coverage by tenor:")
tenor_coverage
# %% [markdown]
# ## 7. Save Daily Data
#
# Save the session-aggregated daily data for downstream use.
# %%
# Create output directory
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
# Save combined daily file
output_path = OUTPUT_DIR / "continuous_daily.parquet"
daily.write_parquet(output_path)
print(f"Saved: {output_path}")
print(f"Size: {output_path.stat().st_size / 1e6:.1f} MB")
# %%
# Also save per-product files for convenience
per_product_dir = OUTPUT_DIR / "by_product"
per_product_dir.mkdir(exist_ok=True)
for product in products:
product_df = daily.filter(pl.col("product") == product)
product_path = per_product_dir / f"{product}.parquet"
product_df.write_parquet(product_path)
print(f"\nSaved per-product files to: {per_product_dir}/")
print(f"Products: {len(products)}")
# %% [markdown]
# ## 8. Using the Daily Data
#
# The daily data is now available via `load_cme_futures()` (daily is the default frequency).
# This loader is defined in `data/__init__.py` and can be used by downstream chapters.
# %%
es_nq_2024 = (
pl.read_parquet(OUTPUT_DIR / "continuous_daily.parquet")
.filter(
pl.col("product").is_in(["ES", "NQ"])
& (pl.col("tenor") == 0)
& (pl.col("session_date") >= pl.lit("2024-01-01").str.to_date())
& (pl.col("session_date") <= pl.lit("2024-12-31").str.to_date())
)
.sort("session_date", "product")
)
print(f"ES + NQ front month, 2024: {len(es_nq_2024)} daily bars")
es_nq_2024.head(10)
# %% [markdown]
# ## Key Takeaways
#
# 1. **CME sessions end at 4 PM CT**, not midnight UTC. The session date is
# the date the session ends — Sunday-evening trading belongs to Monday's
# session.
# 2. **Volume here is 5,463,741 hourly bars across 30 products and 3 tenors**,
# aggregating to 312,859 daily bars over 2011-01-03 through 2025-12-31.
# 3. **Ratio back-adjustment** is applied per (product, tenor) before
# aggregation — for ES front month the cumulative ratio ranges 0.87–1.16
# over 427 rolls, preserving percentage returns across roll boundaries.
# 4. **Full sessions have 23 hourly bars** (23-hour trading day): the 23-bar
# bucket is by far the largest in the bar-count distribution. Shorter
# sessions arise from holidays, deferred tenors with thin trading, and
# partial days.
# 5. **OHLC invariants hold at 100%** on the aggregated daily bars across all
# four checks.
#
# ## Next Steps
#
# - [`06_futures_continuous`](06_futures_continuous.ipynb): Roll detection and
# alternative adjustment methods (Panama / calendar).
# - **Chapter 8**: Feature engineering on daily futures data.
# - **Chapter 16**: Backtesting with session-correct returns.
02_financial_data_universe/06_futures_continuous.ipynb
+10784
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/06_futures_continuous.py
+742
View File
@@ -0,0 +1,742 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.18.1
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Constructing Continuous Futures Contracts
#
# **Docker image**: `ml4t`
#
# **Purpose**: Walk through the construction of a continuous futures price
# series from individual expiring contracts: detect rolls, compare
# adjustment methods (raw, Panama / additive back-adjustment, ratio /
# multiplicative back-adjustment), and validate against the vendor-built
# continuous series.
#
# **Learning objectives**:
#
# - Detect roll dates using volume-based front-month identification (with a
# no-rollback constraint to avoid spurious switches).
# - Apply Panama (additive) back-adjustment to preserve dollar P&L across
# rolls.
# - Apply ratio (multiplicative) back-adjustment to preserve percentage
# returns across rolls.
# - Cross-check constructed continuous prices against Databento's pre-built
# continuous series and quantify the disagreement.
#
# **Book reference**: §2.2 ("The Asset-Class Market Data Landscape" —
# Futures); the methodology comparison underpins the engineering decision
# in §2.2 to store raw contract histories alongside one or more continuous
# variants.
#
# **Prerequisites**: `data` package on `PYTHONPATH`; individual ES contract
# parquet at `ML4T_DATA_PATH/futures/market/individual/ES/data.parquet` and
# the contract-definitions parquet at
# `ML4T_DATA_PATH/futures/market/contract_definitions.parquet`.
# %%
"""Continuous Futures Construction."""
import re
from datetime import UTC, date, datetime
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_cme_futures
from utils import ML4T_DATA_PATH
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ## 1. Understanding the Data
#
# ### 1.1 Load Individual Contracts
# %%
es_individual = load_cme_futures(products=["ES"], frequency="hourly", continuous=False)
print(f"Individual contracts: {es_individual.shape}")
print(f"Unique contracts (by instrument_id): {es_individual['instrument_id'].n_unique()}")
print(f"Date range: {es_individual['timestamp'].min()} to {es_individual['timestamp'].max()}")
print("Sample:")
es_individual.head()
# %%
contract_stats = (
es_individual.group_by("instrument_id")
.agg(
pl.col("timestamp").min().alias("first_trade"),
pl.col("timestamp").max().alias("last_trade"),
pl.col("volume").sum().alias("total_volume"),
pl.len().alias("trading_days"),
)
.sort("first_trade")
)
print(f"Contracts: {len(contract_stats)} (sorted by first trade)")
contract_stats.head(10)
# %% [markdown]
# ### 1.2 Understanding Contract Naming
#
# ES contract symbols follow the pattern: ES + Month Code + Year
#
# Month codes:
# - H = March, M = June, U = September, Z = December (standard quarterly)
# - F = January, G = February, J = April, K = May, N = July, Q = August, V = October, X = November
# %%
_MONTH_CODES = {
"F": 1,
"G": 2,
"H": 3,
"J": 4,
"K": 5,
"M": 6,
"N": 7,
"Q": 8,
"U": 9,
"V": 10,
"X": 11,
"Z": 12,
}
_SYMBOL_RE = re.compile(r"^([A-Z]+)([FGHJKMNQUVXZ])(\d+)$")
def parse_contract_symbol(symbol: str) -> dict:
"""Parse a futures contract symbol like ESH24 or RTYM25 into product / month / year."""
match = _SYMBOL_RE.match(symbol)
if not match:
raise ValueError(f"Cannot parse symbol: {symbol}")
product, month_code, year_str = match.groups()
year = int(year_str)
year = year + 2000 if year < 50 else year + 1900
return {
"product": product,
"month_code": month_code,
"month": _MONTH_CODES[month_code],
"year": year,
}
# %%
defn_path = ML4T_DATA_PATH / "futures" / "market" / "contract_definitions.parquet"
contract_defs = pl.read_parquet(defn_path).filter(pl.col("product") == "ES")
contract_df = (
pl.DataFrame(
[
{**parse_contract_symbol(r["symbol"]), "symbol": r["symbol"]}
for r in contract_defs.iter_rows(named=True)
]
)
.join(contract_defs.select("symbol", "expiration"), on="symbol")
.sort("year", "month")
)
print(f"ES contract definitions: {contract_df.height} contracts")
contract_df.select("symbol", "month_code", "month", "year", "expiration").head(10)
# %% [markdown]
# ### 1.3 Contract Expiration from Symbols
#
# Without a separate definitions file, we can derive expiration information
# from contract symbols. For ES contracts, the pattern is ESH24 (March 2024),
# ESM24 (June 2024), etc.
# %% [markdown]
# `parse_contract_symbol` and the contract-definitions parquet give us actual
# expiration dates. For products where we only see the symbol (no definitions
# file), the expiration can be approximated as the 15th of the contract month
# — close enough for roll detection but not for delivery scheduling.
# %%
es_definition = contract_df.select(
"symbol",
"year",
"month",
pl.struct("year", "month")
.map_elements(lambda x: date(x["year"], x["month"], 15), return_dtype=pl.Date)
.alias("expiration"),
)
print(f"ES definition rows: {es_definition.height}")
es_definition.head(10)
# %% [markdown]
# ## 2. Roll Detection
#
# The "roll" is when we switch from the near-month contract to the next contract.
# There are several approaches:
#
# 1. **Volume-based**: Roll when the next contract has higher daily volume
# 2. **Open Interest-based**: Roll when next contract has higher open interest
# 3. **Fixed Schedule**: Roll N days before expiration (e.g., first Thursday of expiry month)
#
# We'll implement volume-based rolling.
# %%
def identify_front_month(
individual_df: pl.DataFrame, min_outright_price: float = 500.0
) -> pl.DataFrame:
"""Volume-based front-month detection with no-rollback constraint."""
# Filter to outright contracts only (exclude calendar spreads at ~$50-100)
outrights = individual_df.filter(pl.col("close") >= min_outright_price)
# Aggregate to daily volume per contract
daily_volume = (
outrights.with_columns(pl.col("timestamp").dt.date().alias("date"))
.group_by(["date", "instrument_id"])
.agg(pl.col("volume").sum().alias("daily_volume"))
)
daily_leader = (
daily_volume.group_by("date")
.agg(pl.col("instrument_id").sort_by("daily_volume").last().alias("leader"))
.sort("date")
)
# No-rollback constraint — switch to new leaders, never go back
leader_ids = daily_leader["leader"].to_list()
dates = daily_leader["date"].to_list()
used_contracts = {leader_ids[0]}
current_front = leader_ids[0]
front = [current_front]
for i in range(1, len(leader_ids)):
if leader_ids[i] != current_front and leader_ids[i] not in used_contracts:
current_front = leader_ids[i]
used_contracts.add(current_front)
front.append(current_front)
daily_front = pl.DataFrame({"date": dates, "front_symbol": front})
# Expand back to hourly bars
hourly = individual_df.select("timestamp").unique().sort("timestamp")
hourly = hourly.with_columns(pl.col("timestamp").dt.date().alias("date"))
front_month = hourly.join(daily_front, on="date", how="left").drop("date")
front_month = front_month.with_columns(
pl.col("front_symbol").shift(1).alias("prev_front"),
).with_columns(
(pl.col("front_symbol") != pl.col("prev_front")).alias("is_roll"),
)
return front_month
# %%
front_months = identify_front_month(es_individual)
print("Front month identification (2024 sample):")
front_months.filter(pl.col("timestamp") >= datetime(2024, 1, 1, tzinfo=UTC)).head(20)
# %%
roll_dates = front_months.filter(pl.col("is_roll"))
print(f"Total roll events: {len(roll_dates)}")
print("Most recent 10 roll dates:")
roll_dates.tail(10).select("timestamp", "prev_front", "front_symbol")
# %% [markdown]
# ### 2.2 Calendar-Based Roll (Alternative)
#
# An alternative to volume-based rolling is **calendar-based**: roll a fixed number
# of days before contract expiration. This is simpler and more predictable, but may
# not track liquidity as well as volume-based methods.
#
# Common calendar roll schedules:
# - 5 business days before expiry (conservative)
# - First notice day (for physical delivery commodities)
# - 2 weeks before expiry (popular for equity index futures)
# %%
def identify_front_month_calendar(
individual_df: pl.DataFrame,
definition_df: pl.DataFrame,
roll_days_before: int = 5,
) -> pl.DataFrame:
"""Identify front month using calendar-based roll (fixed days before expiry)."""
# Get expiration dates from definitions
# NOTE: Requires individual data to have a "symbol" column with contract names
expirations = definition_df.select(["symbol", "expiration"]).with_columns(
pl.col("expiration").cast(pl.Date).alias("expiry_date")
)
# Join with individual data (requires symbol column)
with_expiry = individual_df.join(expirations, on="symbol", how="left").with_columns(
pl.col("timestamp").cast(pl.Date).alias("trade_date")
)
# Calculate days to expiry
with_expiry = with_expiry.with_columns(
(pl.col("expiry_date") - pl.col("trade_date")).dt.total_days().alias("days_to_expiry")
)
# Filter to contracts with more than roll_days_before to expiry
# Then select the nearest such contract for each day
front_month = (
with_expiry.filter(pl.col("days_to_expiry") > roll_days_before)
.sort(["timestamp", "days_to_expiry"])
.group_by("timestamp")
.first()
.select(["timestamp", pl.col("symbol").alias("front_symbol")])
.sort("timestamp")
)
# Add roll indicators
front_month = front_month.with_columns(
pl.col("front_symbol").shift(1).alias("prev_front"),
).with_columns(
(pl.col("front_symbol") != pl.col("prev_front")).alias("is_roll"),
)
return front_month
# %% [markdown]
# Calendar-based roll detection requires the individual data to carry a symbol
# column that joins to the contract-definitions table. The Databento individual
# parquet uses numeric `instrument_id` rather than ESH24-style symbols, so we
# present the calendar logic above as a teaching reference and use volume-based
# detection for the rest of the notebook.
# %%
volume_rolls = front_months.filter(pl.col("is_roll"))
print(f"Volume-based rolls (ES, 2016-2025): {len(volume_rolls)}")
# %% [markdown]
# **Volume vs Calendar Trade-offs**:
# - **Volume-based**: Follows liquidity naturally, but roll timing varies
# - **Calendar-based**: Predictable timing, easier to automate, but may roll into less liquid contract
#
# For this notebook, we use **volume-based** roll detection as our primary method since it
# better reflects actual market liquidity transitions.
# %% [markdown]
# ## 3. Adjustment Methods
#
# When we roll from contract A to contract B, there's usually a price gap.
# If we don't adjust, our time series will have artificial jumps.
#
# ### 3.1 No Adjustment (Raw)
#
# Simply use prices as-is. Returns calculated on roll dates are invalid.
# %%
def create_continuous_raw(individual_df: pl.DataFrame, front_months: pl.DataFrame) -> pl.DataFrame:
"""Create continuous series with no adjustment (raw prices)."""
# Join individual prices with front month info
continuous = (
individual_df.join(
front_months.select(["timestamp", "front_symbol"]), on="timestamp", how="inner"
)
.filter(pl.col("instrument_id") == pl.col("front_symbol"))
.select(["timestamp", "open", "high", "low", "close", "volume", "instrument_id"])
.sort("timestamp")
)
return continuous
# %%
es_continuous_raw = create_continuous_raw(es_individual, front_months)
print(f"Raw continuous series: {len(es_continuous_raw)} hourly bars")
es_continuous_raw.head(10)
# %% [markdown]
# ### 3.2 Panama (Back-Adjustment)
#
# Add the price gap to all historical prices. This preserves dollar P&L
# but distorts percentage returns for old data.
#
# Gap = Close_new_contract - Close_old_contract
# Adjusted_price = Price + cumulative_gap
#
# Note: We add (not subtract) because we're bringing old prices UP to the
# current contract's level, eliminating the discontinuity at roll dates.
# %%
def _compute_roll_gaps(individual_df: pl.DataFrame, front_months: pl.DataFrame) -> pl.DataFrame:
"""Compute price gaps at each roll date (new - old contract close)."""
roll_info = front_months.filter(pl.col("is_roll"))
prices_lookup = individual_df.select(["timestamp", "instrument_id", "close"])
old_prices = (
roll_info.select(["timestamp", pl.col("prev_front").alias("instrument_id")])
.join(prices_lookup, on=["timestamp", "instrument_id"], how="left")
.rename({"close": "old_close"})
)
new_prices = (
roll_info.select(["timestamp", pl.col("front_symbol").alias("instrument_id")])
.join(prices_lookup, on=["timestamp", "instrument_id"], how="left")
.rename({"close": "new_close"})
)
return (
old_prices.select(["timestamp", "old_close"])
.join(new_prices.select(["timestamp", "new_close"]), on="timestamp", how="inner")
.with_columns((pl.col("new_close") - pl.col("old_close")).alias("gap"))
.select(["timestamp", "gap"])
.drop_nulls()
)
# %% [markdown]
# ### Panama Adjustment
#
# Apply the computed gaps cumulatively backwards through the raw series.
# %%
def create_continuous_panama(
individual_df: pl.DataFrame, front_months: pl.DataFrame
) -> pl.DataFrame:
"""Create continuous series with Panama (back) adjustment.
Uses vectorized Polars joins instead of row-by-row iteration for O(n) complexity.
"""
raw = create_continuous_raw(individual_df, front_months)
roll_info = front_months.filter(pl.col("is_roll"))
if len(roll_info) == 0:
return raw.with_columns(pl.lit(0.0).alias("cumulative_adjustment"))
gaps_df = _compute_roll_gaps(individual_df, front_months)
if len(gaps_df) == 0:
return raw.with_columns(pl.lit(0.0).alias("cumulative_adjustment"))
# Adjustment applies to dates STRICTLY BEFORE each roll date
raw_with_gaps = raw.join(gaps_df, on="timestamp", how="left").with_columns(
pl.col("gap").fill_null(0.0)
)
# Cumulative sum in reverse, shift by 1 to exclude roll date from adjustment
raw_with_gaps = raw_with_gaps.with_columns(
pl.col("gap")
.reverse()
.cum_sum()
.shift(1)
.fill_null(0.0)
.reverse()
.alias("cumulative_adjustment")
)
adjusted = raw_with_gaps.with_columns(
[
(pl.col("open") + pl.col("cumulative_adjustment")).alias("adj_open"),
(pl.col("high") + pl.col("cumulative_adjustment")).alias("adj_high"),
(pl.col("low") + pl.col("cumulative_adjustment")).alias("adj_low"),
(pl.col("close") + pl.col("cumulative_adjustment")).alias("adj_close"),
]
)
return adjusted
# %%
es_continuous_panama = create_continuous_panama(es_individual, front_months)
panama_first = es_continuous_panama["cumulative_adjustment"][0]
print(
f"Panama-adjusted: cumulative_adjustment at the start of the series = {panama_first:+.2f} "
f"(adjusts every historical price up by this amount so the most recent contract is unchanged)"
)
es_continuous_panama.select(
"timestamp", "close", "adj_close", "cumulative_adjustment", "instrument_id"
).head(10)
# %% [markdown]
# ### 3.3 Ratio Adjustment
#
# Multiply historical prices by the ratio of new/old contract prices.
# This preserves percentage returns but distorts dollar amounts.
#
# Ratio = Close_new_contract / Close_old_contract
# Adjusted_price = Price * cumulative_ratio
# %%
def _compute_roll_ratios(individual_df: pl.DataFrame, front_months: pl.DataFrame) -> pl.DataFrame:
"""Compute price ratios (new/old) at each roll date."""
roll_info = front_months.filter(pl.col("is_roll"))
prices_lookup = individual_df.select(["timestamp", "instrument_id", "close"])
old_prices = (
roll_info.select(["timestamp", pl.col("prev_front").alias("instrument_id")])
.join(prices_lookup, on=["timestamp", "instrument_id"], how="left")
.rename({"close": "old_close"})
)
new_prices = (
roll_info.select(["timestamp", pl.col("front_symbol").alias("instrument_id")])
.join(prices_lookup, on=["timestamp", "instrument_id"], how="left")
.rename({"close": "new_close"})
)
return (
old_prices.select(["timestamp", "old_close"])
.join(new_prices.select(["timestamp", "new_close"]), on="timestamp", how="inner")
.filter(pl.col("old_close") != 0)
.with_columns((pl.col("new_close") / pl.col("old_close")).alias("ratio"))
.select(["timestamp", "ratio"])
.drop_nulls()
)
# %% [markdown]
# ### Ratio Adjustment
#
# Apply the computed ratios cumulatively backwards through the raw series.
# %%
def create_continuous_ratio(
individual_df: pl.DataFrame, front_months: pl.DataFrame
) -> pl.DataFrame:
"""Create continuous series with ratio adjustment.
Uses vectorized Polars joins instead of row-by-row iteration for O(n) complexity.
"""
raw = create_continuous_raw(individual_df, front_months)
roll_info = front_months.filter(pl.col("is_roll"))
if len(roll_info) == 0:
return raw.with_columns(pl.lit(1.0).alias("cumulative_ratio"))
ratios_df = _compute_roll_ratios(individual_df, front_months)
if len(ratios_df) == 0:
return raw.with_columns(pl.lit(1.0).alias("cumulative_ratio"))
# Adjustment applies to dates STRICTLY BEFORE each roll date
raw_with_ratios = raw.join(ratios_df, on="timestamp", how="left").with_columns(
pl.col("ratio").fill_null(1.0)
)
# Cumulative product in reverse, shift by 1 to exclude roll date
raw_with_ratios = raw_with_ratios.with_columns(
pl.col("ratio")
.reverse()
.cum_prod()
.shift(1)
.fill_null(1.0)
.reverse()
.alias("cumulative_ratio")
)
adjusted = raw_with_ratios.with_columns(
[
(pl.col("open") * pl.col("cumulative_ratio")).alias("adj_open"),
(pl.col("high") * pl.col("cumulative_ratio")).alias("adj_high"),
(pl.col("low") * pl.col("cumulative_ratio")).alias("adj_low"),
(pl.col("close") * pl.col("cumulative_ratio")).alias("adj_close"),
]
)
return adjusted
# %%
es_continuous_ratio = create_continuous_ratio(es_individual, front_months)
ratio_first = es_continuous_ratio["cumulative_ratio"][0]
print(
f"Ratio-adjusted: cumulative_ratio at the start of the series = {ratio_first:.4f} "
f"(historical prices are scaled up by this factor)"
)
es_continuous_ratio.select(
"timestamp", "close", "adj_close", "cumulative_ratio", "instrument_id"
).head(10)
# %% [markdown]
# ## 4. Validation
#
# Let's compare our construction to DataBento's pre-built continuous series.
#
# We compare our construction against DataBento's production continuous series.
# Both use volume-based roll detection, but the implementations differ in detail
# (daily aggregation window, exact crossover threshold, etc.), so some divergence
# on roll timing is expected — typically by a day or two around the roll date.
# %%
es_databento = load_cme_futures(products=["ES"], tenors=[0], frequency="hourly", continuous=True)
print(f"DataBento continuous: {es_databento.shape}")
es_databento.head()
# %%
comparison = (
es_continuous_raw.select("timestamp", pl.col("close").alias("our_close"))
.join(
es_databento.select("timestamp", pl.col("close").alias("databento_close")),
on="timestamp",
how="inner",
)
.with_columns(
(pl.col("our_close") - pl.col("databento_close")).alias("diff"),
)
)
mean_abs = comparison["diff"].abs().mean()
median_diff = comparison["diff"].median()
max_abs = comparison["diff"].abs().max()
large_diffs = comparison.filter(pl.col("diff").abs() > 1)
print(f"Hours compared: {len(comparison)}")
print(f"Mean absolute difference: ${mean_abs:.2f}")
print(f"Median signed difference: ${median_diff:+.2f}")
print(f"Max absolute difference: ${max_abs:.2f}")
print(
f"Hourly bars with >$1 difference: {len(large_diffs)} ({100 * len(large_diffs) / len(comparison):.1f}%)"
)
comparison.describe()
# %%
print("Sample of bars with large differences:")
large_diffs.head(10)
# %% [markdown]
# Most differences come from roll-timing disagreements — when our volume-based
# detector rolls a day earlier or later than Databento's, the two series report
# the price of a different contract for those hours, and the
# contango/backwardation spread between expiries produces a gap. The median
# signed difference is essentially zero, but mean absolute difference is on the
# order of tens of dollars, with occasional larger gaps around roll dates where
# the two algorithms disagree by more than a day.
# %% [markdown]
# ### 4.1 Visualize the Difference
# %%
# Plot both series
fig = make_subplots(
rows=2, cols=1, shared_xaxes=True, subplot_titles=("Price Comparison", "Difference")
)
comp_pd = comparison.to_pandas()
fig.add_trace(
go.Scatter(x=comp_pd["timestamp"], y=comp_pd["our_close"], name="Our Construction"),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(x=comp_pd["timestamp"], y=comp_pd["databento_close"], name="DataBento"), row=1, col=1
)
fig.add_trace(
go.Scatter(x=comp_pd["timestamp"], y=comp_pd["diff"], name="Difference"), row=2, col=1
)
fig.update_layout(height=600, title="ES Continuous: Our Construction vs DataBento")
fig.show()
# %% [markdown]
# ## 5. Construct + Validate Helper
#
# The construction logic is generic across products. The Databento subscription
# bundled with the book ships individual contract data for ES only; the other
# 29 products are delivered exclusively as pre-built continuous series. We
# therefore wrap the construction-plus-validation in a single function and
# apply it to ES — the only product where individual data is currently
# available on disk.
# %%
def construct_and_validate(product: str) -> dict:
"""Construct a raw continuous series and validate it against the vendor continuous."""
individual = load_cme_futures(products=[product], frequency="hourly", continuous=False)
fronts = identify_front_month(individual)
continuous_raw = create_continuous_raw(individual, fronts)
databento = load_cme_futures(
products=[product], tenors=[0], frequency="hourly", continuous=True
)
cmp = continuous_raw.select("timestamp", pl.col("close").alias("our_close")).join(
databento.select("timestamp", pl.col("close").alias("db_close")),
on="timestamp",
how="inner",
)
diff = (cmp["our_close"] - cmp["db_close"]).abs()
return {
"product": product,
"rows": len(continuous_raw),
"contracts_used": continuous_raw["instrument_id"].n_unique(),
"validation_rows": len(cmp),
"mean_abs_diff": float(diff.mean()),
"max_abs_diff": float(diff.max()),
}
# %%
validation_summary = pl.DataFrame([construct_and_validate("ES")])
print("Construction-vs-vendor validation (ES):")
validation_summary
# %% [markdown]
# ## 6. Production Pipeline
#
# The teaching examples above demonstrate roll detection and adjustment methods on a single product.
# For production use, the pipeline is:
#
# 1. **Download**: Databento provides pre-rolled continuous contracts (hourly OHLCV) for
# front, second, and third month tenors → `data/futures/market/continuous/hourly/`
# 2. **Session aggregation**: [`05_futures_session_aggregation`](05_futures_session_aggregation.ipynb) assigns CME session dates
# and aggregates hourly bars into daily OHLCV → `data/futures/market/continuous/daily/continuous_daily.parquet`
# 3. **Loading**: `load_cme_futures()` reads the daily parquet for downstream analysis
# %% [markdown]
# ---
#
# ## Key Takeaways
#
# 1. **Roll detection (volume-based, ES, 2016-2025)** finds **40 rolls** —
# matching the four quarterly rolls per year × 10 years. The
# no-rollback constraint is necessary because raw daily-volume leadership
# can flicker between contracts during the roll window.
# 2. **Calendar spreads contaminate raw individual data**: CME ships outright
# contracts alongside calendar spreads that trade at the inter-month
# price difference (~$50–100) rather than the index level (~$5,000). The
# `min_outright_price` filter in `identify_front_month` is what prevents
# a high-volume spread from being selected as "front month".
# 3. **Panama (additive) adjustment** for the ES series adds about
# **$625 to the earliest 2016 prices** (so the start-of-history close is
# ~30% above the original quote). This preserves dollar P&L across rolls
# but distorts percentage returns the further back you go.
# 4. **Ratio (multiplicative) adjustment** for the same window has a
# **cumulative ratio of ~1.11 at the start of the series** (about
# +11% scaling). Returns stay correct in percentage terms across the
# whole window — the right choice for IC, momentum features, and any
# statistical analysis.
# 5. **Validation against Databento's continuous** (2,581 daily-aligned bars):
# **mean absolute difference ~$27**, **median signed difference ~$2.50**
# (essentially zero relative to ~$3,800 average price). 2,444 of those
# bars differ by more than $1 (most by a few dollars; **maximum absolute
# gap ~$583**). Differences come almost entirely from roll-timing
# disagreements — when our detector rolls a day earlier or later than the
# vendor's algorithm, the two series report the price of a different
# contract for those hours. The signed median near zero means the
# disagreements wash out: neither algorithm is systematically high or
# low.
#
# ### Adjustment Method Selection
# | Use Case | Recommended Method | Reason |
# |----------|-------------------|--------|
# | Backtesting P&L | Panama (additive) | Preserves dollar gains/losses across rolls |
# | Statistical analysis | Ratio | Preserves percentage returns accurately |
# | Live trading | Raw + position management | Handle rolls in execution layer |
#
# ### Next Steps
#
# - **Chapter 8**: Carry and momentum features built on continuous series.
# - **Chapter 16**: Backtesting with adjusted P&L.
02_financial_data_universe/07_sp500_options_eda.ipynb
+15080
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/07_sp500_options_eda.py
+965
View File
@@ -0,0 +1,965 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # S&P 500 Options Analytics
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Profile the AlgoSeek S&P 500 options analytics dataset for an 8-symbol 2020 EDA
# slice — chain structure, implied-volatility surfaces, data quality, and an early
# look at the predictive content that motivates the options case studies.
#
# ## Learning Objectives
#
# - Read an option chain and locate strikes, expirations, and call/put pairs.
# - Construct and visualize an implied-volatility smile, term structure, and surface.
# - Apply IV-convergence and Greeks-validity filters to clean options data.
# - Quantify a baseline IV-change → forward-return relationship in the cross section.
#
# ## Book Reference
#
# Chapter 2 §2.2 (asset-class market data landscape — derivatives).
#
# ## Prerequisites
#
# - Familiarity with daily OHLC equity data (`01_us_equities_eda`).
# - The AlgoSeek S&P 500 options EDA parquet at `$ML4T_DATA_PATH/sp500_options/`
# (8 representative underlyings: AAPL, AMZN, BA, GOOGL, JPM, KO, MSFT, XOM).
# - The S&P 500 daily-bar parquet covering the same 2020 window.
#
# Loaders used:
#
# | Dataset | Loader | Coverage |
# |---------|--------|----------|
# | S&P 500 options (EDA slice) | `load_sp500_options_eda()` | 2020, 8 underlyings |
# | S&P 500 daily prices | `load_sp500_daily_bars()` | 2020, same 8 underlyings |
# %%
"""S&P 500 Options Analytics — options chain structure, volatility surfaces, and data quality."""
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_sp500_daily_bars, load_sp500_options_eda
# %% tags=["parameters"]
DAILY_START_DATE = "2020-01-01"
# %% [markdown]
# ## 1. Options Primer for ML Practitioners
#
# Before diving into the data, let's establish the key concepts that make options
# data different from—and complementary to—spot market data.
#
# ### 1.1 What is an Option?
#
# An option is a **derivative contract** that gives the holder the right (but not
# obligation) to buy or sell an underlying asset at a specified price (strike)
# by a specified date (expiration).
#
# | Type | Right | Profitable When |
# |------|-------|-----------------|
# | **Call** | Buy at strike | Underlying rises above strike |
# | **Put** | Sell at strike | Underlying falls below strike |
#
# ### 1.2 Why Options Data Matters for ML
#
# Options prices embed **forward-looking information** that spot prices don't:
#
# 1. **Implied Volatility (IV)** - Market's expectation of future volatility
# 2. **IV Skew** - Relative demand for downside vs upside protection
# 3. **Term Structure** - How expectations change across time horizons
# 4. **Greeks** - Sensitivities that quantify risk exposures
#
# This information can predict:
# - Future realized volatility
# - Underlying price movements (via order flow/positioning)
# - Tail risk events (via skew)
#
# ### 1.3 Moneyness: ITM, ATM, OTM
#
# Moneyness describes how an option's strike relates to the current spot price:
#
# | Moneyness | Call (Strike vs Spot) | Put (Strike vs Spot) | Characteristics |
# |-----------|----------------------|---------------------|-----------------|
# | **ITM** (In-the-money) | Strike < Spot | Strike > Spot | Has intrinsic value |
# | **ATM** (At-the-money) | Strike ≈ Spot | Strike ≈ Spot | Highest time value |
# | **OTM** (Out-of-the-money) | Strike > Spot | Strike < Spot | Pure time value |
#
# We typically express moneyness as: **Strike / Spot** (or its log)
# - Moneyness = 1.0 → ATM
# - Moneyness < 1.0 → ITM call / OTM put
# - Moneyness > 1.0 → OTM call / ITM put
#
# ### 1.4 Option Value Components
#
# An option's price decomposes into intrinsic value (immediate exercise payoff) and
# time value (the remainder, reflecting optionality):
#
# $$\text{Price} = \text{Intrinsic} + \text{Time}$$
#
# $$\text{Intrinsic}_{\text{call}} = \max(0,\; S - K), \qquad \text{Intrinsic}_{\text{put}} = \max(0,\; K - S)$$
#
# $$\text{Time} = \text{Price} - \text{Intrinsic}$$
#
# Time value reflects:
# - Time remaining until expiration
# - Expected volatility (IV)
# - Interest rates and dividends
# %% [markdown]
# ## 2. Dataset Overview
#
# ### 2.1 Data Schema
#
# | Field | Type | Description |
# |-------|------|-------------|
# | **Identifiers** | | |
# | `timestamp` | Date | Trading date (observation date, EOD snapshot) |
# | `symbol` | String | Underlying ticker (e.g., "AAPL", "MSFT") |
# | `expiration` | Date | Option expiration date |
# | `strike` | Float64 | Strike price in USD |
# | `call_put` | String | "C" for call, "P" for put |
# | **Prices** | | |
# | `bid` | Float64 | Best bid price at close |
# | `ask` | Float64 | Best ask price at close |
# | `mid_price` | Float64 | Mid-market price: (bid + ask) / 2 |
# | `underlying_price` | Float64 | Underlying stock close price |
# | **Time** | | |
# | `days_to_maturity` | Int32 | Calendar days until expiration |
# | **Greeks** | | |
# | `delta` | Float64 | ∂V/∂S - Price sensitivity to underlying |
# | `gamma` | Float64 | ∂²V/∂S² - Delta sensitivity to underlying |
# | `theta` | Float64 | ∂V/∂t - Time decay ($/day, typically negative) |
# | `vega` | Float64 | ∂V/∂σ - Sensitivity to volatility |
# | `rho` | Float64 | ∂V/∂r - Sensitivity to interest rates |
# | **Volatility** | | |
# | `implied_vol` | Float64 | Black-Scholes implied volatility |
# | `iv_convergence` | String | IV solver status (quality indicator) |
#
# ### 2.2 IV Convergence Codes
#
# The `iv_convergence` field indicates IV computation quality:
#
# | Code | Meaning | Use in Analysis |
# |------|---------|-----------------|
# | `Converged` | IV solver converged normally | [OK] Highest quality |
# | `SmallBid_FlatExtrapol` | Small bid, IV extrapolated | WARNING: Use with caution |
# | `IntrVal_FlatExtrapol` | Deep ITM, IV extrapolated | WARNING: Use with caution |
# | `IntrVal_PutCallPair` | IV from put-call parity | [OK] Usually reliable |
# | `Failed` | IV solver did not converge | [FAIL] Exclude from analysis |
#
# **Best practice**: Filter for `iv_convergence == "Converged"` for clean analysis.
# %%
options = load_sp500_options_eda(
start_date="2020-01-01",
end_date="2020-12-31",
include_greeks=True,
)
print("=== S&P 500 Options Dataset ===")
print(f"Total rows: {len(options):,}")
print(f"Columns: {len(options.columns)}")
print(f"Date range: {options['timestamp'].min()} to {options['timestamp'].max()}")
print(f"Underlyings: {sorted(options['symbol'].unique().to_list())}")
# %%
daily = load_sp500_daily_bars(
symbols=sorted(options["symbol"].unique().to_list()),
start_date=DAILY_START_DATE,
end_date="2020-12-31",
)
print("\n=== S&P 500 Daily Prices ===")
print(f"Total rows: {len(daily):,}")
print(f"Symbols: {daily['symbol'].n_unique()}")
print(f"Date range: {daily['timestamp'].min()} to {daily['timestamp'].max()}")
# %%
# Quick schema preview
print("\n=== Options Schema ===")
options.head(3)
# %% [markdown]
# ## 3. Option Chain Structure
#
# An **option chain** is the full set of options available for one underlying on one day.
# It spans multiple dimensions:
# - **Strikes**: Many price levels around the current spot
# - **Expirations**: Multiple dates from days to years out
# - **Types**: Calls and puts at each strike/expiration
#
# This creates a 3D grid: `(strike × expiration × call_put)`
# %%
# Options per symbol per day - how dense are the chains?
options_per_symbol = options.group_by(["timestamp", "symbol"]).agg(
[
pl.len().alias("n_options"),
(pl.col("call_put") == "C").sum().alias("n_calls"),
(pl.col("call_put") == "P").sum().alias("n_puts"),
pl.col("strike").n_unique().alias("n_strikes"),
pl.col("expiration").n_unique().alias("n_expirations"),
]
)
print("=== Option Chain Density (per symbol per day) ===")
options_per_symbol.select(["n_options", "n_strikes", "n_expirations"]).describe()
# %%
# Visualize: Distribution of chain sizes
fig = px.histogram(
options_per_symbol.to_pandas(),
x="n_options",
nbins=50,
title="Option Chain Size Distribution",
labels={"n_options": "Number of Options per Symbol/Day", "count": "Frequency"},
)
median_options = float(options_per_symbol["n_options"].median())
fig.add_vline(x=median_options, line_dash="dash", line_color="red")
fig.add_annotation(x=median_options, y=0.95, yref="paper", text="Median", showarrow=False)
fig.update_layout(showlegend=False)
fig.show()
# %% [markdown]
# ### 3.1 Single Symbol Deep Dive: AAPL
#
# Let's examine one complete option chain to understand the structure.
# %%
# AAPL on a specific date
sample_date = options["timestamp"].max()
aapl_day = options.filter((pl.col("symbol") == "AAPL") & (pl.col("timestamp") == sample_date))
spot = aapl_day["underlying_price"][0]
print(f"=== AAPL Option Chain ({sample_date}) ===")
print(f"Underlying price: ${spot:.2f}")
print(f"Total options: {len(aapl_day):,}")
print(f" Calls: {aapl_day.filter(pl.col('call_put') == 'C').height:,}")
print(f" Puts: {aapl_day.filter(pl.col('call_put') == 'P').height:,}")
print(f"Expirations: {aapl_day['expiration'].n_unique()}")
print(f"Strikes: {aapl_day['strike'].n_unique()}")
print(f"Strike range: ${aapl_day['strike'].min():.2f} - ${aapl_day['strike'].max():.2f}")
# %%
# Expiration breakdown
exp_breakdown = (
aapl_day.group_by("expiration")
.agg([pl.len().alias("n_options"), pl.col("strike").n_unique().alias("n_strikes")])
.sort("expiration")
)
print("\n=== AAPL Expirations ===")
exp_breakdown.head(10)
# %% [markdown]
# ### 3.2 Option Chain Heatmap
#
# Visualize the entire chain as a heatmap: strikes on y-axis, expirations on x-axis,
# colored by implied volatility. This reveals the **volatility surface** structure.
# %%
# Prepare data for heatmap - calls only, converged IV, reasonable moneyness
aapl_calls = (
aapl_day.filter(
(pl.col("call_put") == "C")
& (pl.col("iv_convergence") == "Converged")
& (pl.col("implied_vol") > 0)
& (pl.col("implied_vol") < 2.0) # Filter outliers
)
.with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
.filter(pl.col("moneyness").is_between(0.7, 1.3)) # Focus on tradeable range
)
# Create pivot for heatmap
heatmap_data = (
aapl_calls.select(["strike", "expiration", "implied_vol"])
.sort(["expiration", "strike"])
.to_pandas()
.pivot(index="strike", columns="expiration", values="implied_vol")
)
# %%
fig = go.Figure(
data=go.Heatmap(
z=heatmap_data.values,
x=[str(c) for c in heatmap_data.columns],
y=heatmap_data.index,
colorscale="Viridis",
colorbar=dict(title="IV"),
)
)
spot_float = float(spot)
fig.add_hline(y=spot_float, line_dash="dash", line_color="white")
fig.add_annotation(
y=spot_float, x=0.95, xref="paper", text=f"Spot: ${spot_float:.0f}", showarrow=False
)
fig.update_layout(
title=f"AAPL Option Chain - Implied Volatility Surface ({sample_date})",
xaxis_title="Expiration",
yaxis_title="Strike ($)",
height=600,
)
fig.show()
# %% [markdown]
# **Reading the heatmap:**
# - Horizontal slice at one strike → Term structure (how IV varies with expiration)
# - Vertical slice at one expiration → IV smile/skew (how IV varies with strike)
# - Darker colors (lower IV) typically at ATM; lighter (higher IV) at wings
# %% [markdown]
# ## 4. Volatility Surface Analysis
#
# The **implied volatility surface** is the core representation for options analytics.
# It captures how IV varies across two dimensions:
# 1. **Moneyness** (strike relative to spot) → IV smile/skew
# 2. **Time to expiration** → IV term structure
#
# ### 4.1 IV Smile and Skew
# %%
# IV smile for nearest expiration
nearest_exp = aapl_calls["expiration"].min()
aapl_smile = aapl_calls.filter(pl.col("expiration") == nearest_exp).sort("strike")
fig = px.scatter(
aapl_smile.to_pandas(),
x="moneyness",
y="implied_vol",
title=f"AAPL IV Smile - Nearest Expiration ({nearest_exp})",
labels={"moneyness": "Moneyness (Strike/Spot)", "implied_vol": "Implied Volatility"},
trendline="lowess",
)
fig.add_vline(x=1.0, line_dash="dash", line_color="gray")
fig.add_annotation(x=1.0, y=0.95, yref="paper", text="ATM", showarrow=False)
fig.show()
# %% [markdown]
# **IV Smile/Skew interpretation:**
# - **Smile**: IV higher for both OTM puts (left) and OTM calls (right) vs ATM
# - **Skew**: Asymmetric - OTM puts typically have higher IV than OTM calls
# - **Why?** Demand for downside protection (crash insurance) exceeds upside speculation
# %%
# Compare smile across multiple expirations
expirations = sorted(aapl_calls["expiration"].unique().to_list())[:4] # First 4
smile_data = aapl_calls.filter(pl.col("expiration").is_in(expirations))
fig = px.scatter(
smile_data.to_pandas(),
x="moneyness",
y="implied_vol",
color="expiration",
title="AAPL IV Smile Across Expirations",
labels={"moneyness": "Moneyness", "implied_vol": "Implied Volatility"},
)
fig.add_vline(x=1.0, line_dash="dash", line_color="gray")
fig.show()
# %% [markdown]
# ### 4.2 IV Term Structure
#
# How does ATM IV vary across expirations?
# %%
# ATM IV term structure (moneyness 0.98-1.02)
atm_term = (
aapl_calls.filter(pl.col("moneyness").is_between(0.98, 1.02))
.group_by("expiration")
.agg(
[
pl.col("implied_vol").mean().alias("iv_atm"),
pl.col("days_to_maturity").first().alias("days"),
]
)
.sort("expiration")
)
fig = px.line(
atm_term.to_pandas(),
x="days",
y="iv_atm",
markers=True,
title=f"AAPL ATM IV Term Structure ({sample_date})",
labels={"days": "Days to Expiration", "iv_atm": "ATM Implied Volatility"},
)
fig.show()
# %% [markdown]
# **Term structure shapes:**
# - **Contango** (upward sloping): Near-term calm, uncertainty further out
# - **Backwardation** (downward sloping): Near-term stress/event expected
# - **Flat**: Consistent expectations across horizons
# %% [markdown]
# ### 4.3 3D Volatility Surface
#
# Combine moneyness and time dimensions into a single surface visualization.
# %%
# Prepare surface data
surface_data = (
aapl_calls.filter(pl.col("days_to_maturity") <= 180) # Focus on <6 months
.select(["moneyness", "days_to_maturity", "implied_vol"])
.to_pandas()
)
# Create 3D surface
fig = go.Figure(
data=[
go.Mesh3d(
x=surface_data["moneyness"],
y=surface_data["days_to_maturity"],
z=surface_data["implied_vol"],
intensity=surface_data["implied_vol"],
colorscale="Viridis",
opacity=0.7,
)
]
)
fig.update_layout(
title=f"AAPL 3D Volatility Surface ({sample_date})",
scene=dict(
xaxis_title="Moneyness",
yaxis_title="Days to Expiration",
zaxis_title="Implied Volatility",
),
height=600,
)
fig.show()
# %% [markdown]
# ## 5. Cross-Sectional Analysis
#
# How does ATM IV differ across the eight underlyings on a single day? The same
# logic scales to the full S&P 500 universe — here we keep the comparison
# tractable on the EDA slice.
# %%
# Compute ATM IV for all symbols on sample date
converged = options.filter(pl.col("iv_convergence") == "Converged")
cross_section = (
converged.filter(pl.col("timestamp") == sample_date)
.with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
.filter(pl.col("moneyness").is_between(0.98, 1.02))
.filter(pl.col("call_put") == "C")
.group_by("symbol")
.agg(
[
pl.col("implied_vol").mean().alias("iv_atm"),
pl.col("underlying_price").first().alias("price"),
]
)
.sort("iv_atm", descending=True)
)
print(f"=== Cross-Sectional ATM IV ({sample_date}) ===")
print(f"Symbols: {len(cross_section)}")
print(f"IV range: {cross_section['iv_atm'].min():.1%} - {cross_section['iv_atm'].max():.1%}")
print(f"IV median: {cross_section['iv_atm'].median():.1%}")
# %%
# Highest IV names
print("\n=== Highest IV Names ===")
cross_section.head(10)
# %%
# Lowest IV names
print("\n=== Lowest IV Names ===")
cross_section.tail(10)
# %%
# IV distribution across universe
fig = px.histogram(
cross_section.to_pandas(),
x="iv_atm",
nbins=40,
title=f"Cross-Sectional ATM IV Distribution ({sample_date})",
labels={"iv_atm": "ATM Implied Volatility", "count": "Number of Symbols"},
)
median_iv = float(cross_section["iv_atm"].median())
fig.add_vline(x=median_iv, line_dash="dash", line_color="red")
fig.add_annotation(x=median_iv, y=0.95, yref="paper", text="Median", showarrow=False)
fig.show()
# %% [markdown]
# ## 6. Time Series Analysis
#
# How does IV evolve over time? The year 2020 provides an excellent case study
# with the COVID crash in March.
# %%
# Daily aggregate IV statistics
daily_iv = (
converged.with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
.filter(pl.col("moneyness").is_between(0.98, 1.02))
.filter(pl.col("call_put") == "C")
.group_by("timestamp")
.agg(
[
pl.col("implied_vol").mean().alias("iv_mean"),
pl.col("implied_vol").median().alias("iv_median"),
pl.col("implied_vol").quantile(0.1).alias("iv_p10"),
pl.col("implied_vol").quantile(0.9).alias("iv_p90"),
pl.col("symbol").n_unique().alias("n_symbols"),
]
)
.sort("timestamp")
)
# %%
# Build the full IV-evolution figure in one cell — splitting the figure across
# two cells produced an intermediate render with no title or axis labels.
fig = go.Figure()
fig.add_trace(
go.Scatter(
x=daily_iv["timestamp"].to_list(),
y=daily_iv["iv_p90"].to_list(),
fill=None,
mode="lines",
line_color="lightblue",
name="P90",
)
)
fig.add_trace(
go.Scatter(
x=daily_iv["timestamp"].to_list(),
y=daily_iv["iv_p10"].to_list(),
fill="tonexty",
mode="lines",
line_color="lightblue",
name="P10-P90 Range",
)
)
fig.add_trace(
go.Scatter(
x=daily_iv["timestamp"].to_list(),
y=daily_iv["iv_median"].to_list(),
mode="lines",
line_color="darkblue",
line_width=2,
name="Median IV",
)
)
fig.add_vline(x="2020-03-16", line_dash="dash", line_color="red")
fig.add_vline(x="2020-03-23", line_dash="dash", line_color="green")
fig.add_annotation(x="2020-03-16", y=0.95, yref="paper", text="COVID Low", showarrow=False)
fig.add_annotation(x="2020-03-23", y=0.90, yref="paper", text="Market Bottom", showarrow=False)
fig.update_layout(
title="S&P 500 Universe ATM IV Evolution (2020)",
xaxis_title="Date",
yaxis_title="Implied Volatility",
height=500,
)
fig.show()
# %% [markdown]
# **Key observations:**
# - The 90th-percentile ATM IV crossed 100% in mid-March 2020 (peak P90 ≈ 115% on
# 2020-03-16); cross-sectional median ATM IV peaked near 70%.
# - IV stayed materially above pre-crash levels well past Q1 — slow mean reversion
# even as spot recovered.
# - The "fear gauge" aspect of IV — sharp spike, slow decay — is visible even on
# this eight-symbol slice.
# %% [markdown]
# ## 7. Execution Cost Proxy: Bid-Ask Spreads
#
# Since we don't have volume or open interest, bid-ask spread serves as our
# primary liquidity/execution cost indicator.
# %%
# Compute spread metrics
spread_analysis = (
converged.with_columns(
[
(pl.col("ask") - pl.col("bid")).alias("spread_abs"),
((pl.col("ask") - pl.col("bid")) / pl.col("mid_price")).alias("spread_pct"),
(pl.col("strike") / pl.col("underlying_price")).alias("moneyness"),
]
)
.filter(pl.col("mid_price") > 0.10) # Filter penny options
.filter(pl.col("spread_pct") < 2.0) # Filter outliers
)
print("=== Bid-Ask Spread Statistics ===")
spread_analysis.select(["spread_abs", "spread_pct"]).describe()
# %%
# Spread by moneyness
spread_by_moneyness = (
spread_analysis.filter(pl.col("moneyness").is_between(0.8, 1.2))
.with_columns((pl.col("moneyness") * 20).round() / 20) # Bucket to 5% increments
.group_by("moneyness")
.agg([pl.col("spread_pct").median().alias("median_spread")])
.sort("moneyness")
)
fig = px.bar(
spread_by_moneyness.to_pandas(),
x="moneyness",
y="median_spread",
title="Median Bid-Ask Spread by Moneyness",
labels={"moneyness": "Moneyness", "median_spread": "Median Spread (%)"},
)
fig.add_vline(x=1.0, line_dash="dash", line_color="gray")
fig.add_annotation(x=1.0, y=0.95, yref="paper", text="ATM", showarrow=False)
fig.show()
# %% [markdown]
# **Spread observations:**
# - ATM options have tightest spreads (most liquid)
# - Spreads widen for OTM options (less liquid)
# - Deep OTM options can have very wide spreads (>50%)
# - **Implication**: Focus on near-ATM for tradeable strategies
# %% [markdown]
# ## 8. Data Quality Assessment
#
# ### 8.1 IV Convergence Rates
# %%
# Convergence statistics
convergence_stats = (
options.group_by("iv_convergence")
.len()
.with_columns((pl.col("len") / pl.sum("len") * 100).alias("pct"))
.sort("len", descending=True)
)
print("=== IV Convergence Status ===")
for row in convergence_stats.iter_rows(named=True):
print(f" {row['iv_convergence']}: {row['len']:,} ({row['pct']:.2f}%)")
# %%
# Horizontal bar chart instead of a pie: 10 convergence categories make a pie
# unreadable (tiny slices overlap their labels). The bar chart sorts by share
# and keeps every label legible.
convergence_pd = convergence_stats.to_pandas().sort_values("pct", ascending=True)
fig = go.Figure(
data=go.Bar(
x=convergence_pd["pct"],
y=convergence_pd["iv_convergence"],
orientation="h",
text=[f"{v:.2f}%" for v in convergence_pd["pct"]],
textposition="outside",
marker_color="#4C72B0",
)
)
fig.update_layout(
title="IV Convergence Status Distribution",
xaxis_title="Share of rows (%)",
yaxis_title="Convergence status",
template="plotly_white",
height=460,
margin=dict(l=170, r=100),
xaxis=dict(range=[0, max(convergence_pd["pct"]) * 1.15]),
)
fig.show()
# %% [markdown]
# ### 8.2 Coverage Analysis
# %%
# Daily symbol coverage
daily_coverage = (
converged.group_by("timestamp")
.agg(
[
pl.col("symbol").n_unique().alias("n_symbols"),
pl.len().alias("n_options"),
]
)
.sort("timestamp")
)
print("=== Daily Coverage (Converged Options) ===")
print(f"Mean symbols/day: {daily_coverage['n_symbols'].mean():.0f}")
print(f"Min symbols/day: {daily_coverage['n_symbols'].min()}")
print(f"Max symbols/day: {daily_coverage['n_symbols'].max()}")
# %%
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
subplot_titles=("Symbols with Converged Options", "Total Converged Options"),
)
coverage_pd = daily_coverage.to_pandas()
fig.add_trace(
go.Scatter(x=coverage_pd["timestamp"], y=coverage_pd["n_symbols"], name="Symbols"),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(x=coverage_pd["timestamp"], y=coverage_pd["n_options"], name="Options"),
row=2,
col=1,
)
fig.update_layout(height=500, title="Options Universe Coverage Over Time")
fig.show()
# %% [markdown]
# ### 8.3 Greeks Validation
# %%
print("=== Greeks Validity Checks ===")
checks = converged.select(
[
# Delta should be [-1, 1]
((pl.col("delta") >= -1.0) & (pl.col("delta") <= 1.0)).mean().alias("delta_in_bounds"),
# Gamma should be non-negative
(pl.col("gamma") >= 0).mean().alias("gamma_non_negative"),
# Vega should be non-negative
(pl.col("vega") >= 0).mean().alias("vega_non_negative"),
# Theta typically negative
(pl.col("theta") <= 0.01).mean().alias("theta_typical"),
# IV should be positive
(pl.col("implied_vol") > 0).mean().alias("iv_positive"),
]
)
for col in checks.columns:
pct = checks[col][0] * 100
status = "PASS" if pct > 99.9 else ("WARN" if pct > 95 else "FAIL")
print(f" [{status}] {col}: {pct:.2f}%")
# %%
print("\n=== Greeks Summary Statistics ===")
converged.select(["delta", "gamma", "theta", "vega", "implied_vol"]).describe()
# %% [markdown]
# ### 8.4 Point-in-Time Validation
# %%
print("=== Point-in-Time Checks ===")
# Expiration must be >= observation date
exp_check = options.filter(pl.col("expiration") < pl.col("timestamp"))
print(f"Expiration < Date violations: {len(exp_check):,}")
# Days to maturity must be non-negative
dtm_check = options.filter(pl.col("days_to_maturity") < 0)
print(f"Negative days_to_maturity: {len(dtm_check):,}")
if len(exp_check) == 0 and len(dtm_check) == 0:
print("[OK] PASSED - No look-ahead bias detected")
else:
print("[FAIL] FAILED - Data integrity issue")
# %% [markdown]
# ## 9. Information Content Preview
#
# Why does options data predict underlying returns? Let's examine the IV-return
# relationship.
# %%
# Compute ATM IV per symbol/date
atm_iv = (
converged.with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
.filter(pl.col("moneyness").is_between(0.98, 1.02))
.filter(pl.col("call_put") == "C")
.with_columns((pl.col("moneyness") - 1.0).abs().alias("atm_distance"))
.sort(["timestamp", "symbol", "atm_distance"])
.group_by(["timestamp", "symbol"])
.first()
.select(["timestamp", "symbol", "implied_vol", "underlying_price"])
.rename({"implied_vol": "iv_atm"})
)
# Join with daily prices and compute returns
panel = (
atm_iv.join(
daily.select(["timestamp", "symbol", "close"]), on=["timestamp", "symbol"], how="inner"
)
.sort(["symbol", "timestamp"])
.with_columns(
[
pl.col("iv_atm").shift(5).over("symbol").alias("iv_atm_lag5"),
pl.col("close").shift(-5).over("symbol").alias("close_fwd5"),
]
)
.with_columns(
[
(pl.col("iv_atm") - pl.col("iv_atm_lag5")).alias("iv_change_5d"),
((pl.col("close_fwd5") / pl.col("close")) - 1).alias("ret_fwd5"),
]
)
.drop_nulls(subset=["iv_change_5d", "ret_fwd5"])
)
# Correlation
correlation = panel.select(pl.corr("iv_change_5d", "ret_fwd5").alias("corr"))[0, 0]
print("=== IV Change vs Forward Return ===")
print(f"Correlation: {correlation:.4f}")
print("Interpretation: Falling IV tends to precede positive returns")
# %%
# Quintile analysis
panel_ranked = (
panel.with_columns(pl.col("iv_change_5d").rank().over("timestamp").alias("iv_rank_raw"))
.with_columns(
(pl.col("iv_rank_raw") / pl.col("iv_rank_raw").max().over("timestamp") * 100).alias(
"iv_pct"
)
)
.with_columns(
pl.when(pl.col("iv_pct") <= 20)
.then(1)
.when(pl.col("iv_pct") <= 40)
.then(2)
.when(pl.col("iv_pct") <= 60)
.then(3)
.when(pl.col("iv_pct") <= 80)
.then(4)
.otherwise(5)
.alias("iv_quintile")
)
)
quintile_returns = (
panel_ranked.group_by("iv_quintile")
.agg(
[
pl.col("ret_fwd5").mean().alias("mean_ret"),
pl.col("ret_fwd5").std().alias("std_ret"),
pl.len().alias("n_obs"),
]
)
.sort("iv_quintile")
)
print("\n=== Forward Returns by IV Change Quintile ===")
print("(Q1 = largest IV decrease, Q5 = largest IV increase)")
quintile_returns
# %%
fig = px.bar(
quintile_returns.to_pandas(),
x="iv_quintile",
y="mean_ret",
title="5-Day Forward Returns by IV Change Quintile",
labels={
"iv_quintile": "IV Change Quintile (1=falling, 5=rising)",
"mean_ret": "Mean 5-Day Return",
},
)
fig.update_layout(xaxis=dict(tickmode="array", tickvals=[1, 2, 3, 4, 5]))
fig.show()
# %% [markdown]
# The sign of the −0.08 IV-change vs forward-return correlation is negative: on
# this 8-symbol 2020 slice, larger IV declines line up with higher 5-day forward
# returns and larger IV increases with lower forward returns. The quintile means
# above show how monotonic the relationship is. This notebook does not test
# statistical significance or out-of-sample stability; the IV-based feature is
# evaluated rigorously via IC analysis in Chapter 9.
# %% [markdown]
# ## 10. Data Quality Summary
# %%
total_rows = len(options)
converged_rows = len(converged)
converged_pct = converged_rows / total_rows * 100
print("=" * 70)
print("DATA QUALITY SUMMARY: S&P 500 OPTIONS")
print("=" * 70)
print("\n1. SCALE")
print(f" Total options records: {total_rows:,}")
print(f" Converged IV records: {converged_rows:,} ({converged_pct:.1f}%)")
print(f" Unique underlyings: {options['symbol'].n_unique()}")
print(f" Date range: {options['timestamp'].min()} to {options['timestamp'].max()}")
print("\n2. COVERAGE")
print(f" Trading days: {daily_coverage['timestamp'].n_unique()}")
print(f" Avg symbols/day: {daily_coverage['n_symbols'].mean():.0f}")
print(f" Avg options/symbol/day: {options_per_symbol['n_options'].mean():.0f}")
print("\n3. DATA QUALITY")
print(f" Point-in-time: {'PASS' if len(exp_check) == 0 else 'FAIL'}")
print(" Greeks validity: See checks above")
print(f" Convergence rate: {converged_pct:.1f}%")
print("\n4. EXECUTION PROXY")
print(
f" Median ATM spread: {spread_by_moneyness.filter(pl.col('moneyness') == 1.0)['median_spread'][0]:.1%}"
)
print("\n5. INFORMATION CONTENT")
print(f" IV-Return Correlation: {correlation:.4f}")
print("\n" + "=" * 70)
print("DATASET SUPPORTS TWO CASE STUDIES:")
print(" sp500_equity_option_analytics — IV features used to trade equities")
print(" sp500_options — short-straddle harvest with daily delta hedge")
print("=" * 70)
# %% [markdown]
# ## Key Takeaways
#
# 1. **Scale (EDA slice)**: ~5.2M option records across 8 representative S&P 500
# underlyings in 2020. The full AlgoSeek panel covers the broader index — this
# notebook intentionally subsamples for fast exploration.
# 2. **Structure**: Each underlying carries dozens of expirations and 100+
# strikes per day; chains are 3-D grids (strike × expiration × call/put).
# 3. **Quality**: 69.6% of rows carry `iv_convergence == "Converged"`. The other
# convergence codes flag extrapolated or solver-failed IVs and should be
# excluded from analysis.
# 4. **Greeks**: Pre-computed Black-Scholes sensitivities are well-bounded — Δ in
# [-1, 1], Γ ≥ 0, ν ≥ 0, θ ≤ 0 — at >99.9% of converged rows.
# 5. **Volatility Surface**: Smile, skew, and term structure are all visible on
# this slice; the 3-D surface compresses both axes into one plot.
# 6. **Information**: 5-day IV changes correlate with 5-day forward equity
# returns at -0.08 on this sample (falling IV → positive returns), motivating
# the IV-based features in Chapter 8.
# 7. **Execution**: Median ATM bid-ask spread is ~6%; deep-OTM spreads widen
# sharply, which constrains tradeable strategies to near-the-money strikes.
#
# ## Data Limitations
#
# - **No volume/open interest**: Cannot filter for liquidity directly.
# - **Daily snapshots only**: No intraday dynamics for gamma scalping.
# - **Spread as proxy**: Bid-ask is the only execution cost indicator available.
#
# ## Next Steps
#
# - `08_options_greeks_computation`: Compute Greeks from scratch and validate
# against the vendor numbers used here.
# - Chapter 8: Build IV surface features for ML.
# - Chapter 9: Model-based feature extraction (PCA, autoencoders).
# - Chapter 12: ML models for equity and options targets.
# - Chapter 16: Backtests using these features.
02_financial_data_universe/08_options_greeks_computation.ipynb
+11629
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/08_options_greeks_computation.py
+875
View File
@@ -0,0 +1,875 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Options Greeks: From Theory to Computation
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Derive Black-Scholes pricing and Greeks from first principles, implement implied
# volatility via root-finding, and validate the computations against the
# vendor-supplied values in the AlgoSeek options dataset.
#
# ## Learning Objectives
#
# - Implement Black-Scholes call/put pricing and verify put-call parity.
# - Solve for implied volatility numerically using Brent's method.
# - Code all five Greeks (Delta, Gamma, Vega, Theta, Rho) and visualize their
# behavior across moneyness and time to expiration.
# - Quantify residuals between in-house and vendor-computed Greeks and explain
# the residual sources.
#
# ## Book Reference
#
# Chapter 2 §2.2 (asset-class market data landscape — derivatives).
#
# ## Prerequisites
#
# - Basic calculus (partial derivatives) and the standard normal distribution.
# - Options terminology from `07_sp500_options_eda`.
# - The AlgoSeek S&P 500 options EDA parquet at `$ML4T_DATA_PATH/equities/market/sp500/options_eda/`.
# %%
"""Options Greeks — Black-Scholes pricing, IV computation, and Greeks validation."""
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from scipy import stats
from scipy.optimize import brentq
from data import load_sp500_options_eda
# %% tags=["parameters"]
# Production defaults
RISK_FREE_RATE = 0.015 # 3-month Treasury rate, ~typical pre-COVID 2020
N_GREEKS_VALIDATE = 500 # Sample size for Greeks comparison
N_IV_VALIDATE = 200 # Sample size for IV recovery comparison
# %% [markdown]
# ## 1. The Black-Scholes Framework
#
# The Black-Scholes model (1973) provides closed-form solutions for European option
# prices under specific assumptions. Understanding these assumptions is critical for
# practitioners - model limitations explain many real-world pricing phenomena.
#
# ### Model Assumptions
#
# | Assumption | Reality | Implication |
# |------------|---------|-------------|
# | Log-normal returns | Fat tails exist | Underprices tail risk |
# | Constant volatility | Vol changes over time | Need to re-estimate σ |
# | No dividends | Stocks pay dividends | Use dividend-adjusted models |
# | No transaction costs | Costs exist | Greeks less useful for small positions |
# | Continuous trading | Markets close | Weekend/overnight gaps |
# | European exercise | Many options are American | Early exercise premium missed |
# | Constant risk-free rate | Rates vary | Use term-matched rates |
#
# Despite these limitations, Black-Scholes remains the industry standard for
# quoting volatility and computing Greeks. The model's tractability outweighs
# its theoretical shortcomings for most practical applications.
# %% [markdown]
# ### The Black-Scholes Formula
#
# For a European call option:
#
# $$C = S \cdot N(d_1) - K \cdot e^{-rT} \cdot N(d_2)$$
#
# For a European put option:
#
# $$P = K \cdot e^{-rT} \cdot N(-d_2) - S \cdot N(-d_1)$$
#
# Where:
# - $S$ = Current stock price
# - $K$ = Strike price
# - $T$ = Time to expiration (in years)
# - $r$ = Risk-free interest rate
# - $\sigma$ = Volatility (annualized standard deviation of log returns)
# - $N(\cdot)$ = Cumulative normal distribution function
#
# And $d_1$, $d_2$ are:
#
# $$d_1 = \frac{\ln(S/K) + (r + \sigma^2/2)T}{\sigma\sqrt{T}}$$
#
# $$d_2 = d_1 - \sigma\sqrt{T}$$
# %%
# Small helper functions for d1 and d2 (tightly coupled, <=5 lines each)
def d1(S: float, K: float, T: float, r: float, sigma: float) -> float:
"""Compute d1 parameter for Black-Scholes formula."""
return (np.log(S / K) + (r + 0.5 * sigma**2) * T) / (sigma * np.sqrt(T))
def d2(S: float, K: float, T: float, r: float, sigma: float) -> float:
"""Compute d2 parameter: d2 = d1 - sigma * sqrt(T)."""
return d1(S, K, T, r, sigma) - sigma * np.sqrt(T)
# %% [markdown]
# ### Call Pricing
# %%
def bs_call_price(S: float, K: float, T: float, r: float, sigma: float) -> float:
"""Black-Scholes price for a European call option."""
if T <= 0:
return max(S - K, 0) # Intrinsic value at expiration
d_1 = d1(S, K, T, r, sigma)
d_2 = d2(S, K, T, r, sigma)
return S * stats.norm.cdf(d_1) - K * np.exp(-r * T) * stats.norm.cdf(d_2)
# %% [markdown]
# ### Put Pricing
# %%
def bs_put_price(S: float, K: float, T: float, r: float, sigma: float) -> float:
"""Black-Scholes price for a European put option."""
if T <= 0:
return max(K - S, 0) # Intrinsic value at expiration
d_1 = d1(S, K, T, r, sigma)
d_2 = d2(S, K, T, r, sigma)
return K * np.exp(-r * T) * stats.norm.cdf(-d_2) - S * stats.norm.cdf(-d_1)
# %% [markdown]
# ### Unified Pricing Function
# %%
def bs_price(
S: float, K: float, T: float, r: float, sigma: float, option_type: str = "call"
) -> float:
"""Black-Scholes option price (call or put)."""
if option_type.lower() == "call":
return bs_call_price(S, K, T, r, sigma)
else:
return bs_put_price(S, K, T, r, sigma)
# %% [markdown]
# ### Verify Put-Call Parity
#
# A fundamental relationship that must hold for European options:
#
# $$C - P = S - K \cdot e^{-rT}$$
#
# This provides a sanity check for our implementation.
# %%
# Test parameters
S, K, T, r, sigma = 100, 100, 0.25, 0.05, 0.20
call_price = bs_call_price(S, K, T, r, sigma)
put_price = bs_put_price(S, K, T, r, sigma)
# Put-call parity check
lhs = call_price - put_price
rhs = S - K * np.exp(-r * T)
print("=== Black-Scholes Implementation Test ===")
print(f"Parameters: S={S}, K={K}, T={T}, r={r}, σ={sigma}")
print(f"\nCall price: ${call_price:.4f}")
print(f"Put price: ${put_price:.4f}")
print("\nPut-Call Parity Check:")
print(f" C - P = {lhs:.6f}")
print(f" S - Ke^(-rT) = {rhs:.6f}")
print(f" Difference: {abs(lhs - rhs):.2e}")
# %% [markdown]
# ## 2. Implied Volatility Computation
#
# Implied volatility (IV) is the volatility value that, when plugged into
# Black-Scholes, produces the observed market price. Since there's no closed-form
# solution, we must solve numerically:
#
# $$\text{Find } \sigma^* \text{ such that } BS(S, K, T, r, \sigma^*) = P_{market}$$
#
# We'll use Brent's method (a robust root-finding algorithm) to solve this.
# %%
def implied_volatility(
market_price: float,
S: float,
K: float,
T: float,
r: float,
option_type: str = "call",
bounds: tuple = (0.001, 5.0),
) -> float | None:
"""Compute implied volatility using Brent's method.
Returns the volatility that makes BS price equal market_price, or None.
"""
if T <= 0:
return None
# Define objective function: BS_price(sigma) - market_price = 0
def objective(sigma):
return bs_price(S, K, T, r, sigma, option_type) - market_price
try:
# Check if solution exists within bounds
f_low = objective(bounds[0])
f_high = objective(bounds[1])
if f_low * f_high > 0:
# No sign change - no solution in bounds
return None
# Brent's method for root finding
iv = brentq(objective, bounds[0], bounds[1], xtol=1e-8)
return iv
except (ValueError, RuntimeError):
return None
# %%
# Test IV computation
test_vol = 0.25
test_call_price = bs_call_price(S, K, T, r, test_vol)
recovered_iv = implied_volatility(test_call_price, S, K, T, r, "call")
print("=== Implied Volatility Test ===")
print(f"Original volatility: {test_vol:.4f}")
print(f"Generated call price: ${test_call_price:.4f}")
print(f"Recovered IV: {recovered_iv:.4f}")
print(f"Error: {abs(test_vol - recovered_iv):.2e}")
# %% [markdown]
# ## 3. The Greeks: Measuring Option Sensitivities
#
# Greeks measure how option prices change with respect to various inputs.
# They're essential for:
# - **Hedging**: Neutralizing unwanted exposures
# - **Risk Management**: Understanding portfolio sensitivities
# - **Trading**: Identifying mispriced options
#
# ### Summary of Greeks
#
# | Greek | Symbol | Measures | Formula |
# |-------|--------|----------|---------|
# | Delta | $\Delta$ | ∂V/∂S | Price sensitivity to underlying |
# | Gamma | $\Gamma$ | ∂²V/∂S² | Delta sensitivity to underlying |
# | Vega | $\mathcal{V}$ | ∂V/∂σ | Price sensitivity to volatility |
# | Theta | $\Theta$ | ∂V/∂t | Price sensitivity to time (decay) |
# | Rho | $\rho$ | ∂V/∂r | Price sensitivity to interest rate |
# %% [markdown]
# ### Delta ($\Delta$)
#
# Delta measures the rate of change of option price with respect to the underlying:
#
# $$\Delta_{call} = N(d_1)$$
# $$\Delta_{put} = N(d_1) - 1 = -N(-d_1)$$
#
# **Interpretation**:
# - Call delta ranges from 0 to 1
# - Put delta ranges from -1 to 0
# - ATM options have |Δ| ≈ 0.5
# - Delta also approximates probability of finishing ITM
# %%
def delta(S: float, K: float, T: float, r: float, sigma: float, option_type: str = "call") -> float:
"""Compute Black-Scholes delta."""
if T <= 0:
if option_type.lower() == "call":
return 1.0 if S > K else 0.0
else:
return -1.0 if S < K else 0.0
d_1 = d1(S, K, T, r, sigma)
if option_type.lower() == "call":
return stats.norm.cdf(d_1)
else:
return stats.norm.cdf(d_1) - 1
# %% [markdown]
# ### Gamma ($\Gamma$)
#
# Gamma measures the rate of change of delta (option's "acceleration"):
#
# $$\Gamma = \frac{N'(d_1)}{S \sigma \sqrt{T}}$$
#
# Where $N'(x)$ is the standard normal PDF.
#
# **Interpretation**:
# - Gamma is highest for ATM options near expiration
# - Same for calls and puts (by put-call parity)
# - High gamma = delta changes rapidly = harder to hedge
# %%
def gamma(S: float, K: float, T: float, r: float, sigma: float) -> float:
"""
Compute Black-Scholes gamma (same for calls and puts).
"""
if T <= 0:
return 0.0
d_1 = d1(S, K, T, r, sigma)
return stats.norm.pdf(d_1) / (S * sigma * np.sqrt(T))
# %% [markdown]
# ### Vega ($\mathcal{V}$)
#
# Vega measures sensitivity to implied volatility:
#
# $$\mathcal{V} = S \sqrt{T} \cdot N'(d_1)$$
#
# **Interpretation**:
# - Usually quoted per 1% change in volatility (divide by 100)
# - Highest for ATM options with longer time to expiration
# - Same for calls and puts
# %%
def vega(S: float, K: float, T: float, r: float, sigma: float) -> float:
"""
Compute Black-Scholes vega.
Returns vega per 1 point (100%) change in volatility.
Divide by 100 for vega per 1% change.
"""
if T <= 0:
return 0.0
d_1 = d1(S, K, T, r, sigma)
return S * np.sqrt(T) * stats.norm.pdf(d_1)
# %% [markdown]
# ### Theta ($\Theta$)
#
# Theta measures time decay - how option value erodes as time passes:
#
# $$\Theta_{call} = -\frac{S \sigma N'(d_1)}{2\sqrt{T}} - rKe^{-rT}N(d_2)$$
#
# $$\Theta_{put} = -\frac{S \sigma N'(d_1)}{2\sqrt{T}} + rKe^{-rT}N(-d_2)$$
#
# **Interpretation**:
# - Usually negative (options lose value over time)
# - Accelerates as expiration approaches
# - Deep ITM puts can have positive theta
# %%
def theta(S: float, K: float, T: float, r: float, sigma: float, option_type: str = "call") -> float:
"""
Compute Black-Scholes theta (per year).
Divide by 365 for daily theta.
"""
if T <= 0:
return 0.0
d_1 = d1(S, K, T, r, sigma)
d_2 = d2(S, K, T, r, sigma)
term1 = -S * sigma * stats.norm.pdf(d_1) / (2 * np.sqrt(T))
if option_type.lower() == "call":
term2 = -r * K * np.exp(-r * T) * stats.norm.cdf(d_2)
else:
term2 = r * K * np.exp(-r * T) * stats.norm.cdf(-d_2)
return term1 + term2
# %% [markdown]
# ### Rho ($\rho$)
#
# Rho measures sensitivity to interest rates:
#
# $$\rho_{call} = KTe^{-rT}N(d_2)$$
# $$\rho_{put} = -KTe^{-rT}N(-d_2)$$
#
# **Interpretation**:
# - Less important for short-dated options
# - Higher rates benefit calls, hurt puts
# - Often the least-monitored Greek
# %%
def rho(S: float, K: float, T: float, r: float, sigma: float, option_type: str = "call") -> float:
"""
Compute Black-Scholes rho (per 1 point change in rate).
Divide by 100 for rho per 1% change.
"""
if T <= 0:
return 0.0
d_2 = d2(S, K, T, r, sigma)
if option_type.lower() == "call":
return K * T * np.exp(-r * T) * stats.norm.cdf(d_2)
else:
return -K * T * np.exp(-r * T) * stats.norm.cdf(-d_2)
# %% [markdown]
# ### All Greeks Summary Function
# %%
def compute_all_greeks(
S: float, K: float, T: float, r: float, sigma: float, option_type: str = "call"
) -> dict:
"""Compute all Greeks for an option."""
return {
"delta": delta(S, K, T, r, sigma, option_type),
"gamma": gamma(S, K, T, r, sigma),
"vega": vega(S, K, T, r, sigma) / 100, # Per 1% vol change
"theta": theta(S, K, T, r, sigma, option_type) / 365, # Daily
"rho": rho(S, K, T, r, sigma, option_type) / 100, # Per 1% rate change
}
# Test the Greeks
test_greeks = compute_all_greeks(S=100, K=100, T=0.25, r=0.05, sigma=0.20)
print("=== Greeks for ATM Call (S=K=100, T=0.25yr, σ=20%) ===")
for greek, value in test_greeks.items():
print(f"{greek.capitalize():>6}: {value:>10.6f}")
# %% [markdown]
# ## 4. Greeks Visualization
#
# Understanding how Greeks behave across different strikes and times to
# expiration is crucial for option traders.
# %%
# Generate data for visualization
strikes = np.linspace(80, 120, 41)
S_0 = 100
r_0 = 0.05
sigma_0 = 0.20
times = [0.25, 0.5, 1.0] # 3mo, 6mo, 1yr
# Compute Greeks across strikes for different expirations
greeks_data = []
for T_val in times:
for K_val in strikes:
greeks = compute_all_greeks(S_0, K_val, T_val, r_0, sigma_0, "call")
greeks_data.append(
{
"strike": K_val,
"moneyness": S_0 / K_val,
"time_to_exp": f"{int(T_val * 12)}M",
"T": T_val,
**greeks,
}
)
greeks_df = pl.DataFrame(greeks_data)
# %% [markdown]
# ### Delta vs Moneyness
#
# Delta transitions from 0 (deep OTM) to 1 (deep ITM), with the steepest
# slope at ATM. Shorter-dated options have sharper transitions.
# %%
fig = px.line(
greeks_df.to_pandas(),
x="strike",
y="delta",
color="time_to_exp",
title="Call Delta vs Strike Price",
labels={"strike": "Strike Price ($)", "delta": "Delta", "time_to_exp": "Expiration"},
)
fig.add_vline(x=100, line_dash="dash", line_color="gray", annotation_text="ATM")
fig.update_layout(height=400)
fig.show()
# %% [markdown]
# ### Gamma Concentration Near ATM
#
# Gamma peaks at ATM and increases dramatically as expiration approaches.
# This is why short-dated ATM options are difficult to hedge.
# %%
fig = px.line(
greeks_df.to_pandas(),
x="strike",
y="gamma",
color="time_to_exp",
title="Gamma vs Strike Price",
labels={"strike": "Strike Price ($)", "gamma": "Gamma", "time_to_exp": "Expiration"},
)
fig.add_vline(x=100, line_dash="dash", line_color="gray", annotation_text="ATM")
fig.update_layout(height=400)
fig.show()
# %% [markdown]
# ### Theta Decay Acceleration
#
# Theta (time decay) accelerates as expiration approaches. Options lose
# value faster in their final weeks.
# %%
fig = px.line(
greeks_df.to_pandas(),
x="strike",
y="theta",
color="time_to_exp",
title="Daily Theta vs Strike Price",
labels={
"strike": "Strike Price ($)",
"theta": "Theta ($/day)",
"time_to_exp": "Expiration",
},
)
fig.add_vline(x=100, line_dash="dash", line_color="gray", annotation_text="ATM")
fig.update_layout(height=400)
fig.show()
# %% [markdown]
# ### Vega Term Structure
#
# Longer-dated options have higher vega - they're more sensitive to
# volatility changes. This makes sense: more time means more opportunity
# for volatility to impact the final payoff.
# %%
fig = px.line(
greeks_df.to_pandas(),
x="strike",
y="vega",
color="time_to_exp",
title="Vega vs Strike Price",
labels={
"strike": "Strike Price ($)",
"vega": "Vega ($/1% vol)",
"time_to_exp": "Expiration",
},
)
fig.add_vline(x=100, line_dash="dash", line_color="gray", annotation_text="ATM")
fig.update_layout(height=400)
fig.show()
# %% [markdown]
# ## 5. Validation Against AlgoSeek Data
#
# Now we validate our Greeks computations against the pre-computed values
# in the AlgoSeek options dataset. We'll use a risk-free rate from FRED.
# %%
options = load_sp500_options_eda(
symbols=["AAPL"],
start_date="2020-01-01",
end_date="2020-12-31",
)
print(f"Loaded {len(options):,} option records")
print(f"Columns: {options.columns}")
# %%
# Filter to liquid options for validation
# ATM options with reasonable time to expiration
options_filtered = options.filter(
(pl.col("days_to_maturity").is_between(20, 90))
& (pl.col("implied_vol").is_not_null())
& (pl.col("implied_vol") > 0.05)
& (pl.col("implied_vol") < 2.0)
& (pl.col("delta").is_not_null())
)
print(f"Filtered to {len(options_filtered):,} options")
options_filtered.head(5)
# %%
# Compute our Greeks for each option
validation_results = []
for row in options_filtered.head(N_GREEKS_VALIDATE).iter_rows(named=True):
S = row["underlying_price"]
K = row["strike"]
T = row["years_to_maturity"]
sigma = row["implied_vol"]
opt_type = "call" if row["call_put"] == "C" else "put"
# Our computed values
our_delta = delta(S, K, T, RISK_FREE_RATE, sigma, opt_type)
our_gamma = gamma(S, K, T, RISK_FREE_RATE, sigma)
our_vega = vega(S, K, T, RISK_FREE_RATE, sigma) / 100
our_theta = theta(S, K, T, RISK_FREE_RATE, sigma, opt_type) / 365
# AlgoSeek values
algoseek_delta = row["delta"]
algoseek_gamma = row["gamma"]
algoseek_vega = row["vega"]
algoseek_theta = row["theta"]
validation_results.append(
{
"symbol": row["symbol"],
"strike": K,
"days_to_exp": row["days_to_maturity"],
"option_type": opt_type,
"iv": sigma,
"our_delta": our_delta,
"algoseek_delta": algoseek_delta,
"our_gamma": our_gamma,
"algoseek_gamma": algoseek_gamma,
"our_vega": our_vega,
"algoseek_vega": algoseek_vega,
"our_theta": our_theta,
"algoseek_theta": algoseek_theta,
}
)
validation_df = pl.DataFrame(validation_results)
# %%
# Calculate validation errors
validation_df = validation_df.with_columns(
delta_error=(pl.col("our_delta") - pl.col("algoseek_delta")).abs(),
gamma_error=(pl.col("our_gamma") - pl.col("algoseek_gamma")).abs(),
vega_error=(pl.col("our_vega") - pl.col("algoseek_vega")).abs(),
theta_error=(pl.col("our_theta") - pl.col("algoseek_theta")).abs(),
)
# Summary statistics
print("=== Greeks Validation Summary ===")
print(f"Options validated: {len(validation_df)}")
print()
for greek in ["delta", "gamma", "vega", "theta"]:
error_col = f"{greek}_error"
stats_row = validation_df.select(
pl.col(error_col).mean().alias("mean"),
pl.col(error_col).median().alias("median"),
pl.col(error_col).max().alias("max"),
)
print(f"{greek.capitalize()}:")
print(f" Mean error: {stats_row['mean'][0]:.6f}")
print(f" Median error: {stats_row['median'][0]:.6f}")
print(f" Max error: {stats_row['max'][0]:.6f}")
print()
# %% [markdown]
# ### Validation Scatter Plots
# %%
# Build all four panels in a single cell — splitting figure construction across
# cells lets papermill flush the inline backend mid-render and capture the
# Delta+Gamma intermediate, leaving Vega/Theta empty in the published figure.
fig = make_subplots(rows=2, cols=2, subplot_titles=["Delta", "Gamma", "Vega", "Theta"])
# Helper: draw scatter + reference 45° line into one panel.
def _add_validation_panel(row: int, col: int, x_col: str, y_col: str, name: str) -> None:
x_vals = validation_df[x_col].to_list()
y_vals = validation_df[y_col].to_list()
fig.add_trace(
go.Scatter(
x=x_vals,
y=y_vals,
mode="markers",
marker=dict(size=4, opacity=0.5),
name=name,
),
row=row,
col=col,
)
lo = min(min(x_vals), min(y_vals))
hi = max(max(x_vals), max(y_vals))
fig.add_trace(
go.Scatter(
x=[lo, hi],
y=[lo, hi],
mode="lines",
line=dict(dash="dash", color="red"),
showlegend=False,
),
row=row,
col=col,
)
_add_validation_panel(1, 1, "algoseek_delta", "our_delta", "Delta")
_add_validation_panel(1, 2, "algoseek_gamma", "our_gamma", "Gamma")
_add_validation_panel(2, 1, "algoseek_vega", "our_vega", "Vega")
_add_validation_panel(2, 2, "algoseek_theta", "our_theta", "Theta")
fig.update_layout(
height=600,
title_text="Our Greeks vs AlgoSeek (perfect = diagonal line)",
showlegend=False,
)
fig.update_xaxes(title_text="AlgoSeek", row=2, col=1)
fig.update_xaxes(title_text="AlgoSeek", row=2, col=2)
fig.update_yaxes(title_text="Our Calculation", row=1, col=1)
fig.update_yaxes(title_text="Our Calculation", row=2, col=1)
fig.show()
# %% [markdown]
# ### Sources of Discrepancy
#
# Small differences between our calculations and AlgoSeek's pre-computed values
# can arise from:
#
# 1. **Risk-free rate**: We used a constant rate; they may use term-matched rates
# 2. **Dividend handling**: We ignored dividends; they may adjust for them
# 3. **American vs European**: Our formulas assume European exercise
# 4. **Numerical precision**: Different root-finding algorithms
# 5. **Timestamp differences**: Greeks change throughout the day
# %% [markdown]
# ## 6. Computing IV from Market Prices
#
# Let's demonstrate computing implied volatility from the observed option
# prices and compare to the provided IV values.
# %%
# Compute IV for a sample of options
iv_validation = []
for row in options_filtered.head(N_IV_VALIDATE).iter_rows(named=True):
S = row["underlying_price"]
K = row["strike"]
T = row["years_to_maturity"]
market_price = row["mid_price"]
opt_type = "call" if row["call_put"] == "C" else "put"
if market_price is not None and market_price > 0:
computed_iv = implied_volatility(market_price, S, K, T, RISK_FREE_RATE, opt_type)
if computed_iv is not None:
iv_validation.append(
{
"symbol": row["symbol"],
"strike": K,
"days_to_exp": row["days_to_maturity"],
"option_type": opt_type,
"market_price": market_price,
"computed_iv": computed_iv,
"algoseek_iv": row["implied_vol"],
}
)
iv_df = pl.DataFrame(iv_validation)
# %%
# Compare IVs
iv_df = iv_df.with_columns(
iv_diff=(pl.col("computed_iv") - pl.col("algoseek_iv")).abs(),
iv_pct_diff=((pl.col("computed_iv") - pl.col("algoseek_iv")).abs() / pl.col("algoseek_iv"))
* 100,
)
print("=== IV Validation Summary ===")
print(f"Options validated: {len(iv_df)}")
print(f"Mean IV difference: {iv_df['iv_diff'].mean():.4f}")
print(f"Median IV difference: {iv_df['iv_diff'].median():.4f}")
print(f"Mean % difference: {iv_df['iv_pct_diff'].mean():.2f}%")
# %%
# Scatter plot
fig = px.scatter(
iv_df.to_pandas(),
x="algoseek_iv",
y="computed_iv",
color="option_type",
title="Computed IV vs AlgoSeek IV",
labels={
"algoseek_iv": "AlgoSeek IV",
"computed_iv": "Our Computed IV",
"option_type": "Type",
},
opacity=0.6,
)
fig.add_trace(
go.Scatter(
x=[0, 1],
y=[0, 1],
mode="lines",
line=dict(dash="dash", color="gray"),
name="Perfect Match",
)
)
fig.update_layout(height=500)
fig.show()
# %% [markdown]
# ## 7. Practical Considerations
#
# ### When Black-Scholes Breaks Down
#
# | Scenario | Problem | Alternative Approach |
# |----------|---------|---------------------|
# | Deep OTM options | Log-normal underprices tails | Use jump-diffusion or local vol |
# | Short-dated ATM | Gamma explosion | Use realized vol, not IV |
# | Dividend stocks | Wrong forward price | Use dividend-adjusted models |
# | American options | Early exercise value | Use binomial tree or approximations |
# | Vol clustering | Constant vol assumption | Use GARCH or stochastic vol |
#
# ### The Greeks in Practice
#
# **For Hedging:**
# - Delta-hedge by trading underlying shares
# - Gamma tells you how often to rebalance
# - Vega exposure from vol moves often dominates
#
# **For Trading:**
# - Greeks help identify relative value
# - High gamma + low premium = potential mispricing
# - Theta/vega ratio useful for vol trades
#
# **For Risk Management:**
# - Aggregate portfolio Greeks
# - Stress test under extreme scenarios
# - Greeks are local approximations - large moves need full repricing
# %% [markdown]
# ## 8. Key Takeaways
#
# 1. Black-Scholes pricing recovers put-call parity to numerical precision
# (~7e-15 on the test parameters); the same code base solves IV via Brent's
# method to ~10⁻¹¹ accuracy on a closed-loop test.
# 2. Greeks computed in-house track the AlgoSeek vendor values closely on
# AAPL options (20–90 DTE, IV ∈ [0.05, 2]): mean absolute errors are
# ~2×10⁻³ for delta and vega, ~7×10⁻⁵ for gamma, ~1×10⁻³ for daily theta.
# 3. The remaining residual reflects modeling choices that the vendor handles
# differently — term-matched risk-free rates, dividend treatment, American
# early-exercise premium, and intraday timestamp drift in the inputs.
# 4. Cross-sectional IV recovery (≈180 options) yields a median absolute IV
# diff of ~5×10⁻⁴ and a mean of ~0.024, with the residual concentrated in
# deep OTM contracts where the loss surface is shallow.
# 5. Greeks are local sensitivities — the visualizations show how Δ steepens
# and Γ peaks near ATM as expiration approaches, which is also the regime
# where the Black-Scholes assumption set is most stressed.
#
# ## Next Steps
#
# - Chapter 8: Build features from options data (IV signals, skew measures).
# - Chapter 9: Evaluate IV-based signals for equity prediction.
# - Chapter 12+: ML models using options-derived features.
# - Chapter 16: Strategy backtests including the `sp500_options` short-straddle
# case study and the `sp500_equity_option_analytics` IV-features case study.
02_financial_data_universe/09_options_continuous.ipynb
+7324
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/09_options_continuous.py
+768
View File
@@ -0,0 +1,768 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Constructing Continuous Options Series
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Demonstrate why a "constant-maturity" option series chains together different
# contracts, produces phantom price jumps at every roll, and contaminates label
# construction. Implement two clean alternatives: same-contract holding returns
# (correct labels) and a roll-zeroed continuous reconstruction (backtestable
# mark-to-market).
#
# ## Learning Objectives
#
# - Detect contract rolls in a 30-day ATM straddle time series and quantify the
# resulting bias against non-roll-day returns.
# - Compute same-contract holding returns by looking up the entry contract's
# exit price in the raw option chain.
# - Build a roll-zeroed continuous price series suitable for backtesting and
# compare it to naive chained returns.
# - Reason about which construction belongs in label generation versus
# transaction-cost accounting.
#
# ## Book Reference
#
# Chapter 2 §2.2 (asset-class market data landscape — derivatives). The futures
# analogue is `06_futures_continuous`; the case-study scale-up is
# `case_studies/sp500_options/02_labels`.
#
# ## Prerequisites
#
# - `07_sp500_options_eda` for option-chain structure.
# - `08_options_greeks_computation` for theta and time decay.
# - The AlgoSeek S&P 500 options EDA parquet at `$ML4T_DATA_PATH/sp500_options/`.
# %%
"""Constructing Continuous Options Series — constant-maturity roll adjustment."""
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_sp500_options_eda
# %% tags=["parameters"]
DEMO_SYMBOL = "AAPL"
DEMO_YEAR = 2019
# %% [markdown]
# ## 1. The Constant-Maturity Construction
#
# A 30-day ATM straddle is a common volatility instrument: you buy (or sell) both
# an ATM call and an ATM put with ~30 days to expiration. To build a daily time
# series, we select the "best" straddle each day — the one closest to 30 DTE and
# 50-delta.
#
# The problem: this selection is **independent** each day. The contract identity
# (strike, expiration) changes whenever a new weekly/monthly option enters the
# 25–35 DTE window or the underlying moves enough to shift which strike is ATM.
# %% [markdown]
# ### 1.1 Load Raw Options for One Symbol
# %%
raw = load_sp500_options_eda(
symbols=[DEMO_SYMBOL],
start_date=f"{DEMO_YEAR}-01-01",
end_date=f"{DEMO_YEAR}-12-31",
).rename({"timestamp": "date"})
print(f"Raw {DEMO_SYMBOL} options ({DEMO_YEAR}): {len(raw):,} rows")
print(f" Dates: {raw['date'].n_unique()}")
print(
f" Unique (strike, expiration): {raw.select(pl.struct('strike', 'expiration')).n_unique():,}"
)
# %% [markdown]
# ### 1.2 Constant-Maturity Straddle Selection
#
# For each trading day, select the ATM straddle closest to 30 DTE.
# This replicates the logic in `materialize_options.py`.
# %%
# Selection parameters for constant-maturity straddle
DTE_WINDOW = (25, 35)
TARGET_DELTA = 0.50
DELTA_TOL = 0.15
MIN_BID = 0.01
MAX_REL_SPREAD = 0.30
# %% [markdown]
# ### Straddle Selection Logic
#
# For each trading day, select the ATM straddle closest to 30 DTE.
# %%
def select_constant_maturity_straddle(raw_options: pl.DataFrame) -> pl.DataFrame:
"""Select the 'best' 30D ATM straddle per day from raw option chains."""
opts = raw_options.with_columns(
pl.col("delta").abs().alias("abs_delta"),
((pl.col("ask") - pl.col("bid")) / pl.col("mid_price").clip(lower_bound=0.01)).alias(
"rel_spread"
),
)
filtered = opts.filter(
pl.col("days_to_maturity").is_between(DTE_WINDOW[0], DTE_WINDOW[1])
& (pl.col("bid") >= MIN_BID)
& (pl.col("rel_spread") <= MAX_REL_SPREAD)
& (pl.col("iv_convergence") == "Converged")
& pl.col("abs_delta").is_between(TARGET_DELTA - DELTA_TOL, TARGET_DELTA + DELTA_TOL)
)
def _best_leg(df: pl.DataFrame, cp: str) -> pl.DataFrame:
leg = df.filter(pl.col("call_put") == cp)
leg = leg.with_columns((pl.col("abs_delta") - TARGET_DELTA).abs().alias("_dd"))
return (
leg.with_columns(pl.col("_dd").rank("ordinal").over("date").alias("_rank"))
.filter(pl.col("_rank") == 1)
.drop(["_rank", "_dd"])
)
call_cols = [
"date",
"strike",
"expiration",
"days_to_maturity",
"underlying_price",
pl.col("mid_price").alias("call_mid"),
pl.col("bid").alias("call_bid"),
pl.col("ask").alias("call_ask"),
pl.col("delta").alias("call_delta"),
pl.col("theta").alias("call_theta"),
]
put_cols = [
"date",
pl.col("mid_price").alias("put_mid"),
pl.col("bid").alias("put_bid"),
pl.col("ask").alias("put_ask"),
pl.col("delta").alias("put_delta"),
pl.col("theta").alias("put_theta"),
]
calls = _best_leg(filtered, "C").select(call_cols)
puts = _best_leg(filtered, "P").select(put_cols)
return (
calls.join(puts, on="date", how="inner")
.with_columns(
(pl.col("call_mid") + pl.col("put_mid")).alias("instr_mid"),
(pl.col("call_theta") + pl.col("put_theta")).alias("instr_theta"),
(pl.col("call_delta") + pl.col("put_delta")).alias("instr_delta"),
)
.sort("date")
)
# %%
cm_straddles = select_constant_maturity_straddle(raw)
print(f"Constant-maturity straddle series: {len(cm_straddles)} days")
print(f" Date range: {cm_straddles['date'].min()} to {cm_straddles['date'].max()}")
cm_straddles.select(["date", "strike", "expiration", "days_to_maturity", "instr_mid"]).head(10)
# %% [markdown]
# ## 2. The Roll Problem
#
# How often does the underlying contract change?
# %%
cm_straddles = cm_straddles.with_columns(
pl.col("expiration").shift(1).alias("prev_exp"),
pl.col("strike").shift(1).alias("prev_strike"),
pl.col("instr_mid").shift(1).alias("prev_mid"),
)
cm_straddles = cm_straddles.with_columns(
((pl.col("expiration") != pl.col("prev_exp")) | (pl.col("strike") != pl.col("prev_strike")))
.fill_null(False)
.alias("is_roll"),
((pl.col("instr_mid") - pl.col("prev_mid")) / pl.col("prev_mid")).alias("daily_return"),
)
n_rolls = cm_straddles.filter(pl.col("is_roll")).height
n_total = len(cm_straddles) - 1 # exclude first row (no return)
print(f"Contract changes: {n_rolls} out of {n_total} days ({n_rolls / n_total:.0%})")
# %% [markdown]
# ### 2.1 Roll-Day vs Non-Roll-Day Returns
#
# If the constant-maturity series were a true continuous instrument, roll days
# and non-roll days should have similar return characteristics. They don't.
# %%
roll_rets = cm_straddles.filter(pl.col("is_roll"))["daily_return"].drop_nulls()
nonroll_rets = cm_straddles.filter(~pl.col("is_roll"))["daily_return"].drop_nulls()
roll_summary = pl.DataFrame(
{
"metric": ["count", "mean_return", "median_return", "pct_positive"],
"roll_days": [
float(roll_rets.len()),
roll_rets.mean(),
roll_rets.median(),
(roll_rets > 0).mean(),
],
"non_roll_days": [
float(nonroll_rets.len()),
nonroll_rets.mean(),
nonroll_rets.median(),
(nonroll_rets > 0).mean(),
],
}
)
roll_summary = roll_summary.with_columns(
(pl.col("roll_days") - pl.col("non_roll_days")).alias("difference"),
)
roll_summary
# %% [markdown]
# The pattern is clear: roll days show systematically positive returns (~+1.3%)
# while non-roll days show negative returns (~-0.9%). This is not a market signal —
# it's the mechanical effect of switching from a time-decayed contract (low
# premium, near expiry) to a fresh contract (high premium, further expiry).
#
# For a short straddle strategy, this phantom "return" would be interpreted as a
# loss, but it's actually the cost of maintaining the constant-maturity position —
# analogous to contango roll cost in futures, but much larger in magnitude.
# %% [markdown]
# ### 2.2 Visualizing the Sawtooth Pattern
# %%
# Prepare data for sawtooth visualization
dates = cm_straddles["date"].to_list()
prices = cm_straddles["instr_mid"].to_list()
dtes = cm_straddles["days_to_maturity"].to_list()
rolls = cm_straddles["is_roll"].to_list()
roll_dates = [d for d, r in zip(dates, rolls, strict=False) if r]
roll_prices = [p for p, r in zip(prices, rolls, strict=False) if r]
nonroll_dates = [d for d, r in zip(dates, rolls, strict=False) if not r]
nonroll_prices = [p for p, r in zip(prices, rolls, strict=False) if not r]
# %%
# Build both panels in a single cell so the inline backend does not flush
# the figure mid-construction.
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
subplot_titles=[
f"{DEMO_SYMBOL} Constant-Maturity Straddle Price",
"Days to Expiration (DTE)",
],
vertical_spacing=0.08,
)
fig.add_trace(
go.Scatter(
x=nonroll_dates,
y=nonroll_prices,
mode="markers",
name="Same contract",
marker=dict(size=4, color="#3498db"),
opacity=0.7,
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=roll_dates,
y=roll_prices,
mode="markers",
name="Contract switch",
marker=dict(size=6, color="#e74c3c", symbol="diamond"),
opacity=0.9,
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=dates,
y=prices,
mode="lines",
name="Price",
line=dict(width=1, color="#95a5a6"),
showlegend=False,
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=dates,
y=dtes,
mode="lines+markers",
name="DTE",
marker=dict(size=3, color="#2ecc71"),
line=dict(width=1),
),
row=2,
col=1,
)
fig.update_yaxes(title_text="Straddle Mid ($)", row=1, col=1)
fig.update_yaxes(title_text="DTE", row=2, col=1)
fig.update_xaxes(title_text="Date", row=2, col=1)
fig.update_layout(template="plotly_white", height=600, legend=dict(x=0.01, y=0.99))
fig.show()
# %% [markdown]
# The sawtooth pattern is visible: price decays for a few days (theta eats the
# premium), then jumps up when a new contract with more time value is selected.
# The DTE panel confirms: DTE drops from ~35 to ~25, then jumps back when the
# contract rolls. **Every price jump at a roll is a phantom return.**
# %% [markdown]
# ## 3. Same-Contract Holding Returns
#
# The correct return for a straddle trade is: enter a specific contract at mid on
# day $t$, exit the **same contract** at mid on day $t + h$. This requires
# looking up that specific contract in the raw option chain $h$ days later.
#
# $$r_{same} = \frac{P_{entry}^{mid}(K, T) - P_{exit}^{mid}(K, T)}{P_{entry}^{mid}(K, T)}$$
#
# where $(K, T)$ identifies the specific strike and expiration.
# %% [markdown]
# ### 3.1 Build Exit Price Lookup from Raw Chain
# %%
HOLDING_PERIOD = 10 # days
def build_exit_lookup(raw_options: pl.DataFrame) -> pl.DataFrame:
"""Build a lookup table: (date, strike, expiration, call_put) → mid_price.
Used to find the exit price of a specific contract h days after entry.
"""
return raw_options.filter(
pl.col("bid") >= MIN_BID,
pl.col("iv_convergence") == "Converged",
).select(
[
"date",
"strike",
"expiration",
"call_put",
"mid_price",
"bid",
"ask",
"delta",
"theta",
"days_to_maturity",
]
)
# %%
# Build lookup from the raw chain
lookup = build_exit_lookup(raw)
print(f"Exit price lookup: {len(lookup):,} rows")
print(
f" Unique contracts: {lookup.select(pl.struct('strike', 'expiration', 'call_put')).n_unique():,}"
)
# %% [markdown]
# ### 3.2 Compute Same-Contract Returns
#
# For each day's constant-maturity straddle, find the same contract's price
# $h$ trading days later in the raw chain.
# %%
# Get the trading calendar (ordered dates)
trading_dates = cm_straddles["date"].unique().sort().to_list()
date_to_idx = {d: i for i, d in enumerate(trading_dates)}
def get_exit_date(entry_date, h: int, cal: list):
"""Get the trading date h business days after entry_date."""
idx = date_to_idx.get(entry_date)
if idx is None or idx + h >= len(cal):
return None
return cal[idx + h]
# Build entry-exit pairs
entries = cm_straddles.select(
[
"date",
"strike",
"expiration",
"instr_mid",
"call_mid",
"put_mid",
"instr_delta",
"is_roll",
]
).rename({"date": "entry_date", "instr_mid": "entry_mid"})
# Add exit dates
exit_dates = [
get_exit_date(d, HOLDING_PERIOD, trading_dates) for d in entries["entry_date"].to_list()
]
entries = entries.with_columns(pl.Series("exit_date", exit_dates).cast(pl.Date))
# Drop entries where exit date is beyond our data
entries = entries.filter(pl.col("exit_date").is_not_null())
print(f"Entry-exit pairs to look up: {len(entries):,}")
# %%
# Look up call exit price
call_exit = lookup.filter(pl.col("call_put") == "C").select(
[
pl.col("date").alias("exit_date"),
"strike",
"expiration",
pl.col("mid_price").alias("call_exit_mid"),
]
)
# Look up put exit price
put_exit = lookup.filter(pl.col("call_put") == "P").select(
[
pl.col("date").alias("exit_date"),
"strike",
"expiration",
pl.col("mid_price").alias("put_exit_mid"),
]
)
# Join: entry contract → exit prices for same (strike, expiration)
same_contract = entries.join(call_exit, on=["exit_date", "strike", "expiration"], how="left").join(
put_exit, on=["exit_date", "strike", "expiration"], how="left"
)
# Compute same-contract straddle exit mid
same_contract = same_contract.with_columns(
(pl.col("call_exit_mid") + pl.col("put_exit_mid")).alias("exit_mid"),
)
# Same-contract return (short straddle: positive = profitable)
same_contract = same_contract.with_columns(
((pl.col("entry_mid") - pl.col("exit_mid")) / pl.col("entry_mid")).alias("same_contract_ret"),
)
# How many lookups succeeded?
found = same_contract.filter(pl.col("exit_mid").is_not_null()).height
print(f"Exit prices found: {found} / {len(same_contract)} ({found / len(same_contract):.1%})")
# %% [markdown]
# ### 3.3 Compare Naive vs Same-Contract Returns
# %%
# Naive return: shift-based on constant-maturity series (what 02_labels.py did)
naive_rets = cm_straddles.with_columns(
pl.col("instr_mid").shift(-1).alias("naive_entry"),
pl.col("instr_mid").shift(-(1 + HOLDING_PERIOD)).alias("naive_exit"),
).with_columns(
((pl.col("naive_entry") - pl.col("naive_exit")) / pl.col("naive_entry")).alias("naive_ret"),
)
# Align for comparison
comparison = (
same_contract.filter(pl.col("same_contract_ret").is_not_null())
.select(["entry_date", "same_contract_ret"])
.join(
naive_rets.select([pl.col("date").alias("entry_date"), "naive_ret"]).drop_nulls(),
on="entry_date",
how="inner",
)
)
print(f"Paired observations: {len(comparison):,}")
naive_vs_same = pl.DataFrame(
{
"metric": ["mean_return", "std", "median_return", "pct_positive_short"],
"naive_chained": [
comparison["naive_ret"].mean(),
comparison["naive_ret"].std(),
comparison["naive_ret"].median(),
(comparison["naive_ret"] > 0).mean(),
],
"same_contract": [
comparison["same_contract_ret"].mean(),
comparison["same_contract_ret"].std(),
comparison["same_contract_ret"].median(),
(comparison["same_contract_ret"] > 0).mean(),
],
}
)
corr = comparison.select(pl.corr("naive_ret", "same_contract_ret").alias("correlation")).item()
print(f"Correlation(naive, same-contract) = {corr:.3f}")
naive_vs_same
# %% [markdown]
# The naive chained returns systematically differ from same-contract returns.
# The correlation tells us how much of the naive signal is genuine vs artifact.
# Any label or model trained on naive returns is learning a mix of actual
# straddle P&L and roll mechanics.
# %%
# Scatter plot: naive vs same-contract
fig = go.Figure()
fig.add_trace(
go.Scatter(
x=comparison["naive_ret"].to_list(),
y=comparison["same_contract_ret"].to_list(),
mode="markers",
marker=dict(size=3, color="#3498db", opacity=0.4),
name="Returns",
)
)
fig.add_trace(
go.Scatter(
x=[-0.5, 0.5],
y=[-0.5, 0.5],
mode="lines",
line=dict(dash="dash", color="gray"),
name="45° line",
)
)
fig.update_layout(
template="plotly_white",
title=f"{DEMO_SYMBOL}: Naive vs Same-Contract {HOLDING_PERIOD}d Returns",
xaxis_title="Naive (chained constant-maturity)",
yaxis_title="Same-contract holding return",
width=600,
height=500,
)
fig.show()
# %% [markdown]
# ## 4. Building a Continuous Series for Backtesting
#
# For backtesting, we need a continuous price series where daily returns reflect
# actual P&L — no phantom jumps at rolls. Unlike futures (which roll quarterly
# and suit Panama adjustment), straddles roll almost daily, so we use a
# **return-based reconstruction**:
#
# 1. On non-roll days: daily return = genuine within-contract P&L
# 2. On roll days: set return to 0 (the roll is a transaction, not a return)
# 3. Reconstruct prices: $P_{adj,t} = P_0 \prod_{i=1}^{t} (1 + r_i)$
#
# Roll costs (bid-ask at roll) are modeled separately as transaction costs in
# Chapter 18.
# %%
def build_continuous_straddle_series(cm_series: pl.DataFrame) -> pl.DataFrame:
"""Build a continuous straddle price series from within-contract returns.
On roll days, the return is set to 0 (roll is a transaction, not a return).
The resulting series has no phantom jumps and its returns reflect actual P&L.
"""
df = cm_series.with_columns(
((pl.col("instr_mid") - pl.col("instr_mid").shift(1)) / pl.col("instr_mid").shift(1)).alias(
"raw_daily_ret"
),
)
# On roll days, set return to 0
df = df.with_columns(
pl.when(pl.col("is_roll"))
.then(0.0)
.otherwise(pl.col("raw_daily_ret"))
.alias("clean_daily_ret"),
)
# Reconstruct price from cumulative returns, starting at the first observed price
start_price = cm_series["instr_mid"][0]
cum_rets = df["clean_daily_ret"].fill_null(0.0).to_list()
adj_prices = [start_price]
for r in cum_rets[1:]:
adj_prices.append(adj_prices[-1] * (1.0 + r))
return df.with_columns(pl.Series("price_adjusted", adj_prices))
# %%
adjusted = build_continuous_straddle_series(cm_straddles)
# Compare roll-day returns: raw vs cleaned
adj_roll = adjusted.filter(pl.col("is_roll"))
adj_nonroll = adjusted.filter(~pl.col("is_roll"))
raw_vs_clean = pl.DataFrame(
{
"metric": ["raw_mean", "clean_mean"],
"roll_days": [
adj_roll["raw_daily_ret"].mean(),
adj_roll["clean_daily_ret"].mean(),
],
"non_roll_days": [
adj_nonroll["raw_daily_ret"].mean(),
adj_nonroll["clean_daily_ret"].mean(),
],
}
)
print(
f"Roll-day bias removed: raw roll mean was {adj_roll['raw_daily_ret'].mean():+.4f}, now 0.0000"
)
raw_vs_clean
# %%
# Visualize raw vs adjusted price series
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
subplot_titles=["Raw Constant-Maturity Series (sawtooth)", "Continuous (roll-zeroed)"],
vertical_spacing=0.08,
)
fig.add_trace(
go.Scatter(
x=adjusted["date"].to_list(),
y=adjusted["instr_mid"].to_list(),
mode="lines",
name="Raw",
line=dict(color="#e74c3c", width=1.5),
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=adjusted["date"].to_list(),
y=adjusted["price_adjusted"].to_list(),
mode="lines",
name="Adjusted",
line=dict(color="#3498db", width=1.5),
),
row=2,
col=1,
)
fig.update_yaxes(title_text="Straddle Mid ($)", row=1, col=1)
fig.update_yaxes(title_text="Adjusted Price ($)", row=2, col=1)
fig.update_layout(
template="plotly_white",
height=500,
title=f"{DEMO_SYMBOL} Straddle: Raw vs Continuous (Roll-Adjusted)",
)
fig.show()
# %% [markdown]
# The adjusted series shows steady theta decay without the sawtooth jumps.
# This is the series suitable for backtesting — roll costs should be modeled
# separately as transaction costs (Chapter 18).
#
# **Note**: This continuous series approximates the actual P&L but is not exact.
# The precise approach — same-contract holding returns (Section 3 above) — is
# what the case study uses for label construction.
# %% [markdown]
# ## 5. Summary: Three Approaches Compared
#
# | Approach | Mechanism | Suitable For |
# |----------|-----------|--------------|
# | Naive (chained) | `shift(-h)` on constant-maturity series | **Nothing** — contaminates labels |
# | Same-contract | Look up entry contract's price h days later | **Labels** — correct per-trade P&L |
# | Continuous (roll-zeroed) | Zero out roll-day returns, rebuild prices | **Backtesting** — smooth mark-to-market |
#
# The same-contract approach (Section 3) requires the raw option chains to
# look up exit prices. The continuous approach (Section 4) only needs the
# constant-maturity series but must detect rolls and zero out their returns.
# %%
# Final comparison: all three 10d forward return approaches
# Naive
adjusted = adjusted.with_columns(
pl.col("instr_mid").shift(-1).alias("raw_entry"),
pl.col("instr_mid").shift(-(1 + HOLDING_PERIOD)).alias("raw_exit"),
).with_columns(
((pl.col("raw_entry") - pl.col("raw_exit")) / pl.col("raw_entry")).alias("naive_fwd_ret"),
)
# Continuous-adjusted
adjusted = adjusted.with_columns(
pl.col("price_adjusted").shift(-1).alias("adj_entry"),
pl.col("price_adjusted").shift(-(1 + HOLDING_PERIOD)).alias("adj_exit"),
).with_columns(
((pl.col("adj_entry") - pl.col("adj_exit")) / pl.col("adj_entry")).alias("cont_fwd_ret"),
)
# Merge with same-contract returns
comparison_all = (
adjusted.select([pl.col("date").alias("entry_date"), "naive_fwd_ret", "cont_fwd_ret"])
.join(
same_contract.select(["entry_date", "same_contract_ret"]),
on="entry_date",
how="left",
)
.drop_nulls()
)
print(f"Paired observations: {len(comparison_all):,}")
three_way = pl.DataFrame(
{
"metric": ["mean", "std", "corr_with_same_contract"],
"naive": [
comparison_all["naive_fwd_ret"].mean(),
comparison_all["naive_fwd_ret"].std(),
comparison_all.select(pl.corr("naive_fwd_ret", "same_contract_ret")).item(),
],
"continuous": [
comparison_all["cont_fwd_ret"].mean(),
comparison_all["cont_fwd_ret"].std(),
comparison_all.select(pl.corr("cont_fwd_ret", "same_contract_ret")).item(),
],
"same_contract": [
comparison_all["same_contract_ret"].mean(),
comparison_all["same_contract_ret"].std(),
1.0,
],
}
)
three_way
# %% [markdown]
# ## Key Takeaways
#
# 1. **Constant-maturity option series chain different contracts**: The "best"
# 30D ATM straddle changes every 1-2 days on average. Each switch creates a
# phantom price jump from time-value reset.
#
# 2. **Roll contamination is systematic**: Roll-day returns average +1.3% vs
# -0.9% on non-roll days. This 2.2pp bias compounds over ~184 roll days/year
# because options decay faster and roll more frequently than futures.
#
# 3. **Same-contract returns are the correct label**: Looking up the entry
# contract's price h days later in the raw option chain gives the actual
# holding P&L. The raw chain data makes this feasible.
#
# 4. **Roll-zeroed reconstruction creates a backtestable series**: Setting
# roll-day returns to zero and rebuilding prices removes phantom jumps while
# preserving genuine within-contract P&L on non-roll days.
#
# 5. **Roll costs are transaction costs, not returns**: The cost of maintaining
# a constant-maturity position (closing old, opening new) should be modeled
# as execution cost in Chapter 18, not embedded in the price series.
#
# ## Next
#
# The S&P 500 options case study
# ([`02_labels`](../case_studies/sp500_options/02_labels.ipynb)) applies the
# same-contract and roll-zeroed constructions at panel scale across the full
# universe and multi-year sample.
02_financial_data_universe/10_crypto_perps_eda.ipynb
+847
View File
@@ -0,0 +1,847 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "34eda0f3",
"metadata": {
"papermill": {
"duration": 0.002064,
"end_time": "2026-06-13T02:33:28.977337+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:28.975273+00:00",
"status": "completed"
}
},
"source": [
"# Crypto Perps — Exploratory Data Analysis\n",
"\n",
"**Docker image**: `ml4t`\n",
"\n",
"## Purpose\n",
"\n",
"Profile the Binance Futures hourly OHLCV dataset for 19 perpetual contracts\n",
"alongside the 8-hourly Premium Index that captures the perpetual–spot\n",
"basis. The notebook anchors data shape, units, coverage, and OHLC integrity\n",
"before the strategy work begins in Chapter 6.\n",
"\n",
"## Learning Objectives\n",
"\n",
"- Load hourly OHLCV and 8-hourly premium index data via the canonical loaders.\n",
"- Document premium-index units (decimal, multiply by 100 for percent).\n",
"- Quantify cross-frequency join coverage between OHLCV and premium data.\n",
"- Run OHLC invariant checks and inspect for time-stamp gaps.\n",
"\n",
"## Book Reference\n",
"\n",
"Chapter 2 §2.2 (asset-class market data landscape — digital assets).\n",
"\n",
"## Prerequisites\n",
"\n",
"- Familiarity with daily OHLCV equity data (`01_us_equities_eda`).\n",
"- The Binance Futures parquet at `$ML4T_DATA_PATH/crypto/` (OHLCV + premium).\n",
"- Methodology continues in `11_crypto_premium_analysis`; the case-study\n",
" pipeline lives under `case_studies/crypto_perps_funding/`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "f2410448",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:28.980599Z",
"iopub.status.busy": "2026-06-13T02:33:28.980491Z",
"iopub.status.idle": "2026-06-13T02:33:30.219451Z",
"shell.execute_reply": "2026-06-13T02:33:30.218950Z"
},
"papermill": {
"duration": 1.241442,
"end_time": "2026-06-13T02:33:30.220191+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:28.978749+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"\"\"\"Crypto Perps EDA — hourly OHLCV and premium index exploration.\"\"\"\n",
"\n",
"import polars as pl\n",
"\n",
"from data import load_crypto_perps, load_crypto_premium\n",
"from utils.data_quality import check_ohlc_invariants, per_asset_stats"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f028d40b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.223486Z",
"iopub.status.busy": "2026-06-13T02:33:30.223403Z",
"iopub.status.idle": "2026-06-13T02:33:30.225251Z",
"shell.execute_reply": "2026-06-13T02:33:30.224946Z"
},
"papermill": {
"duration": 0.004413,
"end_time": "2026-06-13T02:33:30.225693+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.221280+00:00",
"status": "completed"
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"MAX_SYMBOLS = 0 # 0 = all symbols"
]
},
{
"cell_type": "markdown",
"id": "62781cf4",
"metadata": {
"papermill": {
"duration": 0.000842,
"end_time": "2026-06-13T02:33:30.227452+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.226610+00:00",
"status": "completed"
}
},
"source": [
"## 1. Load and Inspect OHLCV\n",
"\n",
"Hourly OHLCV data from Binance Futures for 19 cryptocurrencies.\n",
"Trading is 24/7 (8,760 hours/year vs 252 days for equities)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "b3bc2125",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.229623Z",
"iopub.status.busy": "2026-06-13T02:33:30.229559Z",
"iopub.status.idle": "2026-06-13T02:33:30.269019Z",
"shell.execute_reply": "2026-06-13T02:33:30.268500Z"
},
"papermill": {
"duration": 0.040998,
"end_time": "2026-06-13T02:33:30.269305+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.228307+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== OHLCV Dataset ===\n",
"Shape: (866484, 7)\n",
"Columns: ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'symbol']\n"
]
}
],
"source": [
"ohlcv = load_crypto_perps(frequency=\"1h\")\n",
"\n",
"print(\"=== OHLCV Dataset ===\")\n",
"print(f\"Shape: {ohlcv.shape}\")\n",
"print(f\"Columns: {ohlcv.columns}\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "599855a3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.271848Z",
"iopub.status.busy": "2026-06-13T02:33:30.271778Z",
"iopub.status.idle": "2026-06-13T02:33:30.274032Z",
"shell.execute_reply": "2026-06-13T02:33:30.273658Z"
},
"papermill": {
"duration": 0.003918,
"end_time": "2026-06-13T02:33:30.274317+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.270399+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Schema:\n",
" timestamp: Datetime(time_unit='ms', time_zone='UTC')\n",
" open: Float64\n",
" high: Float64\n",
" low: Float64\n",
" close: Float64\n",
" volume: Float64\n",
" symbol: String\n"
]
}
],
"source": [
"# Schema overview\n",
"print(\"\\nSchema:\")\n",
"for col, dtype in ohlcv.schema.items():\n",
" print(f\" {col}: {dtype}\")"
]
},
{
"cell_type": "markdown",
"id": "51d44bf8",
"metadata": {
"papermill": {
"duration": 0.000888,
"end_time": "2026-06-13T02:33:30.276173+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.275285+00:00",
"status": "completed"
}
},
"source": [
"## 2. Coverage Summary"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "5c9db57a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.278342Z",
"iopub.status.busy": "2026-06-13T02:33:30.278276Z",
"iopub.status.idle": "2026-06-13T02:33:30.285289Z",
"shell.execute_reply": "2026-06-13T02:33:30.284906Z"
},
"papermill": {
"duration": 0.00843,
"end_time": "2026-06-13T02:33:30.285512+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.277082+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Coverage ===\n",
"Number of symbols: 19\n",
"\n",
"Symbols: AAVEUSDT, ADAUSDT, APTUSDT, ATOMUSDT, AVAXUSDT, BNBUSDT, BTCUSDT, COMPUSDT, DOGEUSDT, DOTUSDT, ETHUSDT, INJUSDT, LINKUSDT, MKRUSDT, NEARUSDT, SOLUSDT, SUIUSDT, UNIUSDT, XRPUSDT\n"
]
}
],
"source": [
"# Symbols and date range\n",
"symbols = ohlcv[\"symbol\"].unique().sort().to_list()\n",
"print(\"=== Coverage ===\")\n",
"print(f\"Number of symbols: {len(symbols)}\")\n",
"print(f\"\\nSymbols: {', '.join(symbols)}\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "92032764",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.288070Z",
"iopub.status.busy": "2026-06-13T02:33:30.287999Z",
"iopub.status.idle": "2026-06-13T02:33:30.296853Z",
"shell.execute_reply": "2026-06-13T02:33:30.296505Z"
},
"papermill": {
"duration": 0.010672,
"end_time": "2026-06-13T02:33:30.297260+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.286588+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Date range: 2020-01-01 00:00:00+00:00 to 2025-12-31 23:00:00+00:00\n",
"Unique hours: 52,608\n"
]
}
],
"source": [
"# Overall date range\n",
"date_range = ohlcv.select(\n",
" [\n",
" pl.col(\"timestamp\").min().alias(\"start\"),\n",
" pl.col(\"timestamp\").max().alias(\"end\"),\n",
" pl.col(\"timestamp\").n_unique().alias(\"unique_hours\"),\n",
" ]\n",
")\n",
"\n",
"print(f\"\\nDate range: {date_range['start'][0]} to {date_range['end'][0]}\")\n",
"print(f\"Unique hours: {date_range['unique_hours'][0]:,}\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "c9658965",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.300212Z",
"iopub.status.busy": "2026-06-13T02:33:30.300078Z",
"iopub.status.idle": "2026-06-13T02:33:30.310731Z",
"shell.execute_reply": "2026-06-13T02:33:30.310442Z"
},
"papermill": {
"duration": 0.012868,
"end_time": "2026-06-13T02:33:30.311341+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.298473+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Symbol Statistics (top 5 by volume):\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>rows</th><th>start</th><th>end</th><th>avg_price</th><th>avg_volume</th></tr><tr><td>str</td><td>u32</td><td>datetime[ms, UTC]</td><td>datetime[ms, UTC]</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>&quot;DOGEUSDT&quot;</td><td>48015</td><td>2020-07-10 09:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>0.137195</td><td>2.7592e8</td></tr><tr><td>&quot;XRPUSDT&quot;</td><td>52360</td><td>2020-01-06 08:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>0.89747</td><td>5.2917e7</td></tr><tr><td>&quot;ADAUSDT&quot;</td><td>51880</td><td>2020-01-31 08:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>0.644055</td><td>3.0205e7</td></tr><tr><td>&quot;SUIUSDT&quot;</td><td>23360</td><td>2023-05-03 16:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>1.917873</td><td>1.2587e7</td></tr><tr><td>&quot;NEARUSDT&quot;</td><td>45568</td><td>2020-10-15 08:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>4.297627</td><td>2.1940e6</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5, 6)\n",
"┌──────────┬───────┬─────────────────────────┬─────────────────────────┬───────────┬────────────┐\n",
"│ symbol ┆ rows ┆ start ┆ end ┆ avg_price ┆ avg_volume │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ u32 ┆ datetime[ms, UTC] ┆ datetime[ms, UTC] ┆ f64 ┆ f64 │\n",
"╞══════════╪═══════╪═════════════════════════╪═════════════════════════╪═══════════╪════════════╡\n",
"│ DOGEUSDT ┆ 48015 ┆ 2020-07-10 09:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.137195 ┆ 2.7592e8 │\n",
"│ XRPUSDT ┆ 52360 ┆ 2020-01-06 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.89747 ┆ 5.2917e7 │\n",
"│ ADAUSDT ┆ 51880 ┆ 2020-01-31 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.644055 ┆ 3.0205e7 │\n",
"│ SUIUSDT ┆ 23360 ┆ 2023-05-03 16:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 1.917873 ┆ 1.2587e7 │\n",
"│ NEARUSDT ┆ 45568 ┆ 2020-10-15 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 4.297627 ┆ 2.1940e6 │\n",
"└──────────┴───────┴─────────────────────────┴─────────────────────────┴───────────┴────────────┘"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Per-symbol statistics\n",
"symbol_stats = per_asset_stats(\n",
" ohlcv,\n",
" time_col=\"timestamp\",\n",
" asset_col=\"symbol\",\n",
" price_col=\"close\",\n",
" volume_col=\"volume\",\n",
")\n",
"\n",
"print(\"\\nSymbol Statistics (top 5 by volume):\")\n",
"symbol_stats.sort(\"avg_volume\", descending=True).head(5)"
]
},
{
"cell_type": "markdown",
"id": "74c26d5a",
"metadata": {
"papermill": {
"duration": 0.001876,
"end_time": "2026-06-13T02:33:30.315436+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.313560+00:00",
"status": "completed"
}
},
"source": [
"## 3. Premium Index Data\n",
"\n",
"The premium index measures the spread between perpetual futures and spot prices:\n",
"\n",
"**Premium = (Perpetual Price - Spot Price) / Spot Price**\n",
"\n",
"- Positive premium: Futures above spot (bullish sentiment)\n",
"- Negative premium: Futures below spot (bearish sentiment)\n",
"\n",
"### Units\n",
"\n",
"Premium values are stored as **decimals** (0.001 = 0.1%). When displaying,\n",
"multiply by 100 for percentage representation."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "5bd439f9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.318383Z",
"iopub.status.busy": "2026-06-13T02:33:30.318294Z",
"iopub.status.idle": "2026-06-13T02:33:30.330166Z",
"shell.execute_reply": "2026-06-13T02:33:30.329851Z"
},
"papermill": {
"duration": 0.013671,
"end_time": "2026-06-13T02:33:30.330466+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.316795+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Premium Dataset ===\n",
"Shape: (107839, 6)\n",
"Columns: ['timestamp', 'symbol', 'premium_index_open', 'premium_index_high', 'premium_index_low', 'premium_index_close']\n"
]
}
],
"source": [
"premium = load_crypto_premium(frequency=\"8h\")\n",
"print(\"=== Premium Dataset ===\")\n",
"print(f\"Shape: {premium.shape}\")\n",
"print(f\"Columns: {premium.columns}\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "ac1c5ece",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.333347Z",
"iopub.status.busy": "2026-06-13T02:33:30.333260Z",
"iopub.status.idle": "2026-06-13T02:33:30.336126Z",
"shell.execute_reply": "2026-06-13T02:33:30.335774Z"
},
"papermill": {
"duration": 0.004926,
"end_time": "2026-06-13T02:33:30.336661+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.331735+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Premium Range (decimals):\n",
" Mean: -0.000111 (-0.0111%)\n",
" Std: 0.001137 (0.1137%)\n",
" Min: -0.191547 (-19.1547%)\n",
" Max: 0.012701 (1.2701%)\n"
]
}
],
"source": [
"# Premium range — values are decimals, not percentages\n",
"premium_range = premium.select(\n",
" [\n",
" pl.col(\"premium_index_close\").min().alias(\"min\"),\n",
" pl.col(\"premium_index_close\").max().alias(\"max\"),\n",
" pl.col(\"premium_index_close\").mean().alias(\"mean\"),\n",
" pl.col(\"premium_index_close\").std().alias(\"std\"),\n",
" ]\n",
")\n",
"\n",
"print(\"\\nPremium Range (decimals):\")\n",
"print(f\" Mean: {premium_range['mean'][0]:.6f} ({premium_range['mean'][0] * 100:.4f}%)\")\n",
"print(f\" Std: {premium_range['std'][0]:.6f} ({premium_range['std'][0] * 100:.4f}%)\")\n",
"print(f\" Min: {premium_range['min'][0]:.6f} ({premium_range['min'][0] * 100:.4f}%)\")\n",
"print(f\" Max: {premium_range['max'][0]:.6f} ({premium_range['max'][0] * 100:.4f}%)\")"
]
},
{
"cell_type": "markdown",
"id": "bd327365",
"metadata": {
"papermill": {
"duration": 0.001018,
"end_time": "2026-06-13T02:33:30.338980+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.337962+00:00",
"status": "completed"
}
},
"source": [
"## 4. Data Quality"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b2303360",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.341624Z",
"iopub.status.busy": "2026-06-13T02:33:30.341557Z",
"iopub.status.idle": "2026-06-13T02:33:30.349627Z",
"shell.execute_reply": "2026-06-13T02:33:30.349335Z"
},
"papermill": {
"duration": 0.009977,
"end_time": "2026-06-13T02:33:30.350085+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.340108+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== OHLC Invariants ===\n",
" [OK] high_gte_low: 100.00%\n",
" [OK] high_gte_open: 100.00%\n",
" [OK] high_gte_close: 100.00%\n",
" [OK] low_lte_open: 100.00%\n",
" [OK] low_lte_close: 100.00%\n",
" [OK] volume_non_negative: 100.00%\n"
]
}
],
"source": [
"# OHLC invariants\n",
"invariants = check_ohlc_invariants(ohlcv)\n",
"print(\"=== OHLC Invariants ===\")\n",
"for row in invariants.iter_rows(named=True):\n",
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "81337cc6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.352943Z",
"iopub.status.busy": "2026-06-13T02:33:30.352862Z",
"iopub.status.idle": "2026-06-13T02:33:30.355138Z",
"shell.execute_reply": "2026-06-13T02:33:30.354906Z"
},
"papermill": {
"duration": 0.004193,
"end_time": "2026-06-13T02:33:30.355537+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.351344+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Null values: OHLCV=0, Premium=0\n"
]
}
],
"source": [
"# Check for nulls\n",
"ohlcv_nulls = ohlcv.null_count().sum_horizontal()[0]\n",
"premium_nulls = premium.null_count().sum_horizontal()[0]\n",
"print(f\"\\nNull values: OHLCV={ohlcv_nulls}, Premium={premium_nulls}\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "ed1e0266",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.358207Z",
"iopub.status.busy": "2026-06-13T02:33:30.358135Z",
"iopub.status.idle": "2026-06-13T02:33:30.363600Z",
"shell.execute_reply": "2026-06-13T02:33:30.363206Z"
},
"papermill": {
"duration": 0.007243,
"end_time": "2026-06-13T02:33:30.363925+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.356682+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Gaps > 1 hour in BTC: 0\n"
]
}
],
"source": [
"# Check for gaps > 1 hour (use BTC as reference)\n",
"btc = ohlcv.filter(pl.col(\"symbol\") == \"BTCUSDT\").sort(\"timestamp\")\n",
"btc_gaps = btc.with_columns(pl.col(\"timestamp\").diff().dt.total_hours().alias(\"hours_diff\")).filter(\n",
" pl.col(\"hours_diff\") > 1\n",
")\n",
"\n",
"print(f\"\\nGaps > 1 hour in BTC: {len(btc_gaps)}\")\n",
"if len(btc_gaps) > 0:\n",
" print(\"(Small gaps expected during exchange maintenance)\")"
]
},
{
"cell_type": "markdown",
"id": "ffa0b798",
"metadata": {
"papermill": {
"duration": 0.001072,
"end_time": "2026-06-13T02:33:30.366245+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.365173+00:00",
"status": "completed"
}
},
"source": [
"## 5. Joining OHLCV and Premium\n",
"\n",
"Use left join to preserve all OHLCV rows and identify missing premium coverage."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "c311a381",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.369052Z",
"iopub.status.busy": "2026-06-13T02:33:30.368962Z",
"iopub.status.idle": "2026-06-13T02:33:30.412049Z",
"shell.execute_reply": "2026-06-13T02:33:30.411644Z"
},
"papermill": {
"duration": 0.045382,
"end_time": "2026-06-13T02:33:30.412749+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.367367+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== Join Coverage ===\n",
"OHLCV rows: 866,484\n",
"Premium rows: 107,839\n",
"Combined rows: 866,484\n",
"Missing premium: 758,716 (87.56%)\n",
"Coverage: 12.44%\n"
]
}
],
"source": [
"# Left join to identify missing premium data\n",
"combined = ohlcv.join(premium, on=[\"timestamp\", \"symbol\"], how=\"left\")\n",
"\n",
"# Coverage analysis\n",
"total_rows = len(combined)\n",
"missing_premium = combined.filter(pl.col(\"premium_index_close\").is_null()).height\n",
"coverage_pct = (total_rows - missing_premium) / total_rows * 100\n",
"\n",
"print(\"=== Join Coverage ===\")\n",
"print(f\"OHLCV rows: {len(ohlcv):,}\")\n",
"print(f\"Premium rows: {len(premium):,}\")\n",
"print(f\"Combined rows: {total_rows:,}\")\n",
"print(f\"Missing premium: {missing_premium:,} ({100 - coverage_pct:.2f}%)\")\n",
"print(f\"Coverage: {coverage_pct:.2f}%\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "011f6a3b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:30.417422Z",
"iopub.status.busy": "2026-06-13T02:33:30.417324Z",
"iopub.status.idle": "2026-06-13T02:33:30.434943Z",
"shell.execute_reply": "2026-06-13T02:33:30.434671Z"
},
"papermill": {
"duration": 0.020165,
"end_time": "2026-06-13T02:33:30.435349+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.415184+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Missing premium by symbol (top 5):\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>&quot;BTCUSDT&quot;</td><td>46053</td></tr><tr><td>&quot;ETHUSDT&quot;</td><td>46053</td></tr><tr><td>&quot;XRPUSDT&quot;</td><td>45833</td></tr><tr><td>&quot;LINKUSDT&quot;</td><td>45710</td></tr><tr><td>&quot;ADAUSDT&quot;</td><td>45416</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5, 2)\n",
"┌──────────┬───────┐\n",
"│ symbol ┆ len │\n",
"│ --- ┆ --- │\n",
"│ str ┆ u32 │\n",
"╞══════════╪═══════╡\n",
"│ BTCUSDT ┆ 46053 │\n",
"│ ETHUSDT ┆ 46053 │\n",
"│ XRPUSDT ┆ 45833 │\n",
"│ LINKUSDT ┆ 45710 │\n",
"│ ADAUSDT ┆ 45416 │\n",
"└──────────┴───────┘"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Where does the missing premium concentrate?\n",
"missing_by_symbol = (\n",
" combined.filter(pl.col(\"premium_index_close\").is_null())\n",
" .group_by(\"symbol\")\n",
" .len()\n",
" .sort(\"len\", descending=True)\n",
")\n",
"print(\"Missing premium by symbol (top 5):\")\n",
"missing_by_symbol.head(5)"
]
},
{
"cell_type": "markdown",
"id": "869f500e",
"metadata": {
"papermill": {
"duration": 0.001191,
"end_time": "2026-06-13T02:33:30.437914+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:30.436723+00:00",
"status": "completed"
}
},
"source": [
"## Key Takeaways\n",
"\n",
"1. **24/7 trading**: Crypto runs continuously — 52,608 unique hours across\n",
" six full years (2020-01-01 to 2025-12-31), ~8,760 hours/year for the\n",
" longest-history symbols.\n",
"2. **Universe**: 19 perpetual contracts span 866,484 OHLCV bars; symbol\n",
" coverage is non-uniform because contracts list at different dates.\n",
"3. **Premium units**: Stored as decimals (0.001 = 0.1%); always multiply by\n",
" 100 for percentage display in figures or text.\n",
"4. **Frequency mismatch**: OHLCV is hourly, premium is 8-hourly, so a left\n",
" join lands ~12.4% premium coverage on the hourly grid by design.\n",
"5. **Clean data**: OHLC invariants hold for 100% of records on this\n",
" snapshot; BTC has zero gaps > 1 hour.\n",
"\n",
"## Next Steps\n",
"\n",
"- `11_crypto_premium_analysis`: Premium dynamics, basis seasonality, and\n",
" alignment to the 8-hourly funding cadence.\n",
"- Chapter 8: Feature engineering for premium signals\n",
" (`case_studies/crypto_perps_funding/03_financial_features.py`).\n",
"- Chapter 16: Backtests for the funding-arbitrage case study."
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "tags,-all"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.3"
},
"papermill": {
"default_parameters": {},
"duration": 2.819325,
"end_time": "2026-06-13T02:33:31.153715+00:00",
"environment_variables": {},
"exception": null,
"input_path": "02_financial_data_universe/10_crypto_perps_eda.ipynb",
"output_path": "02_financial_data_universe/10_crypto_perps_eda.ipynb",
"parameters": {},
"start_time": "2026-06-13T02:33:28.334390+00:00",
"version": "2.7.0"
},
"ml4t_provenance": {
"source_py_blob": "ad9e3464a73b7086edefb3b2413e3c50096de365",
"executed_at": "2026-06-13T02:33:31.465291+00:00",
"executor": "cpu-local",
"production": true,
"parameters": {},
"notes": "executed-state finalization batch (2026-06-13 run)"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02_financial_data_universe/10_crypto_perps_eda.py
+231
View File
@@ -0,0 +1,231 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Crypto Perps — Exploratory Data Analysis
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Profile the Binance Futures hourly OHLCV dataset for 19 perpetual contracts
# alongside the 8-hourly Premium Index that captures the perpetual–spot
# basis. The notebook anchors data shape, units, coverage, and OHLC integrity
# before the strategy work begins in Chapter 6.
#
# ## Learning Objectives
#
# - Load hourly OHLCV and 8-hourly premium index data via the canonical loaders.
# - Document premium-index units (decimal, multiply by 100 for percent).
# - Quantify cross-frequency join coverage between OHLCV and premium data.
# - Run OHLC invariant checks and inspect for time-stamp gaps.
#
# ## Book Reference
#
# Chapter 2 §2.2 (asset-class market data landscape — digital assets).
#
# ## Prerequisites
#
# - Familiarity with daily OHLCV equity data (`01_us_equities_eda`).
# - The Binance Futures parquet at `$ML4T_DATA_PATH/crypto/` (OHLCV + premium).
# - Methodology continues in `11_crypto_premium_analysis`; the case-study
# pipeline lives under `case_studies/crypto_perps_funding/`.
# %%
"""Crypto Perps EDA — hourly OHLCV and premium index exploration."""
import polars as pl
from data import load_crypto_perps, load_crypto_premium
from utils.data_quality import check_ohlc_invariants, per_asset_stats
# %% tags=["parameters"]
MAX_SYMBOLS = 0 # 0 = all symbols
# %% [markdown]
# ## 1. Load and Inspect OHLCV
#
# Hourly OHLCV data from Binance Futures for 19 cryptocurrencies.
# Trading is 24/7 (8,760 hours/year vs 252 days for equities).
# %%
ohlcv = load_crypto_perps(frequency="1h")
print("=== OHLCV Dataset ===")
print(f"Shape: {ohlcv.shape}")
print(f"Columns: {ohlcv.columns}")
# %%
# Schema overview
print("\nSchema:")
for col, dtype in ohlcv.schema.items():
print(f" {col}: {dtype}")
# %% [markdown]
# ## 2. Coverage Summary
# %%
# Symbols and date range
symbols = ohlcv["symbol"].unique().sort().to_list()
print("=== Coverage ===")
print(f"Number of symbols: {len(symbols)}")
print(f"\nSymbols: {', '.join(symbols)}")
# %%
# Overall date range
date_range = ohlcv.select(
[
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.col("timestamp").n_unique().alias("unique_hours"),
]
)
print(f"\nDate range: {date_range['start'][0]} to {date_range['end'][0]}")
print(f"Unique hours: {date_range['unique_hours'][0]:,}")
# %%
# Per-symbol statistics
symbol_stats = per_asset_stats(
ohlcv,
time_col="timestamp",
asset_col="symbol",
price_col="close",
volume_col="volume",
)
print("\nSymbol Statistics (top 5 by volume):")
symbol_stats.sort("avg_volume", descending=True).head(5)
# %% [markdown]
# ## 3. Premium Index Data
#
# The premium index measures the spread between perpetual futures and spot prices:
#
# **Premium = (Perpetual Price - Spot Price) / Spot Price**
#
# - Positive premium: Futures above spot (bullish sentiment)
# - Negative premium: Futures below spot (bearish sentiment)
#
# ### Units
#
# Premium values are stored as **decimals** (0.001 = 0.1%). When displaying,
# multiply by 100 for percentage representation.
# %%
premium = load_crypto_premium(frequency="8h")
print("=== Premium Dataset ===")
print(f"Shape: {premium.shape}")
print(f"Columns: {premium.columns}")
# %%
# Premium range — values are decimals, not percentages
premium_range = premium.select(
[
pl.col("premium_index_close").min().alias("min"),
pl.col("premium_index_close").max().alias("max"),
pl.col("premium_index_close").mean().alias("mean"),
pl.col("premium_index_close").std().alias("std"),
]
)
print("\nPremium Range (decimals):")
print(f" Mean: {premium_range['mean'][0]:.6f} ({premium_range['mean'][0] * 100:.4f}%)")
print(f" Std: {premium_range['std'][0]:.6f} ({premium_range['std'][0] * 100:.4f}%)")
print(f" Min: {premium_range['min'][0]:.6f} ({premium_range['min'][0] * 100:.4f}%)")
print(f" Max: {premium_range['max'][0]:.6f} ({premium_range['max'][0] * 100:.4f}%)")
# %% [markdown]
# ## 4. Data Quality
# %%
# OHLC invariants
invariants = check_ohlc_invariants(ohlcv)
print("=== OHLC Invariants ===")
for row in invariants.iter_rows(named=True):
status = "[OK]" if row["valid_pct"] >= 99.99 else "[WARN]"
print(f" {status} {row['check']}: {row['valid_pct']:.2f}%")
# %%
# Check for nulls
ohlcv_nulls = ohlcv.null_count().sum_horizontal()[0]
premium_nulls = premium.null_count().sum_horizontal()[0]
print(f"\nNull values: OHLCV={ohlcv_nulls}, Premium={premium_nulls}")
# %%
# Check for gaps > 1 hour (use BTC as reference)
btc = ohlcv.filter(pl.col("symbol") == "BTCUSDT").sort("timestamp")
btc_gaps = btc.with_columns(pl.col("timestamp").diff().dt.total_hours().alias("hours_diff")).filter(
pl.col("hours_diff") > 1
)
print(f"\nGaps > 1 hour in BTC: {len(btc_gaps)}")
if len(btc_gaps) > 0:
print("(Small gaps expected during exchange maintenance)")
# %% [markdown]
# ## 5. Joining OHLCV and Premium
#
# Use left join to preserve all OHLCV rows and identify missing premium coverage.
# %%
# Left join to identify missing premium data
combined = ohlcv.join(premium, on=["timestamp", "symbol"], how="left")
# Coverage analysis
total_rows = len(combined)
missing_premium = combined.filter(pl.col("premium_index_close").is_null()).height
coverage_pct = (total_rows - missing_premium) / total_rows * 100
print("=== Join Coverage ===")
print(f"OHLCV rows: {len(ohlcv):,}")
print(f"Premium rows: {len(premium):,}")
print(f"Combined rows: {total_rows:,}")
print(f"Missing premium: {missing_premium:,} ({100 - coverage_pct:.2f}%)")
print(f"Coverage: {coverage_pct:.2f}%")
# %%
# Where does the missing premium concentrate?
missing_by_symbol = (
combined.filter(pl.col("premium_index_close").is_null())
.group_by("symbol")
.len()
.sort("len", descending=True)
)
print("Missing premium by symbol (top 5):")
missing_by_symbol.head(5)
# %% [markdown]
# ## Key Takeaways
#
# 1. **24/7 trading**: Crypto runs continuously — 52,608 unique hours across
# six full years (2020-01-01 to 2025-12-31), ~8,760 hours/year for the
# longest-history symbols.
# 2. **Universe**: 19 perpetual contracts span 866,484 OHLCV bars; symbol
# coverage is non-uniform because contracts list at different dates.
# 3. **Premium units**: Stored as decimals (0.001 = 0.1%); always multiply by
# 100 for percentage display in figures or text.
# 4. **Frequency mismatch**: OHLCV is hourly, premium is 8-hourly, so a left
# join lands ~12.4% premium coverage on the hourly grid by design.
# 5. **Clean data**: OHLC invariants hold for 100% of records on this
# snapshot; BTC has zero gaps > 1 hour.
#
# ## Next Steps
#
# - `11_crypto_premium_analysis`: Premium dynamics, basis seasonality, and
# alignment to the 8-hourly funding cadence.
# - Chapter 8: Feature engineering for premium signals
# (`case_studies/crypto_perps_funding/03_financial_features.py`).
# - Chapter 16: Backtests for the funding-arbitrage case study.
02_financial_data_universe/11_crypto_premium_analysis.ipynb
+46863
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/11_crypto_premium_analysis.py
+594
View File
@@ -0,0 +1,594 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# formats: ipynb,py:percent
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Crypto Premium Index: Funding Rate Arbitrage Data
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Explore the Binance perpetual-futures premium index — the per-period deviation between
# perpetual and spot prices that determines the funding rate paid every 8 hours. The
# notebook profiles 19 USDT-margined perpetuals from 2020-01 to 2025-12 and turns the
# raw premium series into estimated funding APY for the funding-arbitrage case study.
#
# ## Learning Objectives
# - Load the 8-hour premium-index panel and read its schema.
# - Characterize the distribution and time-series behavior of BTC premium.
# - Compare premium volatility across majors and altcoins.
# - Translate premium into Binance's clamped funding rate and an annualized return.
#
# ## Book reference
# Chapter 2, §2.2 (asset-class market data — crypto datasets). The funding-arbitrage
# case study built on this dataset lives in `case_studies/crypto_perps_funding/`.
#
# ## Prerequisites
# - Crypto perpetual + premium parquet files materialized under `ML4T_DATA_PATH`.
# - Loader `data.load_crypto_premium`.
#
# ---
# %%
"""Crypto Premium Index — Funding rate arbitrage data exploration."""
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_crypto_premium
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ---
#
# ## Section 1: Understanding the Premium Index
#
# ### What is the Premium Index?
#
# The **Premium Index** measures the deviation between perpetual futures prices and spot prices:
#
# $$\text{Premium Index} = \frac{\text{Perpetual Price} - \text{Spot Price}}{\text{Spot Price}}$$
#
# ### Key Properties:
#
# 1. **Positive Premium**: Perpetual > Spot → Longs pay Shorts (bullish sentiment)
# 2. **Negative Premium**: Perpetual < Spot → Shorts pay Longs (bearish sentiment)
# 3. **Funding Rate**: Derived from premium index, paid every 8 hours on Binance
#
# ### Arbitrage Opportunity
#
# When premium is significantly positive:
# - **Long Spot** + **Short Perpetual** = Collect funding payments
# - Market-neutral position captures the funding rate
#
# When premium is significantly negative:
# - **Short Spot** + **Long Perpetual** = Collect funding payments
# %%
# Load the combined premium index data
premium_df = load_crypto_premium(frequency="8h")
print(f"Total rows: {len(premium_df):,}")
print(f"Columns: {premium_df.columns}")
print("\nSchema:")
for col, dtype in premium_df.schema.items():
print(f" {col}: {dtype}")
# %%
# Overview by asset
symbol_stats = (
premium_df.group_by("symbol")
.agg(
[
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.len().alias("rows"),
pl.col("premium_index_close").mean().alias("avg_premium"),
pl.col("premium_index_close").std().alias("std_premium"),
]
)
.sort("rows", descending=True)
)
symbol_stats
# %%
# Sample data - BTC premium index
btc_premium = premium_df.filter(pl.col("symbol") == "BTCUSDT").sort("timestamp")
print(f"BTC Premium Index: {len(btc_premium):,} 8h observations")
print(f"Date range: {btc_premium['timestamp'].min()} to {btc_premium['timestamp'].max()}")
btc_premium.head(10)
# %% [markdown]
# ---
#
# ## Section 2: Premium Index Distribution
#
# Understanding the distribution of premium values is crucial for:
# 1. Setting entry/exit thresholds for arbitrage
# 2. Risk management (tail events)
# 3. Comparing opportunities across assets
# %%
# BTC Premium distribution
btc_close = btc_premium["premium_index_close"].to_numpy()
# Convert to basis points for readability
btc_close_bps = btc_close * 10000
fig = go.Figure()
fig.add_trace(
go.Histogram(
x=btc_close_bps,
nbinsx=100,
name="BTC Premium",
marker_color="#F7931A",
)
)
# Add vertical lines for mean and +-2 std
mean_val = np.mean(btc_close_bps)
std_val = np.std(btc_close_bps)
fig.add_vline(
x=mean_val, line_dash="dash", line_color="red", annotation_text=f"Mean: {mean_val:.1f} bps"
)
fig.add_vline(
x=mean_val + 2 * std_val,
line_dash="dot",
line_color="green",
annotation_text=f"+2σ: {mean_val + 2 * std_val:.1f} bps",
)
fig.add_vline(
x=mean_val - 2 * std_val,
line_dash="dot",
line_color="green",
annotation_text=f"-2σ: {mean_val - 2 * std_val:.1f} bps",
)
fig.update_layout(
title="BTC Premium Index Distribution (Basis Points)",
xaxis_title="Premium Index (bps)",
yaxis_title="Frequency",
height=400,
template="plotly_white",
)
fig.show()
# %%
print("BTC Premium Statistics:")
print(f" Mean: {mean_val:.2f} bps")
print(f" Std: {std_val:.2f} bps")
print(f" Min: {np.min(btc_close_bps):.2f} bps")
print(f" Max: {np.max(btc_close_bps):.2f} bps")
print(f" Skew: {((btc_close_bps - mean_val) ** 3).mean() / std_val**3:.2f}")
# %%
# Compare premium distributions across major assets
major_symbols = ["BTCUSDT", "ETHUSDT", "SOLUSDT", "BNBUSDT"]
fig = make_subplots(rows=2, cols=2, subplot_titles=major_symbols)
colors = ["#F7931A", "#627EEA", "#00FFA3", "#F3BA2F"]
for idx, (symbol, color) in enumerate(zip(major_symbols, colors, strict=False)):
row = idx // 2 + 1
col = idx % 2 + 1
data = premium_df.filter(pl.col("symbol") == symbol)["premium_index_close"].to_numpy() * 10000
fig.add_trace(
go.Histogram(x=data, nbinsx=50, marker_color=color, name=symbol), row=row, col=col
)
fig.update_layout(
title="Premium Index Distributions (bps) - Major Cryptos",
height=500,
showlegend=False,
template="plotly_white",
)
fig.show()
# %% [markdown]
# ---
#
# ## Section 3: Time Series Analysis
#
# Premium index varies over time based on market sentiment. Let's analyze:
# 1. Long-term trends
# 2. Regime changes (bull vs bear markets)
# 3. Correlation with price movements
# %%
# BTC premium time series.
# Data is on an 8h cadence (3 obs per day), so 30 days = 90 windows.
PERIODS_PER_DAY = 3 # Binance funding interval is 8h
ROLLING_WINDOW_DAYS = 30
btc_bps = btc_premium.with_columns(
(pl.col("premium_index_close") * 10000).alias("premium_bps"),
).with_columns(
pl.col("premium_index_close")
.rolling_mean(window_size=ROLLING_WINDOW_DAYS * PERIODS_PER_DAY)
.alias("rolling_30d"),
)
# %%
# Plot raw 8h premium and 30-day rolling mean
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
vertical_spacing=0.1,
subplot_titles=[
"BTC Premium Index (bps, 8h observations)",
f"{ROLLING_WINDOW_DAYS}-Day Rolling Average",
],
)
fig.add_trace(
go.Scatter(
x=btc_bps["timestamp"].to_list(),
y=btc_bps["premium_bps"].to_list(),
mode="lines",
name="8h Premium",
line=dict(color="#F7931A", width=1),
opacity=0.6,
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=btc_bps["timestamp"].to_list(),
y=(btc_bps["rolling_30d"] * 10000).to_list(),
mode="lines",
name="30-Day Rolling Avg",
line=dict(color="red", width=2),
),
row=2,
col=1,
)
fig.add_hline(y=0, line_dash="dash", line_color="gray", row=1, col=1)
fig.add_hline(y=0, line_dash="dash", line_color="gray", row=2, col=1)
fig.update_layout(height=600, template="plotly_white", showlegend=False)
fig.update_yaxes(title_text="Premium (bps)", row=1, col=1)
fig.update_yaxes(title_text="Premium (bps)", row=2, col=1)
fig.show()
# %%
# Report the observed BTC range so the reader can size the y-axis.
btc_bps_series = btc_bps["premium_bps"]
print(f"BTC premium range: {btc_bps_series.min():.1f} to {btc_bps_series.max():.1f} bps")
# %%
# Identify premium regimes
btc_regimes = btc_premium.with_columns(
[
# Define regimes based on premium level
pl.when(pl.col("premium_index_close") > 0.001)
.then(pl.lit("High Premium (Bullish)"))
.when(pl.col("premium_index_close") < -0.001)
.then(pl.lit("Low Premium (Bearish)"))
.otherwise(pl.lit("Neutral"))
.alias("regime"),
# Year for grouping
pl.col("timestamp").dt.year().alias("year"),
]
)
# Regime distribution by year (counts of 8h periods, ~1095 per full year)
regime_dist = (
btc_regimes.group_by(["year", "regime"])
.agg(pl.len().alias("periods_8h"))
.sort(["year", "regime"])
)
regime_dist.pivot(on="regime", index="year", values="periods_8h").fill_null(0)
# %% [markdown]
# ---
#
# ## Section 4: Cross-Asset Premium Comparison
#
# Different cryptocurrencies have different premium dynamics:
# - **BTC/ETH**: Lower volatility, tighter premiums
# - **Altcoins**: Higher volatility, wider premium swings
#
# This affects arbitrage opportunity selection.
# %%
# Calculate premium statistics for all assets
premium_stats = (
premium_df.group_by("symbol")
.agg(
[
pl.col("premium_index_close").mean().alias("mean_premium"),
pl.col("premium_index_close").std().alias("std_premium"),
pl.col("premium_index_close").min().alias("min_premium"),
pl.col("premium_index_close").max().alias("max_premium"),
# Percentage of time premium > 10 bps (profitable arbitrage threshold)
(pl.col("premium_index_close").abs() > 0.001).mean().alias("pct_above_10bps"),
]
)
.sort("std_premium", descending=True)
)
# Convert to basis points for display
premium_stats_bps = premium_stats.with_columns(
[
(pl.col("mean_premium") * 10000).round(2).alias("mean_bps"),
(pl.col("std_premium") * 10000).round(2).alias("std_bps"),
(pl.col("min_premium") * 10000).round(2).alias("min_bps"),
(pl.col("max_premium") * 10000).round(2).alias("max_bps"),
(pl.col("pct_above_10bps") * 100).round(1).alias("pct_above_10bps"),
]
).select(["symbol", "mean_bps", "std_bps", "min_bps", "max_bps", "pct_above_10bps"])
premium_stats_bps
# %%
# Scatter: Premium volatility vs mean premium
fig = px.scatter(
premium_stats_bps.to_pandas(),
x="std_bps",
y="mean_bps",
size="pct_above_10bps",
color="symbol",
hover_name="symbol",
title="Premium Volatility vs Mean Premium",
labels={
"std_bps": "Premium Volatility (bps)",
"mean_bps": "Mean Premium (bps)",
"pct_above_10bps": "% Time > 10bps",
},
)
fig.update_layout(
height=600,
template="plotly_white",
legend=dict(
orientation="h",
yanchor="top",
y=-0.15,
xanchor="center",
x=0.5,
),
margin=dict(b=120),
)
fig.show()
print("\nInterpretation:")
print("- Top-right quadrant: High volatility, positive bias (bullish altcoins)")
print("- Larger bubbles: More arbitrage opportunities (premium often > 10bps)")
# %%
# Monthly premium heatmap
# Note: Some months have extreme values (e.g., SOL during FTX collapse at -72 bps)
# We clip the color scale at ±20 bps for better visualization of typical patterns
monthly_premium = (
premium_df.with_columns([pl.col("timestamp").dt.strftime("%Y-%m").alias("month")])
.group_by(["symbol", "month"])
.agg(pl.col("premium_index_close").mean().alias("avg_premium"))
)
# Pivot for heatmap
heatmap_data = monthly_premium.pivot(on="month", index="symbol", values="avg_premium").sort(
"symbol"
)
# Get month columns in order
month_cols = sorted([c for c in heatmap_data.columns if c != "symbol"])
assets = heatmap_data["symbol"].to_list()
# Extract values for heatmap
z_values = heatmap_data.select(month_cols).to_numpy() * 10000 # Convert to bps
# Clip color scale at ±20 bps for better visualization
COLOR_CLIP_BPS = 20
# %%
fig = go.Figure(
data=go.Heatmap(
z=z_values,
x=month_cols,
y=assets,
colorscale="RdBu",
zmid=0,
zmin=-COLOR_CLIP_BPS,
zmax=COLOR_CLIP_BPS,
colorbar=dict(title="Premium (bps)"),
)
)
fig.update_layout(
title=f"Monthly Average Premium by Asset (bps, color clipped at ±{COLOR_CLIP_BPS})",
xaxis_title="Month",
yaxis_title="Symbol",
height=600,
template="plotly_white",
)
fig.show()
# Report extremes that exceed color scale (shown as saturated colors)
extremes = (
monthly_premium.filter(pl.col("avg_premium").abs() * 10000 > COLOR_CLIP_BPS)
.with_columns((pl.col("avg_premium") * 10000).round(1).alias("avg_bps"))
.select(["symbol", "month", "avg_bps"])
.sort("avg_bps")
)
extremes
# %% [markdown]
# ---
#
# ## Section 5: Funding Rate Estimation
#
# Binance calculates funding rates from premium index every 8 hours:
#
# $$\text{Funding Rate} = \text{clamp}(\text{Premium Index}, -0.05\%, 0.05\%) + \text{Interest Rate}$$
#
# Where Interest Rate ≈ 0.01% (0.03%/day).
#
# **Annualized Return** from funding collection:
# $$\text{APY} = \text{Funding Rate} \times 3 \times 365$$
# %%
# Calculate estimated funding rates using native Polars expressions
# Formula: funding_rate = clamp(premium, -0.05%, 0.05%) + interest_rate
# Interest rate ≈ 0.01% per 8h (0.0001)
INTEREST_RATE = 0.0001
btc_funding = btc_premium.with_columns(
# Clamp premium to [-0.05%, 0.05%] and add interest rate
(pl.col("premium_index_close").clip(-0.0005, 0.0005) + INTEREST_RATE).alias("est_funding_rate"),
).with_columns(
# Annualized return: 3 funding periods/day * 365 days * 100 for percentage
(pl.col("est_funding_rate") * 3 * 365 * 100).alias("annualized_pct"),
)
avg_funding_rate = float(btc_funding["est_funding_rate"].mean())
ann_min = float(btc_funding["annualized_pct"].min())
ann_max = float(btc_funding["annualized_pct"].max())
ann_mean = float(btc_funding["annualized_pct"].mean())
print("BTC Estimated Funding Rate Analysis:")
print(f" Average funding rate (per 8h): {avg_funding_rate * 100:.4f}%")
print(f" Annualized return (avg): {ann_mean:.1f}%")
print(f" Annualized return (max): {ann_max:.1f}%")
print(f" Annualized return (min): {ann_min:.1f}%")
# %%
# Visualize annualized funding returns over time
# Note: Funding rate is clamped to ±0.05% per period, so annualized range is bounded
# to approximately ±55% (3 periods/day × 365 days × 0.05%)
fig = go.Figure()
fig.add_trace(
go.Scatter(
x=btc_funding["timestamp"].to_list(),
y=btc_funding["annualized_pct"].to_list(),
mode="lines",
name="Annualized Funding Return",
line=dict(color="#F7931A", width=1),
)
)
# Add horizontal lines for reference
fig.add_hline(y=0, line_dash="dash", line_color="gray")
fig.add_hline(y=20, line_dash="dot", line_color="green", annotation_text="20% APY")
fig.add_hline(y=-20, line_dash="dot", line_color="red", annotation_text="-20% APY")
y_padding = 10
fig.update_layout(
title="BTC Estimated Annualized Funding Return (%)",
xaxis_title="Date",
yaxis_title="Annualized Return (%)",
yaxis=dict(range=[ann_min - y_padding, ann_max + y_padding]),
height=400,
template="plotly_white",
)
fig.show()
print(f"Annualized funding return range: {ann_min:.1f}% to {ann_max:.1f}%")
# %%
# 8h periods where the estimated funding APY exceeds ±20%.
# Note: the funding-rate clamp pins per-period funding at ±0.0005 + 0.0001 interest,
# so the APY ceiling is 3 × 365 × 0.0006 × 100 ≈ 65.7% (and floor ≈ −43.8%);
# the top rows therefore all sit at the clamp.
high_conviction = btc_funding.filter(pl.col("annualized_pct").abs() > 20)
print(
f"High-conviction periods (|APY| > 20%): {len(high_conviction):,} of {len(btc_funding):,} 8h periods"
)
print(f"Share of total: {len(high_conviction) / len(btc_funding) * 100:.1f}%")
(
high_conviction.sort("annualized_pct", descending=True)
.head(10)
.select(["timestamp", "premium_index_close", "est_funding_rate", "annualized_pct"])
)
# %% [markdown]
# ---
#
# ## Section 6: Using the CryptoDataManager
#
# The ml4t-data library provides a `CryptoDataManager` for convenient access to the premium index data.
# %%
# Using the CryptoDataManager (requires ml4t-data library)
# This demonstrates the programmatic API for loading crypto data
from ml4t.data.crypto import CryptoDataManager # noqa: F401
# CryptoDataManager provides a clean API for loading crypto data
# For this notebook, we use direct parquet loading as shown above
print("CryptoDataManager API available from ml4t-data library.")
print("For this analysis, we use direct parquet loading for simplicity.")
# %% [markdown]
# ---
#
# ## Key Takeaways
#
# Profile of the Binance premium-index panel underpinning the funding-arbitrage
# case study.
#
# ### Quantitative Findings
# - **Panel scale**: 107,839 8h observations across 19 USDT-margined perpetuals,
# 2020-01-01 → 2025-12-31. Coverage ranges from BTC/ETH (6,555 obs) down to
# SUIUSDT (2,920 obs from May 2023).
# - **Slight short bias**: All 19 symbols have a *negative* mean premium
# (between −0.03 and −2.6 bps; MKR is essentially flat), so on average
# perpetuals trade *below* spot — the raw funding flow is from shorts to
# longs before adding the interest-rate baseline.
# - **Volatility spectrum**: BTC has the tightest premium (std 5.6 bps).
# ETH/ADA/DOT cluster at 6–7 bps (~1.2× BTC). The wide-tail altcoins are
# SOL (std 36.7 bps, min −1,915 bps during the FTX collapse), XRP/UNI/COMP
# (10–11 bps), reflecting episodic dislocation rather than steady-state
# volatility.
# - **Arbitrage frequency**: |premium| > 10 bps in 5–14 % of 8h periods
# depending on the symbol (BTC 5.0 %, COMP/ATOM 13–14 %).
# - **Funding APY**: Binance's clamped funding rate (±0.05 % + 0.01 % interest)
# bounds the BTC annualized return at +65.7 % / −43.8 %. Realised mean is
# −5.7 % over 2020-25; the clamp is hit in 82.9 % of 8h periods (driven by
# the interest-rate baseline pushing |APY| above 20 % whenever premium is
# small).
#
# ### Implications for the Funding-Arbitrage Case Study
# - **Direction matters**: The negative mean premium means a *delta-neutral
# short-spot / long-perpetual* leg captures the structural funding flow on
# average for these symbols; the mirror trade only profits during transient
# bullish dislocations.
# - **Asset selection**: Wide-tail altcoins (SOL, XRP, COMP) offer the largest
# per-event funding but expose the strategy to extreme premium tails.
# BTC/ETH provide a tighter, more reliable funding stream.
# - **Regime awareness**: The 30-day rolling premium swings between bull (2021)
# and bear (2022) regimes; static thresholds will mis-fire — see the
# `case_studies/crypto_perps_funding/` pipeline for the regime-aware signal
# used downstream.
#
# **Next**: `12_fx_pairs_eda` profiles the third 24/7-adjacent dataset —
# G10 FX pairs at 4h cadence — completing the global market-data tour.
02_financial_data_universe/12_fx_pairs_eda.ipynb
+854
View File
@@ -0,0 +1,854 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "0e6cf5e5",
"metadata": {
"papermill": {
"duration": 0.001987,
"end_time": "2026-06-13T02:33:42.350184+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:42.348197+00:00",
"status": "completed"
}
},
"source": [
"# FX Pairs — Exploratory Data Analysis\n",
"\n",
"**Docker image**: `ml4t`\n",
"\n",
"## Purpose\n",
"Profile the OANDA 20-pair, 4-hour FX dataset that anchors the FX case study.\n",
"FX is OTC: there is no central tape, so quotes and reported volumes are\n",
"venue-specific. The notebook surveys coverage, quote conventions, OHLC\n",
"integrity, and the 4h→daily aggregation used by downstream chapters.\n",
"\n",
"## Learning Objectives\n",
"- Load and inspect the 4-hour OHLC + indicative-volume panel for 20 pairs.\n",
"- Distinguish direct (USD-quoted), indirect (USD-base), and cross pairs.\n",
"- Read FX volume as an OANDA indicator, not an authoritative tape.\n",
"- Aggregate 4h bars to UTC-day daily bars and read the gap-pattern signal.\n",
"\n",
"## Book reference\n",
"Chapter 2, §2.2 (asset-class market data — foreign exchange). The FX case\n",
"study built on this dataset lives in `case_studies/fx_pairs/`.\n",
"\n",
"## Prerequisites\n",
"- OANDA 4h FX parquet files materialized under `ML4T_DATA_PATH`.\n",
"- Loader `data.load_fx_pairs`."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "75a4c694",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:42.353826Z",
"iopub.status.busy": "2026-06-13T02:33:42.353718Z",
"iopub.status.idle": "2026-06-13T02:33:43.581552Z",
"shell.execute_reply": "2026-06-13T02:33:43.581002Z"
},
"papermill": {
"duration": 1.230464,
"end_time": "2026-06-13T02:33:43.582246+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:42.351782+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"\"\"\"FX Pairs — Exploratory data analysis of OANDA currency pair data.\"\"\"\n",
"\n",
"import polars as pl\n",
"\n",
"from data import load_fx_pairs\n",
"from utils.data_quality import check_ohlc_invariants, per_asset_stats"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "26b9a4ca",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.586308Z",
"iopub.status.busy": "2026-06-13T02:33:43.586220Z",
"iopub.status.idle": "2026-06-13T02:33:43.588185Z",
"shell.execute_reply": "2026-06-13T02:33:43.587686Z"
},
"papermill": {
"duration": 0.004652,
"end_time": "2026-06-13T02:33:43.588663+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.584011+00:00",
"status": "completed"
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"# Production defaults — Papermill injects overrides for CI\n",
"# (No tunable knobs: this notebook EDAs the full 12-pair universe via the\n",
"# canonical load_fx_pairs() API; there is no MAX_SYMBOLS / START_DATE knob to expose.)"
]
},
{
"cell_type": "markdown",
"id": "d3ace9f3",
"metadata": {
"papermill": {
"duration": 0.000828,
"end_time": "2026-06-13T02:33:43.590412+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.589584+00:00",
"status": "completed"
}
},
"source": [
"## 1. Load and Inspect"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e092607c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.592603Z",
"iopub.status.busy": "2026-06-13T02:33:43.592534Z",
"iopub.status.idle": "2026-06-13T02:33:43.645036Z",
"shell.execute_reply": "2026-06-13T02:33:43.644439Z"
},
"papermill": {
"duration": 0.054124,
"end_time": "2026-06-13T02:33:43.645352+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.591228+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== FX Dataset ===\n",
"Shape: (478640, 7)\n",
"Columns: ['timestamp', 'symbol', 'open', 'high', 'low', 'close', 'volume']\n",
"Date range: 2011-01-02 14:00:00 to 2025-12-31 18:00:00\n"
]
}
],
"source": [
"fx_4h = load_fx_pairs(frequency=\"4h\")\n",
"\n",
"print(\"=== FX Dataset ===\")\n",
"print(f\"Shape: {fx_4h.shape}\")\n",
"print(f\"Columns: {fx_4h.columns}\")\n",
"print(f\"Date range: {fx_4h['timestamp'].min()} to {fx_4h['timestamp'].max()}\")"
]
},
{
"cell_type": "markdown",
"id": "226404df",
"metadata": {
"papermill": {
"duration": 0.000883,
"end_time": "2026-06-13T02:33:43.647282+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.646399+00:00",
"status": "completed"
}
},
"source": [
"### Volume Disclaimer\n",
"\n",
"**Important**: FX is an OTC market. Volume figures are indicative estimates from OANDA,\n",
"not authoritative exchange data. Do not interpret FX volume the same way as equity volume."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "582f10dd",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.649708Z",
"iopub.status.busy": "2026-06-13T02:33:43.649581Z",
"iopub.status.idle": "2026-06-13T02:33:43.653412Z",
"shell.execute_reply": "2026-06-13T02:33:43.652994Z"
},
"papermill": {
"duration": 0.00548,
"end_time": "2026-06-13T02:33:43.653645+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.648165+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>symbol</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>datetime[ms]</td><td>str</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>i64</td></tr></thead><tbody><tr><td>2011-01-02 18:00:00</td><td>&quot;AUD_JPY&quot;</td><td>83.0</td><td>83.332</td><td>82.8</td><td>82.875</td><td>986</td></tr><tr><td>2011-01-02 22:00:00</td><td>&quot;AUD_JPY&quot;</td><td>82.877</td><td>83.015</td><td>82.698</td><td>82.796</td><td>4378</td></tr><tr><td>2011-01-03 02:00:00</td><td>&quot;AUD_JPY&quot;</td><td>82.8</td><td>82.98</td><td>82.782</td><td>82.924</td><td>1934</td></tr><tr><td>2011-01-03 06:00:00</td><td>&quot;AUD_JPY&quot;</td><td>82.926</td><td>83.03</td><td>82.826</td><td>83.03</td><td>3949</td></tr><tr><td>2011-01-03 10:00:00</td><td>&quot;AUD_JPY&quot;</td><td>83.032</td><td>83.158</td><td>82.986</td><td>83.01</td><td>5296</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5, 7)\n",
"┌─────────────────────┬─────────┬────────┬────────┬────────┬────────┬────────┐\n",
"│ timestamp ┆ symbol ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ datetime[ms] ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ i64 │\n",
"╞═════════════════════╪═════════╪════════╪════════╪════════╪════════╪════════╡\n",
"│ 2011-01-02 18:00:00 ┆ AUD_JPY ┆ 83.0 ┆ 83.332 ┆ 82.8 ┆ 82.875 ┆ 986 │\n",
"│ 2011-01-02 22:00:00 ┆ AUD_JPY ┆ 82.877 ┆ 83.015 ┆ 82.698 ┆ 82.796 ┆ 4378 │\n",
"│ 2011-01-03 02:00:00 ┆ AUD_JPY ┆ 82.8 ┆ 82.98 ┆ 82.782 ┆ 82.924 ┆ 1934 │\n",
"│ 2011-01-03 06:00:00 ┆ AUD_JPY ┆ 82.926 ┆ 83.03 ┆ 82.826 ┆ 83.03 ┆ 3949 │\n",
"│ 2011-01-03 10:00:00 ┆ AUD_JPY ┆ 83.032 ┆ 83.158 ┆ 82.986 ┆ 83.01 ┆ 5296 │\n",
"└─────────────────────┴─────────┴────────┴────────┴────────┴────────┴────────┘"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fx_4h.head()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "97391cf2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.656035Z",
"iopub.status.busy": "2026-06-13T02:33:43.655974Z",
"iopub.status.idle": "2026-06-13T02:33:43.661039Z",
"shell.execute_reply": "2026-06-13T02:33:43.660623Z"
},
"papermill": {
"duration": 0.006656,
"end_time": "2026-06-13T02:33:43.661267+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.654611+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Currency pairs (20):\n",
" AUD_JPY\n",
" AUD_NZD\n",
" AUD_USD\n",
" CAD_JPY\n",
" CHF_JPY\n",
" EUR_AUD\n",
" EUR_CAD\n",
" EUR_CHF\n",
" EUR_GBP\n",
" EUR_JPY\n",
" EUR_USD\n",
" GBP_AUD\n",
" GBP_CHF\n",
" GBP_JPY\n",
" GBP_USD\n",
" NZD_JPY\n",
" NZD_USD\n",
" USD_CAD\n",
" USD_CHF\n",
" USD_JPY\n"
]
}
],
"source": [
"# Available pairs\n",
"pairs = fx_4h[\"symbol\"].unique().sort().to_list()\n",
"print(f\"\\nCurrency pairs ({len(pairs)}):\")\n",
"for pair in pairs:\n",
" print(f\" {pair}\")"
]
},
{
"cell_type": "markdown",
"id": "95116a76",
"metadata": {
"papermill": {
"duration": 0.000895,
"end_time": "2026-06-13T02:33:43.663133+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.662238+00:00",
"status": "completed"
}
},
"source": [
"### Symbol Normalization\n",
"\n",
"The data uses underscore format (`EUR_USD`). The canonical format for this dataset is\n",
"concatenated (`EURUSD`). Here's how to convert:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "aea13dc3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.665439Z",
"iopub.status.busy": "2026-06-13T02:33:43.665377Z",
"iopub.status.idle": "2026-06-13T02:33:43.677943Z",
"shell.execute_reply": "2026-06-13T02:33:43.677539Z"
},
"papermill": {
"duration": 0.014182,
"end_time": "2026-06-13T02:33:43.678196+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.664014+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Symbol normalization example:\n",
" Raw format: EUR_USD, USD_JPY, GBP_USD\n",
" Canonical: EURUSD, USDJPY, GBPUSD\n",
"\n",
"Normalized pairs: ['AUDJPY', 'AUDNZD', 'AUDUSD', 'CADJPY', 'CHFJPY', 'EURAUD'] ...\n"
]
}
],
"source": [
"# Normalize symbols: EUR_USD → EURUSD (canonical format)\n",
"# The raw file uses underscores; we normalize to concatenated format for downstream joins\n",
"fx = fx_4h.with_columns(pl.col(\"symbol\").str.replace(\"_\", \"\").alias(\"symbol\"))\n",
"\n",
"print(\"Symbol normalization example:\")\n",
"print(\" Raw format: EUR_USD, USD_JPY, GBP_USD\")\n",
"print(\" Canonical: EURUSD, USDJPY, GBPUSD\")\n",
"print(f\"\\nNormalized pairs: {fx['symbol'].unique().sort().to_list()[:6]} ...\")"
]
},
{
"cell_type": "markdown",
"id": "a1437f5f",
"metadata": {
"papermill": {
"duration": 0.000954,
"end_time": "2026-06-13T02:33:43.680241+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.679287+00:00",
"status": "completed"
}
},
"source": [
"## 2. Coverage Summary"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "bb1c9769",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.682634Z",
"iopub.status.busy": "2026-06-13T02:33:43.682558Z",
"iopub.status.idle": "2026-06-13T02:33:43.693125Z",
"shell.execute_reply": "2026-06-13T02:33:43.692663Z"
},
"papermill": {
"duration": 0.012245,
"end_time": "2026-06-13T02:33:43.693424+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.681179+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (20, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>rows</th><th>start</th><th>end</th><th>avg_price</th><th>avg_volume</th></tr><tr><td>str</td><td>u32</td><td>datetime[ms]</td><td>datetime[ms]</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>&quot;GBPAUD&quot;</td><td>23945</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>1.795271</td><td>27719.096262</td></tr><tr><td>&quot;GBPJPY&quot;</td><td>23924</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>156.656959</td><td>22954.817673</td></tr><tr><td>&quot;EURCAD&quot;</td><td>23921</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>1.44887</td><td>21140.860081</td></tr><tr><td>&quot;GBPCHF&quot;</td><td>23924</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>1.302496</td><td>18534.499415</td></tr><tr><td>&quot;CHFJPY&quot;</td><td>23919</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>122.492732</td><td>18296.437686</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>&quot;NZDUSD&quot;</td><td>23943</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>0.702921</td><td>7312.247797</td></tr><tr><td>&quot;EURCHF&quot;</td><td>23925</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>1.099216</td><td>6110.44698</td></tr><tr><td>&quot;EURGBP&quot;</td><td>23922</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>0.846666</td><td>5989.497952</td></tr><tr><td>&quot;AUDUSD&quot;</td><td>23941</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>0.789923</td><td>5563.880832</td></tr><tr><td>&quot;USDCHF&quot;</td><td>23927</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>0.93156</td><td>5375.534835</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (20, 6)\n",
"┌────────┬───────┬─────────────────────┬─────────────────────┬────────────┬──────────────┐\n",
"│ symbol ┆ rows ┆ start ┆ end ┆ avg_price ┆ avg_volume │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ u32 ┆ datetime[ms] ┆ datetime[ms] ┆ f64 ┆ f64 │\n",
"╞════════╪═══════╪═════════════════════╪═════════════════════╪════════════╪══════════════╡\n",
"│ GBPAUD ┆ 23945 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.795271 ┆ 27719.096262 │\n",
"│ GBPJPY ┆ 23924 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 156.656959 ┆ 22954.817673 │\n",
"│ EURCAD ┆ 23921 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.44887 ┆ 21140.860081 │\n",
"│ GBPCHF ┆ 23924 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.302496 ┆ 18534.499415 │\n",
"│ CHFJPY ┆ 23919 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 122.492732 ┆ 18296.437686 │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ NZDUSD ┆ 23943 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.702921 ┆ 7312.247797 │\n",
"│ EURCHF ┆ 23925 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.099216 ┆ 6110.44698 │\n",
"│ EURGBP ┆ 23922 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.846666 ┆ 5989.497952 │\n",
"│ AUDUSD ┆ 23941 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.789923 ┆ 5563.880832 │\n",
"│ USDCHF ┆ 23927 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.93156 ┆ 5375.534835 │\n",
"└────────┴───────┴─────────────────────┴─────────────────────┴────────────┴──────────────┘"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Per-pair statistics (using normalized symbols)\n",
"pairs = fx[\"symbol\"].unique().sort().to_list()\n",
"\n",
"pair_stats = per_asset_stats(\n",
" fx,\n",
" time_col=\"timestamp\",\n",
" asset_col=\"symbol\",\n",
" price_col=\"close\",\n",
" volume_col=\"volume\",\n",
")\n",
"\n",
"pair_stats.sort(\"avg_volume\", descending=True)"
]
},
{
"cell_type": "markdown",
"id": "736a507b",
"metadata": {
"papermill": {
"duration": 0.001009,
"end_time": "2026-06-13T02:33:43.695620+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.694611+00:00",
"status": "completed"
}
},
"source": [
"## 3. Quote Conventions\n",
"\n",
"FX pairs follow **BASE/QUOTE** convention:\n",
"\n",
"| Pair | Interpretation | USD Strength |\n",
"|------|----------------|--------------|\n",
"| EUR/USD = 1.10 | 1 EUR costs 1.10 USD | Down = USD stronger |\n",
"| USD/JPY = 150 | 1 USD costs 150 JPY | Up = USD stronger |\n",
"| EUR/GBP = 0.86 | 1 EUR costs 0.86 GBP | Cross rate (no USD) |\n",
"\n",
"To create a consistent \"USD index\", you must **invert** EUR/USD and GBP/USD."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "1248d371",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.698207Z",
"iopub.status.busy": "2026-06-13T02:33:43.698121Z",
"iopub.status.idle": "2026-06-13T02:33:43.701090Z",
"shell.execute_reply": "2026-06-13T02:33:43.700792Z"
},
"papermill": {
"duration": 0.004731,
"end_time": "2026-06-13T02:33:43.701341+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.696610+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (9, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>convention</th><th>meaning</th><th>usd_strength</th></tr><tr><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>&quot;EURUSD&quot;</td><td>&quot;Direct&quot;</td><td>&quot;USD per EUR&quot;</td><td>&quot;invert for USD strength&quot;</td></tr><tr><td>&quot;GBPUSD&quot;</td><td>&quot;Direct&quot;</td><td>&quot;USD per GBP&quot;</td><td>&quot;invert for USD strength&quot;</td></tr><tr><td>&quot;AUDUSD&quot;</td><td>&quot;Direct&quot;</td><td>&quot;USD per AUD&quot;</td><td>&quot;invert for USD strength&quot;</td></tr><tr><td>&quot;NZDUSD&quot;</td><td>&quot;Direct&quot;</td><td>&quot;USD per NZD&quot;</td><td>&quot;invert for USD strength&quot;</td></tr><tr><td>&quot;USDJPY&quot;</td><td>&quot;Indirect&quot;</td><td>&quot;JPY per USD&quot;</td><td>&quot;direct USD strength&quot;</td></tr><tr><td>&quot;USDCHF&quot;</td><td>&quot;Indirect&quot;</td><td>&quot;CHF per USD&quot;</td><td>&quot;direct USD strength&quot;</td></tr><tr><td>&quot;USDCAD&quot;</td><td>&quot;Indirect&quot;</td><td>&quot;CAD per USD&quot;</td><td>&quot;direct USD strength&quot;</td></tr><tr><td>&quot;EURGBP&quot;</td><td>&quot;Cross&quot;</td><td>&quot;GBP per EUR&quot;</td><td>&quot;no USD&quot;</td></tr><tr><td>&quot;EURJPY&quot;</td><td>&quot;Cross&quot;</td><td>&quot;JPY per EUR&quot;</td><td>&quot;no USD&quot;</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (9, 4)\n",
"┌────────┬────────────┬─────────────┬─────────────────────────┐\n",
"│ symbol ┆ convention ┆ meaning ┆ usd_strength │\n",
"│ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ str ┆ str │\n",
"╞════════╪════════════╪═════════════╪═════════════════════════╡\n",
"│ EURUSD ┆ Direct ┆ USD per EUR ┆ invert for USD strength │\n",
"│ GBPUSD ┆ Direct ┆ USD per GBP ┆ invert for USD strength │\n",
"│ AUDUSD ┆ Direct ┆ USD per AUD ┆ invert for USD strength │\n",
"│ NZDUSD ┆ Direct ┆ USD per NZD ┆ invert for USD strength │\n",
"│ USDJPY ┆ Indirect ┆ JPY per USD ┆ direct USD strength │\n",
"│ USDCHF ┆ Indirect ┆ CHF per USD ┆ direct USD strength │\n",
"│ USDCAD ┆ Indirect ┆ CAD per USD ┆ direct USD strength │\n",
"│ EURGBP ┆ Cross ┆ GBP per EUR ┆ no USD │\n",
"│ EURJPY ┆ Cross ┆ JPY per EUR ┆ no USD │\n",
"└────────┴────────────┴─────────────┴─────────────────────────┘"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Quote convention classification (using canonical symbols).\n",
"QUOTE_CONVENTIONS = {\n",
" \"EURUSD\": (\"Direct\", \"USD per EUR\", \"invert for USD strength\"),\n",
" \"GBPUSD\": (\"Direct\", \"USD per GBP\", \"invert for USD strength\"),\n",
" \"AUDUSD\": (\"Direct\", \"USD per AUD\", \"invert for USD strength\"),\n",
" \"NZDUSD\": (\"Direct\", \"USD per NZD\", \"invert for USD strength\"),\n",
" \"USDJPY\": (\"Indirect\", \"JPY per USD\", \"direct USD strength\"),\n",
" \"USDCHF\": (\"Indirect\", \"CHF per USD\", \"direct USD strength\"),\n",
" \"USDCAD\": (\"Indirect\", \"CAD per USD\", \"direct USD strength\"),\n",
" \"EURGBP\": (\"Cross\", \"GBP per EUR\", \"no USD\"),\n",
" \"EURJPY\": (\"Cross\", \"JPY per EUR\", \"no USD\"),\n",
"}\n",
"\n",
"quote_conventions = pl.DataFrame(\n",
" [\n",
" {\"symbol\": p, \"convention\": c, \"meaning\": m, \"usd_strength\": n}\n",
" for p, (c, m, n) in QUOTE_CONVENTIONS.items()\n",
" if p in pairs\n",
" ]\n",
")\n",
"quote_conventions"
]
},
{
"cell_type": "markdown",
"id": "f6e41bbb",
"metadata": {
"papermill": {
"duration": 0.0011,
"end_time": "2026-06-13T02:33:43.703547+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.702447+00:00",
"status": "completed"
}
},
"source": [
"## 4. Data Quality"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "4476f44a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.706165Z",
"iopub.status.busy": "2026-06-13T02:33:43.706091Z",
"iopub.status.idle": "2026-06-13T02:33:43.713408Z",
"shell.execute_reply": "2026-06-13T02:33:43.712933Z"
},
"papermill": {
"duration": 0.009113,
"end_time": "2026-06-13T02:33:43.713719+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.704606+00:00",
"status": "completed"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (6, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>check</th><th>valid_pct</th><th>applicable_rows</th><th>total_rows</th></tr><tr><td>str</td><td>f64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>&quot;high_gte_low&quot;</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>&quot;high_gte_open&quot;</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>&quot;high_gte_close&quot;</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>&quot;low_lte_open&quot;</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>&quot;low_lte_close&quot;</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>&quot;volume_non_negative&quot;</td><td>100.0</td><td>478640</td><td>478640</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (6, 4)\n",
"┌─────────────────────┬───────────┬─────────────────┬────────────┐\n",
"│ check ┆ valid_pct ┆ applicable_rows ┆ total_rows │\n",
"│ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ f64 ┆ i64 ┆ i64 │\n",
"╞═════════════════════╪═══════════╪═════════════════╪════════════╡\n",
"│ high_gte_low ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
"│ high_gte_open ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
"│ high_gte_close ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
"│ low_lte_open ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
"│ low_lte_close ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
"│ volume_non_negative ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
"└─────────────────────┴───────────┴─────────────────┴────────────┘"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# OHLC invariants\n",
"invariants = check_ohlc_invariants(fx)\n",
"invariants"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "9c27cc3c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.717828Z",
"iopub.status.busy": "2026-06-13T02:33:43.717708Z",
"iopub.status.idle": "2026-06-13T02:33:43.724040Z",
"shell.execute_reply": "2026-06-13T02:33:43.723475Z"
},
"papermill": {
"duration": 0.009097,
"end_time": "2026-06-13T02:33:43.724412+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.715315+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Gaps > 24 hours (EURUSD): 747 (should be weekends only)\n"
]
}
],
"source": [
"# Check for weekend gaps (expected in FX, which trades 24/5)\n",
"eurusd = fx.filter(pl.col(\"symbol\") == \"EURUSD\").sort(\"timestamp\")\n",
"\n",
"eurusd_gaps = eurusd.with_columns(\n",
" pl.col(\"timestamp\").diff().dt.total_hours().alias(\"hours_since_prev\")\n",
")\n",
"\n",
"large_gaps = eurusd_gaps.filter(pl.col(\"hours_since_prev\") > 24)\n",
"print(f\"\\nGaps > 24 hours (EURUSD): {len(large_gaps)} (should be weekends only)\")"
]
},
{
"cell_type": "markdown",
"id": "b6e1c1bb",
"metadata": {
"papermill": {
"duration": 0.001464,
"end_time": "2026-06-13T02:33:43.727484+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.726020+00:00",
"status": "completed"
}
},
"source": [
"## 5. Daily Aggregation\n",
"\n",
"Aggregate 4-hour bars to daily for consistency with other datasets.\n",
"\n",
"**Note**: This uses UTC midnight boundaries. FX daily bars are conventionally defined\n",
"by a session cutoff (often 5pm New York). For production, align to your broker's\n",
"convention. This simple calendar-day aggregation is sufficient for exploration."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1c719ab9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T02:33:43.730960Z",
"iopub.status.busy": "2026-06-13T02:33:43.730762Z",
"iopub.status.idle": "2026-06-13T02:33:43.773847Z",
"shell.execute_reply": "2026-06-13T02:33:43.773341Z"
},
"papermill": {
"duration": 0.045315,
"end_time": "2026-06-13T02:33:43.774176+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.728861+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Daily aggregation (UTC boundaries) — shape: (94642, 7)\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>timestamp</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>str</td><td>datetime[ms]</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>i64</td></tr></thead><tbody><tr><td>&quot;EURUSD&quot;</td><td>2025-12-26 00:00:00</td><td>1.17873</td><td>1.17968</td><td>1.17616</td><td>1.1771</td><td>99014</td></tr><tr><td>&quot;EURUSD&quot;</td><td>2025-12-28 00:00:00</td><td>1.1768</td><td>1.17864</td><td>1.17654</td><td>1.17842</td><td>10811</td></tr><tr><td>&quot;EURUSD&quot;</td><td>2025-12-29 00:00:00</td><td>1.1784</td><td>1.17891</td><td>1.17494</td><td>1.17751</td><td>110397</td></tr><tr><td>&quot;EURUSD&quot;</td><td>2025-12-30 00:00:00</td><td>1.1775</td><td>1.17798</td><td>1.17419</td><td>1.17426</td><td>83060</td></tr><tr><td>&quot;EURUSD&quot;</td><td>2025-12-31 00:00:00</td><td>1.17424</td><td>1.17595</td><td>1.172</td><td>1.17459</td><td>92505</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5, 7)\n",
"┌────────┬─────────────────────┬─────────┬─────────┬─────────┬─────────┬────────┐\n",
"│ symbol ┆ timestamp ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ str ┆ datetime[ms] ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ i64 │\n",
"╞════════╪═════════════════════╪═════════╪═════════╪═════════╪═════════╪════════╡\n",
"│ EURUSD ┆ 2025-12-26 00:00:00 ┆ 1.17873 ┆ 1.17968 ┆ 1.17616 ┆ 1.1771 ┆ 99014 │\n",
"│ EURUSD ┆ 2025-12-28 00:00:00 ┆ 1.1768 ┆ 1.17864 ┆ 1.17654 ┆ 1.17842 ┆ 10811 │\n",
"│ EURUSD ┆ 2025-12-29 00:00:00 ┆ 1.1784 ┆ 1.17891 ┆ 1.17494 ┆ 1.17751 ┆ 110397 │\n",
"│ EURUSD ┆ 2025-12-30 00:00:00 ┆ 1.1775 ┆ 1.17798 ┆ 1.17419 ┆ 1.17426 ┆ 83060 │\n",
"│ EURUSD ┆ 2025-12-31 00:00:00 ┆ 1.17424 ┆ 1.17595 ┆ 1.172 ┆ 1.17459 ┆ 92505 │\n",
"└────────┴─────────────────────┴─────────┴─────────┴─────────┴─────────┴────────┘"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Daily aggregation (must be sorted for group_by_dynamic)\n",
"fx_daily = (\n",
" fx.sort(\"symbol\", \"timestamp\")\n",
" .group_by_dynamic(\"timestamp\", every=\"1d\", group_by=\"symbol\")\n",
" .agg(\n",
" [\n",
" pl.col(\"open\").first(),\n",
" pl.col(\"high\").max(),\n",
" pl.col(\"low\").min(),\n",
" pl.col(\"close\").last(),\n",
" pl.col(\"volume\").sum(),\n",
" ]\n",
" )\n",
")\n",
"\n",
"print(f\"Daily aggregation (UTC boundaries) — shape: {fx_daily.shape}\")\n",
"fx_daily.filter(pl.col(\"symbol\") == \"EURUSD\").tail(5)"
]
},
{
"cell_type": "markdown",
"id": "28013e00",
"metadata": {
"papermill": {
"duration": 0.001196,
"end_time": "2026-06-13T02:33:43.776700+00:00",
"exception": false,
"start_time": "2026-06-13T02:33:43.775504+00:00",
"status": "completed"
}
},
"source": [
"## Key Takeaways\n",
"\n",
"Profile of the OANDA 4h FX panel that anchors the FX case study.\n",
"\n",
"### Quantitative Findings\n",
"- **Panel scale**: 478,640 4h observations across 20 currency pairs spanning\n",
" 2011-01-02 → 2025-12-31. Each pair has ~23,920–23,945 4h bars.\n",
"- **Liquidity tiers (by indicative OANDA volume)**: GBPAUD, GBPJPY, EURCAD,\n",
" GBPCHF and CHFJPY top the table at 18k–28k contracts/4h; the bottom of\n",
" the universe (NZDUSD, EURCHF, EURGBP, AUDUSD, USDCHF) sits at 5k–7k.\n",
" These rankings are *OANDA-specific* — interbank-market liquidity for\n",
" EURUSD/USDJPY is the largest globally but is not visible to a single\n",
" retail venue.\n",
"- **OHLC integrity**: 100% of 4h bars satisfy all six invariants\n",
" (high ≥ low/open/close, low ≤ open/close, volume ≥ 0).\n",
"- **Session gaps**: 747 EURUSD inter-bar gaps exceed 24 h, matching the\n",
" ~770 weekend closes over 14 years — confirming the 24/5 calendar.\n",
"- **Daily roll-up**: UTC-boundary aggregation produces 94,642 daily rows\n",
" across the panel (≈4,732 trading days × 20 pairs).\n",
"\n",
"### Implications for Practitioners\n",
"- **Volume**: Treat as a relative liquidity indicator across pairs on this\n",
" venue, not as an interbank tape.\n",
"- **Quote inversion**: USD strength composites must invert direct pairs\n",
" (EUR/USD, GBP/USD, AUD/USD, NZD/USD); USD-base and cross pairs do not\n",
" need inversion.\n",
"- **Daily session convention**: UTC-day aggregation is convenient for joins\n",
" with the equity/crypto panels but is *not* a tradable session boundary;\n",
" broker-specific 5pm-NY cutoffs are wired in `case_studies/fx_pairs/`\n",
" downstream.\n",
"\n",
"**Next**: `13_data_quality_framework` profiles the cross-asset DQ checks\n",
"that consume this panel and the others built up so far."
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "tags,-all"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.3"
},
"papermill": {
"default_parameters": {},
"duration": 2.888138,
"end_time": "2026-06-13T02:33:44.593918+00:00",
"environment_variables": {},
"exception": null,
"input_path": "02_financial_data_universe/12_fx_pairs_eda.ipynb",
"output_path": "02_financial_data_universe/12_fx_pairs_eda.ipynb",
"parameters": {},
"start_time": "2026-06-13T02:33:41.705780+00:00",
"version": "2.7.0"
},
"ml4t_provenance": {
"source_py_blob": "8f3afbd31633079f8ac78d86f3bc5a9cf4b326ad",
"executed_at": "2026-06-13T02:33:44.819485+00:00",
"executor": "cpu-local",
"production": true,
"parameters": {},
"notes": "executed-state finalization batch (2026-06-13 run)"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02_financial_data_universe/12_fx_pairs_eda.py
+230
View File
@@ -0,0 +1,230 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # FX Pairs — Exploratory Data Analysis
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Profile the OANDA 20-pair, 4-hour FX dataset that anchors the FX case study.
# FX is OTC: there is no central tape, so quotes and reported volumes are
# venue-specific. The notebook surveys coverage, quote conventions, OHLC
# integrity, and the 4h→daily aggregation used by downstream chapters.
#
# ## Learning Objectives
# - Load and inspect the 4-hour OHLC + indicative-volume panel for 20 pairs.
# - Distinguish direct (USD-quoted), indirect (USD-base), and cross pairs.
# - Read FX volume as an OANDA indicator, not an authoritative tape.
# - Aggregate 4h bars to UTC-day daily bars and read the gap-pattern signal.
#
# ## Book reference
# Chapter 2, §2.2 (asset-class market data — foreign exchange). The FX case
# study built on this dataset lives in `case_studies/fx_pairs/`.
#
# ## Prerequisites
# - OANDA 4h FX parquet files materialized under `ML4T_DATA_PATH`.
# - Loader `data.load_fx_pairs`.
# %%
"""FX Pairs — Exploratory data analysis of OANDA currency pair data."""
import polars as pl
from data import load_fx_pairs
from utils.data_quality import check_ohlc_invariants, per_asset_stats
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# (No tunable knobs: this notebook EDAs the full 12-pair universe via the
# canonical load_fx_pairs() API; there is no MAX_SYMBOLS / START_DATE knob to expose.)
# %% [markdown]
# ## 1. Load and Inspect
# %%
fx_4h = load_fx_pairs(frequency="4h")
print("=== FX Dataset ===")
print(f"Shape: {fx_4h.shape}")
print(f"Columns: {fx_4h.columns}")
print(f"Date range: {fx_4h['timestamp'].min()} to {fx_4h['timestamp'].max()}")
# %% [markdown]
# ### Volume Disclaimer
#
# **Important**: FX is an OTC market. Volume figures are indicative estimates from OANDA,
# not authoritative exchange data. Do not interpret FX volume the same way as equity volume.
# %%
fx_4h.head()
# %%
# Available pairs
pairs = fx_4h["symbol"].unique().sort().to_list()
print(f"\nCurrency pairs ({len(pairs)}):")
for pair in pairs:
print(f" {pair}")
# %% [markdown]
# ### Symbol Normalization
#
# The data uses underscore format (`EUR_USD`). The canonical format for this dataset is
# concatenated (`EURUSD`). Here's how to convert:
# %%
# Normalize symbols: EUR_USD → EURUSD (canonical format)
# The raw file uses underscores; we normalize to concatenated format for downstream joins
fx = fx_4h.with_columns(pl.col("symbol").str.replace("_", "").alias("symbol"))
print("Symbol normalization example:")
print(" Raw format: EUR_USD, USD_JPY, GBP_USD")
print(" Canonical: EURUSD, USDJPY, GBPUSD")
print(f"\nNormalized pairs: {fx['symbol'].unique().sort().to_list()[:6]} ...")
# %% [markdown]
# ## 2. Coverage Summary
# %%
# Per-pair statistics (using normalized symbols)
pairs = fx["symbol"].unique().sort().to_list()
pair_stats = per_asset_stats(
fx,
time_col="timestamp",
asset_col="symbol",
price_col="close",
volume_col="volume",
)
pair_stats.sort("avg_volume", descending=True)
# %% [markdown]
# ## 3. Quote Conventions
#
# FX pairs follow **BASE/QUOTE** convention:
#
# | Pair | Interpretation | USD Strength |
# |------|----------------|--------------|
# | EUR/USD = 1.10 | 1 EUR costs 1.10 USD | Down = USD stronger |
# | USD/JPY = 150 | 1 USD costs 150 JPY | Up = USD stronger |
# | EUR/GBP = 0.86 | 1 EUR costs 0.86 GBP | Cross rate (no USD) |
#
# To create a consistent "USD index", you must **invert** EUR/USD and GBP/USD.
# %%
# Quote convention classification (using canonical symbols).
QUOTE_CONVENTIONS = {
"EURUSD": ("Direct", "USD per EUR", "invert for USD strength"),
"GBPUSD": ("Direct", "USD per GBP", "invert for USD strength"),
"AUDUSD": ("Direct", "USD per AUD", "invert for USD strength"),
"NZDUSD": ("Direct", "USD per NZD", "invert for USD strength"),
"USDJPY": ("Indirect", "JPY per USD", "direct USD strength"),
"USDCHF": ("Indirect", "CHF per USD", "direct USD strength"),
"USDCAD": ("Indirect", "CAD per USD", "direct USD strength"),
"EURGBP": ("Cross", "GBP per EUR", "no USD"),
"EURJPY": ("Cross", "JPY per EUR", "no USD"),
}
quote_conventions = pl.DataFrame(
[
{"symbol": p, "convention": c, "meaning": m, "usd_strength": n}
for p, (c, m, n) in QUOTE_CONVENTIONS.items()
if p in pairs
]
)
quote_conventions
# %% [markdown]
# ## 4. Data Quality
# %%
# OHLC invariants
invariants = check_ohlc_invariants(fx)
invariants
# %%
# Check for weekend gaps (expected in FX, which trades 24/5)
eurusd = fx.filter(pl.col("symbol") == "EURUSD").sort("timestamp")
eurusd_gaps = eurusd.with_columns(
pl.col("timestamp").diff().dt.total_hours().alias("hours_since_prev")
)
large_gaps = eurusd_gaps.filter(pl.col("hours_since_prev") > 24)
print(f"\nGaps > 24 hours (EURUSD): {len(large_gaps)} (should be weekends only)")
# %% [markdown]
# ## 5. Daily Aggregation
#
# Aggregate 4-hour bars to daily for consistency with other datasets.
#
# **Note**: This uses UTC midnight boundaries. FX daily bars are conventionally defined
# by a session cutoff (often 5pm New York). For production, align to your broker's
# convention. This simple calendar-day aggregation is sufficient for exploration.
# %%
# Daily aggregation (must be sorted for group_by_dynamic)
fx_daily = (
fx.sort("symbol", "timestamp")
.group_by_dynamic("timestamp", every="1d", group_by="symbol")
.agg(
[
pl.col("open").first(),
pl.col("high").max(),
pl.col("low").min(),
pl.col("close").last(),
pl.col("volume").sum(),
]
)
)
print(f"Daily aggregation (UTC boundaries) — shape: {fx_daily.shape}")
fx_daily.filter(pl.col("symbol") == "EURUSD").tail(5)
# %% [markdown]
# ## Key Takeaways
#
# Profile of the OANDA 4h FX panel that anchors the FX case study.
#
# ### Quantitative Findings
# - **Panel scale**: 478,640 4h observations across 20 currency pairs spanning
# 2011-01-02 → 2025-12-31. Each pair has ~23,920–23,945 4h bars.
# - **Liquidity tiers (by indicative OANDA volume)**: GBPAUD, GBPJPY, EURCAD,
# GBPCHF and CHFJPY top the table at 18k–28k contracts/4h; the bottom of
# the universe (NZDUSD, EURCHF, EURGBP, AUDUSD, USDCHF) sits at 5k–7k.
# These rankings are *OANDA-specific* — interbank-market liquidity for
# EURUSD/USDJPY is the largest globally but is not visible to a single
# retail venue.
# - **OHLC integrity**: 100% of 4h bars satisfy all six invariants
# (high ≥ low/open/close, low ≤ open/close, volume ≥ 0).
# - **Session gaps**: 747 EURUSD inter-bar gaps exceed 24 h, matching the
# ~770 weekend closes over 14 years — confirming the 24/5 calendar.
# - **Daily roll-up**: UTC-boundary aggregation produces 94,642 daily rows
# across the panel (≈4,732 trading days × 20 pairs).
#
# ### Implications for Practitioners
# - **Volume**: Treat as a relative liquidity indicator across pairs on this
# venue, not as an interbank tape.
# - **Quote inversion**: USD strength composites must invert direct pairs
# (EUR/USD, GBP/USD, AUD/USD, NZD/USD); USD-base and cross pairs do not
# need inversion.
# - **Daily session convention**: UTC-day aggregation is convenient for joins
# with the equity/crypto panels but is *not* a tradable session boundary;
# broker-specific 5pm-NY cutoffs are wired in `case_studies/fx_pairs/`
# downstream.
#
# **Next**: `13_data_quality_framework` profiles the cross-asset DQ checks
# that consume this panel and the others built up so far.
02_financial_data_universe/13_data_quality_framework.ipynb
+2069
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/13_data_quality_framework.py
+656
View File
@@ -0,0 +1,656 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Data Quality Framework: Validation, Anomalies, and Remediation
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Demonstrate the four pillars of a production data-quality pipeline applied to
# US equities daily OHLCV: structural validation (OHLC invariants, nulls,
# duplicates), anomaly detection (return outliers, volume spikes, price
# staleness), distribution drift (PSI), and ingestion hygiene (gaps,
# duplicates, corporate-action detection). All checks come from the
# `ml4t.data.validation` and `ml4t.data.anomaly` modules.
#
# ## Learning Objectives
# - Run `OHLCVValidator` and read its issue report.
# - Compare MAD / Z-score / IQR thresholds for return-outlier detection.
# - Compute Population Stability Index for drift monitoring and read its bins.
# - Detect ingestion gaps, duplicates, and likely corporate actions, and wire
# the pieces together into a quarantine-aware pipeline.
#
# ## Book reference
# Chapter 2, §2.3 (data quality framework). Downstream chapters that consume
# the cleaned panel: `14_point_in_time_validation` (bitemporal hygiene),
# `15_survivorship_bias_detection`, `17_complete_pipeline`.
#
# ## Prerequisites
# - Quandl/Wiki US equities parquet materialized under `ML4T_DATA_PATH`
# (the legacy dataset; ends 2018-03-27).
# - Loader `data.load_us_equities`.
# - Library packages `ml4t.data.validation` and `ml4t.data.anomaly`.
# %%
"""Data Quality Framework — Validation, anomaly detection, and remediation."""
import warnings
warnings.filterwarnings("ignore")
import logging
from datetime import datetime
from pathlib import Path
import numpy as np
import polars as pl
from ml4t.data.anomaly import (
AnomalyManager,
PriceStalenessDetector,
ReturnOutlierDetector,
VolumeSpikeDetector,
)
from ml4t.data.anomaly.config import (
AnomalyConfig,
PriceStalenessConfig,
ReturnOutlierConfig,
VolumeSpikeConfig,
)
from ml4t.data.validation import OHLCVValidator
from data import load_us_equities
from utils.paths import get_output_dir
# The library emits per-symbol DEBUG logs that drown the cell output;
# silence to WARNING so the notebook prints only call results.
logging.getLogger("ml4t.data.anomaly").setLevel(logging.WARNING)
def _to_date(value: object) -> object:
"""Normalize a polars timestamp scalar to a datetime.date for printing."""
return value.date() if hasattr(value, "date") else value
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %%
OUTPUT_DIR = get_output_dir(2, "quality")
# Five large-cap symbols from the legacy Wiki/Quandl dataset (1962–2018).
# FB (not META) is the canonical Facebook ticker in this vintage.
SYMBOLS = ["AAPL", "MSFT", "GOOGL", "NVDA", "FB"]
# %% [markdown]
# ## Load Sample Data
#
# Use the pre-2018 Wiki/Quandl US equities panel. The validation and anomaly
# detectors operate on any DataFrame with `timestamp / open / high / low /
# close / volume`.
# %%
wiki_df = load_us_equities()
print(
f"US equities loaded: {len(wiki_df):,} rows; "
f"{wiki_df['timestamp'].min()} → {wiki_df['timestamp'].max()}"
)
datasets = {
symbol: (
wiki_df.lazy()
.filter(pl.col("symbol") == symbol)
.select(["timestamp", "symbol", "open", "high", "low", "close", "volume"])
.collect()
)
for symbol in SYMBOLS
}
missing = [s for s, df in datasets.items() if df.is_empty()]
if missing:
raise RuntimeError(f"Symbols missing from Wiki/Quandl dataset: {missing}")
per_symbol_rows = pl.DataFrame(
{"symbol": list(datasets), "rows": [len(df) for df in datasets.values()]}
)
per_symbol_rows
# %% [markdown]
# ---
#
# ## Part 1: OHLC Invariant Validation
#
# **OHLC Invariants** are mathematical relationships that MUST hold for valid
# market data:
#
# | Invariant | Meaning |
# |-----------------|----------------------|
# | High ≥ Low | by definition |
# | High ≥ Open, Close | high is the maximum |
# | Low ≤ Open, Close | low is the minimum |
# | Prices > 0 | no negative prices |
# | Volume ≥ 0 | no negative volume |
#
# Violations indicate provider errors, transmission corruption, or incorrect
# adjustments.
# %%
validator = OHLCVValidator(
check_nulls=True,
check_price_consistency=True,
check_negative_prices=True,
check_negative_volume=True,
check_duplicate_timestamps=True,
check_chronological_order=True,
check_price_staleness=True,
check_extreme_returns=True,
max_return_threshold=0.5, # flag |return| > 50%
staleness_threshold=5, # flag 5+ identical-price days
)
validation_summary = pl.DataFrame(
[
{
"symbol": sym,
"passed": (r := validator.validate(df)).passed,
"issues": len(r.issues),
"critical": r.critical_count,
"errors": r.error_count,
}
for sym, df in datasets.items()
]
)
validation_summary
# %% [markdown]
# ### Validation on Dirty Data
#
# Inject three known faults into AAPL and re-run the validator to see what it
# catches.
# %%
clean_df = datasets["AAPL"]
highs = clean_df["high"].to_numpy().copy()
lows = clean_df["low"].to_numpy().copy()
volumes = clean_df["volume"].to_numpy().copy()
closes = clean_df["close"].to_numpy().copy()
# Fault 1: high < low at rows 10–12
highs[10:13] = lows[10:13] - 1.0
# Fault 2: negative volume at row 20
volumes[20] = -1000
# Fault 3: null close at row 30
closes[30] = np.nan
dirty_df = pl.DataFrame(
{
"timestamp": clean_df["timestamp"],
"symbol": clean_df["symbol"],
"open": clean_df["open"],
"high": highs,
"low": lows,
"close": closes,
"volume": volumes,
}
)
dirty_result = validator.validate(dirty_df)
print(f"Validation passed: {dirty_result.passed}")
print(f"Critical: {dirty_result.critical_count}, Errors: {dirty_result.error_count}")
dirty_issues = pl.DataFrame(
[
{
"severity": issue.severity.name,
"check": issue.check,
"rows": issue.row_count or 0,
"message": issue.message,
}
for issue in dirty_result.issues
]
)
dirty_issues
# %% [markdown]
# ---
#
# ## Part 2: Anomaly Detection
#
# Validation checks data correctness; anomaly detection finds unusual patterns
# in otherwise structurally-valid data:
#
# | Detector | What it finds | Method |
# |-----------------------|-------------------------------|------------------------|
# | ReturnOutlierDetector | flash crashes, splits, pumps | MAD, Z-score, IQR |
# | VolumeSpikeDetector | unusual trading activity | rolling Z-score |
# | PriceStalenessDetector| data gaps, illiquid securities| consecutive-unchanged |
# %% [markdown]
# ### ReturnOutlierDetector
#
# Flags returns whose magnitude exceeds a threshold under three statistics:
# **MAD** (median absolute deviation, robust to extreme tails), **Z-score**
# (Gaussian, threshold-sensitive to fat tails), and **IQR** (interquartile
# range, distribution-free).
# %%
sample_df = datasets["AAPL"]
methods = ["mad", "zscore", "iqr"]
method_summary = pl.DataFrame(
[
{
"method": m,
"anomalies": len(
ReturnOutlierDetector(
config=ReturnOutlierConfig(method=m, threshold=3.0, min_samples=20)
).detect(sample_df, symbol="AAPL")
),
}
for m in methods
]
)
method_summary
# %% [markdown]
# MAD flags the most events because it is more sensitive in the tails of a
# heavy-tailed return distribution. The biggest "anomalies" in the AAPL series
# below are *real* events: stock splits and an earnings-driven crash, not
# data-quality issues. The detector cannot distinguish — that is the operator's
# job downstream.
# %%
mad_anomalies = ReturnOutlierDetector(
config=ReturnOutlierConfig(method="mad", threshold=3.0, min_samples=20)
).detect(sample_df, symbol="AAPL")
top_mad = pl.DataFrame(
[
{
"date": _to_date(a.timestamp),
"return_pct": float(a.value), # value is already a percentage
}
for a in sorted(mad_anomalies, key=lambda x: abs(x.value), reverse=True)[:5]
]
)
top_mad
# %% [markdown]
# ### VolumeSpikeDetector
#
# Flags rolling-window volume Z-scores above a threshold.
# %%
volume_anomalies = VolumeSpikeDetector(
config=VolumeSpikeConfig(window=20, threshold=3.0, min_volume=0, min_samples=20)
).detect(sample_df, symbol="AAPL")
top_volume = pl.DataFrame(
[
{
"date": _to_date(a.timestamp),
"volume": int(a.value),
"ratio_vs_avg": (
a.value / a.metadata["average_volume"] if a.metadata.get("average_volume") else None
),
}
for a in sorted(volume_anomalies, key=lambda x: x.value, reverse=True)[:5]
]
)
print(f"Found {len(volume_anomalies)} AAPL volume spikes")
top_volume
# %% [markdown]
# ### PriceStalenessDetector
#
# Flags runs of consecutive identical prices — a strong signal of feed
# outages or illiquid securities.
# %%
stale_anomalies = PriceStalenessDetector(
config=PriceStalenessConfig(max_unchanged_days=3, check_close_only=False)
).detect(sample_df, symbol="AAPL")
print(f"AAPL stale-price periods (≥4 consecutive days): {len(stale_anomalies)}")
# %% [markdown]
# ### AnomalyManager: Production Pipeline
#
# `AnomalyManager` orchestrates the three detectors and provides batch
# analysis across symbols.
# %%
anomaly_config = AnomalyConfig(
enabled=True,
report_severity_threshold="warning",
return_outliers=ReturnOutlierConfig(method="mad", threshold=3.0),
volume_spikes=VolumeSpikeConfig(window=20, threshold=3.0),
price_staleness=PriceStalenessConfig(max_unchanged_days=5),
)
manager = AnomalyManager(config=anomaly_config)
reports = manager.analyze_batch(datasets)
batch_summary = pl.DataFrame(
[
{
"symbol": sym,
"total_anomalies": len(rep.anomalies),
"critical": len(rep.get_critical_anomalies()),
}
for sym, rep in reports.items()
]
)
batch_summary
# %% [markdown]
# ---
#
# ## Part 3: Population Stability Index (PSI)
#
# **PSI** measures distribution drift — whether recent data follows the same
# distribution as a historical baseline. Useful for detecting regime changes,
# data-source switches, and market-structure shifts.
#
# Bin the baseline series into deciles, recompute the same bin boundaries on
# the current series, and sum
#
# $$\mathrm{PSI} = \sum_i (p_i^{\text{current}} - p_i^{\text{baseline}})
# \log \frac{p_i^{\text{current}}}{p_i^{\text{baseline}}}$$
#
# | PSI | Interpretation |
# |-----------|---------------------------------|
# | < 0.1 | no significant change |
# | 0.1–0.25 | moderate shift, investigate |
# | > 0.25 | significant shift, action needed|
# %%
def calculate_psi(
baseline: pl.Series, current: pl.Series, n_bins: int = 10, epsilon: float = 1e-6
) -> tuple[float, pl.DataFrame]:
"""Population Stability Index between baseline and current distributions."""
baseline_clean = baseline.drop_nulls()
current_clean = current.drop_nulls()
percentiles = [i * 100 / n_bins for i in range(n_bins + 1)]
bin_edges = [baseline_clean.quantile(p / 100) for p in percentiles]
# Ensure unique edges
unique_edges = [bin_edges[0]]
for edge in bin_edges[1:]:
if edge <= unique_edges[-1]:
edge = unique_edges[-1] + epsilon
unique_edges.append(edge)
baseline_counts = np.histogram(baseline_clean.to_numpy(), bins=unique_edges)[0]
current_counts = np.histogram(current_clean.to_numpy(), bins=unique_edges)[0]
baseline_pct = np.maximum(baseline_counts / len(baseline_clean), epsilon)
current_pct = np.maximum(current_counts / len(current_clean), epsilon)
psi_values = (current_pct - baseline_pct) * np.log(current_pct / baseline_pct)
breakdown = pl.DataFrame(
{
"bin": list(range(1, n_bins + 1)),
"baseline_pct": baseline_pct.round(4),
"current_pct": current_pct.round(4),
"psi_contribution": psi_values.round(4),
}
)
return float(np.sum(psi_values)), breakdown
# %%
df = datasets["AAPL"].with_columns((pl.col("close").pct_change() * 100).alias("return_pct"))
midpoint = len(df) // 2
baseline_returns = df["return_pct"][:midpoint]
current_returns = df["return_pct"][midpoint:]
psi_value, psi_breakdown = calculate_psi(baseline_returns, current_returns)
psi_severity = (
"no significant change"
if psi_value < 0.1
else "moderate shift"
if psi_value < 0.25
else "significant shift"
)
print(
f"AAPL daily-return PSI: {psi_value:.4f} ({psi_severity})\n"
f"Baseline: {_to_date(df['timestamp'][0])} → {_to_date(df['timestamp'][midpoint])}\n"
f"Current: {_to_date(df['timestamp'][midpoint])} → {_to_date(df['timestamp'][-1])}"
)
psi_breakdown
# %% [markdown]
# ---
#
# ## Part 4: Data Hygiene
#
# Beyond validation and anomaly detection, ingestion hygiene covers gap
# detection, deduplication, and corporate-action signaling.
# %% [markdown]
# ### Gap Detection
#
# Holidays produce expected 3–4 calendar-day gaps; provider outages and
# delistings produce longer ones. The 9/11 close (2001-09-11 → 09-17) is the
# only > 5-day gap visible in this universe.
# %%
def detect_gaps(df: pl.DataFrame, max_gap_days: int = 5) -> pl.DataFrame:
"""Return rows whose gap from the previous timestamp exceeds `max_gap_days`."""
return (
df.sort("timestamp")
.with_columns(
pl.col("timestamp").diff().dt.total_days().alias("days_since_prev"),
pl.col("timestamp").shift(1).alias("prev_timestamp"),
)
.filter(pl.col("days_since_prev") > max_gap_days)
.select(["prev_timestamp", "timestamp", "days_since_prev"])
)
# %%
gap_rows = []
for symbol, df in datasets.items():
gaps = detect_gaps(df, max_gap_days=5)
for row in gaps.iter_rows(named=True):
gap_rows.append(
{
"symbol": symbol,
"prev_date": _to_date(row["prev_timestamp"]),
"next_date": _to_date(row["timestamp"]),
"days": int(row["days_since_prev"]),
}
)
gap_table = (
pl.DataFrame(gap_rows)
if gap_rows
else pl.DataFrame({"symbol": [], "prev_date": [], "next_date": [], "days": []})
)
gap_table
# %% [markdown]
# ### Deduplication
#
# Duplicates appear when a provider re-sends overlapping date ranges in
# incremental updates. The choice of `keep="first"` vs `"last"` depends on
# whether you trust the original feed or the correction.
# %%
sample = datasets["AAPL"].head(100)
df_with_dups = pl.concat([sample, sample.head(10)]).sort("timestamp")
print(
f"Original: {len(df_with_dups)} rows; keep first: "
f"{df_with_dups.unique(subset=['timestamp'], keep='first').shape[0]} rows; "
f"keep last: {df_with_dups.unique(subset=['timestamp'], keep='last').shape[0]} rows"
)
# %% [markdown]
# ### Corporate-Action Detection
#
# Splits, reverse splits, and large special dividends produce overnight
# returns that trip the >25 % threshold below. The Wiki/Quandl panel here is
# *not* split-adjusted, so AAPL splits (1987-06-16 2:1, 2000-06-21 2:1,
# 2005-02-28 2:1, 2014-06-09 7:1) and the GOOGL Class C distribution
# (2014-04-03) appear as flagged events.
# %%
def detect_corporate_actions(df: pl.DataFrame, threshold: float = 0.25) -> pl.DataFrame:
"""Flag overnight returns whose magnitude exceeds `threshold`."""
return (
df.with_columns(pl.col("close").shift(1).alias("prev_close"))
.with_columns(((pl.col("open") / pl.col("prev_close")) - 1).alias("overnight_return"))
.filter(pl.col("overnight_return").abs() > threshold)
.select(["timestamp", "prev_close", "open", "overnight_return"])
)
# %%
events_rows = []
for symbol, df in datasets.items():
for row in detect_corporate_actions(df, threshold=0.25).iter_rows(named=True):
events_rows.append(
{
"symbol": symbol,
"date": _to_date(row["timestamp"]),
"prev_close": round(row["prev_close"], 2),
"open": round(row["open"], 2),
"overnight_pct": round(row["overnight_return"] * 100, 1),
}
)
events_table = pl.DataFrame(events_rows)
print(f"Potential corporate-action events across {len(datasets)} symbols: {len(events_table)}")
events_table.head(10)
# %% [markdown]
# ---
#
# ## Part 5: Production Quality Pipeline
#
# Wire validation, anomaly detection, and deduplication together with
# quarantine routing for critical failures.
# %%
def quality_check_pipeline(
df: pl.DataFrame,
symbol: str,
quarantine_dir: Path,
anomaly_cfg: AnomalyConfig | None = None,
) -> tuple[pl.DataFrame, dict]:
"""Run validation → anomaly detection → dedup; quarantine on critical issues."""
results: dict = {
"symbol": symbol,
"timestamp": datetime.now().isoformat(),
"input_rows": len(df),
"actions": [],
}
validation = OHLCVValidator().validate(df)
results["validation_issues"] = len(validation.issues)
results["actions"].append(
f"{'PASS' if validation.passed else 'FAIL'} Validation: {len(validation.issues)} issues"
)
if not validation.passed:
critical = [i for i in validation.issues if i.severity.value == "critical"]
if critical:
quarantine_dir.mkdir(parents=True, exist_ok=True)
path = quarantine_dir / f"{symbol}_{datetime.now():%Y%m%d_%H%M%S}.parquet"
df.write_parquet(path)
results["actions"].append(f"QUARANTINED: {path.name}")
mgr = AnomalyManager(config=anomaly_cfg) if anomaly_cfg else AnomalyManager()
report = mgr.analyze(df, symbol)
results["anomaly_count"] = len(report.anomalies)
results["actions"].append(f"Anomalies: {len(report.anomalies)}")
duplicates = len(df) - df["timestamp"].n_unique()
if duplicates > 0:
df = df.unique(subset=["timestamp"], keep="last")
results["actions"].append(f"Removed {duplicates} duplicates")
results["output_rows"] = len(df)
results["status"] = (
"PASS" if validation.passed and not report.get_critical_anomalies() else "REVIEW"
)
return df, results
# %%
quarantine_dir = OUTPUT_DIR / "quarantine"
pipeline_rows = []
for symbol, df in datasets.items():
_, result = quality_check_pipeline(df, symbol, quarantine_dir, anomaly_config)
pipeline_rows.append(
{
"symbol": symbol,
"status": result["status"],
"input_rows": result["input_rows"],
"output_rows": result["output_rows"],
"validation_issues": result["validation_issues"],
"anomalies": result["anomaly_count"],
}
)
pipeline_summary = pl.DataFrame(pipeline_rows)
pipeline_summary
# %% [markdown]
# ---
#
# ## Key Takeaways
#
# Quality-pipeline profile across five large-cap symbols (AAPL, MSFT, GOOGL,
# NVDA, FB) on the legacy Wiki/Quandl panel (1962-01 → 2018-03).
#
# ### Quantitative Findings
# - **Structural validation**: All five symbols pass the OHLCVValidator with
# only an `extreme_returns` warning — that warning fires on the same large
# moves the corporate-action detector flags below.
# - **Synthetic-fault detection**: After injecting `high < low`, negative
# volume, and a null close into AAPL, the validator returns `passed=False`
# with 4 errors / 0 critical / 1 warning, identifying every fault.
# - **Return-outlier detector method spread (AAPL, threshold = 3.0)**:
# MAD = 336 events, Z-score = 83, IQR = 69. MAD is most sensitive in the
# tails because the AAPL return distribution is heavy-tailed; the top MAD
# events are the four AAPL stock splits and the 2000-09-29 earnings crash.
# - **Volume spikes**: 180 AAPL events at threshold 3.0; the largest is
# 2014-09-09 (~3.2× the 20-day average), the eve of the iPhone 6 launch.
# - **Price staleness**: 0 events on AAPL — the legacy panel is clean for
# this symbol.
# - **PSI (AAPL daily returns, halves split)**: 0.118 ⇒ moderate shift.
# Baseline 1980-12-12 → 1999-07-21, current 1999-07-21 → 2018-03-27 — the
# regime change between the two halves is detectable but not extreme.
# - **Hygiene**: The only >5-day gap is the 2001-09-10 → 09-17 NYSE closure
# following 9/11, present in the three symbols listed before that date
# (AAPL, MSFT, NVDA); GOOGL and FB IPO'd later. Across the universe the
# corporate-action detector flags 27 events at the 25% threshold,
# dominated by AAPL/MSFT/NVDA splits and the GOOGL Class C distribution
# (2014-04-03).
#
# ### Implications for Practitioners
# - **Pre-adjustment matters**: An unadjusted historical panel makes the
# anomaly detectors fire on real corporate events. Either adjust upstream
# or maintain a corporate-action whitelist that the pipeline consults
# before quarantining.
# - **MAD over Z-score**: For heavy-tailed financial returns, MAD's tail
# sensitivity is a feature; the operator must classify each event rather
# than rely on a single auto-threshold.
# - **PSI as alarm, not classifier**: A moderate shift between two 19-year
# halves is unsurprising; PSI is most useful at the rolling-30-day
# timescale where regime changes manifest faster.
#
# **Next**: `14_point_in_time_validation` adds the temporal dimension
# (bitemporal queries) on top of the structural checks shown here.
02_financial_data_universe/14_point_in_time_validation.ipynb
+2978
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/14_point_in_time_validation.py
+468
View File
@@ -0,0 +1,468 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Point-in-Time (PIT) Data Validation
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Demonstrate the operational checks needed to keep a backtest free of
# lookahead bias on real ETF and macroeconomic data. Three threads run
# through the notebook: feature-time vs decision-time alignment, a
# correlation-based heuristic for catching obvious feature leakage, and the
# bitemporal (vintage) view of macro data via the FRED provider.
#
# ## Learning Objectives
# - Distinguish *event time* (when something happened) from *knowledge time*
# (when we learned about it), and the bitemporal axis for macro releases.
# - Show why a centered moving average leaks future information visually,
# and why a level-vs-return correlation heuristic is a *weak* leakage
# detector that misses obvious cases.
# - Build the corrected scale-invariant heuristic (feature-return ↔
# future-return) that does fire for `close.pct_change(-1)` features.
# - Query FRED with `vintage_date` to see how the GDP advance estimate
# differs from the revised value.
#
# ## Book reference
# Chapter 2, §2.3 (data quality framework — point-in-time correctness and
# bitemporal data). The figure below is the §2.3 illustration.
#
# ## Prerequisites
# - ETF parquet files materialized under `ML4T_DATA_PATH`.
# - Macro parquet (FRED snapshot) materialized under `ML4T_DATA_PATH` or
# loadable via `data.load_macro`.
# - `FRED_API_KEY` environment variable for the live vintage query in §5
# (free key at https://fred.stlouisfed.org/docs/api/api_key.html).
# %%
"""Point-in-Time Data Validation."""
import os
from datetime import datetime
import plotly.graph_objects as go
import polars as pl
from ml4t.data.providers import FREDProvider
from data import load_etfs, load_macro
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ## 1. Load Real Market Data
#
# SPY daily bars (2006-01-03 → 2025-12-31) anchor every example below.
# %%
spy = load_etfs().filter(pl.col("symbol") == "SPY").sort("timestamp")
print(f"SPY data: {len(spy)} rows; {spy['timestamp'].min()} → {spy['timestamp'].max()}")
spy.head()
# %% [markdown]
# ## 2. Lookahead Bias: A Visual Demonstration
#
# A *centered* moving average computed at time $T$ averages prices in
# $[T-2, T+2]$ — three of those five inputs are unavailable at $T$. The
# trailing window only uses $[T-4, T]$, which is what a live system can
# actually compute.
# %%
spy_ma = (
spy.with_columns(
pl.col("close").rolling_mean(window_size=20).alias("ma20_trailing"),
pl.col("close").rolling_mean(window_size=20, center=True).alias("ma20_centered"),
)
.drop_nulls()
.tail(120)
)
# %%
fig = go.Figure()
fig.add_trace(
go.Scatter(
x=spy_ma["timestamp"].to_list(),
y=spy_ma["close"].to_list(),
name="SPY close",
line=dict(color="black", width=1),
)
)
fig.add_trace(
go.Scatter(
x=spy_ma["timestamp"].to_list(),
y=spy_ma["ma20_trailing"].to_list(),
name="20d trailing MA (PIT-correct)",
line=dict(color="#1f77b4", width=2),
)
)
fig.add_trace(
go.Scatter(
x=spy_ma["timestamp"].to_list(),
y=spy_ma["ma20_centered"].to_list(),
name="20d centered MA (uses future)",
line=dict(color="#d62728", width=2, dash="dash"),
)
)
fig.update_layout(
title="Trailing vs Centered Moving Average — Centered Knows the Future",
xaxis_title="Date",
yaxis_title="SPY price",
height=420,
template="plotly_white",
)
fig.show()
# %% [markdown]
# The dashed centered MA tracks SPY more tightly because it averages
# tomorrow's price into today's smoother. A live trading system cannot
# reproduce that line, so any backtest using it will overstate skill.
# %% [markdown]
# ## 3. Why the Naïve Correlation Heuristic Fails
#
# A natural-but-wrong leakage test correlates the *level* of a feature with
# next-period *return*. The two live on different distributions: prices are
# (close to) random walks while returns are mean-zero noise, so even a
# feature constructed from tomorrow's close has a ~0 correlation with the
# next-day return.
# %%
def naive_leakage_corr(df: pl.DataFrame, feature: str, price: str = "close") -> float:
"""Correlation between a level feature and the next-period price return.
Reproduces the textbook-warning heuristic that *seems* like it should
catch lookahead but rarely does because it compares incompatible scales.
"""
enriched = df.with_columns(
(pl.col(price).shift(-1) / pl.col(price) - 1).alias("next_ret")
).drop_nulls([feature, "next_ret"])
return float(enriched.select(pl.corr(feature, "next_ret")).item())
# %%
naive_features = spy.with_columns(
pl.col("close").rolling_mean(window_size=5).alias("ma5_trailing"),
pl.col("close").rolling_mean(window_size=5, center=True).alias("ma5_centered"),
pl.col("close").shift(-1).alias("tomorrow_close"),
).drop_nulls()
naive_results = pl.DataFrame(
[
{"feature": f, "naive_corr_with_next_return": naive_leakage_corr(naive_features, f)}
for f in ["ma5_trailing", "ma5_centered", "tomorrow_close"]
]
)
naive_results
# %% [markdown]
# All three features score near zero — the heuristic flunks both the trailing
# (clean) and the lookahead (centered, tomorrow's close) examples. The
# problem is the units: a ~$500 price level has no scale relationship with a
# ±1 % daily return. The fix is a **scale-invariant** test.
# %% [markdown]
# ## 4. A Correct Scale-Invariant Leakage Test
#
# Convert any feature to its own period-over-period return, then correlate
# with the next-period price return. A feature constructed from $\mathrm{close}_{T+k}$
# now exposes itself: when $k \geq 1$ the feature-return at $T$ literally
# *is* the price return some periods ahead, so the correlation jumps to ~1.
# %%
def leakage_corr_scale_invariant(df: pl.DataFrame, feature: str, price: str = "close") -> float:
"""Correlation between a feature's own return and the next price return.
Most useful when `feature` is a price-like (level) series — for return-
shaped features the pct_change is unstable near zero, so the caller
should pass them through directly instead.
"""
enriched = (
df.with_columns(
pl.col(feature).pct_change().alias("feat_ret"),
(pl.col(price).shift(-1) / pl.col(price) - 1).alias("next_ret"),
)
.drop_nulls(["feat_ret", "next_ret"])
.filter(pl.col("feat_ret").is_finite() & pl.col("next_ret").is_finite())
)
if enriched.is_empty():
return float("nan")
return float(enriched.select(pl.corr("feat_ret", "next_ret")).item())
# %%
fixed_results = pl.DataFrame(
[
{
"feature": f,
"scale_invariant_corr": leakage_corr_scale_invariant(naive_features, f),
}
for f in ["ma5_trailing", "ma5_centered", "tomorrow_close"]
]
)
fixed_results
# %% [markdown]
# The same three features now sort cleanly: the trailing MA stays low, the
# centered MA's correlation jumps because three of its five inputs sit in
# the future, and `tomorrow_close` (which is literally the next day's price)
# saturates near 1. This is the leakage detector you actually want.
# %% [markdown]
# ## 5. Signal–Trade Lag: Trading Days, Not Calendar Days
#
# Daily strategies that close-of-day on $T$ must trade no earlier than the
# open of $T+1$. The library helper `validate_signal_trade_lag` shifts by
# *rows* (trading days) so weekends and holidays drop out automatically.
# %%
def validate_signal_trade_lag(signals: pl.DataFrame, execution_lag: int = 1) -> pl.DataFrame:
"""Tag each signal with the earliest tradable date `execution_lag` rows ahead."""
return signals.sort("timestamp").with_columns(
pl.col("timestamp").alias("signal_date"),
pl.col("timestamp").shift(-execution_lag).alias("earliest_execution"),
)
# %%
spy_with_signal = spy.with_columns(
(pl.col("close") / pl.col("close").shift(5) - 1).alias("momentum_signal")
)
validate_signal_trade_lag(spy_with_signal, execution_lag=1).select(
["timestamp", "close", "momentum_signal", "signal_date", "earliest_execution"]
).head(8)
# %% [markdown]
# ## 6. Macro Data Freshness and Forward-Fill
#
# Macro series have very different cadences. Always forward-fill — never
# back-fill — and remember that the timestamp must represent the *release*
# date, not the period the data describes, for the fill to be PIT-safe.
# %%
macro = load_macro()
macro = macro.rename({col: col.lower() for col in macro.columns})
date_col = "timestamp" if "timestamp" in macro.columns else "date"
macro_summary = pl.DataFrame(
[
{
"series": s,
"non_null": macro.filter(pl.col(s).is_not_null()).height,
"first": macro.filter(pl.col(s).is_not_null())[date_col].min(),
"last": macro.filter(pl.col(s).is_not_null())[date_col].max(),
}
for s in ["dgs10", "unrate", "vixcls"]
if s in macro.columns
]
)
macro_summary
# %%
unrate = (
macro.select([date_col, "unrate"])
.filter(pl.col(date_col).dt.year() >= 2023)
.with_columns(pl.col("unrate").forward_fill().alias("unrate_filled"))
.filter(
(pl.col(date_col) >= datetime(2023, 1, 1)) & (pl.col(date_col) <= datetime(2023, 3, 31))
)
)
unrate.head(10)
# %% [markdown]
# ## 7. Bitemporal GDP via FRED `vintage_date`
#
# Macro data is revised. The FRED API's `realtime_start` / `realtime_end`
# parameters return the values **as known at** a chosen historical date.
# `FREDProvider.fetch_ohlcv` exposes that as `vintage_date`.
#
# The cell below requires `FRED_API_KEY` (free signup); if the key is
# missing the fetch fails loudly rather than substituting a synthetic
# illustration.
# %%
FRED_API_KEY = os.getenv("FRED_API_KEY")
if not FRED_API_KEY:
raise RuntimeError(
"FRED_API_KEY not set. Get a free key at "
"https://fred.stlouisfed.org/docs/api/api_key.html and export it."
)
provider = FREDProvider()
gdp_early = provider.fetch_ohlcv(
"GDP",
"2023-01-01",
"2023-09-30",
frequency="quarterly",
vintage_date="2023-11-01",
)
gdp_late = provider.fetch_ohlcv(
"GDP",
"2023-01-01",
"2023-09-30",
frequency="quarterly",
vintage_date="2024-06-01",
)
provider.close()
revisions = (
gdp_early.select(["timestamp", pl.col("close").alias("vintage_2023_11_01")])
.join(
gdp_late.select(["timestamp", pl.col("close").alias("vintage_2024_06_01")]),
on="timestamp",
)
.with_columns(
(
(pl.col("vintage_2024_06_01") - pl.col("vintage_2023_11_01"))
/ pl.col("vintage_2023_11_01")
* 100
)
.round(3)
.alias("revision_pct")
)
)
revisions
# %% [markdown]
# Q1 2023 and Q2 2023 are unchanged between the two vintages, but the Q3
# 2023 *advance* level (released 2023-10-26 and visible to a 2023-11-01
# query) was revised down ~0.05 % by mid-2024. That is small for GDP but
# would compound across many series in a macro-driven backtest.
# %% [markdown]
# ## 8. PIT Validator Walkthrough
#
# Wrap the scale-invariant leakage test (above), a date-monotonicity check,
# and a gap audit into a single class so it can be run over a feature
# matrix in one call.
# %%
class PITValidator:
"""Point-in-time validation for daily price-and-feature panels."""
LEAK_THRESHOLD = 0.30 # |corr| above which we flag
def __init__(self, df: pl.DataFrame, date_col: str = "timestamp"):
self.df = df
self.date_col = date_col
self.violations: list[dict] = []
def check_future_leakage(self, feature_col: str, price_col: str = "close") -> dict:
corr = leakage_corr_scale_invariant(self.df, feature_col, price_col)
severity = "HIGH" if abs(corr) > self.LEAK_THRESHOLD else "LOW"
result = {
"feature": feature_col,
"scale_invariant_corr": round(corr, 4),
"severity": severity,
"violation": severity == "HIGH",
}
if result["violation"]:
self.violations.append(result)
return result
def check_date_gaps(self, max_gap_days: int = 5) -> dict:
gaps = (
self.df.sort(self.date_col)
.with_columns(pl.col(self.date_col).diff().dt.total_days().alias("gap_days"))
.filter(pl.col("gap_days") > max_gap_days)
)
return {
"rows": len(self.df),
"gaps_above_threshold": len(gaps),
"max_gap_days": int(gaps["gap_days"].max()) if len(gaps) > 0 else 0,
}
def check_monotonic_dates(self) -> dict:
diffs = self.df[self.date_col].diff().drop_nulls().dt.total_days()
is_sorted = bool((diffs >= 0).all())
return {"is_monotonic": is_sorted, "violation": not is_sorted}
# %%
spy_features = spy.with_columns(
pl.col("close").rolling_mean(window_size=5).alias("ma5_trailing"),
pl.col("close").rolling_mean(window_size=20).alias("ma20_trailing"),
pl.col("close").rolling_mean(window_size=5, center=True).alias("ma5_centered_BAD"),
pl.col("close").shift(-1).alias("tomorrow_close_BAD"),
).drop_nulls()
validator = PITValidator(spy_features)
validation_table = pl.DataFrame(
[
validator.check_future_leakage(f)
for f in [
"ma5_trailing",
"ma20_trailing",
"ma5_centered_BAD",
"tomorrow_close_BAD",
]
]
)
validation_table
# %%
gap_check = validator.check_date_gaps(max_gap_days=5)
mono_check = validator.check_monotonic_dates()
print(
f"Date gaps over 5 calendar days: {gap_check['gaps_above_threshold']} "
f"(max {gap_check['max_gap_days']} d); monotonic={mono_check['is_monotonic']}"
)
print(f"Violations flagged: {len(validator.violations)}")
# %% [markdown]
# ## Key Takeaways
#
# Operational PIT checks on SPY (2006-2025) and the FRED GDP series.
#
# ### Quantitative Findings
# - **Naïve heuristic (level vs return correlation)** scores |0.0049|,
# |0.0100|, and |0.0189| for trailing-MA, centered-MA and `tomorrow_close`
# respectively — all near zero, so it cannot distinguish leakage from clean
# construction on this panel and is not a useful detector by itself.
# - **Scale-invariant heuristic (feature-return vs next-return)** scores
# ~0 for the trailing MA-5 and momentum, jumps for the centered MA-5,
# and saturates near 1.0 for `tomorrow_close` — exactly the ordering one
# expects when leakage is real.
# - **PIT validator** flags `ma5_centered_BAD` and `tomorrow_close_BAD` at
# the 0.30 threshold; trailing MA-5/20 pass. (Return-shaped features like
# 5-day momentum should be checked by direct correlation rather than the
# pct_change-based helper, which destabilizes near zero.)
# - **GDP vintage**: Q3 2023 GDP-level revision between vintages
# 2023-11-01 and 2024-06-01 is −0.049 % ($27 623.5 → $27 610.1B). Q1/Q2
# are unchanged. Small in absolute terms, but every macro series in a
# backtest has its own revision profile.
# - **Macro panel cadence**: `unrate` carries 9,497 daily forward-filled
# rows (2000-01-01 → 2025-12-31) but the underlying release is monthly —
# forward-fill across calendar days only is PIT-safe when the timestamp
# represents the release date, not the observation period.
#
# ### Implications for Practitioners
# - **Construction matters more than detection**: the centered-MA leak is
# visible in the figure but invisible to the naïve correlation test.
# Detection is a backstop, not a substitute for code review.
# - **Always express features in scale-invariant form** when running
# automated leakage scans.
# - **Use vintage queries for any macro indicator that is revised** (GDP,
# payrolls, inflation). Revisions inflate apparent strategy quality if the
# final value is used for a trade decision the backtest maker only could
# have observed at the advance value.
#
# **Next**: `15_survivorship_bias_detection` adds the
# universe-membership dimension to the temporal correctness shown here.
02_financial_data_universe/15_survivorship_bias_detection.ipynb
+22714
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/15_survivorship_bias_detection.py
+1220
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/16_provider_comparison.ipynb
+20631
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/16_provider_comparison.py
+802
View File
@@ -0,0 +1,802 @@
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Provider Comparison: Multi-Source Data Acquisition
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Demonstrate the `ml4t.data.providers` unified `fetch_ohlcv()` interface
# across YahooFinance, WikiPrices and FRED, then build a fallback strategy
# and a price-level provider-comparison report. The notebook seeds the
# ETF rotation universe used by the ETFs case study downstream.
#
# ## Learning Objectives
# - Fetch OHLCV from multiple providers with one call signature.
# - Build a priority-ordered fallback fetcher for production resilience.
# - Quantify provider disagreement (row counts, price-difference stats)
# so multi-source stitching is auditable.
# - Use FRED (vintage-aware) to pull macro series for risk overlays.
#
# ## Book reference
# Chapter 2, §2.3 (multi-source stitching). The ETF universe materialised
# here feeds the `case_studies/etfs/` pipeline.
#
# ## Prerequisites
# - Network access for live YahooFinance + FRED calls (the notebook is a
# provider-acquisition demo — these are the only places in the
# publication pass where live API calls are intentional).
# - WikiPrices parquet under `ML4T_DATA_PATH/equities/market/us_equities/us_equities.parquet`
# for the historical leg.
# - `FRED_API_KEY` set (free signup) for §8.
# %%
"""Provider Comparison — Multi-source data acquisition with ml4t-data providers."""
import os
from dataclasses import dataclass
from datetime import datetime, timedelta
from pathlib import Path
import numpy as np
import plotly.graph_objects as go
import polars as pl
from ml4t.data.providers import WikiPricesProvider, YahooFinanceProvider
from ml4t.data.providers.fred import FREDProvider
HAS_FRED = bool(os.getenv("FRED_API_KEY"))
if not HAS_FRED:
raise RuntimeError(
"FRED_API_KEY not set. Get a free key at "
"https://fred.stlouisfed.org/docs/api/api_key.html and export it."
)
# Reproducibility: a fixed as-of date keeps outputs stable between book
# editions. Bump when the book is revised.
AS_OF_DATE = "2025-01-15"
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ---
#
# ## Section 1: Understanding the Provider Architecture
#
# ml4t-data provides a **unified interface** for fetching data from multiple sources. All providers inherit from `BaseProvider` and implement the same `fetch_ohlcv()` method.
#
# ### Key Benefits:
# 1. **Consistency**: Same API regardless of data source
# 2. **Validation**: Automatic OHLC invariant checks
# 3. **Polars Output**: 10-100x faster than pandas alternatives
# 4. **Rate Limiting**: Built-in throttling to avoid API bans
# 5. **Circuit Breaker**: Automatic failure detection and recovery
# %%
# The ml4t-data provider landscape
# Note: This notebook demonstrates Yahoo, WikiPrices, and FRED. Other providers
# are available but require API keys and are covered in asset-class-specific notebooks.
providers_info = pl.DataFrame(
{
"provider": [
"YahooFinanceProvider",
"WikiPricesProvider",
"FREDProvider",
"EODHDProvider",
"BinanceAPIProvider",
],
"asset_class": [
"US Equities, ETFs",
"US Equities (Historical)",
"Economic Indicators",
"Global Equities",
"Crypto",
],
"api_key_required": [
"No",
"No (local file)",
"Yes (free)",
"Yes (free tier)",
"No",
],
"date_range": ["~30 years", "1962-2018", "50+ years", "~20 years", "~5 years"],
"demonstrated_in": [
"this notebook",
"this notebook",
"this notebook",
"10_crypto_perps_eda",
"10_crypto_perps_eda",
],
}
)
providers_info
# %% [markdown]
# ---
#
# ## Section 2: Fetching Data with Yahoo Finance
#
# Yahoo Finance is the easiest starting point - no API key required. Let's fetch our ETF universe for the momentum strategy.
# %%
# Define our ETF universe for the rotation strategy
ETF_UNIVERSE = ["SPY", "QQQ", "IWM", "EFA", "EEM", "TLT", "GLD"]
# Date range: 5 years of daily data (relative to AS_OF_DATE for reproducibility)
end_date = AS_OF_DATE
start_date = (datetime.strptime(AS_OF_DATE, "%Y-%m-%d") - timedelta(days=5 * 365)).strftime(
"%Y-%m-%d"
)
print(f"Fetching data from {start_date} to {end_date}")
print(f"ETF Universe: {ETF_UNIVERSE}")
# %%
# Create Yahoo Finance provider
yahoo = YahooFinanceProvider()
# Fetch SPY as our primary example
spy_data = yahoo.fetch_ohlcv("SPY", start_date, end_date)
print(f"Fetched {len(spy_data)} rows of SPY data")
print(f"Date range: {spy_data['timestamp'].min()} to {spy_data['timestamp'].max()}")
print("\nSample data:")
print(spy_data.head())
# %%
# Examine the schema - ml4t-data provides consistent column names
print("Schema (consistent across all providers):")
for name, dtype in spy_data.schema.items():
print(f" {name}: {dtype}")
# %%
# Fetch the full ETF universe — fail loudly on missing tickers
etf_data = {symbol: yahoo.fetch_ohlcv(symbol, start_date, end_date) for symbol in ETF_UNIVERSE}
print(
f"Fetched {len(etf_data)}/{len(ETF_UNIVERSE)} ETFs · "
f"{sum(len(d) for d in etf_data.values()):,} total daily rows"
)
# %%
# Combine into a single DataFrame with symbol column
combined_dfs = []
for symbol, df in etf_data.items():
combined_dfs.append(df.with_columns(pl.lit(symbol).alias("symbol")))
etf_universe_df = pl.concat(combined_dfs).sort(["symbol", "timestamp"])
(
etf_universe_df.group_by("symbol")
.agg(
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.len().alias("rows"),
pl.col("close").last().alias("last_close"),
)
.sort("symbol")
)
# %% [markdown]
# ---
#
# ## Section 3: Historical Data with WikiPrices
#
# For long-term backtests (30+ years), Yahoo Finance has limitations. WikiPrices provides
# long-horizon U.S. equity history (including many delisted names) from roughly 1962 to 2018.
#
# **Key Advantage**: Includes delisted companies, but you still need explicit survivorship
# checks and delisting return handling (see `08_survivorship_bias_detection`).
# %%
from utils import ML4T_DATA_PATH
WIKI_PATHS = [
# Primary: canonical layout under ML4T_DATA_PATH
ML4T_DATA_PATH / "equities" / "market" / "us_equities" / "us_equities.parquet",
# Docker mount (same nested layout)
Path("/data/equities/market/us_equities/us_equities.parquet"),
# Test fixture (CI subset)
Path("/app/tests/fixtures/data/equities/market/us_equities/us_equities.parquet"),
Path("tests/fixtures/data/equities/market/us_equities/us_equities.parquet"),
]
wiki = None
wiki_path_used = None
wiki_load_errors = []
# %% [markdown]
# ### WikiPrices Schema Adapter
#
# When WikiPrices data has been canonicalized to `symbol`/`timestamp` columns,
# this adapter provides the same `fetch_ohlcv()` interface as the library provider.
# %%
class CanonicalWikiPricesAdapter:
"""Adapter for canonicalized local Wiki Prices parquet."""
def __init__(self, parquet_path: Path):
self.parquet_path = Path(parquet_path)
def fetch_ohlcv(self, symbol: str, start: str, end: str) -> pl.DataFrame:
start_date = datetime.strptime(start, "%Y-%m-%d").date()
end_date = datetime.strptime(end, "%Y-%m-%d").date()
return (
pl.scan_parquet(self.parquet_path)
.filter(
(pl.col("symbol") == symbol)
& pl.col("timestamp").is_between(start_date, end_date, closed="both")
)
.select(
[
pl.col("timestamp").cast(pl.Datetime("us")).alias("timestamp"),
"open",
"high",
"low",
"close",
"volume",
"adj_close",
"adj_open",
"adj_high",
"adj_low",
"adj_volume",
"ex_dividend",
"split_ratio",
]
)
.sort("timestamp")
.collect()
)
def list_available_symbols(self) -> list[str]:
return (
pl.scan_parquet(self.parquet_path)
.select(pl.col("symbol").unique().sort())
.collect()
.to_series()
.to_list()
)
def close(self) -> None:
"""Mirror provider lifecycle API."""
return None
# %% [markdown]
# ### Load WikiPrices Provider
#
# Try multiple paths to find WikiPrices data (local install, Docker, CI fixtures).
# %%
for wiki_path in WIKI_PATHS:
if wiki_path.exists():
print(f" Found: {wiki_path}")
try:
wiki = WikiPricesProvider(parquet_path=wiki_path)
wiki_path_used = wiki_path
break
except Exception as e:
# File exists but failed to load - this is a real error, not silent skip
wiki_load_errors.append((wiki_path, str(e)))
print(f" ERROR loading {wiki_path}: {e}")
# If we found files but couldn't load any of them, that's a bug - fail loudly
if wiki is None and wiki_load_errors:
error_msg = "WikiPrices files found but failed to load:\n"
for path, err in wiki_load_errors:
error_msg += f" {path}: {err}\n"
raise RuntimeError(error_msg)
# %%
if wiki is None:
raise RuntimeError(
f"WikiPrices parquet not found in any of: {[str(p) for p in WIKI_PATHS]}. "
"Materialise it via WikiPricesProvider.download(api_key=<nasdaq>) first."
)
print(f"WikiPrices loaded from: {wiki_path_used}")
# Fetch long-term AAPL history; the canonical local schema may need the
# adapter wrapper if the file uses asset/date instead of symbol/timestamp.
try:
aapl_historical = wiki.fetch_ohlcv("AAPL", "1990-01-01", "2018-03-27")
except Exception as e:
if 'unable to find column "ticker"' not in str(e):
raise
print("Detected canonical schema; switching to CanonicalWikiPricesAdapter.")
wiki = CanonicalWikiPricesAdapter(wiki_path_used)
aapl_historical = wiki.fetch_ohlcv("AAPL", "1990-01-01", "2018-03-27")
print(
f"AAPL data: {len(aapl_historical)} rows · "
f"{aapl_historical['timestamp'].min()} → {aapl_historical['timestamp'].max()}"
)
aapl_historical.head()
# %%
# WikiPrices includes delisted companies — critical for avoiding survivorship bias.
available_symbols = wiki.list_available_symbols()
print(f"WikiPrices contains {len(available_symbols):,} symbols; sample: {available_symbols[:10]}")
# %% [markdown]
# ---
#
# ## Section 4: Multi-Provider Fallback Strategy
#
# For production systems, we need a robust strategy that handles:
# 1. API failures
# 2. Rate limits
# 3. Data gaps
# 4. Historical coverage
#
# The **fallback pattern** tries providers in order until one succeeds.
# %%
@dataclass
class FetchResult:
"""Result of a multi-provider fetch attempt."""
success: bool
data: pl.DataFrame | None
provider_used: str | None
providers_tried: list[str]
error_messages: dict[str, str]
# %% [markdown]
# ### Fallback Fetch
#
# Try multiple providers in priority order; return the first successful result.
# %%
def fetch_with_fallback(
symbol: str,
start: str,
end: str,
providers: list,
provider_names: list[str],
) -> FetchResult:
"""
Fetch data trying multiple providers in order.
Parameters
----------
symbol : str
Ticker symbol
start, end : str
Date range (YYYY-MM-DD)
providers : list
Provider instances to try
provider_names : list[str]
Names for logging
Returns
-------
FetchResult
Result with data and metadata
"""
errors = {}
tried = []
for provider, name in zip(providers, provider_names, strict=False):
tried.append(name)
try:
data = provider.fetch_ohlcv(symbol, start, end)
if len(data) > 0:
return FetchResult(
success=True,
data=data,
provider_used=name,
providers_tried=tried,
error_messages=errors,
)
else:
errors[name] = "Empty result"
except Exception as e:
errors[name] = str(e)
return FetchResult(
success=False,
data=None,
provider_used=None,
providers_tried=tried,
error_messages=errors,
)
# %%
# Try Yahoo first, then WikiPrices as fallback
providers = [yahoo, wiki]
provider_names = ["Yahoo Finance", "WikiPrices"]
result = fetch_with_fallback("AAPL", "2020-01-01", "2020-12-31", providers, provider_names)
if not result.success:
raise RuntimeError(f"All providers failed: {result.error_messages}")
print(
f"Success via {result.provider_used}: {len(result.data):,} rows; tried {result.providers_tried}"
)
# %% [markdown]
# ---
#
# ## Section 5: Combining Historical and Recent Data
#
# For 30+ year backtests, we combine:
# 1. **WikiPrices** (1962-2018) - includes delisted names
# 2. **Yahoo Finance** (2018-present) - current data
#
# This gives us the best of both worlds.
# %%
def build_complete_history(
symbol: str,
yahoo_provider,
wiki_provider=None,
start_date: str = "1990-01-01",
end_date: str = None,
) -> pl.DataFrame:
"""Build complete OHLCV history combining WikiPrices (pre-2018) and Yahoo Finance (2018+)."""
if end_date is None:
end_date = AS_OF_DATE
parts = []
# WikiPrices cutoff
wiki_end = "2018-03-27"
# Part 1: Historical data from WikiPrices
if wiki_provider and start_date < wiki_end:
try:
historical = wiki_provider.fetch_ohlcv(symbol, start_date, wiki_end)
if len(historical) > 0:
parts.append(historical)
print(f" WikiPrices: {len(historical)} rows ({start_date} to {wiki_end})")
except Exception as e:
print(f" WikiPrices: {e}")
# Part 2: Recent data from Yahoo Finance
yahoo_start = "2018-03-28" if start_date < wiki_end else start_date
try:
recent = yahoo_provider.fetch_ohlcv(symbol, yahoo_start, end_date)
if len(recent) > 0:
parts.append(recent)
print(f" Yahoo Finance: {len(recent)} rows ({yahoo_start} to {end_date})")
except Exception as e:
print(f" Yahoo Finance: {e}")
if not parts:
raise ValueError(f"No data found for {symbol}")
# Standardize columns and types before concatenation
# Both providers should have: timestamp, open, high, low, close, volume
standard_cols = ["timestamp", "open", "high", "low", "close", "volume"]
standardized = []
for df in parts:
# Select only standard columns (ensure all have same schema)
df_std = df.select(standard_cols)
# Normalize types: timestamp to us precision, numerics to Float64
df_std = df_std.with_columns(
pl.col("timestamp").cast(pl.Datetime("us")),
pl.col("open", "high", "low", "close").cast(pl.Float64),
pl.col("volume").cast(
pl.Float64
), # Volume can be Int64 or Float64 depending on provider
)
standardized.append(df_std)
# Combine and deduplicate
combined = pl.concat(standardized).sort("timestamp").unique("timestamp")
print(f" Combined: {len(combined)} rows total")
return combined
# %%
# Build complete AAPL history (1990-now)
aapl_complete = build_complete_history("AAPL", yahoo, wiki, "1990-01-01")
print(
f"Combined: {len(aapl_complete):,} rows · "
f"{aapl_complete['timestamp'].min()} → {aapl_complete['timestamp'].max()}"
)
# %% [markdown]
# ---
#
# ## Section 6: Provider Data Comparison
#
# Yahoo Finance and WikiPrices both expose a close price, but they apply different
# adjustment conventions and cover different date ranges, so the two series can
# diverge sharply. This section demonstrates how to:
#
# 1. Align data from multiple providers by date
# 2. Quantify differences (row counts, price discrepancies)
# 3. Identify systematic vs random variations
# 4. Diagnose the causes (splits, dividends, timing)
#
# **Why this matters**: Multi-source strategies must understand when providers disagree.
# Small discrepancies compound over backtests; large ones signal data errors.
# %%
def compare_providers_detailed(
symbol: str,
start: str,
end: str,
provider_a,
provider_b,
name_a: str = "Provider A",
name_b: str = "Provider B",
) -> dict:
"""
Detailed comparison of two data providers for a symbol.
Returns dict with:
- summary: high-level stats
- aligned: row-by-row comparison DataFrame
- discrepancies: rows where providers disagree significantly
"""
try:
data_a = provider_a.fetch_ohlcv(symbol, start, end)
data_b = provider_b.fetch_ohlcv(symbol, start, end)
except Exception as e:
return {"error": str(e), "summary": None, "aligned": None, "discrepancies": None}
# Normalize timestamps to date for joining (providers may have different time components)
data_a = data_a.with_columns(pl.col("timestamp").dt.date().alias("timestamp"))
data_b = data_b.with_columns(pl.col("timestamp").dt.date().alias("timestamp"))
# Inner join on date to get overlapping rows
aligned = data_a.select(["timestamp", "open", "high", "low", "close", "volume"]).join(
data_b.select(["timestamp", "open", "high", "low", "close", "volume"]),
on="timestamp",
suffix="_b",
)
# Calculate differences
aligned = aligned.with_columns(
[
((pl.col("close") - pl.col("close_b")) / pl.col("close_b") * 100).alias(
"close_diff_pct"
),
((pl.col("volume") - pl.col("volume_b")) / pl.col("volume_b") * 100).alias(
"volume_diff_pct"
),
]
)
# Identify significant discrepancies (>0.1% price difference)
discrepancies = aligned.filter(pl.col("close_diff_pct").abs() > 0.1)
# Summary statistics
summary = {
"symbol": symbol,
"period": f"{start} to {end}",
f"{name_a}_rows": len(data_a),
f"{name_b}_rows": len(data_b),
"aligned_rows": len(aligned),
"missing_in_a": len(data_b) - len(aligned),
"missing_in_b": len(data_a) - len(aligned),
"mean_close_diff_pct": aligned["close_diff_pct"].mean() if len(aligned) > 0 else None,
"max_close_diff_pct": aligned["close_diff_pct"].abs().max() if len(aligned) > 0 else None,
"discrepancy_days": len(discrepancies),
"exact_matches": len(aligned.filter(pl.col("close_diff_pct").abs() < 0.01)),
}
return {"summary": summary, "aligned": aligned, "discrepancies": discrepancies}
# %%
comparison_result = compare_providers_detailed(
"AAPL", "2017-01-01", "2017-12-31", yahoo, wiki, "Yahoo", "WikiPrices"
)
if comparison_result.get("error"):
raise RuntimeError(f"Comparison failed: {comparison_result['error']}")
summary = comparison_result["summary"]
summary_df = pl.DataFrame([summary])
summary_df
# %%
# Show top discrepancies (if any)
disc = comparison_result["discrepancies"]
if len(disc) > 0:
disc.select(["timestamp", "close", "close_b", "close_diff_pct"]).head(5)
else:
print("No significant Yahoo↔WikiPrices price discrepancies for AAPL 2017.")
# %% [markdown]
# ### What Real Provider Discrepancies Look Like
#
# When comparing Yahoo Finance to WikiPrices on historical data, common findings include:
#
# | Pattern | Typical Magnitude | Cause |
# |---------|-------------------|-------|
# | **Exact match** | <0.01% | Same underlying source |
# | **Small drift** | 0.01-0.1% | Rounding, adjustment timing |
# | **Step change** | 1-10% | Different split adjustment date |
# | **Systematic offset** | Consistent % | Different dividend treatment |
#
# **Key insight**: Raw quotes share the same exchange source, but *adjustments*
# diverge — and that divergence can dominate the comparison. The AAPL 2017 run
# above is a case in point: all 249 overlapping days differ by ~77% (zero exact
# matches) because Yahoo's close retroactively reflects Apple's 2020 4:1 split,
# while WikiPrices ends in 2018 and never applied it. Always align the adjustment
# basis (and watch the coverage window) before diffing two providers.
# %% [markdown]
# ---
#
# ## Section 7: ETF Universe Performance
#
# We visualize the 7 core ETFs defined at the start of this notebook. These represent
# distinct asset classes for the rotation strategy:
#
# | Symbol | Asset Class | Role in Portfolio |
# |--------|-------------|-------------------|
# | SPY | US Large Cap | Risk-on equity |
# | QQQ | US Tech | High-beta growth |
# | IWM | US Small Cap | Cyclical exposure |
# | EFA | Intl Developed | Geographic diversification |
# | EEM | Emerging Markets | Growth/risk allocation |
# | TLT | Long Treasury | Flight-to-quality |
# | GLD | Gold | Inflation/crisis hedge |
#
# A rotation strategy switches between these based on momentum signals (Chapter 7).
# %%
# Calculate normalized prices (start = 100)
fig = go.Figure()
for symbol, df in etf_data.items():
# Normalize to 100 at start
normalized = (df["close"] / df["close"][0]) * 100
fig.add_trace(
go.Scatter(
x=df["timestamp"].to_list(),
y=normalized.to_list(),
name=symbol,
mode="lines",
)
)
fig.update_layout(
title=f"Core ETF Universe Performance ({len(etf_data)} ETFs, Normalized to 100)",
xaxis_title="Date",
yaxis_title="Normalized Price",
height=500,
template="plotly_white",
legend=dict(orientation="h", yanchor="bottom", y=1.02),
)
# Add horizontal line at 100
fig.add_hline(y=100, line_dash="dash", line_color="gray", opacity=0.5)
fig.show()
# %%
# Calculate performance statistics
performance_stats = []
for symbol, df in etf_data.items():
returns = df["close"].pct_change().drop_nulls()
total_return = (df["close"][-1] / df["close"][0] - 1) * 100
annual_return = ((1 + total_return / 100) ** (252 / len(df)) - 1) * 100
volatility = returns.std() * np.sqrt(252) * 100
sharpe = (annual_return - 2) / volatility if volatility > 0 else 0 # Assume 2% risk-free
performance_stats.append(
{
"symbol": symbol,
"total_return_pct": total_return,
"annual_return_pct": annual_return,
"volatility_pct": volatility,
"sharpe_ratio": sharpe,
"trading_days": len(df),
}
)
performance_df = pl.DataFrame(performance_stats).sort("annual_return_pct", descending=True)
performance_df
# %% [markdown]
# ---
#
# ## Section 8: Economic Data with FREDProvider
#
# For macro regime detection and risk management, FRED (Federal Reserve Economic Data)
# provides 800,000+ economic time series. Key series for trading include:
#
# | Series | Description | Frequency |
# |--------|-------------|-----------|
# | VIXCLS | VIX Volatility Index | Daily |
# | DGS10 | 10-Year Treasury Yield | Daily |
# | T10Y2Y | Yield Curve Slope | Daily |
# | UNRATE | Unemployment Rate | Monthly |
# | ICSA | Initial Jobless Claims | Weekly |
#
# **Get free API key**: https://fred.stlouisfed.org/docs/api/api_key.html
# %%
# Pull VIX and 10Y Treasury from FRED for the same window
fred = FREDProvider()
vix = fred.fetch_ohlcv("VIXCLS", "2023-01-01", "2024-01-01")
treasury_10y = fred.fetch_ohlcv("DGS10", "2023-01-01", "2024-01-01")
fred.close()
vix_mean = float(vix["close"].mean())
vix_max = float(vix["close"].max())
last_yield = float(treasury_10y.filter(pl.col("close").is_not_null())["close"][-1])
print(
f"VIX 2023: {len(vix):,} obs · mean {vix_mean:.1f}, max {vix_max:.1f} · "
f"DGS10 last yield {last_yield:.2f}%"
)
vix.head()
# %% [markdown]
# ## Key Takeaways
#
# Multi-source data acquisition profile for the 7-ETF rotation universe.
#
# ### Quantitative Findings
# - **ETF universe** (5 years, 1,256 trading days/symbol): QQQ
# +138.5 % total return / +19.1 % annualised / Sharpe 0.66 leads;
# TLT −28.6 % / −6.5 % / Sharpe −0.47 trails. EEM essentially flat
# (+0.85 % total, Sharpe −0.08). The 5-year window covers a
# bond-bear / equity-bull regime — useful for the rotation case study
# to learn that "diversification" requires non-equity diversifiers
# beyond duration alone.
# - **WikiPrices coverage**: ~3,200 historical symbols (1962-2018)
# including delisted names — required for any backtest that wants to
# avoid survivorship bias on legacy data.
# - **Yahoo↔WikiPrices comparison (AAPL 2017)**: zero exact matches — all
# 249 overlapping days differ by ~77% because Yahoo's auto-adjusted close
# reflects Apple's 2020 4:1 split while WikiPrices ends in 2018 and predates
# it. The helper surfaces these price-difference statistics so adjustment-basis
# mismatches are caught row-by-row rather than glossed.
# - **AAPL stitched history**: WikiPrices + Yahoo joins to ~8,800 daily
# rows (1990-now) without duplication after the 2018-03-27 cutoff.
# - **FRED**: VIXCLS and DGS10 fetched in the same call signature as
# the equity providers — the unified API generalises cleanly to
# macro overlays.
#
# ### Implications for Practitioners
# - **One signature, many sources**: The unified `fetch_ohlcv()` makes
# provider swaps cheap. Treat the provider as configuration, not as
# bespoke per-source code.
# - **Fallback over fail-fast** for production data acquisition: prefer
# the next provider over a missing day, but log the source used so
# the audit trail is intact.
# - **Comparison is the contract**: never trust two providers blindly —
# run a row-by-row diff on overlapping windows and treat any
# adjustment-method mismatch as a bug, not a feature.
#
# **Next**: `17_complete_pipeline` consumes this universe end-to-end
# (ingestion → quality gate → storage → query); `18_data_management`
# adds the `DataManager`/`Universe`/`HiveStorage` layer for production.
# %%
# Close provider sessions
yahoo.close()
wiki.close()
print(f" - Total rows: {len(etf_universe_df):,}")
02_financial_data_universe/17_complete_pipeline.ipynb
+2105
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/17_complete_pipeline.py
+833
View File
@@ -0,0 +1,833 @@
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Complete Data Pipeline: Multi-Source Acquisition to ML-Ready Data
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Wire together the chapter's components — multi-provider acquisition,
# OHLC validation, source attribution, and efficient storage — into two
# end-to-end pipelines: an equity pipeline that stitches WikiPrices
# (1990-2018) with Yahoo (2018-now), and a crypto pipeline that consumes
# the local Binance hourly perpetuals panel. The notebook then shows how
# `ml4t.data.DataManager` collapses the same logic into a few lines.
#
# ## Learning Objectives
# - Build a multi-source equity pipeline and validate at every boundary.
# - Detect 24/7 coverage gaps and add session features for crypto data.
# - Tag every row with its source so multi-source data is auditable.
# - Replace the explicit pipeline with the `DataManager` + `Universe` +
# `HiveStorage` higher-level API that ships with `ml4t.data`.
#
# ## Book reference
# Chapter 2, §2.3 (multi-source stitching + production pipelines). The
# pipeline outputs feed the ETF and crypto-perps case studies.
#
# ## Prerequisites
# - WikiPrices parquet under
# `ML4T_DATA_PATH/equities/market/us_equities/us_equities.parquet`.
# - Crypto-perps hourly parquet loadable via `data.load_crypto_perps`.
# - Live YahooFinance access (the equity pipeline calls it for the
# 2018-now leg — same convention as `16_provider_comparison`).
#
# ## Note on universe choice
# WikiPrices covers individual US stocks (~3,200 tickers, 1962-2018) but
# **not** index ETFs. For the equity pipeline below the demo universe
# uses three large-caps (AAPL, MSFT, JPM) so the WikiPrices-Yahoo stitch
# is non-trivial; the ETF rotation case study itself uses Yahoo-only
# from 2008 onwards (see `case_studies/etfs/`).
# %%
"""Complete Data Pipeline — Multi-source acquisition to ML-ready data."""
from datetime import datetime
from pathlib import Path
from typing import Any
import matplotlib.pyplot as plt
import polars as pl
from ml4t.data.providers import WikiPricesProvider, YahooFinanceProvider
from data import load_crypto_perps
from utils import DATA_DIR
from utils.paths import get_output_dir
# Reproducibility: a fixed AS_OF_DATE keeps outputs stable between book editions.
AS_OF_DATE = "2025-01-15"
print(f"Data path: {DATA_DIR}")
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ---
#
# ## 1. Pipeline Architecture
#
# A production pipeline has three stages:
#
# | Stage | Inputs | Outputs |
# |-------|--------|---------|
# | **Acquire** | Yahoo Finance, WikiPrices, Binance | raw OHLCV per source |
# | **Process** | raw OHLCV | validated, normalized, tagged frames |
# | **Store** | tagged frames | partitioned Parquet keyed by symbol/date |
#
# Four design principles drive the rest of the notebook:
#
# 1. **Provider independence** — the orchestrator shouldn't care which
# source each row came from.
# 2. **Validate at boundaries** — every entry/exit point checks OHLC
# invariants, gap thresholds, and duplicate timestamps.
# 3. **Source attribution** — every row carries a `source` column so
# downstream debugging stays sane.
# 4. **Incremental updates** — re-runs fetch only what is new (the
# `DataManager` API in §5 handles this transparently).
# %% [markdown]
# ---
#
# ## 2. Equity Pipeline (Stocks: AAPL / MSFT / JPM)
#
# ### Requirements
#
# - **Universe**: AAPL, MSFT, JPM (chosen because they have full
# WikiPrices coverage; pure ETFs like SPY only have Yahoo data here).
# - **History**: 1990-present (~35 years).
# - **Frequency**: Daily.
# - **Adjustments**: Split + dividend (total return).
# %% [markdown]
# ### Combine Multi-Source Data
#
# Wiki Prices ends at 2018-03-27; Yahoo Finance continues from there.
# This function stitches the two sources at the boundary, tagging each
# row with its origin for downstream auditability.
# %%
def _combine_pipeline_sources(
historical: pl.DataFrame | None,
recent: pl.DataFrame | None,
symbol: str,
wiki_end: str = "2018-03-27",
) -> pl.DataFrame:
"""Combine Wiki Prices and Yahoo Finance data at the provider boundary."""
if historical is None and recent is None:
raise ValueError(f"No data available for {symbol}")
if historical is None:
return recent.with_columns(pl.lit("yahoo").alias("source"))
if recent is None:
return historical.with_columns(pl.lit("wiki").alias("source"))
# Both sources - combine with proper boundary
wiki_end_dt = datetime.strptime(wiki_end, "%Y-%m-%d").date()
historical = historical.filter(pl.col("timestamp").dt.date() <= wiki_end_dt)
recent = recent.filter(pl.col("timestamp").dt.date() > wiki_end_dt)
# Back-adjust the historical Wiki series for any splits that occurred AFTER
# the Wiki cutoff (e.g., AAPL 4-for-1 on 2020-08-31). Wiki's adjusted prices
# only back-adjust splits available within its own coverage window; Yahoo's
# adjusted prices retroactively account for every later split. Without this
# rescale, the stitch produces a visible step at the source boundary. We
# rescale historical OHLC by the ratio of (Yahoo first close) / (Wiki last
# close); both series are already dividend-adjusted, so this isolates the
# post-boundary split factor. Volume scales inversely to price (a 4-for-1
# split divides price by 4 and multiplies volume by 4), so the historical
# volume column is divided by the same factor to stay consistent with
# Yahoo's retroactively-split-adjusted volume on the recent side.
if len(historical) > 0 and len(recent) > 0:
wiki_last_close = historical.sort("timestamp")["close"][-1]
yahoo_first_close = recent.sort("timestamp")["close"][0]
if wiki_last_close and wiki_last_close > 0 and yahoo_first_close and yahoo_first_close > 0:
scale = yahoo_first_close / wiki_last_close
historical = historical.with_columns(
[
(pl.col("open") * scale).alias("open"),
(pl.col("high") * scale).alias("high"),
(pl.col("low") * scale).alias("low"),
(pl.col("close") * scale).alias("close"),
(pl.col("volume") / scale).alias("volume"),
]
)
historical = historical.with_columns(pl.lit("wiki").alias("source"))
recent = recent.with_columns(pl.lit("yahoo").alias("source"))
common_cols = list(set(historical.columns) & set(recent.columns))
combined = pl.concat([historical.select(common_cols), recent.select(common_cols)]).sort(
"timestamp"
)
return combined
# %% [markdown]
# ### Validate Pipeline Data
#
# Check OHLC invariants, detect zero/negative prices, flag date gaps
# beyond five calendar days, and catch duplicate timestamps.
# %%
def _validate_pipeline_data(df: pl.DataFrame, symbol: str) -> dict[str, Any]:
"""Run data quality validation checks on OHLCV data."""
issues = []
# OHLC invariants
ohlc_valid = (
(df["high"] >= df["low"]).all()
and (df["high"] >= df["open"]).all()
and (df["high"] >= df["close"]).all()
and (df["low"] <= df["open"]).all()
and (df["low"] <= df["close"]).all()
)
if not ohlc_valid:
issues.append("OHLC invariant violations")
# Zero/negative prices
if (df["close"] <= 0).any():
issues.append("Zero or negative prices")
# Date gaps
dates = df["timestamp"].sort().to_list()
max_gap_days = max(
(dates[i] - dates[i - 1]).days for i in range(1, len(dates)) if len(dates) > 1
)
if max_gap_days > 5:
issues.append(f"Date gap of {max_gap_days} days")
# Duplicate dates
if df["timestamp"].n_unique() != len(df):
issues.append(f"Duplicate dates: {len(df) - df['timestamp'].n_unique()}")
return {
"symbol": symbol,
"n_rows": len(df),
"date_range": (df["timestamp"].min(), df["timestamp"].max()),
"max_gap_days": max_gap_days if len(dates) > 1 else 0,
"issues": issues,
"is_valid": len(issues) == 0,
}
# %% [markdown]
# ### Fetch Historical Data from Wiki Prices
#
# Reads adjusted OHLCV from the Wiki Prices parquet, adapting to whichever
# column-naming schema the on-disk file uses (`symbol`/`asset`/`ticker`).
# %%
def _fetch_wiki_historical(
wiki_path: Path, symbol: str, start_date: str, end_date: str
) -> pl.DataFrame | None:
"""Fetch historical data from Wiki Prices parquet with schema auto-detection."""
schema = set(pl.scan_parquet(wiki_path).collect_schema().names())
start_dt = datetime.strptime(start_date, "%Y-%m-%d").date()
end_dt = datetime.strptime(end_date, "%Y-%m-%d").date()
# Determine entity and time column names from on-disk schema
if "symbol" in schema:
entity_col, time_col = "symbol", "timestamp"
elif "asset" in schema:
entity_col, time_col = "asset", "timestamp"
else:
entity_col, time_col = "ticker", "date"
df = (
pl.scan_parquet(wiki_path)
.filter(
(pl.col(entity_col) == symbol)
& (pl.col(time_col) >= pl.lit(start_dt))
& (pl.col(time_col) <= pl.lit(end_dt))
)
.select(
[
pl.col(time_col).cast(pl.Datetime("us")).alias("timestamp"),
pl.col("adj_open").alias("open"),
pl.col("adj_high").alias("high"),
pl.col("adj_low").alias("low"),
pl.col("adj_close").alias("close"),
pl.col("adj_volume").alias("volume"),
]
)
.sort("timestamp")
.collect()
)
return df if len(df) > 0 else None
# %% [markdown]
# ### Process a Single ETF Symbol
#
# Drives one symbol through fetch, combine, validate, and tag. Returns
# a result dict with the data and validation report.
# %%
def _process_etf_symbol(
pipeline: "ETFMomentumPipeline", symbol: str, start_date: str = "1990-01-01"
) -> dict[str, Any]:
"""Process a single symbol through the complete pipeline."""
print(f"\n{symbol}:")
# Stage 1: Fetch
historical = pipeline.fetch_historical(symbol, start_date)
recent = pipeline.fetch_recent(symbol, "2018-03-28")
wiki_rows = len(historical) if historical is not None else 0
yahoo_rows = len(recent) if recent is not None else 0
print(f" Fetched: Wiki={wiki_rows:,}, Yahoo={yahoo_rows:,}")
# Stage 2: Combine
try:
combined = pipeline.combine_sources(historical, recent, symbol)
except ValueError as e:
return {"symbol": symbol, "error": str(e)}
# Stage 3: Validate
validation = _validate_pipeline_data(combined, symbol)
if validation["issues"]:
for issue in validation["issues"]:
print(f" [WARN] {issue}")
pipeline.stats["validation_errors"] += len(validation["issues"])
else:
print(f" Validated: {len(combined):,} rows, no issues")
# Stage 4: Add metadata
combined = combined.with_columns(
[pl.lit(symbol).alias("symbol"), pl.lit(datetime.now()).alias("processed_at")]
)
pipeline.stats["symbols_processed"] += 1
pipeline.stats["total_rows"] += len(combined)
return {"symbol": symbol, "data": combined, "validation": validation}
# %% [markdown]
# ### Run ETF Pipeline
#
# Iterates over the symbol universe, processing each through fetch/combine/validate,
# then prints a summary of processed symbols, total rows, and validation issues.
# %%
def _run_etf_pipeline(
pipeline: "ETFMomentumPipeline", symbols: list[str] | None = None
) -> dict[str, Any]:
"""Run the complete ETF pipeline for all symbols."""
symbols = symbols or pipeline.UNIVERSE
print("=" * 60)
print("ETF Momentum Pipeline")
print("=" * 60)
print(f"Universe: {symbols}")
results = {}
for symbol in symbols:
results[symbol] = pipeline.process_symbol(symbol)
print("\n" + "-" * 60)
print(
f"Summary: {pipeline.stats['symbols_processed']} symbols, "
f"{pipeline.stats['total_rows']:,} rows, "
f"{pipeline.stats['validation_errors']} issues"
)
return results
# %% [markdown]
# ### ETF Momentum Pipeline
#
# Orchestrates multi-source fetch (Wiki Prices + Yahoo Finance),
# data combination, validation, and metadata tagging for each symbol.
# %%
class ETFMomentumPipeline:
"""Equity pipeline: combines WikiPrices (1962-2018) with Yahoo Finance (2018-present).
Demo universe uses individual stocks because WikiPrices does not include
index ETFs. The book's ETF rotation case study uses Yahoo-only data from
2008 onwards (see `case_studies/etfs/`).
"""
UNIVERSE = ["AAPL", "MSFT", "JPM"]
WIKI_END_DATE = "2018-03-27"
END_DATE = AS_OF_DATE
def __init__(self, wiki_path: Path | None = None, storage_path: Path | None = None):
"""Initialize pipeline with data sources."""
from utils import ML4T_DATA_PATH
self.wiki_path = (
wiki_path
or ML4T_DATA_PATH / "equities" / "market" / "us_equities" / "us_equities.parquet"
)
self.storage_path = storage_path or get_output_dir(2, "etf_pipeline")
self.yahoo_provider = YahooFinanceProvider()
self.stats = {"symbols_processed": 0, "total_rows": 0, "validation_errors": 0}
def fetch_historical(self, symbol: str, start_date: str) -> pl.DataFrame | None:
"""Fetch data from Wiki Prices (pre-2018)."""
if not self.wiki_path.exists():
raise FileNotFoundError(f"Wiki Prices parquet not found at {self.wiki_path}")
return _fetch_wiki_historical(self.wiki_path, symbol, start_date, self.WIKI_END_DATE)
def fetch_recent(self, symbol: str, start_date: str) -> pl.DataFrame | None:
"""Fetch data from Yahoo Finance (2018-present)."""
df = self.yahoo_provider.fetch_ohlcv(
symbol=symbol, start=start_date, end=self.END_DATE, frequency="1d"
)
return df if len(df) > 0 else None
def combine_sources(
self, historical: pl.DataFrame | None, recent: pl.DataFrame | None, symbol: str
) -> pl.DataFrame:
"""Combine Wiki Prices and Yahoo Finance data."""
return _combine_pipeline_sources(historical, recent, symbol, self.WIKI_END_DATE)
def process_symbol(self, symbol: str, start_date: str = "1990-01-01") -> dict[str, Any]:
"""Process a single symbol through the complete pipeline."""
return _process_etf_symbol(self, symbol, start_date)
def run(self, symbols: list[str] | None = None) -> dict[str, Any]:
"""Run the complete pipeline for all symbols."""
return _run_etf_pipeline(self, symbols)
# %%
# Run the equity pipeline on the demo universe
etf_pipeline = ETFMomentumPipeline()
etf_results = etf_pipeline.run()
# %%
# Visualize the stitched data — one panel per symbol
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
for ax, (symbol, result) in zip(axes, etf_results.items(), strict=True):
df = result["data"]
for source, color in [("wiki", "#1f77b4"), ("yahoo", "#ff7f0e")]:
src = df.filter(pl.col("source") == source)
label = "Wiki Prices" if source == "wiki" else "Yahoo Finance"
if len(src) > 0:
ax.plot(
src["timestamp"].to_list(),
src["close"].to_list(),
label=label,
alpha=0.8,
linewidth=1,
color=color,
)
ax.axvline(
datetime(2018, 3, 27), color="red", linestyle="--", alpha=0.5, label="Source transition"
)
ax.set_title(f"{symbol}: stitched WikiPrices + Yahoo")
ax.set_xlabel("Date")
ax.set_ylabel("Close ($)")
ax.legend(loc="upper left")
ax.set_yscale("log")
plt.tight_layout()
plt.show()
# %% [markdown]
# ---
#
# ## 3. Case Study 2: Crypto Funding Rate Pipeline
#
# ### Requirements
#
# - **Universe**: BTCUSDT, ETHUSDT (perpetual futures)
# - **History**: 2020-present
# - **Frequency**: Hourly OHLCV
# - **Coverage**: 24/7 (no market hours)
# %% [markdown]
# ### Validate 24/7 Coverage
#
# Crypto markets run continuously, so we check for hourly gaps rather than
# the weekday/holiday gaps expected in equity data.
# %%
def _validate_crypto_coverage(df: pl.DataFrame, symbol: str) -> dict[str, Any]:
"""Validate 24/7 coverage for crypto data."""
issues = []
df = df.sort("timestamp")
timestamps = df["timestamp"].to_list()
expected_hours = len(timestamps) - 1 if len(timestamps) > 1 else 0
missing_hours = 0
for i in range(1, len(timestamps)):
gap_hours = (timestamps[i] - timestamps[i - 1]).total_seconds() / 3600
if gap_hours > 1.5:
missing_hours += int(gap_hours) - 1
coverage_pct = (
(expected_hours - missing_hours) / expected_hours * 100 if expected_hours > 0 else 0
)
if coverage_pct < 99.0:
issues.append(f"Coverage {coverage_pct:.1f}% ({missing_hours} hours missing)")
if df["timestamp"].n_unique() != len(df):
issues.append(f"Duplicate timestamps: {len(df) - df['timestamp'].n_unique()}")
return {
"symbol": symbol,
"n_rows": len(df),
"hours_coverage": expected_hours,
"missing_hours": missing_hours,
"coverage_pct": coverage_pct,
"issues": issues,
"is_valid": len(issues) == 0,
}
# %% [markdown]
# ### Add Session Features
#
# Derive time-of-day and day-of-week columns used downstream for
# funding-window analysis and weekend/weekday volume comparisons.
# %%
def _add_crypto_session_features(df: pl.DataFrame) -> pl.DataFrame:
"""Add time-based features for crypto data."""
return df.with_columns(
[
pl.col("timestamp").dt.hour().alias("hour_utc"),
pl.col("timestamp").dt.weekday().alias("day_of_week"),
(pl.col("timestamp").dt.hour() // 8 * 8).alias("funding_window"),
pl.col("timestamp").dt.date().alias("session_date"),
(pl.col("timestamp").dt.weekday() >= 5).alias("is_weekend"),
]
)
# %% [markdown]
# ### Process a Single Crypto Symbol
#
# Drives one symbol through fetch, coverage validation, feature enrichment,
# and source tagging. Returns a result dict with data and validation report.
# %%
def _process_crypto_symbol(
pipeline: "CryptoFundingRatePipeline", symbol: str, start_date: str = "2020-01-01"
) -> dict[str, Any]:
"""Process a single crypto symbol through the pipeline."""
print(f"\n{symbol}:")
df = pipeline.fetch_ohlcv(symbol, start_date)
validation = _validate_crypto_coverage(df, symbol)
print(f" Rows: {len(df):,}, Coverage: {validation['coverage_pct']:.1f}%")
if validation["issues"]:
for issue in validation["issues"]:
print(f" [WARN] {issue}")
df = _add_crypto_session_features(df)
df = df.with_columns([pl.lit(symbol).alias("symbol"), pl.lit("local").alias("source")])
pipeline.stats["symbols_processed"] += 1
pipeline.stats["total_rows"] += len(df)
pipeline.stats["hours_of_data"] += validation["hours_coverage"]
return {"symbol": symbol, "data": df, "validation": validation}
# %% [markdown]
# ### Fetch Crypto OHLCV
#
# Filters the pre-loaded hourly crypto DataFrame for a single symbol
# starting from the requested date.
# %%
def _fetch_crypto_ohlcv(crypto_data: pl.DataFrame, symbol: str, start_date: str) -> pl.DataFrame:
"""Filter the hourly crypto panel for one symbol from `start_date`."""
start_dt = datetime.strptime(start_date, "%Y-%m-%d")
df = crypto_data.filter(
(pl.col("symbol") == symbol) & (pl.col("timestamp").dt.date() >= start_dt.date())
).drop("symbol")
if df.is_empty():
raise RuntimeError(f"No crypto data for {symbol} from {start_date}")
return df
# %% [markdown]
# ### Run Crypto Pipeline
#
# Iterates over the crypto symbol universe, processing each through fetch,
# validation, and feature enrichment, then prints a summary.
# %%
def _run_crypto_pipeline(
pipeline: "CryptoFundingRatePipeline",
symbols: list[str] | None = None,
start_date: str = "2023-01-01",
) -> dict:
"""Run the complete crypto pipeline for all symbols."""
symbols = symbols or pipeline.UNIVERSE
print("=" * 60)
print("Crypto Funding Rate Pipeline")
print("=" * 60)
print(f"Universe: {symbols}")
results = {}
for symbol in symbols:
results[symbol] = pipeline.process_symbol(symbol, start_date)
print("\n" + "-" * 60)
print(
f"Summary: {pipeline.stats['symbols_processed']} symbols, "
f"{pipeline.stats['total_rows']:,} rows"
)
return results
# %% [markdown]
# ### Crypto Funding Rate Pipeline
#
# Orchestrates fetch, validation, and feature enrichment for hourly crypto
# perpetual-futures data loaded from local parquet storage.
# %%
class CryptoFundingRatePipeline:
"""
Data pipeline for Crypto Funding Rate strategy.
Uses pre-downloaded hourly crypto data from local storage.
"""
UNIVERSE = ["BTCUSDT", "ETHUSDT", "SOLUSDT", "BNBUSDT"]
END_DATE = AS_OF_DATE
def __init__(self, storage_path: Path | None = None):
"""Initialize pipeline."""
self.storage_path = storage_path or get_output_dir(2, "crypto_pipeline")
self.crypto_data = load_crypto_perps(frequency="1h")
print(f"Loaded {len(self.crypto_data):,} crypto records")
self.stats = {"symbols_processed": 0, "total_rows": 0, "hours_of_data": 0}
def fetch_ohlcv(self, symbol: str, start_date: str) -> pl.DataFrame | None:
"""Fetch OHLCV data from local parquet."""
return _fetch_crypto_ohlcv(self.crypto_data, symbol, start_date)
def process_symbol(self, symbol: str, start_date: str = "2020-01-01") -> dict[str, Any]:
"""Process a single crypto symbol."""
return _process_crypto_symbol(self, symbol, start_date)
def run(self, symbols: list[str] | None = None, start_date: str = "2023-01-01") -> dict:
"""Run the complete pipeline."""
return _run_crypto_pipeline(self, symbols, start_date)
# %%
# Run the crypto pipeline
crypto_pipeline = CryptoFundingRatePipeline()
crypto_results = crypto_pipeline.run(symbols=["BTCUSDT", "ETHUSDT"], start_date="2024-01-01")
# %%
# Visualize crypto data patterns: price series + hourly/daily volume profile
btc_data = crypto_results["BTCUSDT"]["data"]
fig, axes = plt.subplots(1, 3, figsize=(16, 5))
# Panel 1: last 30 days of close
recent_data = btc_data.tail(24 * 30)
axes[0].plot(
recent_data["timestamp"].to_list(),
recent_data["close"].to_list(),
linewidth=0.8,
alpha=0.8,
color="#2E86AB",
)
axes[0].set_ylabel("Close ($)")
axes[0].set_title("BTCUSDT: last 30 days")
axes[0].tick_params(axis="x", rotation=45)
# Panel 2: average volume by hour, with funding windows highlighted
hourly_volume = (
btc_data.group_by("hour_utc").agg(pl.col("volume").mean().alias("avg_volume")).sort("hour_utc")
)
axes[1].bar(
hourly_volume["hour_utc"].to_list(),
hourly_volume["avg_volume"].to_list(),
color="#2E86AB",
alpha=0.7,
)
for funding_hour in [0, 8, 16]:
axes[1].axvline(funding_hour, color="red", linestyle="--", alpha=0.5)
axes[1].set_ylabel("Average volume")
axes[1].set_xlabel("Hour (UTC)")
axes[1].set_title("Volume by hour (funding windows in red)")
# Panel 3: average volume by weekday (weekend in red)
daily_volume = (
btc_data.group_by("day_of_week")
.agg(pl.col("volume").mean().alias("avg_volume"))
.sort("day_of_week")
)
days = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
colors = ["#2E86AB"] * 5 + ["#E94F37"] * 2
axes[2].bar(range(7), daily_volume["avg_volume"].to_list(), color=colors, alpha=0.7)
axes[2].set_xticks(range(7))
axes[2].set_xticklabels(days)
axes[2].set_ylabel("Average volume")
axes[2].set_title("Volume by day (weekend in red)")
plt.tight_layout()
plt.show()
# %% [markdown]
# ---
#
# ## 4. Storage Best Practices
#
# ### Partitioned Parquet Storage
#
# For large datasets, partition by time and symbol:
#
# ```
# output/
# └── etf_momentum/
# ├── year=2024/
# │ ├── SPY.parquet
# │ ├── QQQ.parquet
# │ └── TLT.parquet
# └── year=2023/
# └── ...
# ```
#
# **Benefits**:
# - **Partition pruning**: Only read needed date ranges
# - **Incremental writes**: Add new data without rewriting
# - **Parallel reads**: Process multiple partitions concurrently
# %%
# Save pipeline outputs (example)
print("Storage Example:")
print(f" ETF output: {etf_pipeline.storage_path}")
print(f" Crypto output: {crypto_pipeline.storage_path}")
# %% [markdown]
# ---
#
# ## 5. Simplifying with DataManager
#
# The pipelines above show every step explicitly. In practice, ml4t-data's
# `DataManager` and `Universe` classes handle most of this boilerplate.
# Here's how the ETF pipeline looks using the higher-level API:
# %%
from ml4t.data import DataManager
from ml4t.data.storage import HiveStorage
from ml4t.data.storage.backend import StorageConfig
from ml4t.data.universe import Universe
from ml4t.data.update_manager import GapDetector
from ml4t.data.validation import OHLCVValidator
# One-line universe instead of hardcoded list
Universe.add_custom("etf_momentum", ["SPY", "QQQ", "IWM", "TLT", "GLD"])
print(f"Universe: {Universe.get('etf_momentum')}")
# DataManager with storage — fetch, store, and validate in one step
dm_config = StorageConfig(base_path=get_output_dir(2, "dm_pipeline"), compression="zstd")
dm_storage = HiveStorage(config=dm_config)
dm = DataManager(storage=dm_storage, enable_validation=True)
# Fetch and store all symbols
for symbol in Universe.get("etf_momentum"):
key = dm.load(symbol, "2024-01-01", AS_OF_DATE, provider="yahoo")
meta = dm.get_metadata(symbol)
rows = meta.get("row_count", "?") if meta else "?"
print(f" {symbol}: {rows} rows → {key}")
# %%
# Validate and check gaps — compare to the manual validate_data() above
validator = OHLCVValidator(max_return_threshold=0.5)
gap_detector = GapDetector(exclude_weekends=True)
for symbol in Universe.get("etf_momentum"):
key = f"equities/daily/{symbol}"
if dm_storage.exists(key):
df = dm_storage.read(key).collect()
result = validator.validate(df)
gaps = gap_detector.detect_gaps(df, frequency="daily")
status = "OK" if result.passed else f"{result.error_count} issues"
gap_status = f"{len(gaps)} gaps" if gaps else "complete"
print(f" {symbol}: {len(df)} rows, {status}, {gap_status}")
# %% [markdown]
# The DataManager version replaces ~100 lines of manual provider instantiation,
# data combination, and validation with a few lines. Under the hood it uses
# the same providers and validation — see notebooks `17_data_management` and
# `18_incremental_updates` for the full walkthrough.
# %% [markdown]
# ## Key Takeaways
#
# Two end-to-end pipelines wired through the validation/storage stack
# from chapter §2.3.
#
# ### Quantitative Findings
# - **Equity stitch (AAPL/MSFT/JPM, 1990-now)**: each symbol's stitched
# panel is ~8,800 daily rows — 7,113-7,114 from WikiPrices
# (1990 → 2018-03-27) + 1,711 from Yahoo (2018-03-28 → 2025-01-15),
# for 26,474 total rows across the three symbols. The validator
# flags 3 issues — one >5-day gap per symbol corresponding to the
# 2001-09-10 → 09-17 NYSE closure following 9/11 — which is the
# expected behaviour for the calendar-day gap heuristic on
# pre-2018 data.
# - **Equity-pipeline ETF caveat**: the *case study* universe
# (SPY/QQQ/IWM/TLT/GLD) is index ETFs, which are **not** in WikiPrices.
# Running the same pipeline on those tickers produces 0 WikiPrices
# rows + Yahoo-only data starting 2018-03-28 — the multi-source
# stitch is real for stocks, vacuous for ETFs.
# - **Crypto pipeline (BTCUSDT, ETHUSDT from 2024-01-01)**: 17,544 hours
# per symbol, 100 % coverage of the expected 24/7 window, zero
# duplicate timestamps.
# - **DataManager equivalent** (§5): the 5-symbol Yahoo-only fetch +
# HiveStorage write + OHLCVValidator + GapDetector loop reproduces
# the manual pipeline above in ~20 lines instead of ~150.
#
# ### Implications for Practitioners
# - **Tag the source**: a `source` column makes provider mismatches
# debuggable and migrations safe.
# - **Match validators to cadence**: equity validators reject >5-day
# gaps as anomalies; crypto validators expect every hour and reject
# any >1.5 h gap.
# - **Promote to DataManager once the pipeline stabilises**: explicit
# class-based pipelines are great for teaching, but the production
# path is the higher-level API (storage + validation + universe
# built in).
#
# **Next**: `18_data_management` walks the `DataManager`, `Universe`,
# and `HiveStorage` API in depth; `19_incremental_updates` shows the
# gap-detection + update-strategy patterns that make daily refreshes
# cheap.
02_financial_data_universe/18_data_management.ipynb
+1531
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/18_data_management.py
+526
View File
@@ -0,0 +1,526 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Data Management: From Download to Production Pipeline
#
# **Docker image**: `ml4t`
#
# **Chapter 2: The Financial Data Universe**
#
# Previous notebooks fetched and validated data. This notebook shows how to
# **manage** it at scale using ml4t-data's production features:
#
# - **DataManager**: Unified entry point for fetching, storing, and updating
# - **Universe**: Predefined symbol lists (S&P 500, NASDAQ 100, etc.)
# - **HiveStorage**: Partitioned Parquet for fast queries and incremental writes
# - **Incremental Updates**: Keep data fresh without re-downloading history
# - **CLI**: Command-line interface for scripted workflows
#
# ## Learning Objectives
#
# By completing this notebook, you will:
# 1. Use `DataManager` as a single entry point for all data operations
# 2. Load predefined universes with the `Universe` class
# 3. Store and query data with Hive-partitioned Parquet
# 4. Perform incremental updates and detect gaps
# 5. Use the `ml4t-data` CLI for scripted workflows
#
# ## Why This Matters
#
# A one-time download is fine for a tutorial. A trading system needs:
# - **Daily updates** that only fetch new data (10x faster than full refresh)
# - **Partitioned storage** that supports fast date-range queries
# - **Gap detection** to ensure completeness before backtesting
# - **Metadata tracking** so you know what you have and when it was updated
#
# > **ml4t-data docs**: See the [Incremental Updates Guide](https://ml4trading.io/docs/data/user-guide/incremental-updates/)
# > and [Storage Guide](https://ml4trading.io/docs/data/user-guide/storage/) for full reference.
#
# **Prerequisites**: ml4t-data installed; live network access for Yahoo Finance.
# %% [markdown]
# ## Setup
# %%
"""Data Management — DataManager, Universe, HiveStorage, and incremental updates."""
import logging
import shutil
from datetime import datetime
from pathlib import Path
import polars as pl
import structlog
# ml4t-data emits structured debug logs on every fetch/store; route them
# through stdlib logging at WARNING so the notebook output stays focused on
# the demonstration.
structlog.configure(
wrapper_class=structlog.make_filtering_bound_logger(logging.WARNING),
)
# ml4t-data core imports
from ml4t.data import DataManager
from ml4t.data.storage import HiveStorage
from ml4t.data.storage.backend import StorageConfig
from ml4t.data.universe import Universe
from utils.paths import get_output_dir
# Working directory for this notebook's storage examples. Wipe any artifacts
# from a previous run so the demo is fully reproducible.
STORAGE_DIR = get_output_dir(2, "data_management")
if STORAGE_DIR.exists():
shutil.rmtree(STORAGE_DIR)
STORAGE_DIR.mkdir(parents=True, exist_ok=True)
print(f"Storage directory: {STORAGE_DIR}")
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ---
#
# ## 1. DataManager: The Unified Entry Point
#
# `DataManager` abstracts away provider selection, storage, and updates
# behind a single interface. Compare:
#
# ```python
# # Without DataManager (manual)
# provider = YahooFinanceProvider()
# df = provider.fetch_ohlcv("AAPL", "2024-01-01", "2024-12-31", "daily")
#
# # With DataManager (unified)
# dm = DataManager()
# df = dm.fetch("AAPL", "2024-01-01", "2024-12-31")
# ```
#
# The real power shows with batch operations, storage integration, and updates.
# %% [markdown]
# ### Fetch: Single Symbol
# %%
# DataManager without storage — pure fetch mode
dm = DataManager()
# Fetch a single symbol (defaults to Yahoo Finance for equities)
aapl = dm.fetch("AAPL", "2024-01-01", "2024-12-31", provider="yahoo")
print(f"AAPL: {aapl.shape[0]} rows, {aapl.shape[1]} columns")
print(f"Date range: {aapl['timestamp'].min().date()} to {aapl['timestamp'].max().date()}")
print(f"Columns: {aapl.columns}")
aapl.head(3)
# %% [markdown]
# ### Batch Fetch: Multiple Symbols
#
# `batch_load` fetches multiple symbols in parallel and returns a single
# stacked DataFrame — the standard multi-asset format used throughout the book.
# %%
# Fetch 5 ETFs in parallel
etf_symbols = ["SPY", "QQQ", "IWM", "TLT", "GLD"]
etf_data = dm.batch_load(
symbols=etf_symbols,
start="2024-01-01",
end="2024-12-31",
provider="yahoo",
max_workers=4,
)
print(f"Combined: {etf_data.shape[0]:,} rows across {etf_data['symbol'].n_unique()} symbols")
etf_data.group_by("symbol").len().sort("symbol")
# %% [markdown]
# ---
#
# ## 2. Universe: Predefined Symbol Lists
#
# Instead of maintaining symbol lists in YAML or hardcoding them, ml4t-data
# ships curated universes that stay current with index rebalances.
# %%
# List available universes
print("Available universes:")
for name in Universe.list_universes():
symbols = Universe.get(name)
print(f" {name}: {len(symbols)} symbols")
# %%
# Access a universe directly
sp500 = Universe.SP500
print(f"\nS&P 500: {len(sp500)} symbols")
print(f"First 10: {sp500[:10]}")
print(f"Last 10: {sp500[-10:]}")
# %%
# Use with DataManager.batch_load_universe for one-line loading
# (fetches all 503 S&P 500 symbols — use a smaller slice for demo)
sp500_sample = dm.batch_load(
symbols=sp500[:5],
start="2024-06-01",
end="2024-12-31",
provider="yahoo",
)
print(
f"S&P 500 sample: {sp500_sample.shape[0]:,} rows, {sp500_sample['symbol'].n_unique()} symbols"
)
# %%
# Custom universes for your strategy
Universe.add_custom("etf_momentum", ["SPY", "QQQ", "IWM", "EFA", "EEM", "TLT", "GLD"])
Universe.add_custom("crypto_arb", ["BTCUSDT", "ETHUSDT", "SOLUSDT", "BNBUSDT"])
print("\nCustom universes registered:")
for name in ["etf_momentum", "crypto_arb"]:
print(f" {name}: {Universe.get(name)}")
# %% [markdown]
# ---
#
# ## 3. HiveStorage: Partitioned Parquet
#
# For data you'll query repeatedly, Hive-partitioned Parquet is the storage
# layer used throughout ml4t-data. The HiveStorage backend collapses the
# logical key `equities/daily/AAPL` to a filesystem-safe directory name and
# nests Hive-style year/month partitions underneath:
#
# ```
# hive_demo/
# ├── .metadata/
# │ └── equities_daily_AAPL.json
# └── equities_daily_AAPL/
# ├── year=2024/month=1/data.parquet
# ├── year=2024/month=2/data.parquet
# └── ...
# ```
#
# **Benefits over flat files**:
# - **Partition pruning**: Query "last 30 days" reads 1 file, not all of history
# - **Incremental writes**: New data appends without rewriting existing partitions
# - **Metadata tracking**: Know when each symbol was last updated
# %% [markdown]
# ### DataManager with Storage
# %%
# Initialize storage
storage_config = StorageConfig(
base_path=STORAGE_DIR / "hive_demo",
compression="zstd",
partition_granularity="month",
)
storage = HiveStorage(config=storage_config)
# DataManager with storage — enables load/update/metadata operations
dm_stored = DataManager(storage=storage)
# %% [markdown]
# ### Load and Store
#
# `DataManager.load()` fetches from the provider and writes to Hive
# partitions in one call. The storage key encodes the asset class, frequency,
# and symbol.
# %%
symbols = ["AAPL", "MSFT", "GOOGL"]
stored_keys = {}
for symbol in symbols:
key = dm_stored.load(symbol, "2023-01-01", "2024-12-31", provider="yahoo")
stored_keys[symbol] = key
print(f" Stored {symbol} → key: {key}")
# %% [markdown]
# ### Query Stored Data
# %%
# List what's in storage. `storage.list_keys()` walks the on-disk layout, so it
# reports every symbol regardless of metadata-file contents.
stored_symbols = sorted(storage.list_keys())
print(f"Symbols in storage: {stored_symbols}")
# Read back with a date-range filter — only the matching month=k partitions
# touch disk, so reading 2024 from a 2-year archive halves the I/O.
aapl_2024 = storage.read(
stored_keys["AAPL"],
start_date=datetime(2024, 1, 1),
end_date=datetime(2024, 12, 31),
).collect()
print(f"\nAAPL 2024 only: {len(aapl_2024)} rows (partition-pruned)")
print(f"Date range: {aapl_2024['timestamp'].min().date()} to {aapl_2024['timestamp'].max().date()}")
# %% [markdown]
# ### Metadata
# %%
# Check metadata for stored symbols
for symbol in symbols:
meta = dm_stored.get_metadata(symbol)
if meta:
print(f"\n{symbol}:")
for k, v in list(meta.items())[:5]:
print(f" {k}: {v}")
# %% [markdown]
# ### Inspect Partition Structure
# %%
# See the actual file layout
hive_root = STORAGE_DIR / "hive_demo"
parquet_files = sorted(hive_root.rglob("*.parquet"))
print(f"Total Parquet files: {len(parquet_files)}")
print("\nExample partition paths (first 8):")
for f in parquet_files[:8]:
rel = f.relative_to(hive_root)
size_kb = f.stat().st_size / 1024
print(f" {rel} ({size_kb:.1f} KB)")
# %% [markdown]
# ---
#
# ## 4. Incremental Updates
#
# The key workflow: download history once, then **update daily** with only new data.
# %% [markdown]
# ### Update a Symbol
# %%
# update() checks what's already stored and only fetches new data
for symbol in symbols:
key = dm_stored.update(symbol, lookback_days=7, provider="yahoo")
print(f" Updated {symbol} → {key}")
# Verify data is current
for symbol in symbols:
meta = dm_stored.get_metadata(symbol)
if meta and "last_updated" in meta:
print(f" {symbol} last updated: {meta['last_updated']}")
# %% [markdown]
# ### Understanding Update Strategies
#
# ml4t-data supports four update strategies:
#
# | Strategy | Behavior | Use Case |
# |----------|----------|----------|
# | `INCREMENTAL` | Only fetch data after last stored timestamp | Daily updates (default) |
# | `APPEND_ONLY` | Never modify existing rows | Audit-safe archives |
# | `FULL_REFRESH` | Replace all data for the symbol | Recovery after corruption |
# | `BACKFILL` | Fill gaps in historical data | Fix missing periods |
#
# The default `INCREMENTAL` strategy is correct for most workflows.
# `DataManager.update()` uses it automatically.
# %% [markdown]
# ### Gap Detection
#
# Before backtesting, verify data completeness. The IncrementalUpdater
# can detect missing trading days.
# %%
from ml4t.data.update_manager import GapDetector
# Pass `exclude_weekends=True` so Saturdays and Sundays don't count as gaps.
# The cached series here is calendar-dense (each non-trading day carries the
# prior close forward), so the detector reports no gaps. For a sparse,
# trading-days-only feed it would instead flag every missing session, including
# holidays — without an exchange calendar it cannot tell a holiday from a true
# gap, so pair it with a calendar-aware check for end-of-day pipelines.
gap_detector = GapDetector(exclude_weekends=True)
for symbol, key in stored_keys.items():
df = storage.read(key).collect()
gaps = gap_detector.detect_gaps(df, frequency="daily")
if gaps:
print(f"{symbol}: {len(gaps)} gap(s) detected")
for gap in gaps[:3]:
print(f" {gap['start'].date()} to {gap['end'].date()} ({gap['size_days']} days)")
else:
print(f"{symbol}: No gaps (complete)")
# %% [markdown]
# ---
#
# ## 5. Command-Line Interface
#
# ml4t-data includes a CLI for scripted workflows and cron jobs.
# Here are the key commands:
#
# ### Fetch Data
# ```bash
# # Single symbol
# ml4t-data fetch AAPL --start 2024-01-01 --end 2024-12-31
#
# # Multiple symbols
# ml4t-data fetch SPY QQQ IWM TLT --provider yahoo --output data/etfs.parquet
# ```
#
# ### Update Stored Data
# ```bash
# # Update a symbol (incremental — only fetches new data)
# ml4t-data update AAPL --storage-path ./data
#
# # Update all stored symbols
# ml4t-data update --all --storage-path ./data
# ```
#
# ### Validate Data Quality
# ```bash
# # Run OHLCV validation on stored data
# ml4t-data validate ./data/etfs.parquet
# ```
#
# ### List Available Data
# ```bash
# # List symbols in storage
# ml4t-data list --storage-path ./data
#
# # List available providers
# ml4t-data info --providers
# ```
#
# ### Automated Daily Updates (Cron)
# ```bash
# # Daily at 6 PM EST (after US market close), Monday-Friday
# 0 18 * * 1-5 cd ~/ml4t && ml4t-data update --all --storage-path ./data >> logs/update.log 2>&1
# ```
# %% [markdown]
# ---
#
# ## 6. Putting It Together: Production Data Pipeline
#
# Here's the complete workflow combining everything above — the pattern
# used by the book's `data/download_all.py` orchestrator.
# %%
def production_pipeline(
universe_name: str,
start: str,
end: str,
storage_path: Path,
) -> pl.DataFrame:
"""Fetch, store, validate, and assemble a stacked DataFrame for a universe.
The same pattern drives `data/download_all.py` for every asset class —
only the universe and provider differ.
"""
from ml4t.data.validation import OHLCVValidator
symbols = Universe.get(universe_name)
print(f"Universe '{universe_name}': {len(symbols)} symbols")
config = StorageConfig(base_path=storage_path, compression="zstd")
store = HiveStorage(config=config)
manager = DataManager(storage=store, enable_validation=True)
stored = {}
for symbol in symbols:
stored[symbol] = manager.load(symbol, start, end, provider="yahoo")
print(f"Fetched: {len(stored)} symbols")
validator = OHLCVValidator(max_return_threshold=0.5)
issues = 0
for symbol, key in stored.items():
df = store.read(key).collect()
result = validator.validate(df)
if not result.passed:
issues += result.error_count
print(f" {symbol}: {result.error_count} validation issues")
print(f"Validated: {issues} total issue(s) across {len(stored)} symbols")
frames = [
store.read(key).collect().with_columns(pl.lit(symbol).alias("symbol"))
for symbol, key in stored.items()
]
combined = pl.concat(frames)
print(f"Result: {combined.shape[0]:,} rows, {combined['symbol'].n_unique()} symbols")
return combined
# %%
# Run pipeline on a small universe
pipeline_output = production_pipeline(
universe_name="etf_momentum",
start="2024-01-01",
end="2024-12-31",
storage_path=STORAGE_DIR / "pipeline_demo",
)
pipeline_output.head()
# %% [markdown]
# A single validation issue per symbol on this 2024 ETF panel comes from the
# `OHLCVValidator(max_return_threshold=0.5)` flagging the largest 1-day move
# in each series — a sanity check, not a data error. The validator surfaces
# candidates; downstream code decides whether to drop, winsorize, or pass
# through. Section 2.6 (data quality) covers the trade-offs.
# %% [markdown]
# ---
#
# ## Summary
#
# | Component | Purpose | Key Method |
# |-----------|---------|------------|
# | **DataManager** | Unified entry point | `fetch()`, `batch_load()`, `load()`, `update()` |
# | **Universe** | Predefined symbol lists | `Universe.SP500`, `Universe.get("nasdaq100")` |
# | **HiveStorage** | Partitioned Parquet | `read()`, `write()`, partition pruning |
# | **GapDetector** | Gap detection in time series | `detect_gaps()`, `detect_gaps_in_storage()` |
# | **CLI** | Scripted workflows & cron | `ml4t-data fetch`, `ml4t-data update` |
#
# ### The ml4t-data Workflow
#
# ```
# 1. Initial load: dm.load("AAPL", "2020-01-01", "2024-12-31")
# 2. Daily update: dm.update("AAPL", lookback_days=7)
# 3. Gap check: gap_detector.detect_gaps(df, frequency="daily")
# 4. Batch load: dm.batch_load_universe("sp500", start, end)
# 5. Automate: cron + ml4t-data update --all
# ```
#
# ### Key Takeaways
#
# - **One entry point, many providers.** `DataManager.fetch()` hides whether the
# bytes come from Yahoo, Binance, AlgoSeek, or local Hive parquet; the user
# code does not change when providers do.
# - **`load()` is cache-first, `fetch()` is provider-first.** Use `load()` for
# research / backtesting (fast, offline, deterministic); use `fetch()` only
# when the cache must be refreshed.
# - **Universes are first-class.** `Universe.SP500` and friends keep symbol
# lists out of notebook code and version-controlled in the library.
# - **Gap detection is a separate concern.** `GapDetector` runs against
# already-stored data; missing trading days surface as findings, not silent
# nulls.
# - **The CLI is the production surface.** Cron-driven `ml4t-data update --all`
# is the same code path the notebook exercises.
#
# ### Further Reading
#
# - **Incremental updates**: `19_incremental_updates` walks the update strategies
# from this notebook in detail and shows how to schedule them.
# - **Storage formats**: `20_storage_benchmark_file` compares Parquet, CSV, and HDF5;
# `21_storage_benchmark_database` benchmarks Postgres-backed alternatives.
# - **Data quality**: `13_data_quality_framework` covers validation and anomaly detection.
# - **Provider comparison**: `16_provider_comparison` demonstrates multi-source acquisition.
# - **ml4t-data docs**: [ml4trading.io/docs/data/](https://ml4trading.io/docs/data/)
02_financial_data_universe/19_incremental_updates.ipynb
+956
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/19_incremental_updates.py
+372
View File
@@ -0,0 +1,372 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Incremental Updates: Keeping Market Data Fresh
#
# **Docker image**: `ml4t`
#
# **Purpose**: Walk through the update lifecycle — initial load, daily delta,
# gap detection, and a small health dashboard — so the same flow that runs
# in `data/download_all.py --update` is visible end-to-end.
#
# **Learning objectives**:
# 1. Compare full-refresh and incremental update costs.
# 2. Drive a daily delta with `DataManager.update(..., fill_gaps=False)` and
# understand why the `fill_gaps=True` default is unsafe for OHLCV data.
# 3. Verify completeness with `GapDetector(exclude_weekends=True)`.
# 4. Render a 3-panel health dashboard (volume, freshness, issues).
#
# **Book reference**: §2.4 (storing data) — operational counterpart to the
# storage benchmarks in notebooks 20 and 21.
#
# **Prerequisites**: ml4t-data installed; live network access for Yahoo Finance.
#
# **Why incremental updates**: a full refresh of 500 symbols × 10 years pulls
# ~1.26M rows; an incremental run pulls roughly the trading days since the
# last fetch. The gap is two orders of magnitude in network and disk I/O.
# %% [markdown]
# ## Setup
# %%
"""Incremental Updates — Keeping market data fresh with daily delta fetches."""
import logging
import shutil
from datetime import datetime
import matplotlib.pyplot as plt
import polars as pl
import structlog
# Quiet ml4t-data's structured logger so the notebook focuses on demo output.
structlog.configure(
wrapper_class=structlog.make_filtering_bound_logger(logging.WARNING),
)
from ml4t.data import DataManager
from ml4t.data.storage import HiveStorage
from ml4t.data.storage.backend import StorageConfig
from ml4t.data.update_manager import GapDetector
from ml4t.data.validation import OHLCVValidator
from utils.paths import get_output_dir
from utils.style import COLORS
# Storage for this notebook's demos. Wipe any prior-run artifacts so the
# initial-load + update sequence is reproducible.
DEMO_DIR = get_output_dir(2, "incremental_updates")
if DEMO_DIR.exists():
shutil.rmtree(DEMO_DIR)
DEMO_DIR.mkdir(parents=True, exist_ok=True)
print(f"Demo storage: {DEMO_DIR}")
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ---
#
# ## 1. Full Refresh vs Incremental Update
#
# Let's demonstrate the difference with a concrete example.
# %% [markdown]
# ### Step 1: Initial Load (Full History)
# %%
# Initialize storage and DataManager
config = StorageConfig(base_path=DEMO_DIR / "updates_demo", compression="zstd")
storage = HiveStorage(config=config)
dm = DataManager(storage=storage)
symbols = ["AAPL", "MSFT", "GOOGL"]
print("=== Initial Load (Full History) ===")
for symbol in symbols:
dm.load(symbol, "2023-01-01", "2024-12-31", provider="yahoo")
meta = dm.get_metadata(symbol)
print(f" {symbol}: stored ({meta['row_count']} rows)")
# %% [markdown]
# ### Step 2: Incremental Update (Only New Data)
#
# `DataManager.update()` reads the last timestamp in storage, refetches a small
# overlap (`lookback_days=7`) plus everything since, and merges. We pass
# `fill_gaps=False` deliberately — the library default `fill_gaps=True` runs a
# calendar-unaware gap detector that treats every weekend and US holiday as a
# missing trading day and forward-fills it, silently inflating each symbol's
# row count by hundreds of phantom bars. For OHLCV data, the right behavior is
# to leave non-trading days absent and rely on a calendar-aware completeness
# check downstream.
# %%
print("=== Incremental Update ===")
for symbol in symbols:
dm.update(symbol, lookback_days=7, provider="yahoo", fill_gaps=False)
meta = dm.get_metadata(symbol)
print(f" {symbol}: updated ({meta['row_count']} rows)")
# %% [markdown]
# Each symbol started with 502 rows (2023-01 through 2024-12). The update
# fetches the 7-day overlap plus all sessions through today and merges; the
# resulting row count equals the original 502 plus exactly the new trading
# sessions — no synthetic weekend/holiday rows.
# %% [markdown]
# ---
#
# ## 2. Update Strategies
#
# ml4t-data supports four strategies for different scenarios:
# %%
pl.DataFrame(
{
"strategy": ["INCREMENTAL", "APPEND_ONLY", "FULL_REFRESH", "BACKFILL"],
"behavior": [
"Fetch data after last stored timestamp",
"Add new rows; never modify existing",
"Re-download and replace all data",
"Fetch missing periods inside existing range",
],
"use_case": [
"Daily updates (default, fastest)",
"Audit-safe archives",
"Recovery after corruption",
"Patch holes in historical data",
],
}
)
# %% [markdown]
# The `IncrementalUpdater` class provides low-level control over these strategies.
# For most use cases, `DataManager.update()` (which uses `INCREMENTAL`) is sufficient.
# %% [markdown]
# ---
#
# ## 3. Gap Detection
#
# Before running a backtest, verify that your data is complete.
# Gaps can occur from failed downloads, provider outages, or holiday handling.
# %%
gap_detector = GapDetector(exclude_weekends=True)
print("=== Gap Detection ===")
for symbol in symbols:
key = f"equities/daily/{symbol}"
df = storage.read(key).collect()
gaps = gap_detector.detect_gaps(df, frequency="daily")
if gaps:
print(f"{symbol}: {len(gaps)} gap(s) found")
for gap in gaps[:5]:
print(f" {gap['start'].date()} -> {gap['end'].date()} ({gap['size_days']} days)")
if len(gaps) > 5:
print(f" ... and {len(gaps) - 5} more")
else:
print(f"{symbol}: complete (no gaps)")
# %% [markdown]
# `exclude_weekends=True` filters Saturdays and Sundays. US market holidays
# (MLK Day, Good Friday, Thanksgiving, etc.) still register as one-day gaps
# because the detector has no exchange calendar — fine for routine update
# health checks, but pair with a calendar-aware completeness check before
# trusting the result for backtest panels.
# %% [markdown]
# ---
#
# ## 4. Data Health Dashboard
#
# In production, monitor data health across your entire universe.
# %%
def data_health_report(storage: HiveStorage, symbols: list[str]) -> pl.DataFrame:
"""Per-symbol freshness, gap count, and validation issue count."""
validator = OHLCVValidator(max_return_threshold=0.5)
detector = GapDetector(exclude_weekends=True)
now = datetime.now()
rows = []
for symbol in symbols:
df = storage.read(f"equities/daily/{symbol}").collect()
last_date = df["timestamp"].max().replace(tzinfo=None)
days_stale = (now - last_date).days
gaps = detector.detect_gaps(df, frequency="daily")
result = validator.validate(df)
rows.append(
{
"symbol": symbol,
"status": "stale" if days_stale > 5 else "fresh",
"rows": len(df),
"last_date": last_date.date(),
"days_stale": days_stale,
"gaps": len(gaps),
"issues": result.error_count if not result.passed else 0,
}
)
return pl.DataFrame(rows)
# %%
report = data_health_report(storage, symbols)
report
# %%
fig, axes = plt.subplots(1, 3, figsize=(14, 4), constrained_layout=True)
axes[0].barh(report["symbol"].to_list(), report["rows"].to_list(), color=COLORS["blue"])
axes[0].set_xlabel("Rows")
axes[0].set_title("Data Volume")
stale_days = report["days_stale"].to_list()
freshness_color = [
COLORS["positive"] if d <= 3 else COLORS["amber"] if d <= 7 else COLORS["negative"]
for d in stale_days
]
# Plot bars; zero-width bars (days_stale == 0) get a small marker so the
# panel is not blank when every symbol is fresh.
axes[1].barh(report["symbol"].to_list(), stale_days, color=freshness_color)
zero_mask = [i for i, d in enumerate(stale_days) if d == 0]
if zero_mask:
axes[1].scatter(
[0] * len(zero_mask),
[report["symbol"].to_list()[i] for i in zero_mask],
color=[freshness_color[i] for i in zero_mask],
s=80,
marker="o",
zorder=3,
label="Fresh (0 days)",
)
axes[1].axvline(3, color=COLORS["positive"], linestyle="--", alpha=0.6, label="3-day threshold")
axes[1].axvline(7, color=COLORS["amber"], linestyle="--", alpha=0.6, label="7-day threshold")
axes[1].set_xlim(left=-0.5)
axes[1].set_xlabel("Days Since Update")
axes[1].set_title("Data Freshness")
axes[1].legend(fontsize=8, loc="lower right")
gaps = report["gaps"].to_list()
issues = report["issues"].to_list()
axes[2].barh(report["symbol"].to_list(), gaps, color=COLORS["negative"], label="Gaps")
axes[2].barh(
report["symbol"].to_list(), issues, left=gaps, color=COLORS["amber"], label="Validation"
)
axes[2].set_xlabel("Count")
axes[2].set_title("Data Issues")
axes[2].legend(fontsize=8, loc="lower right")
fig.suptitle("Data Health Dashboard")
plt.show()
# %% [markdown]
# ---
#
# ## 5. The Book's Update Workflow
#
# The book's `data/download_all.py` script implements exactly this pattern
# for all asset classes. Run it with `--update` to extend datasets to the present:
#
# ```bash
# # Initial download (run once)
# python data/download_all.py
#
# # Update to present (run daily/weekly)
# python data/download_all.py --update
# ```
#
# Under the hood, `download_all.py` uses:
# - `ETFDataManager.from_config("data/etfs/config.yaml")` → `manager.update()`
# - `CryptoDataManager.from_config("data/crypto/config.yaml")` → `manager.update()`
# - `MacroDataManager.from_config("data/macro/config.yaml")` → `manager.download_treasury_yields()`
# - `FuturesDataManager.from_config("data/futures/config.yaml")` → `manager.download_all()`
#
# Each manager reads its YAML config for symbols, date ranges, and provider settings,
# then updates only what's new.
#
# ### Config-Driven Downloads
#
# The ETF config at `data/etfs/config.yaml` defines:
# ```yaml
# etfs:
# provider: yahoo
# start: '2006-01-01'
# end: '2025-12-31'
# frequency: daily
# tickers:
# us_equity_broad:
# symbols: [SPY, QQQ, IWM, ...]
# us_sectors:
# symbols: [XLB, XLC, XLE, ...]
# # ... 9 categories, 100 ETFs total
# ```
#
# When you run `--update` in 2026+, it extends data beyond the configured end date
# to the present — no config changes needed.
# %% [markdown]
# ---
#
# ## Summary
#
# ### Key Patterns
#
# | Pattern | Command | When |
# |---------|---------|------|
# | Initial load | `dm.load(symbol, start, end)` | First time |
# | Daily update | `dm.update(symbol, lookback_days=7)` | Every trading day |
# | Gap check | `updater.detect_gaps(df, "daily")` | Before backtesting |
# | Full refresh | `UpdateStrategy.FULL_REFRESH` | After data corruption |
# | Batch update | `download_all.py --update` | Cron job |
#
# ### Production Checklist
#
# 1. **Initial download**: `python data/download_all.py` (run once, ~10 min)
# 2. **Schedule updates**: Add `download_all.py --update` to cron (daily at 6 PM)
# 3. **Monitor health**: Check freshness, gaps, and validation before backtests
# 4. **Validate data**: Use `OHLCVValidator` on every load (see `13_data_quality_framework`)
#
# ### Key Takeaways
#
# - **`lookback_days` is the operating lever**, not the symbol set. Set it once
# to cover provider revision windows (Yahoo retro-adjusts ~5 trading days; CRSP
# ~30) and the strategy generalizes across instruments.
# - **Gap detection runs against stored data, not stream data.** A daily cron
# that finishes with `detect_gaps()` catches missed trading days before any
# backtest reads stale partitions.
# - **Full refresh is the recovery path.** Use `UpdateStrategy.FULL_REFRESH`
# when validation flags a regression; never patch a corrupted parquet in place.
# - **Configurable end dates eliminate config drift.** Symbol-list YAMLs use
# today-as-default; `--update` extends data beyond the file's `end:` field
# without needing edits each calendar year.
# - **Cron + idempotent CLI is the production interface.** The same
# `download_all.py --update` works in a notebook, a CI run, and a 6 PM cron.
#
# Next: `20_storage_benchmark_file` compares file-format throughput for the
# parquet write path implicit in every update; `21_storage_benchmark_database`
# extends the same comparison to database engines.
#
# ### Cross-References
#
# - **Data quality**: `13_data_quality_framework` — validation and anomaly detection.
# - **DataManager basics**: `18_data_management` — fetch, batch, Universe, storage.
# - **Storage benchmarks**: `20_storage_benchmark_file` (file formats) and
# `21_storage_benchmark_database` (database engines).
# - **Download scripts**: `data/download_all.py` — the book's orchestrator.
# - **ml4t-data docs**: [ml4trading.io/docs/data/user-guide/incremental-updates/](https://ml4trading.io/docs/data/user-guide/incremental-updates/)
02_financial_data_universe/20_storage_benchmark_file.ipynb
+2140
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/20_storage_benchmark_file.py
+435
View File
@@ -0,0 +1,435 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # File-Format Storage Benchmark
#
# **Docker image**: `benchmark`
#
# **Purpose**: Compare CSV, Parquet, Feather (Arrow IPC), and HDF5 on the same
# 1 M-row OHLCV panel along three axes — write time, read time (with forced
# materialization), and on-disk size — so the trade-offs in §2.4 are
# reproducible end to end.
#
# **Learning objectives**:
# 1. Generate a deterministic OHLCV benchmark panel at the L scale that
# chapter §2.4 cites (100 symbols × 10,000 daily rows = 1,000,000 rows).
# 2. Time write and read for each format with `gc.collect()` between runs.
# 3. Force materialization on Feather / HDF5 reads so memory-mapped or lazy
# reads don't masquerade as instant.
# 4. Quantify the columnar-projection win (read 2 columns vs 9).
# 5. Render a 3-panel comparison (read time / write time / file size).
#
# **Book reference**: §2.4 — file-based storage benchmarks.
#
# **Prerequisites**: PyTables (HDF5 backend) is only present in the
# `benchmark` Docker image. Run locally with `uv run`, or via
# `docker compose --profile benchmark run --rm benchmark python …`.
# %% [markdown]
# ## Setup
# %%
"""File-format storage benchmark — CSV / Parquet / Feather / HDF5 at L scale."""
import gc
import os
import time
# %% tags=["parameters"]
# Production scale follows chapter §2.4 prose ("L scale, ~1 M OHLCV rows,
# 100 MB in-memory"). Override via Papermill for CI: BENCHMARK_SCALE = "S".
BENCHMARK_SCALE = "L"
# %%
# storage_benchmarks reads BENCHMARK_SCALE at import time, so the env var
# must be set before the import below.
os.environ["BENCHMARK_SCALE"] = BENCHMARK_SCALE
import pandas as pd
import plotly.graph_objects as go
import polars as pl
# PyTables is the HDF5 backend; raise loudly if the image is wrong.
import tables # noqa: F401
from plotly.subplots import make_subplots
from utils.paths import get_output_dir
from utils.storage_benchmarks import (
ACTIVE_SCALE,
BENCHMARK_DIR,
N_ROWS_PER_SYMBOL,
N_SYMBOLS,
BenchmarkResult,
estimate_memory_mb,
force_materialize_pandas,
force_materialize_polars,
generate_ohlcv_data,
save_benchmark_results,
time_operation,
validate_result,
)
from utils.style import COLORS
OUTPUT_DIR = get_output_dir(2, "storage_benchmark")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
# %% [markdown]
# ## 1. Generate the Benchmark Panel
#
# `generate_ohlcv_data` returns a deterministic OHLCV panel: per-symbol random
# walks under a fixed seed so format comparisons aren't muddled by data drift
# across runs. At L scale that's 100 symbols × 10,000 rows = 1 M total rows.
# %%
ohlcv_df = generate_ohlcv_data(n_symbols=N_SYMBOLS, n_rows=N_ROWS_PER_SYMBOL)
ohlcv_pandas = ohlcv_df.to_pandas()
panel_summary = pl.DataFrame(
{
"field": [
"scale",
"symbols",
"rows per symbol",
"total rows",
"in-memory size (Polars, MB)",
"in-memory size (pandas, MB)",
],
"value": [
ACTIVE_SCALE,
f"{N_SYMBOLS:,}",
f"{N_ROWS_PER_SYMBOL:,}",
f"{len(ohlcv_df):,}",
f"{estimate_memory_mb(ohlcv_df):.2f}",
f"{estimate_memory_mb(ohlcv_pandas):.2f}",
],
}
)
panel_summary
# %%
total_rows = len(ohlcv_df)
results: list[BenchmarkResult] = []
# %% [markdown]
# ## 2. CSV — Universal Baseline
#
# CSV is the row-oriented baseline: human-readable, no compression, no schema.
# Every other format is judged against it.
# %%
csv_path = BENCHMARK_DIR / f"ohlcv_{ACTIVE_SCALE.lower()}.csv"
gc.collect()
write_time, _ = time_operation(lambda: ohlcv_df.write_csv(csv_path))
csv_size = csv_path.stat().st_size
results.append(BenchmarkResult("CSV", "write", write_time, csv_size, total_rows))
def read_csv_materialized() -> pl.DataFrame:
return force_materialize_polars(pl.read_csv(csv_path))
gc.collect()
read_time, csv_result = time_operation(read_csv_materialized)
validate_result(csv_result, total_rows, "CSV read")
results.append(BenchmarkResult("CSV", "read", read_time, csv_size, total_rows))
def read_csv_columnar() -> pl.DataFrame:
return force_materialize_polars(pl.read_csv(csv_path, columns=["close", "volume"]))
gc.collect()
columnar_time, _ = time_operation(read_csv_columnar)
results.append(BenchmarkResult("CSV", "columnar_read", columnar_time, csv_size, total_rows))
# %% [markdown]
# ## 3. Parquet — Compressed Columnar Standard
#
# Parquet's row-group layout, dictionary encoding, and Snappy compression make
# it the default for analytical workloads. Column projection reads only the
# row-group chunks for the requested columns.
# %%
parquet_path = BENCHMARK_DIR / f"ohlcv_{ACTIVE_SCALE.lower()}.parquet"
gc.collect()
write_time, _ = time_operation(lambda: ohlcv_df.write_parquet(parquet_path))
parquet_size = parquet_path.stat().st_size
results.append(BenchmarkResult("Parquet", "write", write_time, parquet_size, total_rows))
def read_parquet_materialized() -> pl.DataFrame:
return force_materialize_polars(pl.read_parquet(parquet_path))
gc.collect()
read_time, parquet_result = time_operation(read_parquet_materialized)
validate_result(parquet_result, total_rows, "Parquet read")
results.append(BenchmarkResult("Parquet", "read", read_time, parquet_size, total_rows))
def read_parquet_columnar() -> pl.DataFrame:
return force_materialize_polars(pl.read_parquet(parquet_path, columns=["close", "volume"]))
gc.collect()
columnar_time, _ = time_operation(read_parquet_columnar)
results.append(BenchmarkResult("Parquet", "columnar_read", columnar_time, parquet_size, total_rows))
# %% [markdown]
# ## 4. Feather (Arrow IPC) — Zero-Copy Interchange
#
# Feather opens the file by memory-mapping it; the bare `read_ipc` call returns
# almost instantly because no bytes have been read yet. We capture both the
# raw open time and the time to actually materialize the columns into memory.
# Only the materialized number is comparable across formats.
# %%
feather_path = BENCHMARK_DIR / f"ohlcv_{ACTIVE_SCALE.lower()}.feather"
gc.collect()
write_time, _ = time_operation(lambda: ohlcv_df.write_ipc(feather_path))
feather_size = feather_path.stat().st_size
results.append(BenchmarkResult("Feather", "write", write_time, feather_size, total_rows))
gc.collect()
start = time.perf_counter()
_ = pl.read_ipc(feather_path) # raw handle — memory-mapped, not materialized
raw_handle_time = time.perf_counter() - start
def read_feather_materialized() -> pl.DataFrame:
return force_materialize_polars(pl.read_ipc(feather_path))
gc.collect()
read_time, feather_result = time_operation(read_feather_materialized)
validate_result(feather_result, total_rows, "Feather read")
results.append(BenchmarkResult("Feather", "read", read_time, feather_size, total_rows))
def read_feather_columnar() -> pl.DataFrame:
return force_materialize_polars(pl.read_ipc(feather_path, columns=["close", "volume"]))
gc.collect()
columnar_time, _ = time_operation(read_feather_columnar)
results.append(BenchmarkResult("Feather", "columnar_read", columnar_time, feather_size, total_rows))
# %% [markdown]
# ## 5. HDF5 — Legacy Scientific Container
#
# HDF5 keeps a foothold in research codebases that predate Parquet. The
# `fixed` format used by `pandas.HDFStore` doesn't support column projection,
# so we record the columnar read as the same as the full read for fairness.
# %%
hdf5_path = BENCHMARK_DIR / f"ohlcv_{ACTIVE_SCALE.lower()}.h5"
def write_hdf5() -> None:
with pd.HDFStore(hdf5_path, mode="w") as store:
store["ohlcv"] = ohlcv_pandas
gc.collect()
write_time, _ = time_operation(write_hdf5)
hdf5_size = hdf5_path.stat().st_size
results.append(BenchmarkResult("HDF5", "write", write_time, hdf5_size, total_rows))
def read_hdf5_materialized() -> pd.DataFrame:
with pd.HDFStore(hdf5_path, mode="r") as store:
df = store["ohlcv"]
return force_materialize_pandas(df)
gc.collect()
read_time, hdf5_result = time_operation(read_hdf5_materialized)
validate_result(hdf5_result, total_rows, "HDF5 read")
results.append(BenchmarkResult("HDF5", "read", read_time, hdf5_size, total_rows))
# Fixed-format HDF5 has no column projection — record the full-read time
# so the comparison plot still has a value for the format.
results.append(BenchmarkResult("HDF5", "columnar_read", read_time, hdf5_size, total_rows))
# %% [markdown]
# ## 6. Results Summary
# %%
results_df = pl.DataFrame(
[
{
"format": r.name,
"operation": r.operation,
"time_s": r.time_seconds,
"size_mb": r.size_bytes / 1e6,
"throughput_M_rows_s": r.rows_per_second / 1e6,
}
for r in results
]
)
# %%
write_summary = (
results_df.filter(pl.col("operation") == "write")
.select(["format", "time_s", "size_mb", "throughput_M_rows_s"])
.sort("time_s")
)
write_summary
# %%
read_summary = (
results_df.filter(pl.col("operation") == "read")
.select(["format", "time_s", "throughput_M_rows_s"])
.sort("time_s")
)
read_summary
# %%
columnar_summary = (
results_df.filter(pl.col("operation") == "columnar_read")
.select(["format", "time_s", "throughput_M_rows_s"])
.sort("time_s")
)
columnar_summary
# %%
full_read = results_df.filter(pl.col("operation") == "read").select(["format", "time_s"])
col_read = (
results_df.filter(pl.col("operation") == "columnar_read")
.select(["format", "time_s"])
.rename({"time_s": "columnar_time_s"})
)
projection_speedup = (
full_read.join(col_read, on="format")
.with_columns(speedup=pl.col("time_s") / pl.col("columnar_time_s"))
.select(["format", "time_s", "columnar_time_s", "speedup"])
.sort("speedup", descending=True)
)
projection_speedup
# %% [markdown]
# Memory-mapped reads still need to be materialized before they're useful;
# the raw `read_ipc` handle time below is excluded from the comparison and
# listed only so the gap to the materialized Feather read is visible.
# %%
print(f"Feather raw handle (memory-mapped, not materialized): {raw_handle_time:.4f} s")
# %% [markdown]
# ## 7. Visualisation
# %%
fig = make_subplots(
rows=1,
cols=3,
subplot_titles=[
"Read time (lower is better)",
"Write time (lower is better)",
"File size (smaller is better)",
],
horizontal_spacing=0.12,
)
read_data = results_df.filter(pl.col("operation") == "read").sort("time_s")
write_data = results_df.filter(pl.col("operation") == "write").sort("time_s")
size_data = results_df.filter(pl.col("operation") == "write").sort("size_mb")
fig.add_trace(
go.Bar(
y=read_data["format"].to_list(),
x=read_data["time_s"].to_list(),
orientation="h",
marker_color=COLORS["blue"],
text=[f"{t:.3f}s" for t in read_data["time_s"].to_list()],
textposition="outside",
),
row=1,
col=1,
)
fig.add_trace(
go.Bar(
y=write_data["format"].to_list(),
x=write_data["time_s"].to_list(),
orientation="h",
marker_color=COLORS["amber"],
text=[f"{t:.3f}s" for t in write_data["time_s"].to_list()],
textposition="outside",
),
row=1,
col=2,
)
fig.add_trace(
go.Bar(
y=size_data["format"].to_list(),
x=size_data["size_mb"].to_list(),
orientation="h",
marker_color=COLORS["slate"],
text=[f"{s:.1f} MB" for s in size_data["size_mb"].to_list()],
textposition="outside",
),
row=1,
col=3,
)
fig.update_xaxes(title_text="Seconds (log)", row=1, col=1, type="log")
fig.update_xaxes(title_text="Seconds (log)", row=1, col=2, type="log")
fig.update_xaxes(title_text="MB", row=1, col=3)
_scale_word = {"S": "Small", "M": "Medium", "L": "Large"}.get(ACTIVE_SCALE, ACTIVE_SCALE)
fig.update_layout(
title_text=f"File-Format Comparison ({_scale_word} scale, {total_rows:,} rows)",
height=400,
showlegend=False,
paper_bgcolor=COLORS["bg_light"],
plot_bgcolor=COLORS["bg_light"],
# Wider right margin so 'XX.X MB' / 'X.XXs' value labels don't crop.
margin=dict(l=60, r=80, t=70, b=50),
)
fig.show()
# %% [markdown]
# ## Key Takeaways
#
# - **Memory-mapping is not a read.** The Feather raw handle returns in
# microseconds because no data has been touched. The materialized read is
# the apples-to-apples comparison.
# - **Columnar projection multiplies wins.** Reading two columns versus all
# nine is roughly proportional to the column-count ratio for Parquet and
# Feather; CSV barely benefits because every row must be parsed in full.
# - **Compression collapses on disk, not in RAM.** Parquet writes the panel
# at roughly a quarter of the CSV size; the in-memory footprint after
# read-back is identical because both formats land in Arrow buffers.
# - **HDF5 fixed format is single-shot.** It can read or write the entire
# panel but offers no column projection.
#
# ### Format Picks
#
# - **Long-term storage, cloud, cross-language**: Parquet.
# - **Local interchange between Python tools**: Feather.
# - **Legacy scientific Python pipelines that need append**: HDF5.
# - **Human inspection or small exports**: CSV.
#
# ### Cross-References
#
# - **Database engines** for the same panel: `21_storage_benchmark_database`.
# - **Daily data lifecycle on Parquet**: `19_incremental_updates`.
# - **Library-level storage primitives**: `18_data_management`.
# %%
save_benchmark_results(results, "formats")
02_financial_data_universe/21_storage_benchmark_database.ipynb
+3581
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/21_storage_benchmark_database.py
+1373
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/22_pandas_polars_benchmark.ipynb
+5156
View File
File diff suppressed because one or more lines are too long
02_financial_data_universe/22_pandas_polars_benchmark.py
+1476
View File
File diff suppressed because it is too large Load Diff
02_financial_data_universe/README.md
+101
View File
@@ -0,0 +1,101 @@
# Chapter 2: The Financial Data Universe
The chapter gives readers the conceptual map they need before touching any dataset. Its key contribution is not just the market / fundamental / alternative taxonomy, but the claim that every dataset embeds definitions about timestamps, adjustments, identifiers, and revisions, and that these choices determine what the data actually means in research.
## Learning Objectives
* Distinguish among market, fundamental, and alternative data, and explain how dataset definitions shape what each source means in research and trading applications
* Compare the observability, conventions, and engineering constraints of major asset classes, and identify how market structure changes what can be measured and modeled
* Apply a financial data quality framework to diagnose common failure modes, especially point-in-time violations, survivorship bias, corporate action errors, and identifier mismatches
* Conduct vendor due diligence across data quality, legal and compliance, and technical and commercial dimensions
* Choose storage and query architectures that fit research and production needs, including when to use partitioned files, embedded analytical databases, or server-based systems
## Sections
### 2.1 A Modern Taxonomy of Financial Data
This section gives readers the conceptual map they need before touching any dataset. Its key contribution is not just the market / fundamental / alternative taxonomy, but the claim that every dataset embeds definitions about timestamps, adjustments, identifiers, and revisions, and that these choices determine what the data actually means in research.
### 2.2 The Asset-Class Market Data Landscape
This section broadens the discussion from data categories to the practical reality that "price," "liquidity," and even "the dataset" mean different things across equities, ETPs, futures, options, digital assets, FX, fixed income, swaps, and commodities. Its value is comparative: it helps readers understand why engineering choices are inseparable from market structure.
- [`01_us_equities_eda`](01_us_equities_eda.ipynb) — This notebook introduces the Wiki Prices dataset - a survivorship-bias-free collection of US equity prices. Understanding survivorship bias is critical for realistic backtesting.
- [`02_corporate_actions`](02_corporate_actions.ipynb) — This notebook demonstrates how stock splits and dividends break historical price series, and shows the industry-standard backward adjustment methodology used by major data vendors. Correctly adjusting for corporate actions is essential for any ML model using return-based features.
- [`03_etfs_eda`](03_etfs_eda.ipynb) — This notebook introduces the 50-ETF universe that serves as the foundation for the ETF Rotational Momentum case study throughout the book. We explore the schema, coverage, categories, and data quality characteristics.
- [`04_cme_futures_eda`](04_cme_futures_eda.ipynb) — This notebook introduces the CME futures dataset shipped with the book. It demonstrates the data structure, coverage, and key concepts for working with futures data.
- [`05_futures_session_aggregation`](05_futures_session_aggregation.ipynb) — This notebook converts hourly continuous futures data (stored in UTC) to session-aware daily bars. CME futures sessions end at 4:00 PM Central Time, so daily bars must respect this boundary—not midnight UTC.
- [`06_futures_continuous`](06_futures_continuous.ipynb) — This notebook tackles one of the most critical challenges in futures analysis: creating continuous price series from individual expiring contracts. We implement roll detection algorithms and adjustment methods (Panama, ratio) to eliminate artificial price gaps while preserving accurate return characteristics.
- [`07_sp500_options_eda`](07_sp500_options_eda.ipynb) — This notebook provides a comprehensive exploration of the AlgoSeek S&P 500 Options Analytics dataset. Options data is fundamentally different from spot market data—it contains forward-looking information about expected volatility, directional sentiment, and tail risk that isn't directly observable in underlying prices.
- [`08_options_greeks_computation`](08_options_greeks_computation.ipynb) — This notebook derives and implements the Black-Scholes option pricing framework from first principles. We compute implied volatility and all Greeks, then validate our calculations against the pre-computed values in the AlgoSeek options data.
- [`09_options_continuous`](09_options_continuous.ipynb) — Options are time-decaying instruments. Unlike equities or futures, an option's price reflects both the value of the underlying exposure and the remaining time to expiration.
- [`10_crypto_perps_eda`](10_crypto_perps_eda.ipynb) — This notebook introduces the cryptocurrency dataset from Binance Futures. We explore hourly OHLCV data and the Premium Index (perpetual futures vs spot spread) that forms the basis for the Crypto Premium Arbitrage case study.
- [`11_crypto_premium_analysis`](11_crypto_premium_analysis.ipynb) — This notebook demonstrates how to work with Binance perpetual futures premium index data - the foundation for funding rate arbitrage strategies. We load, explore, and analyze premium dynamics across major cryptocurrencies to identify potential arbitrage opportunities.
- [`12_fx_pairs_eda`](12_fx_pairs_eda.ipynb) — This notebook introduces the FX dataset from OANDA. FX markets are OTC with no centralized exchange, so prices aggregate from multiple liquidity providers.
### 2.3 Data Sourcing: The Due Diligence Framework
This is the chapter's risk-control core. It explains that many apparent research successes are manufactured by data defects, then organizes due diligence around general quality dimensions, finance-specific failure modes, vendor evaluation, and internal governance. The section turns abstract warnings into operational rules: point-in-time correctness, survivorship handling, corporate action methodology, identifier integrity, legal rights, and reproducible versioning.
- [`13_data_quality_framework`](13_data_quality_framework.ipynb) — The ml4t-data library provides purpose-built tools for financial data quality. This notebook demonstrates the complete data quality workflow: Uses us_equities data.
- [`14_point_in_time_validation`](14_point_in_time_validation.ipynb) — Point-in-time correctness is essential for valid backtesting. Using information that wasn't available at decision time creates lookahead bias - making backtests look better than they would perform in live trading.
- [`15_survivorship_bias_detection`](15_survivorship_bias_detection.ipynb) — Survivorship bias is arguably the most dangerous form of data contamination in quantitative finance. This notebook uses real historical data from the US equities dataset (US Equities, originally Quandl WIKI) to demonstrate, detect, and quantify survivorship bias.
- [`16_provider_comparison`](16_provider_comparison.ipynb) — ML4T Third Edition - Chapter 2: The Financial Data...
### 2.4 Data Storage
This section translates data discipline into infrastructure decisions. Rather than promoting a single stack, it explains how storage choice depends on access patterns, scale, concurrency, and operational maturity, and it benchmarks the trade-offs among file formats, embedded engines, and server databases. It gives readers a practical default architecture for modern research workflows while also teaching when more complex systems are justified.
- [`17_complete_pipeline`](17_complete_pipeline.ipynb) — This notebook demonstrates end-to-end data pipelines, bringing together concepts from this chapter: Uses crypto_perps, wiki_provider data.
- [`18_data_management`](18_data_management.ipynb) — Previous notebooks fetched and validated data. This notebook shows how to manage it at scale using ml4t-data's production features: Uses universe data.
- [`19_incremental_updates`](19_incremental_updates.ipynb) — The previous notebook introduced DataManager and HiveStorage. This notebook focuses on the update workflow — the core reason ml4t-data exists: Uses all, treasury_yields data.
- [`20_storage_benchmark_file`](20_storage_benchmark_file.ipynb) — Focus: Pure file format comparison (no query engines) Technologies: CSV, Parquet, Feather (Arrow IPC), HDF5 Operations: Write, Read (with forced materialization), Columnar...
- [`21_storage_benchmark_database`](21_storage_benchmark_database.ipynb) — > Docker required: This notebook depends on the benchmark environment and > database services.
- [`22_pandas_polars_benchmark`](22_pandas_polars_benchmark.ipynb) — DataFrame-engine comparison (pandas vs Polars) across read, filter, groupby, join, and lazy operations on synthetic financial data at S/M/L scales. Backs the in-memory engine-choice recommendation in §2.4.
## Running the Notebooks
```bash
# From the repository root
uv run python 02_financial_data_universe/<notebook>.py
# Test mode (reduced data via Papermill)
uv run pytest tests/test_notebooks.py -v -k "02_financial_data_universe"
```
### NB21 storage_benchmark_database prerequisites
`21_storage_benchmark_database` exercises 7 storage engines (DuckDB, SQLite, TimescaleDB, ClickHouse, QuestDB, PostgreSQL, Polars Parquet). Of these, **DuckDB and SQLite run locally** with no additional setup; the other five require the benchmark service stack to be running first:
```bash
docker compose --profile benchmark up -d
```
Without these services, NB21 silently falls back to the 2-engine subset (DuckDB + SQLite only) and the comparative server-database results in §2.4 will not be reproduced. Bring the stack down with `docker compose --profile benchmark down` when finished.
## References
- **William Beaver et al.** (2007). [Delisting returns and their effect on accounting-based market anomalies](https://doi.org/10.1016/j.jacceco.2006.12.002). *Journal of Accounting and Economics*.
- **Florian Berg et al.** (2022). [Aggregate Confusion: The Divergence of ESG Ratings*](https://doi.org/10.1093/rof/rfac033). *Review of Finance*.
- **Mark M. Carhart et al.** (2002). [Mutual Fund Survivorship](https://doi.org/10.1093/rfs/15.5.1439). *The Review of Financial Studies*.
- **Lin William Cong et al.** (2023). [Crypto Wash Trading](https://doi.org/10.1287/mnsc.2021.02709). *Management Science*.
- **David Easley et al.** (2021). [Microstructure in the Machine Age](https://doi.org/10.1093/rfs/hhaa078). *The Review of Financial Studies*.
- **B. Espen Eckbo and Markus Lithell** (2025). [Merger-Driven Listing Dynamics](https://doi.org/10.1017/S0022109023001394). *Journal of Financial and Quantitative Analysis*.
- **Gene Ekster and Petter N. Kolm** (2020). [Alternative Data in Investment Management: Usage, Challenges and Valuation](https://doi.org/10.2139/ssrn.3715828).
- **Edwin J. Elton et al.** (1996). [Survivor Bias and Mutual Fund Performance](https://doi.org/10.1093/rfs/9.4.1097). *The Review of Financial Studies*.
- **Kingsley Y L Fong et al.** (2017). [What Are the Best Liquidity Proxies for Global Research?*](https://doi.org/10.1093/rof/rfx003). *Review of Finance*.
- **Songrun He et al.** (2024). [Fundamentals of Perpetual Futures](https://doi.org/10.48550/arXiv.2212.06888).
- **Jacques Joubert et al.** (2024). [The Three Types of Backtests](https://doi.org/10.2139/ssrn.4897573).
- **Nina Karnaukh et al.** (2015). [Understanding FX Liquidity](https://doi.org/10.2139/ssrn.2329738).
- **Gueorgui S. Konstantinov** (2025). [On Systematic Currency Management](https://doi.org/10.3905/jpm.2025.1.724). *The Journal of Portfolio Management*.
- **John Lehoczky and Mark Schervish** (2018). [Overview and History of Statistics for Equity Markets](https://doi.org/10.1146/annurev-statistics-031017-100518). *Annual Review of Statistics and Its Application*.
- **Alex Lipton and Marcos Lopez de Prado** (2020). [Three Quant Lessons from COVID-19](https://doi.org/10.2139/ssrn.3580185).
- **Tim Loughran and Bill McDonald** (2020). [Textual Analysis in Finance](https://doi.org/10.1146/annurev-financial-012820-032249). *Annual Review of Financial Economics*.
- **Yin Luo et al.** (2014). Seven Sins of Quantitative Investing.
- **Igor Makarov and Antoinette Schoar** (2020). [Trading and arbitrage in cryptocurrency markets](https://doi.org/10.1016/j.jfineco.2019.07.001). *Journal of Financial Economics*.
- **Hunter Ng et al.** (2025). [Price Discovery and Trading in Prediction Markets](https://doi.org/10.2139/ssrn.5331995).
- **Maureen O’Hara** (2015). [High frequency market microstructure](https://doi.org/10.1016/j.jfineco.2015.01.003). *Journal of Financial Economics*.
- **Marcos Lopez de Prado** (2018). Advances in Financial Machine Learning. *John Wiley & Sons*.
- **SEC** (2020). [Staff Report on Algorithmic Trading in U.S. Capital Markets](https://www.sec.gov/file/staff-report-algorithmic-trading-us-capital-markets).
- **Tyler Shumway** (1997). [The Delisting Bias in CRSP Data](https://doi.org/10.1111/j.1540-6261.1997.tb03818.x). *The Journal of Finance*.
- **David Vidal-Tomás** (2022). [Which cryptocurrency data sources should scholars use?](https://doi.org/10.1016/j.irfa.2022.102061). *International Review of Financial Analysis*.
03_market_microstructure/01_itch_parser.ipynb
+1119
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/01_itch_parser.py
+576
View File
@@ -0,0 +1,576 @@
# ---
# jupyter:
# jupytext:
# formats: ipynb,py:percent
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.1
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # NASDAQ TotalView-ITCH: Order Book Data Parsing
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# This notebook demonstrates how to parse NASDAQ's TotalView-ITCH binary protocol.
# Understanding MBO (message-by-order) data is foundational for microstructure-based ML features.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Understand the ITCH 5.0 binary message format
# - Parse ITCH messages using Python's `struct` module
# - Load pre-parsed ITCH data for analysis
# - Choose between Python (educational) and Rust (production) parsers
#
# ## Cross-References
#
# - **Downstream**: `02_itch_lob_reconstruction` (builds order book from these messages)
# - **Related**: `09_databento_mbo_analysis` (alternative MBO data source)
#
# ## Data Requirements
#
# ITCH sample data can be downloaded using:
# ```bash
# python data/equities/market/microstructure/nasdaq_itch_download.py --list # List available files
# python data/equities/market/microstructure/nasdaq_itch_download.py # Download default sample
# ```
#
# Files are ~5GB compressed from: https://emi.nasdaq.com/ITCH/
# %% [markdown]
# ## ITCH Message Types
#
# The [ITCH v5.0 specification](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf) defines 20+ message types:
#
# | Type | Name | Description |
# |------|------|-------------|
# | **S** | System Event | Market open/close events |
# | **R** | Stock Directory | Ticker information and characteristics |
# | **H** | Trading Action | Trading halts, pauses, and resumptions |
# | **Y** | Reg SHO Restriction | Short sale price test restrictions |
# | **L** | Market Participant | Market maker positions |
# | **V** | MWCB Decline Level | Market-wide circuit breaker levels |
# | **W** | MWCB Status | Circuit breaker breach status |
# | **A** | Add Order | New limit order enters the book |
# | **F** | Add Order (MPID) | Same as A, with market participant ID |
# | **E** | Order Executed | Partial/full execution against standing order |
# | **C** | Order Executed w/Price | Execution at different price (hidden orders) |
# | **X** | Order Cancel | Partial cancellation |
# | **D** | Order Delete | Full removal from book |
# | **U** | Order Replace | Modify price/size (cancel + add) |
# | **P** | Trade | Non-displayed execution |
# | **Q** | Cross Trade | Opening/closing cross |
# | **B** | Broken Trade | Trade cancellation |
# | **I** | NOII | Net Order Imbalance Indicator (auction) |
# | **J** | LULD Auction Collar | Limit up-limit down price bands |
# | **K** | IPO Quoting Period | IPO quotation timing |
#
# By combining these messages chronologically, we can reconstruct the order book at any point in time.
# %%
"""NASDAQ TotalView-ITCH: Order Book Data Parsing — parse ITCH binary protocol into structured messages."""
import gzip
import os
import shutil
import struct
import warnings
from collections import Counter, defaultdict
from datetime import date, datetime
from pathlib import Path
from time import time
warnings.filterwarnings("ignore")
import polars as pl
from itch_message_specs import (
FMT_DICT,
MESSAGE_SPECS,
NT_DICT,
flush_to_parquet,
parse_price4,
parse_timestamp,
print_message_formats,
)
from tqdm.auto import tqdm
from data import load_nasdaq_itch
# %% tags=["parameters"]
SKIP_PARSING = False
# %%
# Data paths
# Canonical parsed messages location (both read and write target)
MESSAGE_DIR = load_nasdaq_itch(get_base_path=True)
MESSAGE_DIR.mkdir(parents=True, exist_ok=True)
# Raw ITCH binary input (from download script)
ITCH_RAW_DIR = MESSAGE_DIR.parent / "raw"
print(f"Raw ITCH data (input): {ITCH_RAW_DIR}")
print(f"Parsed messages (output): {MESSAGE_DIR}")
_raw_present = ITCH_RAW_DIR.exists() and next(ITCH_RAW_DIR.iterdir(), None) is not None
print(f"Raw data exists: {_raw_present}")
if not _raw_present:
print(" (run NB00_itch_download.py first to fetch the ITCH binary)")
# %% [markdown]
# ## 1. Message Specifications
#
# Each ITCH message has a fixed binary structure. The format is defined in `itch_message_specs.py`
# using Python's `struct` module. Format codes:
# - `H` = unsigned short (2 bytes)
# - `I` = unsigned int (4 bytes)
# - `Q` = unsigned long long (8 bytes)
# - `s` = char (1 byte), `Ns` = N chars
# - `>` = big-endian byte order
# %%
# Show message formats (loaded from utils/itch_message_specs.py)
print_message_formats()
# %% [markdown]
# ## 2. Binary Parsing Example
#
# Let's demonstrate how binary parsing works by creating and parsing a sample Add Order message.
# %%
# Create a sample Add Order message to demonstrate parsing
# Format: >HH6sQsI8sI (big-endian)
sample_add_order = struct.pack(
">HH6sQsI8sI",
1234, # stock_locate
5678, # tracking_number
b"\x00\x00\x00\x00\x00\x01", # timestamp (1 nanosecond)
9876543210, # order_reference_number
b"B", # buy_sell_indicator
100, # shares
b"AAPL ", # stock (padded to 8 chars)
1500000, # price (150.0000 in price4 format)
)
print(f"Raw Add Order message ({len(sample_add_order)} bytes):")
print(f" Hex: {sample_add_order.hex()}")
# %%
# Parse the binary data using our struct format
parsed = struct.unpack(FMT_DICT["A"], sample_add_order)
add_order = NT_DICT["A"]._make(parsed)
print("Parsed Add Order Message:")
print("-" * 40)
for field, value in add_order._asdict().items():
if isinstance(value, bytes):
value = value.decode("ascii").strip()
print(f" {field:25}: {value}")
# %%
# Apply conversions using helper functions from utils.itch_message_specs
ts_ns = parse_timestamp(add_order.timestamp)
price = parse_price4(add_order.price)
print(f"Timestamp: {ts_ns:,} nanoseconds = {ts_ns / 1e9:.9f} seconds after midnight")
print(f"Price: ${price:.4f}")
# %% [markdown]
# ## 3. Loading Pre-Parsed ITCH Data
#
# If you've already parsed ITCH data (using the Rust parser or Python parser), you can load
# the pre-parsed messages directly. This is the recommended approach for analysis.
# %%
# Check what data is available locally
print("ITCH Data Pipeline Status:")
print("-" * 50)
# Step 1: Raw binary from download
raw_files = []
if ITCH_RAW_DIR.exists():
raw_files = list(ITCH_RAW_DIR.glob("*.gz")) + list(ITCH_RAW_DIR.glob("*.bin"))
print(f"Raw binary files: {len(raw_files)}")
for f in raw_files:
print(f" {f.name} ({f.stat().st_size / 1e9:.2f} GB)")
# Step 2: Parsed messages (single uppercase letter = message type)
parsed_types = (
[
d
for d in sorted(MESSAGE_DIR.iterdir())
if d.is_dir() and len(d.name) == 1 and d.name.isupper()
]
if MESSAGE_DIR.exists()
else []
)
parsed_with_data = [d for d in parsed_types if list(d.glob("*.parquet"))]
print(f"Parsed message types: {len(parsed_with_data)}")
for msg_dir in parsed_with_data:
name = MESSAGE_SPECS.get(msg_dir.name, {}).get("name", "Unknown")
n_files = len(list(msg_dir.glob("*.parquet")))
print(f" {msg_dir.name} ({name}): {n_files} files")
# %%
# Validate: at minimum we need parsed data to continue
trade_dir = MESSAGE_DIR / "P"
assert trade_dir.exists() and list(trade_dir.glob("*.parquet")), (
f"No parsed ITCH data at {MESSAGE_DIR}.\n"
"To set up the data pipeline:\n"
" 1. Download raw data: uv run python data/equities/market/microstructure/nasdaq_itch_download.py\n"
" 2. Parse (this notebook, Section 4) or use Rust parser (Section 6)\n"
" 3. Parsed messages go to: data/equities/market/microstructure/nasdaq_itch/messages/"
)
trades = pl.read_parquet(trade_dir / "*.parquet")
print(f"\nLoaded {len(trades):,} trade messages")
print(f"Columns: {trades.columns}")
# %% [markdown]
# ## 4. Full Parser Implementation
#
# This Python parser is for **educational purposes**. For production use with large files,
# use the Rust parser (see Section 6) which provides order-of-magnitude speedups.
# %% [markdown]
# ### Parser Helpers
#
# We split the parser into three functions: `_read_frame` reads one binary message
# frame, `_decode_message` unpacks and converts it, and `parse_itch_file` orchestrates
# the loop with buffered Parquet writes.
# %%
def _read_frame(f, pbar) -> tuple[str, bytes] | None:
"""Read one ITCH message frame: 2-byte length + 1-byte type + payload.
Returns (msg_type, payload) on success, or None on EOF/truncation.
"""
# 2-byte big-endian length prefix (message size including type byte)
length_bytes = f.read(2)
if len(length_bytes) < 2:
return None
pbar.update(2)
msg_size = int.from_bytes(length_bytes, "big")
# 1-byte message type
msg_type_byte = f.read(1)
if len(msg_type_byte) < 1:
print(f"\nWarning: Truncated message at byte {f.tell()}, expected type byte")
return None
pbar.update(1)
msg_type = msg_type_byte.decode("ascii")
# Payload (msg_size includes type byte, so payload is msg_size - 1)
payload = f.read(msg_size - 1)
if len(payload) < msg_size - 1:
print(f"\nWarning: Truncated payload for message type {msg_type}")
return None
pbar.update(msg_size - 1)
return msg_type, payload
# %% [markdown]
# Decode binary payload into a Python dict, converting raw timestamp bytes
# to nanosecond integers and byte strings to stripped ASCII.
# %%
def _decode_message(msg_type: str, payload: bytes) -> dict | None:
"""Unpack binary payload into a dict, converting timestamps and strings.
Returns parsed message dict, or None on struct error.
"""
try:
parsed = struct.unpack(FMT_DICT[msg_type], payload)
msg = NT_DICT[msg_type]._make(parsed)._asdict()
except struct.error:
return None
# Convert timestamp: nanoseconds since midnight
if "timestamp" in msg:
msg["timestamp"] = int.from_bytes(msg["timestamp"], "big")
# Decode string fields
for field, value in msg.items():
if isinstance(value, bytes):
msg[field] = value.decode("ascii").strip()
return msg
# %% [markdown]
# The main parser reads the binary file sequentially, buffering decoded messages
# and flushing to Parquet periodically to bound memory usage.
# %% — single function body, helpers already extracted
def parse_itch_file(
itch_file: Path,
trading_day: date,
output_dir: Path,
max_buffered_messages: int = 10_000_000,
max_messages: int | None = None,
) -> dict[str, int]:
"""Parse ITCH binary file and store messages as Parquet.
Args:
itch_file: Path to binary ITCH file (.bin, not .gz).
trading_day: Trading date for timestamp construction.
output_dir: Directory for Parquet output (one subdir per message type).
max_buffered_messages: Flush threshold (total buffered messages).
max_messages: Optional limit for testing.
Returns:
Dictionary with message type counts.
"""
# Midnight timestamp for the trading day (ITCH timestamps are nanoseconds offset)
base_ts = datetime(trading_day.year, trading_day.month, trading_day.day)
file_counters: dict[str, int] = defaultdict(int)
buffers = defaultdict(list)
counts = Counter()
file_size = itch_file.stat().st_size
start_time = time()
with (
itch_file.open("rb") as f,
tqdm(total=file_size, desc="Parsing ITCH", unit="B", unit_scale=True) as pbar,
):
while True:
if max_messages and sum(counts.values()) >= max_messages:
print(f"\nLimit reached: {max_messages:,} messages")
break
frame = _read_frame(f, pbar)
if frame is None:
break
msg_type, payload = frame
counts[msg_type] += 1
if msg_type not in FMT_DICT:
continue
msg = _decode_message(msg_type, payload)
if msg is None:
continue
# Check for end of messages
if msg_type == "S" and msg.get("event_code") == "C":
print("\nEnd of Messages")
flush_to_parquet(buffers, output_dir, base_ts, file_counters)
break
buffers[msg_type].append(msg)
# Periodic flush
if sum(len(v) for v in buffers.values()) >= max_buffered_messages:
flush_to_parquet(buffers, output_dir, base_ts, file_counters)
# Final flush
if any(buffers.values()):
flush_to_parquet(buffers, output_dir, base_ts, file_counters)
elapsed = time() - start_time
total = sum(counts.values())
print(f"Parsed {total:,} messages in {elapsed:.1f}s ({total / elapsed:,.0f} msg/s)")
return dict(counts)
# %%
# Locate ITCH data file (skip if SKIP_PARSING is set)
if SKIP_PARSING:
print("SKIP_PARSING=True: skipping ITCH binary parsing (uses pre-parsed data)")
itch_file = None
counts = {}
else:
# Clear any existing parsed data to avoid schema conflicts
# (Different parser versions may produce different schemas)
# Set ITCH_KEEP_EXISTING=1 to skip cleanup and use existing data
clear_existing = os.environ.get("ITCH_KEEP_EXISTING", "0") != "1"
if clear_existing and MESSAGE_DIR.exists() and list(MESSAGE_DIR.glob("*/part-*.parquet")):
print(f"Clearing existing parsed data in {MESSAGE_DIR}")
print(" (Set ITCH_KEEP_EXISTING=1 to keep existing data)")
shutil.rmtree(MESSAGE_DIR)
MESSAGE_DIR.mkdir(parents=True, exist_ok=True)
# Find ITCH file (compressed or uncompressed)
gz_files = list(ITCH_RAW_DIR.glob("*.gz")) if ITCH_RAW_DIR.exists() else []
bin_files = list(ITCH_RAW_DIR.glob("*.bin")) if ITCH_RAW_DIR.exists() else []
if not gz_files and not bin_files:
raise FileNotFoundError(
f"No raw ITCH binary found at {ITCH_RAW_DIR}.\n"
"Download first:\n"
" uv run python data/equities/market/microstructure/nasdaq_itch_download.py"
)
# %%
# Decompress if needed and extract trading date
if not SKIP_PARSING and (gz_files or bin_files):
# Prefer uncompressed, otherwise decompress
if bin_files:
itch_file = bin_files[0]
print(f"Found uncompressed: {itch_file.name}")
else:
gz_file = gz_files[0]
itch_file = gz_file.with_suffix(".bin")
if not itch_file.exists():
print(f"Decompressing {gz_file.name}...")
with gzip.open(gz_file, "rb") as f_in, open(itch_file, "wb") as f_out:
shutil.copyfileobj(f_in, f_out)
print(f"Created: {itch_file.name} ({itch_file.stat().st_size / 1e9:.1f} GB)")
else:
print(f"Found: {itch_file.name}")
# Extract trading date from filename (format: MMDDYYYY.NASDAQ_ITCH50.bin)
date_str = itch_file.stem.split(".")[0]
if len(date_str) == 8 and date_str.isdigit():
trading_day = date(int(date_str[4:8]), int(date_str[:2]), int(date_str[2:4]))
else:
trading_day = date(2020, 1, 30) # Fallback
print(f"Trading day: {trading_day}")
# Parse ITCH file (full-day parse: ~22 min on the reference machine)
if itch_file and itch_file.exists():
counts = parse_itch_file(
itch_file=itch_file,
trading_day=trading_day,
output_dir=MESSAGE_DIR,
# max_messages=1_000_000, # Remove this line for full parse
)
print("\nMessage counts:")
for msg_type, count in sorted(counts.items(), key=lambda x: -x[1]):
name = MESSAGE_SPECS.get(msg_type, {}).get("name", "Unknown")
print(f" {msg_type} ({name:25}): {count:>10,}")
# %% [markdown]
# ## 5. Message Type Analysis
#
# After parsing, we can analyze message distributions.
# %%
# Message type distribution — use lazy scan to count without loading all data
print("Parsed Message Types:")
print("-" * 50)
for msg_dir in sorted(MESSAGE_DIR.iterdir()):
if (
msg_dir.is_dir()
and len(msg_dir.name) == 1
and msg_dir.name.isupper()
and list(msg_dir.glob("*.parquet"))
):
count = pl.scan_parquet(msg_dir / "*.parquet").select(pl.len()).collect().item()
name = MESSAGE_SPECS.get(msg_dir.name, {}).get("name", "Unknown")
print(f" {msg_dir.name} ({name:25}): {count:>12,} messages")
# %%
# Schema compatibility check — verify we can read each message type
print("Schema Compatibility Check:")
print("-" * 50)
for msg_dir in sorted(MESSAGE_DIR.iterdir()):
if (
msg_dir.is_dir()
and len(msg_dir.name) == 1
and msg_dir.name.isupper()
and list(msg_dir.glob("*.parquet"))
):
try:
sample = pl.scan_parquet(msg_dir / "*.parquet").head(5).collect()
name = MESSAGE_SPECS.get(msg_dir.name, {}).get("name", "Unknown")
print(f" [OK] {msg_dir.name} ({name:25}): cols={list(sample.columns)[:4]}...")
except Exception as e:
print(f" [FAIL] {msg_dir.name}: {e}")
# %% [markdown]
# ## 6. Production Parsing with Rust
#
# The Python parser above is educational but slow for full-day files.
# For production use, we provide a **Rust parser** that is an order of magnitude faster.
#
# **Repository**: [github.com/ml4t/itch-parser](https://github.com/ml4t/itch-parser)
#
# ### Performance Characteristics
#
# | Aspect | Python | Rust |
# |--------|--------|------|
# | Speed | Baseline | **10-20× faster** |
# | Memory | High (buffers in RAM) | Low (streaming) |
# | Use case | Learning, debugging | Production pipelines |
#
# Actual speedups depend on disk I/O and CPU. The Rust parser uses memory-mapped
# I/O and zero-copy parsing, which provides substantial gains on modern hardware.
#
# ### Installation
#
# ```bash
# # Clone the repository
# git clone https://github.com/ml4t/itch-parser.git
# cd itch-parser
#
# # Build release binary
# cargo build --release
# ```
#
# ### Usage
#
# ```bash
# # Parse ITCH file (works with .gz or uncompressed)
# ./target/release/itch_parser <input_file> <output_dir> <MMDDYYYY>
#
# # Example
# ./target/release/itch_parser data/01302020.NASDAQ_ITCH50.gz ./messages 01302020
# ```
#
# Output is identical Parquet files partitioned by message type, compatible with
# the Python code in this notebook and downstream analysis.
#
# ### When to Use Which
#
# | Use Case | Recommendation |
# |----------|---------------|
# | Learning the protocol | Python (this notebook) |
# | Debugging parse issues | Python |
# | Processing a single day | Either |
# | Multi-day backtesting | **Rust** |
# | Production pipeline | **Rust** |
# %% [markdown]
# ## Key Takeaways
#
# 1. **ITCH Protocol**: Binary message-by-order format with nanosecond precision
# 2. **Message Types**: A/F (add), E/C (execute), X (cancel), D (delete), U (replace)
# 3. **Price Format**: Integers with 4 implied decimals (150.0000 → 1500000)
# 4. **Parser Choice**: Python for learning, Rust for production (20× faster)
#
# ### Next Steps
#
# - **Order Book Reconstruction**: `02_itch_lob_reconstruction`
# - **Trading Activity Overview**: `05_itch_trading_activity` (includes E/C enrichment)
#
# ---
#
# ## Reference
#
# Bouchaud, J.-P., Bonart, J., Donier, J., & Gould, M. (2018).
# *Trades, Quotes and Prices: Financial Markets Under the Microscope*.
# Cambridge University Press.
# [https://doi.org/10.1017/9781009028943](https://doi.org/10.1017/9781009028943)
03_market_microstructure/02_itch_lob_reconstruction.ipynb
+1237
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/02_itch_lob_reconstruction.py
+528
View File
@@ -0,0 +1,528 @@
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Order Book Reconstruction from NASDAQ ITCH Messages
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Reconstruct the **limit order book (LOB)** for a single symbol-day from NASDAQ
# ITCH message-by-order (MBO) data, producing a one-second-resolution top-of-book
# snapshot series (best bid/ask price and size) with per-second order-flow
# imbalance (OFI). The reconstruction tracks full per-order book state internally;
# the emitted snapshots record the best bid and ask.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Process ITCH `A`/`F`/`D`/`X`/`E`/`C`/`U` messages into book state with correct
# handling of Replace (`U`) chains and order-pool tracking.
# - Validate reconstruction quality via spread positivity (>99% non-crossed).
# - Generate per-second OFI as the building block for short-horizon flow signals.
# - Choose between order-by-order tracking and price-level aggregation, and
# understand the trade-offs.
#
# ## Book reference
#
# Section §3.3, *From Raw Messages to the Limit Order Book*.
#
# ## Prerequisites
#
# - Parsed ITCH message parquets at `data/equities/market/microstructure/nasdaq_itch/messages/`
# (output of `01_itch_parser` or the Rust parser).
# - Familiarity with §3.2 (data feed taxonomy) and §3.3 (LOB reconstruction
# algorithm).
#
# **Output**: per-second LOB snapshots saved to
# `output/ch03/nasdaq_itch/order_book/{SYMBOL}/lob_snapshots.parquet`.
# %% [markdown]
# ## Setup
# %%
"""Order Book Reconstruction from NASDAQ ITCH Messages — build limit order book from MBO data."""
import os
import warnings
from datetime import datetime
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
from limit_orderbook import (
get_stock_locate_mapping,
load_itch_messages,
reconstruct_lob_with_ofi,
)
from data.equities.loader import load_nasdaq_itch
from utils.paths import get_output_dir
# %%
# Input: Pre-parsed ITCH messages from canonical data location
ITCH_DIR = load_nasdaq_itch(get_base_path=True)
# Output: LOB snapshots organized by symbol
OUTPUT_DIR = get_output_dir(3, "nasdaq_itch") / "order_book"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
print(f"Input messages: {ITCH_DIR}")
print(f"Output directory: {OUTPUT_DIR}")
# %% tags=["parameters"]
# Configuration — Papermill injects overrides for CI
SYMBOL = "AAPL"
TRADING_DATE = "2020-01-30"
START_TIME = "09:30:00"
END_TIME = "16:00:00"
MAX_MESSAGES = 0 # 0 = all messages
# %%
# Symbol can be overridden via environment variable for batch processing
SYMBOL = os.environ.get("ITCH_SYMBOL", SYMBOL)
# Normalize MAX_MESSAGES: 0 means no limit
if MAX_MESSAGES == 0:
MAX_MESSAGES = None
# %%
# Validate parsed ITCH data — produced by 01_itch_parser or Rust parser
assert ITCH_DIR.exists(), (
f"Parsed ITCH data not found at {ITCH_DIR}.\n"
"Run the ITCH pipeline first:\n"
" 1. Download: uv run python data/equities/market/microstructure/nasdaq_itch_download.py\n"
" 2. Parse: Run 01_itch_parser.py (Section 4) or Rust parser (Section 6)"
)
msg_types = sorted([d.name for d in ITCH_DIR.iterdir() if d.is_dir()])
assert len(msg_types) > 0, f"No message types in {ITCH_DIR} — run 01_itch_parser first"
print(f"Available message types: {msg_types}")
# %% [markdown]
# ## 1. Understanding ITCH Message Types
#
# NASDAQ ITCH 5.0 provides message-by-order (MBO) data with these key types:
#
# | Type | Name | Description |
# |------|------|-------------|
# | **A** | Add Order | New limit order enters the book |
# | **F** | Add Order (MPID) | Same as A, with market participant ID |
# | **E** | Order Executed | Partial/full execution |
# | **C** | Order Executed w/Price | Execution at different price (hidden orders) |
# | **X** | Order Cancel | Partial cancellation |
# | **D** | Order Delete | Full removal from book |
# | **U** | Order Replace | Modify price/size (cancel + add) |
# | **P** | Trade | Non-displayed execution |
#
# **Price format**: Integer with 4 implied decimal places (e.g., 3212000 = $321.20)
#
# **Timezone note**: ITCH timestamps are nanoseconds since midnight in US/Eastern (exchange local time).
# The data is timezone-naive; convert to America/New_York before cross-source joins.
# %% [markdown]
# ## 2. Load Messages for Target Symbol
#
# We load all message types needed for LOB reconstruction and **pre-join** them
# to attach price and side information to D/X/E messages.
# %%
# %%
# Get the stock_locate ID for our symbol from R messages
stock_map = get_stock_locate_mapping(ITCH_DIR)
assert SYMBOL in stock_map, f"Symbol {SYMBOL} not found in stock directory"
symbol_locate = stock_map[SYMBOL]
print(f"Symbol {SYMBOL} has stock_locate = {symbol_locate}")
# %%
# Add orders (A and F types) - have 'stock' column
add_a = load_itch_messages(ITCH_DIR, "A", symbol=SYMBOL, max_messages=MAX_MESSAGES)
add_f = load_itch_messages(ITCH_DIR, "F", symbol=SYMBOL, max_messages=MAX_MESSAGES)
# Combine A and F (select common columns)
common_cols = [
"stock_locate",
"tracking_number",
"timestamp",
"order_reference_number",
"buy_sell_indicator",
"shares",
"stock",
"price",
]
add_orders = pl.concat([add_a.select(common_cols), add_f.select(common_cols)])
# D, X, E, C, U messages don't have 'stock' column - use stock_locate filtering
deletes = load_itch_messages(ITCH_DIR, "D", stock_locate=symbol_locate, max_messages=MAX_MESSAGES)
cancels = load_itch_messages(ITCH_DIR, "X", stock_locate=symbol_locate, max_messages=MAX_MESSAGES)
executions = load_itch_messages(
ITCH_DIR, "E", stock_locate=symbol_locate, max_messages=MAX_MESSAGES
)
executions_c = load_itch_messages(
ITCH_DIR, "C", stock_locate=symbol_locate, max_messages=MAX_MESSAGES
)
replaces = load_itch_messages(ITCH_DIR, "U", stock_locate=symbol_locate, max_messages=MAX_MESSAGES)
# P messages (trades) have 'stock' column
trades = load_itch_messages(ITCH_DIR, "P", symbol=SYMBOL, max_messages=MAX_MESSAGES)
# %%
# =========================================================================
# CRITICAL: Pre-join D/X/E messages with Add orders to get price and side
# =========================================================================
# This is essential for correct LOB reconstruction. D/X/E messages only have
# order_reference_number - they need to be joined with Add orders to get
# the price and side for each order.
order_cols = ["order_reference_number", "buy_sell_indicator", "price", "shares"]
orders_lookup = add_orders.select(order_cols).rename({"shares": "original_shares"})
# =========================================================================
# IMPORTANT: Do NOT filter D/X/E/C messages based on order lookup!
# =========================================================================
# Orders can be created by U (Replace) messages, not just A/F. The chain:
# A → U → U → E → D
# means E and D reference an order created by U, which isn't in orders_lookup.
# We look up order info from submitted_orders during reconstruction anyway.
# The join here is ONLY for backward compatibility with code that expects
# these columns - but we DON'T filter based on join results.
#
# Note: The original pre-join approach assumed all orders start with A/F,
# which is wrong for ~95% of replace chains.
# %%
# Join D messages (but DON'T filter - order may be from U chain)
if len(deletes) > 0:
deletes = deletes.join(
orders_lookup,
on="order_reference_number",
how="left",
)
# NOT filtered - order info looked up from pool during reconstruction
# Join X messages (but DON'T filter)
if len(cancels) > 0:
cancels = cancels.join(
orders_lookup,
on="order_reference_number",
how="left",
)
# NOT filtered
# Join E messages (but DON'T filter)
if len(executions) > 0:
executions = executions.join(
orders_lookup,
on="order_reference_number",
how="left",
)
# NOT filtered
# %%
# Join C messages (but DON'T filter)
if executions_c is not None and len(executions_c) > 0:
executions_c = executions_c.join(
orders_lookup,
on="order_reference_number",
how="left",
)
# NOT filtered
# U (Replace) messages: DON'T filter on join!
# Replace chains can be: A → U → U → U (order created by U, then replaced again)
# The original_order_reference_number may reference an order created by a previous U,
# not an A/F add. We look up the original order from the pool during reconstruction.
if replaces is not None and len(replaces) > 0:
# Join to get original_side for orders that started with A/F
replaces = replaces.join(
orders_lookup.rename(
{
"order_reference_number": "original_order_reference_number",
"price": "original_price",
"buy_sell_indicator": "original_side",
"original_shares": "replaced_shares",
}
),
on="original_order_reference_number",
how="left",
)
# Note: Do NOT filter on original_price.is_not_null() - many U messages
# reference orders created by other U messages, not A/F. The side is
# looked up from the order pool during reconstruction.
# %%
print(f"\nSymbol: {SYMBOL}")
print(f"Trading Date: {TRADING_DATE}")
print("\nMessage counts:")
print(f" Add orders (A+F): {len(add_orders):,}")
print(f" Deletes (D): {len(deletes):,}")
print(f" Cancels (X): {len(cancels):,}")
print(f" Executions (E): {len(executions):,}")
print(f" Replaces (U): {len(replaces) if replaces is not None else 0:,}")
print(f" Trades (P): {len(trades):,}")
# %% [markdown]
# Inspect the structure of `Add` messages — `order_reference_number` is the join
# key that lets later `D`/`X`/`E` messages locate the price level and side they
# affect.
# %%
add_orders.head(5)
# %%
print(f"Price range: ${add_orders['price'].min():.2f} - ${add_orders['price'].max():.2f}")
# %% [markdown]
# ## 3. Order Book Reconstruction Algorithm
#
# ### Why Price-Level Aggregation?
#
# We use **price-level aggregation** rather than order-by-order tracking:
#
# ```
# book = {
# "B": {price1: total_shares, price2: total_shares, ...}, # Bids
# "S": {price1: total_shares, price2: total_shares, ...}, # Asks
# }
# ```
#
# **Processing rules for each message type**:
# - **A/F (Add)**: Add shares to book at price level
# - **D (Delete)**: Subtract original order's shares from price level
# - **X (Cancel)**: Subtract cancelled shares from price level
# - **E (Execute)**: Subtract executed shares from price level
# - **U (Replace)**: Subtract original shares at old price, add new shares at new price
#
# This approach is more robust than order-by-order tracking because it doesn't
# require tracking individual order IDs through their lifecycle.
#
# **Note**: The `reconstruct_lob_with_ofi` function is imported from `limit_orderbook` to enable
# code reuse with `14_itch_bar_sampling` (Lee-Ready trade classification).
# %% [markdown]
# ## 4. Run Reconstruction
# %%
# Use configured time window (defaults to full RTH: 09:30-16:00)
# Note: We load ALL messages from start of day up to end_time because
# delete/execute messages reference orders placed hours earlier in pre-market.
start_time = datetime.strptime(f"{TRADING_DATE} {START_TIME}", "%Y-%m-%d %H:%M:%S")
end_time = datetime.strptime(f"{TRADING_DATE} {END_TIME}", "%Y-%m-%d %H:%M:%S")
# CRITICAL: Include ALL messages from start of day up to end_time
# Delete/Execute messages at 9:31 reference Add orders from 4:00 AM pre-market.
# Without the full message history, we can't track remaining shares correctly.
#
# The order pool is built incrementally:
# - 4:00 AM: Add order 12345, 500 shares
# - 5:00 AM: Execute 200 shares (remaining = 300)
# - 9:31 AM: Delete order 12345 (remove 300 shares, not 500!)
#
# Snapshots are only generated after start_time.
add_all = add_orders.filter(pl.col("timestamp") <= end_time)
del_all = deletes.filter(pl.col("timestamp") <= end_time)
can_all = cancels.filter(pl.col("timestamp") <= end_time)
exec_all = executions.filter(pl.col("timestamp") <= end_time)
exec_c_all = (
executions_c.filter(pl.col("timestamp") <= end_time) if executions_c is not None else None
)
rep_all = replaces.filter(pl.col("timestamp") <= end_time) if replaces is not None else None
print(f"Messages for LOB reconstruction (up to {end_time.time()}):")
print(f" Add orders: {len(add_all):,}")
print(f" Deletes: {len(del_all):,}")
print(f" Cancels: {len(can_all):,}")
print(f" Executions (E): {len(exec_all):,}")
print(f" Executions (C): {len(exec_c_all) if exec_c_all is not None else 0:,}")
print(f" Replaces: {len(rep_all) if rep_all is not None else 0:,}")
# %%
# Reconstruct LOB with OFI using Numba-accelerated function
# This computes Order Flow Imbalance during the reconstruction pass:
# OFI = (Bid Adds - Bid Removes) - (Ask Adds - Ask Removes)
lob = reconstruct_lob_with_ofi(
add_all,
del_all,
can_all,
exec_all,
executions_c=exec_c_all,
replaces=rep_all,
n_levels=5,
snapshot_freq="1s",
)
# Filter to RTH snapshots only (reconstruction processes all messages from start of day)
if len(lob) > 0:
lob = lob.filter(pl.col("timestamp") >= start_time)
assert len(lob) > 0, (
f"LOB reconstruction returned 0 snapshots for {SYMBOL} on {TRADING_DATE}. "
"Check that the trading date has parsed ITCH messages on disk."
)
# %%
print(f"LOB snapshots: {len(lob):,}")
# %%
lob.head()
# %% [markdown]
# ### Spread validity
#
# A correctly reconstructed book has positive spreads almost everywhere — crossed
# quotes (`bid > ask`) signal lost messages or order-pool errors.
# %%
valid_count = (lob["spread"] > 0).sum()
crossed_count = (lob["spread"] < 0).sum()
print(f"Valid spreads (spread > 0): {valid_count:,} ({valid_count / len(lob) * 100:.1f}%)")
print(f"Crossed quotes (spread < 0): {crossed_count:,} ({crossed_count / len(lob) * 100:.1f}%)")
# %% [markdown]
# ### Order Flow Imbalance per second
#
# OFI = (bid adds − bid removes) − (ask adds − ask removes), aggregated to one
# second. Cumulative OFI tracks net buying/selling pressure within the trading
# day.
# %%
lob.select(
pl.col("ofi").mean().alias("mean"),
pl.col("ofi").std().alias("std"),
pl.col("ofi").quantile(0.5).alias("median"),
pl.col("ofi").min().alias("min"),
pl.col("ofi").max().alias("max"),
)
# %% [markdown]
# ## 5. Visualize Order Book Dynamics
#
# For detailed spread and imbalance analysis over time, see **`03_itch_lob_analysis`**.
# This section focuses on market depth which shows the reconstructed book structure.
# %%
# Create output directory for symbol
symbol_dir = OUTPUT_DIR / SYMBOL
symbol_dir.mkdir(parents=True, exist_ok=True)
# %% [markdown]
# ### 5.1 Market Depth
# %%
lob_pd = lob.to_pandas().set_index("timestamp")
ofi_1m = (
lob_pd[["ofi", "bid_size_0", "ask_size_0"]]
.resample("1min")
.agg({"ofi": "sum", "bid_size_0": "mean", "ask_size_0": "mean"})
)
fig, axes = plt.subplots(2, 1, figsize=(14, 8), sharex=True)
ax1 = axes[0]
# Per-minute OFI (signed). Positive values indicate net buying pressure
# within the minute, negative net selling. Per-minute is more readable
# than the cumulative trace because the opening cross dominates the
# cumulant's scale.
ofi_series = ofi_1m["ofi"].fillna(0)
ofi_series.plot(ax=ax1, color="#1E3A5F", linewidth=1.0, label="1-min OFI")
ax1.axhline(0, color="gray", linestyle="--", linewidth=0.5)
# Y-range from the 1st/99th percentiles to keep extreme minutes from
# squashing the rest of the session.
_vals = ofi_series.to_numpy()
if len(_vals):
_lo, _hi = np.nanpercentile(_vals, [1, 99])
_pad = max(abs(_lo), abs(_hi)) * 0.15
ax1.set_ylim(_lo - _pad, _hi + _pad)
ax1.set_title(f"{SYMBOL} per-minute order-flow imbalance (signed)")
ax1.set_ylabel("OFI (shares per minute)")
ax1.legend(loc="upper right")
ax2 = axes[1]
ofi_1m["bid_size_0"].plot(ax=ax2, label="Bid Depth (L0)", color="green", alpha=0.7)
ofi_1m["ask_size_0"].plot(ax=ax2, label="Ask Depth (L0)", color="red", alpha=0.7)
ax2.set_title(f"{SYMBOL} top-of-book depth — bid and ask sizes evolve through the session")
ax2.set_ylabel("Shares")
ax2.legend()
plt.tight_layout()
plt.show()
# %% [markdown]
# For depth imbalance analysis and return predictability, see **`03_itch_lob_analysis`** Section 7.
# %% [markdown]
# ## 6. Save Results
# %%
output_file = symbol_dir / "lob_snapshots.parquet"
lob.write_parquet(output_file)
print(f"Saved LOB snapshots to: {output_file}")
print(f"Shape: {lob.shape}")
# %% [markdown]
# ## Key Takeaways
#
# ### Order Pool Tracking is Essential
#
# **Critical insight**: Proper order book reconstruction requires tracking the
# *remaining* shares per order, not just the original order data.
#
# The key challenge is handling message chains correctly:
#
# ```
# A (Add 500 shares) → E (Execute 100) → E (Execute 200) → D (Delete remaining)
# ```
#
# After two executions, 200 shares remain. The Delete message must remove 200
# shares, not the original 500. This requires maintaining an order pool that
# tracks remaining shares after each execution.
#
# ### Replace (U) Message Chains
#
# Replace messages create new order references: `A → U → U → U`
#
# When an order is replaced, the *new* order_reference_number becomes canonical.
# Subsequent D/E/X messages reference orders created by U, not just A/F adds.
# This means ~95% of Replace messages would be lost if we only tracked A/F orders.
#
# ### Technical Summary
#
# 1. **Order pool tracking**: Track remaining shares per order_reference_number
# 2. **U creates new orders**: Replace messages spawn new order references
# 3. **C messages use original price**: Execute-with-price updates book at order's price
# 4. **P messages don't affect book**: Non-displayable/hidden order executions
#
# ### Next Steps
#
# - **`03_itch_lob_analysis`**: Spread dynamics, OFI predictability, liquidity spectrum
# - **Chapter 8**: Feature engineering using LOB metrics
# - **Chapter 19**: Price impact modeling using depth and imbalance
#
# ---
#
# ## References
#
# - Bouchaud, J.-P., Bonart, J., Donier, J., & Gould, M. (2018).
# *Trades, Quotes and Prices: Financial Markets Under the Microscope*.
# Cambridge University Press.
# [https://doi.org/10.1017/9781009028943](https://doi.org/10.1017/9781009028943)
#
# - Gould, M. D., Porter, M. A., Williams, S., McDonald, M., Fenn, D. J., & Howison, S. D. (2013).
# "Limit order books." *Quantitative Finance*, 13(11), 1709-1742.
# [https://doi.org/10.1080/14697688.2013.803148](https://doi.org/10.1080/14697688.2013.803148)
03_market_microstructure/03_itch_lob_analysis.ipynb
+2649
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/03_itch_lob_analysis.py
+1369
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/04_itch_order_lifecycle_analysis.ipynb
+2402
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/04_itch_order_lifecycle_analysis.py
+1287
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/05_itch_trading_activity.ipynb
+1669
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/05_itch_trading_activity.py
+770
View File
@@ -0,0 +1,770 @@
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.1
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Trading Activity Overview: NASDAQ Market Structure
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Provide the chapter's market-wide ITCH context: aggregate message-type
# composition, dollar-volume concentration across tickers, and the canonical
# enriched-trade parquet that `06_itch_intraday_patterns` and
# `07_itch_stylized_facts` reuse.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Tally ITCH message types across a multi-day sample and place add/cancel/
# execute frequencies in proportion.
# - Build a stock-attributed trade table by joining `E`/`C`/`X` messages to the
# `R`-based `stock_locate → stock` directory and to `A`/`F` add-order
# attributes (with `U` replace lineage).
# - Quantify dollar-volume concentration (Pareto-style) over the universe.
#
# ## Book reference
#
# Section §3.3, *From Raw Messages to the Limit Order Book* — the
# market-wide statistics paragraph cites this notebook.
#
# ## Prerequisites
#
# - Parsed ITCH message parquets at `data/equities/market/microstructure/nasdaq_itch/messages/`
# (output of `01_itch_parser` or the Rust parser).
# - Notebooks 06 and 07 read the canonical trade table this notebook writes
# under `output/ch03/nasdaq_itch/trading_activity/`.
#
# ---
# %% [markdown]
# ## 2. Setup and Configuration
# %%
"""Trading Activity Overview — high-level view of NASDAQ trading activity using TotalView-ITCH data."""
import warnings
from pathlib import Path
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import matplotlib.ticker as mtick
import numpy as np
import polars as pl
import pyarrow.dataset as ds
import seaborn as sns
from IPython.display import display
from data import load_nasdaq_itch
from utils.paths import get_output_dir
sns.set_style("whitegrid")
# %% tags=["parameters"]
# Production defaults
MAX_ROWS = 0
# %%
# Configuration - Unified output directory structure
# All ITCH-related outputs under a single chapter directory
NASDAQ_ITCH_OUTPUT = get_output_dir(3, "nasdaq_itch")
# Input: Parsed messages from canonical loader path
MESSAGE_DIR = load_nasdaq_itch(get_base_path=True)
# Output: This notebook's analysis outputs
# Trade summary is consumed by notebooks 06 and 07
OUTPUT_DIR = NASDAQ_ITCH_OUTPUT / "trading_activity"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
ROW_LIMIT = MAX_ROWS or None
print(f"Input directory (messages): {MESSAGE_DIR}")
print(f"Output directory (analysis): {OUTPUT_DIR}")
if not MESSAGE_DIR.exists():
print(f"\nWARNING: Message directory not found: {MESSAGE_DIR}")
print(" Run 01_itch_parser first to parse ITCH data.")
available = []
else:
available = sorted([d.name for d in MESSAGE_DIR.iterdir() if d.is_dir()])
print(f"\nAvailable message types: {available}")
# Check if we have message data to analyze
HAS_MESSAGE_DATA = len(available) > 0
if not HAS_MESSAGE_DATA:
raise RuntimeError(
"No parsed ITCH message data found.\n"
"This notebook requires output from 01_itch_parser (full parse ~60min).\n"
f"Expected directory: {MESSAGE_DIR}"
)
# %% [markdown]
# ## 3. Counting Messages by Type
#
# Each message type is stored in its own Parquet subdirectory. Let's see how many messages
# of each type exist in our dataset.
# %%
def count_parquet_rows(base_dir: Path) -> dict[str, int]:
"""
Count total rows (messages) in each Parquet subdirectory.
Args:
base_dir: Directory containing subfolders like A/, C/, E/, etc.
Returns:
Mapping from subfolder name (e.g. 'A') to total row count.
"""
message_counts = {}
for sub in sorted(base_dir.iterdir()):
if sub.is_dir():
try:
dset = ds.dataset(sub.as_posix(), format="parquet")
total_rows = sum(frag.metadata.num_rows for frag in dset.get_fragments())
message_counts[sub.name] = total_rows
except (OSError, FileNotFoundError) as e:
print(f"Error reading {sub.name}: {e}")
except Exception as e:
# Log unexpected errors for debugging
print(f"Unexpected error reading {sub.name}: {type(e).__name__}: {e}")
return message_counts
# %%
if HAS_MESSAGE_DATA:
message_summary = count_parquet_rows(MESSAGE_DIR)
# Convert to pandas for plotting (matplotlib expects pandas)
import pandas as pd
fig, ax = plt.subplots(figsize=(8, 5))
pd.Series(message_summary).sort_values().plot.barh(ax=ax)
ax.set_title("Message Counts by Type")
ax.set_xlabel("Number of Messages")
plt.tight_layout()
plt.show()
# Print summary
print("\nMessage Type Summary:")
for msg_type, count in sorted(message_summary.items(), key=lambda x: -x[1]):
print(f" {msg_type}: {count:>12,}")
# %% [markdown]
# ### Message Type Reference
#
# | Type | Name | Description |
# |------|------|-------------|
# | **A** | Add Order | New limit order enters the book |
# | **F** | Add Order (Attributed) | Same as A, with market participant ID |
# | **E** | Order Executed | Partial/full execution |
# | **C** | Order Executed w/Price | Execution at different price |
# | **X** | Order Cancel | Partial cancellation |
# | **D** | Order Delete | Full removal from book |
# | **P** | Trade | Non-displayed execution |
# | **Q** | Cross Trade | Opening/closing cross |
# %% [markdown]
# ## 4. Execution Attribution: Enriching E/C Messages
#
# E (Order Executed) and C (Order Executed with Price) messages contain `stock_locate`
# but not the stock symbol. We enrich them by joining to:
#
# 1. **R messages**: `stock_locate → stock` mapping (basic attribution)
# 2. **A/F messages**: `order_ref → (price, side)` (execution quality analysis)
#
# This enables filtering trades by stock symbol and analyzing execution quality.
#
# **Note**: This enrichment is performed here in notebook 05, not notebook 01.
# The enriched files are saved to `messages/enriched/` and reused by downstream
# notebooks (04, 06, 07) that need stock-level execution data.
# %%
def _build_stock_directory(message_dir: Path) -> pl.DataFrame | None:
"""Load stock_locate -> stock mapping from R (Stock Directory) messages."""
r_path = message_dir / "R"
if not r_path.exists():
print("No R (Stock Directory) messages found. Cannot enrich.")
return None
print("Building stock_locate -> stock mapping from R messages...")
stock_directory = (
pl.scan_parquet(r_path / "*.parquet").select("stock_locate", "stock").collect()
)
print(f" {len(stock_directory):,} stocks in directory")
return stock_directory
# %% [markdown]
# ### Build Order Attributes
# Map order references to their price, side, and stock from A/F messages.
# %%
def _build_order_attrs(message_dir: Path) -> pl.DataFrame | None:
"""Build order_ref -> (price, side, stock) mapping from A/F messages."""
print("Building order_ref -> attributes mapping from A/F messages...")
a_path = message_dir / "A"
if not a_path.exists():
print("No A (Add Order) messages found. Cannot enrich.")
return None
orders_a = pl.scan_parquet(a_path / "*.parquet").select(
"order_reference_number",
"price",
"buy_sell_indicator",
"stock",
)
# F messages have same structure plus attribution (MPID)
f_path = message_dir / "F"
if f_path.exists() and list(f_path.glob("*.parquet")):
orders_f = pl.scan_parquet(f_path / "*.parquet").select(
"order_reference_number",
"price",
"buy_sell_indicator",
"stock",
)
order_attrs = pl.concat([orders_a, orders_f]).collect()
else:
order_attrs = orders_a.collect()
print(f" {len(order_attrs):,} orders with attributes")
return order_attrs
# %% [markdown]
# ### Apply Order Replacements
# Incorporate U (Replace) messages so new order references inherit original attributes.
# %%
def _apply_replacements(message_dir: Path, order_attrs: pl.DataFrame) -> pl.DataFrame:
"""Incorporate U (Replace) messages so new_order_ref inherits original attributes."""
u_path = message_dir / "U"
if not (u_path.exists() and list(u_path.glob("*.parquet"))):
return order_attrs
print("Processing U (Replace) messages for order lineage...")
replacements = (
pl.scan_parquet(u_path / "*.parquet")
.select(
"original_order_reference_number",
"new_order_reference_number",
"price", # U messages have updated price
)
.collect()
)
# For each replacement, look up the original order's attributes
replaced = (
replacements.join(
order_attrs.select("order_reference_number", "buy_sell_indicator", "stock"),
left_on="original_order_reference_number",
right_on="order_reference_number",
how="left",
)
.select(
pl.col("new_order_reference_number").alias("order_reference_number"),
"price", # Use the NEW price from U message
"buy_sell_indicator",
"stock",
)
.drop_nulls()
)
print(f" {len(replaced):,} replacement orders tracked")
return pl.concat([order_attrs, replaced])
# %% [markdown]
# ### Enrich E Messages
# Add stock symbol and order attributes to Order Executed messages.
# %%
def _enrich_e_messages(
message_dir: Path, enriched_dir: Path, stock_directory: pl.DataFrame, order_attrs: pl.DataFrame
) -> int | None:
"""Enrich E (Order Executed) messages with stock and order attributes."""
e_path = message_dir / "E"
if not e_path.exists():
return None
print("Enriching E (Order Executed) messages...")
executions_e = pl.scan_parquet(e_path / "*.parquet").collect()
# Join to get stock from stock_locate
enriched_e = executions_e.join(stock_directory, on="stock_locate", how="left")
# Join to get order attributes (price, side)
enriched_e = enriched_e.join(
order_attrs.select(
"order_reference_number",
pl.col("price").alias("order_price"),
pl.col("buy_sell_indicator").alias("side"),
),
on="order_reference_number",
how="left",
)
# E messages execute at the order's limit price
enriched_e = enriched_e.with_columns(pl.col("order_price").alias("execution_price"))
enriched_e.write_parquet(enriched_dir / "E.parquet")
matched = enriched_e.filter(pl.col("stock").is_not_null()).height
count = len(enriched_e)
print(f" {count:,} executions enriched ({matched:,} with stock match)")
return count
# %% [markdown]
# ### Enrich C Messages
# Add stock symbol and order attributes to Order Executed with Price messages.
# %%
def _enrich_c_messages(
message_dir: Path, enriched_dir: Path, stock_directory: pl.DataFrame, order_attrs: pl.DataFrame
) -> int | None:
"""Enrich C (Order Executed with Price) messages — includes price improvement."""
c_path = message_dir / "C"
if not c_path.exists():
return None
print("Enriching C (Order Executed with Price) messages...")
executions_c = pl.scan_parquet(c_path / "*.parquet").collect()
enriched_c = executions_c.join(stock_directory, on="stock_locate", how="left")
enriched_c = enriched_c.join(
order_attrs.select(
"order_reference_number",
pl.col("price").alias("order_price"),
pl.col("buy_sell_indicator").alias("side"),
),
on="order_reference_number",
how="left",
)
# C messages have execution_price - can compute price improvement
enriched_c = enriched_c.with_columns(
(pl.col("execution_price").cast(pl.Int64) - pl.col("order_price").cast(pl.Int64)).alias(
"price_improvement_raw"
)
)
enriched_c.write_parquet(enriched_dir / "C.parquet")
count = len(enriched_c)
print(f" {count:,} executions with price enriched")
return count
# %% [markdown]
# ### Enrich X Messages
# Add stock symbol to Order Cancel messages.
# %%
def _enrich_x_messages(
message_dir: Path, enriched_dir: Path, stock_directory: pl.DataFrame
) -> int | None:
"""Enrich X (Order Cancel) messages with stock symbol."""
x_path = message_dir / "X"
if not x_path.exists():
return None
print("Enriching X (Order Cancel) messages...")
cancels = pl.scan_parquet(x_path / "*.parquet").collect()
enriched_x = cancels.join(stock_directory, on="stock_locate", how="left")
enriched_x.write_parquet(enriched_dir / "X.parquet")
count = len(enriched_x)
print(f" {count:,} cancellations enriched")
return count
# %% [markdown]
# ### Enrich Execution Messages
# Orchestrate enrichment of E/C/X messages with stock symbols and order attributes.
# %%
def enrich_execution_messages(message_dir: Path) -> dict[str, int]:
"""
Enrich E/C/X messages with stock symbol and order attributes.
Creates enriched Parquet files that enable:
- Filtering executions by stock symbol
- Execution quality analysis (fill price vs limit price)
- Fill rate analysis by order characteristics
Returns count of enriched messages by type.
"""
enriched_dir = message_dir / "enriched"
enriched_dir.mkdir(exist_ok=True)
stock_directory = _build_stock_directory(message_dir)
if stock_directory is None:
return {}
order_attrs = _build_order_attrs(message_dir)
if order_attrs is None:
return {}
order_attrs = _apply_replacements(message_dir, order_attrs)
counts = {}
for label, fn in [
("E", lambda: _enrich_e_messages(message_dir, enriched_dir, stock_directory, order_attrs)),
("C", lambda: _enrich_c_messages(message_dir, enriched_dir, stock_directory, order_attrs)),
("X", lambda: _enrich_x_messages(message_dir, enriched_dir, stock_directory)),
]:
result = fn()
if result is not None:
counts[label] = result
print(f"\nEnriched files saved to: {enriched_dir}")
return counts
# %%
# Run enrichment if parsed data exists and enriched files don't
if HAS_MESSAGE_DATA and (MESSAGE_DIR / "R").exists():
enriched_dir = MESSAGE_DIR / "enriched"
# Check if enrichment already done
if not (enriched_dir / "E.parquet").exists():
print("Running execution enrichment...")
enrichment_counts = enrich_execution_messages(MESSAGE_DIR)
print("\n=== Enrichment Summary ===")
for msg_type, count in enrichment_counts.items():
print(f" {msg_type}: {count:,} messages")
else:
print(f"Enriched files already exist in {enriched_dir}")
print(" Delete the directory and re-run to regenerate.")
# %% [markdown]
# ## 5. Trade Volume and Value by Ticker
#
# We analyze **executions** (trades) from message types that reflect actual trades:
# - `'C'`: Order Executed with Price
# - `'E'`: Order Executed
# - `'P'`: Trade (regular)
# - `'Q'`: Cross Trade
# %%
def _unify_columns(df: pl.DataFrame, msg_type: str) -> pl.DataFrame | None:
"""Normalize column names across ITCH message types to a common schema."""
cols = {c.lower(): c for c in df.columns}
# Unify 'shares' column
if "executed_shares" in cols:
df = df.with_columns(pl.col(cols["executed_shares"]).cast(pl.Float64).alias("shares"))
elif "shares" in cols:
df = df.with_columns(pl.col(cols["shares"]).cast(pl.Float64).alias("shares"))
else:
return None
# Unify 'price' column
if "execution_price" in cols:
df = df.with_columns(pl.col(cols["execution_price"]).cast(pl.Float64).alias("price"))
elif "cross_price" in cols:
df = df.with_columns(pl.col(cols["cross_price"]).cast(pl.Float64).alias("price"))
elif "price" in cols:
df = df.with_columns(pl.col(cols["price"]).cast(pl.Float64).alias("price"))
else:
return None
# Handle timestamp
if "timestamp" in cols:
df = df.with_columns(pl.col(cols["timestamp"]).alias("timestamp"))
# Handle ticker (may be 'stock' or 'ticker')
if "ticker" in cols:
df = df.with_columns(pl.col(cols["ticker"]).alias("ticker"))
elif "stock" in cols:
df = df.with_columns(pl.col(cols["stock"]).alias("ticker"))
else:
return None
df = df.with_columns(pl.lit(msg_type).alias("msg_type"))
keep_cols = ["timestamp", "ticker", "shares", "price", "msg_type"]
existing = [c for c in keep_cols if c in df.columns]
return df.select(existing)
# %% [markdown]
# ### Load Single Message Type
# Load one ITCH execution message type from enriched or raw parquet files.
# %%
def _load_single_msg_type(
base_dir: Path, msg_type: str, max_rows: int | None
) -> pl.DataFrame | None:
"""Load a single ITCH execution message type (C, E, P, or Q)."""
enriched_file = base_dir / "enriched" / f"{msg_type}.parquet"
msg_folder = base_dir / msg_type
try:
if enriched_file.exists():
lf = pl.scan_parquet(enriched_file)
if max_rows:
lf = lf.head(max_rows)
df = lf.collect()
elif msg_folder.is_dir():
lf = pl.scan_parquet(msg_folder / "*.parquet")
if max_rows:
lf = lf.head(max_rows)
df = lf.collect()
else:
return None
if len(df) == 0:
return None
return _unify_columns(df, msg_type)
except (OSError, FileNotFoundError, pl.exceptions.ComputeError) as e:
print(f"Error loading {msg_type}: {e}")
except Exception as e:
print(f"Unexpected error loading {msg_type}: {type(e).__name__}: {e}")
return None
# %% [markdown]
# ### Normalize Trades
# Concatenate trade DataFrames, clean tickers, normalize ITCH prices, and compute trade value.
# %%
def _normalize_trades(trades: list[pl.DataFrame], type_counts: dict[str, int]) -> pl.DataFrame:
"""Concat trade DataFrames, clean tickers, normalize prices, compute value."""
all_trades = pl.concat(trades)
all_trades = all_trades.drop_nulls(subset=["ticker", "shares", "price"])
# Strip ITCH ticker padding (8-char fixed width with trailing spaces)
all_trades = all_trades.with_columns(pl.col("ticker").str.strip_chars().alias("ticker"))
# Normalize prices from ITCH price4 format (divide by 10000)
# Only normalize if prices appear to be scaled integers (median > 10000)
median_price = all_trades.select(pl.col("price").median()).item()
if median_price is not None and median_price > 10000:
all_trades = all_trades.with_columns((pl.col("price") / 10000).alias("price"))
all_trades = all_trades.with_columns((pl.col("shares") * pl.col("price")).alias("value"))
# Print message type breakdown
if type_counts:
print("Execution message type breakdown:")
for mt, count in sorted(type_counts.items()):
print(f" {mt}: {count:>12,}")
return all_trades
# %% [markdown]
# ### Load Executions
# Load and combine C, E, P, Q execution messages into a unified trades DataFrame.
# %%
def load_executions(base_dir: Path, max_rows: int | None = None) -> pl.DataFrame:
"""
Load execution data from C, E, P, Q message types using Polars.
For E and C messages, prefers enriched files (with stock symbol).
Falls back to raw files for P/Q which already have stock.
Returns DataFrame with columns: timestamp, ticker, shares, price, value, msg_type
"""
trades = []
type_counts = {}
for msg_type in ["C", "E", "P", "Q"]:
df = _load_single_msg_type(base_dir, msg_type, max_rows)
if df is not None:
type_counts[msg_type] = len(df)
trades.append(df)
if not trades:
return pl.DataFrame(
schema={
"timestamp": pl.Datetime,
"ticker": pl.Utf8,
"shares": pl.Float64,
"price": pl.Float64,
"value": pl.Float64,
"msg_type": pl.Utf8,
}
)
return _normalize_trades(trades, type_counts)
# %%
if HAS_MESSAGE_DATA:
trade_df = load_executions(MESSAGE_DIR, max_rows=ROW_LIMIT)
print(f"\nLoaded {len(trade_df):,} trades total")
print(trade_df.schema)
# Price sanity check - ITCH prices may be scaled (often 1/10000)
# The parser should normalize to dollars; verify with sample prices
if "price" in trade_df.columns and len(trade_df) > 0:
price_stats = trade_df.select(
[
pl.col("price").min().alias("min"),
pl.col("price").median().alias("median"),
pl.col("price").max().alias("max"),
]
).row(0, named=True)
print("\nPrice sanity check (should be plausible dollar values):")
print(
f" Min: ${price_stats['min']:.2f}, Median: ${price_stats['median']:.2f}, Max: ${price_stats['max']:.2f}"
)
if price_stats["median"] > 100000:
print(" WARNING: Prices appear scaled - check parser normalization!")
# Timestamp type check
if "timestamp" in trade_df.columns and len(trade_df) > 0:
ts_dtype = trade_df.schema["timestamp"]
print(f"\nTimestamp dtype: {ts_dtype}")
# In Polars, check dtype by comparing base_type or string representation
is_datetime = (
ts_dtype.base_type() == pl.Datetime if hasattr(ts_dtype, "base_type") else False
)
if not is_datetime:
print(" WARNING: Timestamps may need conversion for time-based analysis")
print("\nSample trades:")
display(trade_df.head().to_pandas())
# %% [markdown]
# ### 4.1. Aggregate Trades by Ticker
#
# Summarize total shares and dollar value to see which tickers dominate activity.
# %%
if HAS_MESSAGE_DATA and len(trade_df) > 0:
trade_summary = trade_df.group_by("ticker").agg(
[
pl.col("shares").sum().alias("total_shares"),
pl.col("value").sum().alias("total_value"),
pl.col("shares").count().alias("trade_count"),
]
)
# Calculate shares
total_value = trade_summary.select(pl.col("total_value").sum()).item()
total_shares = trade_summary.select(pl.col("total_shares").sum()).item()
trade_summary = trade_summary.with_columns(
[
(pl.col("total_value") / total_value).alias("share_of_value"),
(pl.col("total_shares") / total_shares).alias("share_of_volume"),
]
)
trade_summary = trade_summary.sort("total_value", descending=True)
print("Top 10 Tickers by Dollar Volume:")
display(trade_summary.head(10).to_pandas())
# Concentration analysis
top_50 = trade_summary.head(50)
sum_top_50 = top_50.select(pl.col("share_of_value").sum()).item()
print(f"\nConcentration: Top 50 tickers account for {sum_top_50:.1%} of total dollar volume")
# %%
if HAS_MESSAGE_DATA and len(trade_df) > 0:
# Plot cumulative concentration
cum_val = trade_summary.select(pl.col("share_of_value").cum_sum()).to_series().to_numpy()
fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(np.arange(len(cum_val)), cum_val, linewidth=2)
ax.axhline(0.5, color="red", linestyle="--", alpha=0.7, label="50% of value")
ax.axhline(0.8, color="orange", linestyle="--", alpha=0.7, label="80% of value")
# Find how many tickers needed for 50% and 80%
n_50 = np.searchsorted(cum_val, 0.5) + 1
n_80 = np.searchsorted(cum_val, 0.8) + 1
ax.axvline(n_50, color="red", linestyle=":", alpha=0.5)
ax.axvline(n_80, color="orange", linestyle=":", alpha=0.5)
ax.set_title("Concentration of Traded Value Across Tickers")
ax.set_ylabel("Cumulative Share of Dollar Volume")
ax.set_xlabel("Ticker Rank by Value")
ax.yaxis.set_major_formatter(mtick.PercentFormatter(1.0))
ax.legend()
ax.set_xlim(0, min(500, len(cum_val)))
sns.despine()
plt.tight_layout()
plt.show()
print(f"Tickers needed for 50% of value: {n_50}")
print(f"Tickers needed for 80% of value: {n_80}")
# %% [markdown]
# **Key Insight**: A small fraction of tickers accounts for most of the traded value.
# This concentration is typical of equity markets - a few highly liquid names dominate activity.
# %% [markdown]
# ## Save Output for Downstream Notebooks
#
# Save the trade summary for use by `06_itch_intraday_patterns` and
# `07_itch_stylized_facts`.
# %%
if HAS_MESSAGE_DATA and len(trade_df) > 0:
# Save trade summary for downstream notebooks
output_path = OUTPUT_DIR / "trade_summary.parquet"
trade_summary.write_parquet(output_path)
print(f"Saved trade summary to {output_path}")
# Save canonical trades table for notebooks 06/07
# This is the single source of truth for trade extraction,
# using enriched E/C files where available
trades_path = OUTPUT_DIR / "trades.parquet"
trade_df.write_parquet(trades_path)
print(f"Saved canonical trades to {trades_path}")
print(f" {len(trade_df):,} trades across {trade_df['ticker'].n_unique():,} tickers")
# %% [markdown]
# ## Key Takeaways
#
# 1. **Message volume**: ITCH generates hundreds of millions of messages per day
# 2. **Concentration**: A small number of stocks dominate trading activity
# 3. **Top 50 stocks** account for nearly half of total dollar volume
#
# **Next**: See `06_itch_intraday_patterns` for time-of-day analysis.
#
# ---
#
# ## Reference
#
# Bouchaud, J.-P., Bonart, J., Donier, J., & Gould, M. (2018).
# *Trades, Quotes and Prices: Financial Markets Under the Microscope*.
# Cambridge University Press.
# [https://doi.org/10.1017/9781009028943](https://doi.org/10.1017/9781009028943)
03_market_microstructure/06_itch_intraday_patterns.ipynb
+919
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/06_itch_intraday_patterns.py
+431
View File
@@ -0,0 +1,431 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Intraday Patterns: Volume and Volatility Dynamics
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Quantify the intraday volume U-shape across high-, medium-, and low-liquidity
# NASDAQ tickers using ITCH-derived trade data, and produce the comparative
# 30-minute-resolution figures that §3.1 and §3.3 reference.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Resample tick-level trades into 30-minute volume bars and recognize the
# open/close hump versus midday lull.
# - Compare intraday patterns across liquidity tiers (TSLA / mid-tier / illiquid)
# and quantify the open-vs-midday volume ratio.
# - Connect the U-shape to feature engineering choices in Chapter 8 (time-of-day
# features, volume-normalized signals).
#
# ## Book reference
#
# Section §3.1 (intraday-flow narrative) and Section §3.3 (stylized-facts
# subsection on intraday U-shape).
#
# ## Prerequisites
#
# - The canonical enriched-trade parquet at
# `output/ch03/nasdaq_itch/trading_activity/trades.parquet` and the matching
# `trade_summary.parquet` (used for liquidity-tier symbol selection); both
# are produced by `05_itch_trading_activity`.
#
# ---
# %% [markdown]
# ## Setup
# %%
"""Intraday Patterns — volume and volatility dynamics from NASDAQ ITCH data."""
import warnings
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import polars as pl
import seaborn as sns
from utils.paths import get_output_dir
sns.set_style("whitegrid")
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
MAX_SYMBOLS = 0 # 0 = all
# %%
# Configuration - Unified output directory structure
# All ITCH-related outputs under a single chapter directory
NASDAQ_ITCH_OUTPUT = get_output_dir(3, "nasdaq_itch")
# Input: Parsed messages from notebook 01
MESSAGE_DIR = NASDAQ_ITCH_OUTPUT / "messages"
# Input: Trade summary from notebook 05 (trading_activity_overview)
TRADING_ACTIVITY_DIR = NASDAQ_ITCH_OUTPUT / "trading_activity"
print(f"Input directory (messages): {MESSAGE_DIR}")
print(f"Input directory (trade summary): {TRADING_ACTIVITY_DIR}")
if not MESSAGE_DIR.exists():
print(f"\nWARNING: Message directory not found: {MESSAGE_DIR}")
print(" Run 01_itch_parser first.")
# %% [markdown]
# ## Load Trade Data
#
# Load outputs from `05_itch_trading_activity`:
# - `trade_summary.parquet`: Aggregated stats by ticker (for symbol selection)
# - `trades.parquet`: Canonical tick-level trades (for analysis)
#
# Using the canonical trades file ensures we include enriched E/C data
# (executions with stock attribution added by notebook 01).
# %%
# Load trade summary and canonical trades from notebook 05
TRADE_SUMMARY_PATH = TRADING_ACTIVITY_DIR / "trade_summary.parquet"
TRADES_PATH = TRADING_ACTIVITY_DIR / "trades.parquet"
# Load trade summary for ticker selection
if TRADE_SUMMARY_PATH.exists():
trade_summary = pl.read_parquet(TRADE_SUMMARY_PATH)
# Sort explicitly by value to ensure correct selection
trade_summary = trade_summary.sort("total_value", descending=True)
# Gate the low-liquidity tier on a minimum number of trades so the
# comparison panel is not degenerate (single-trade tickers produce
# collapsed price/volume axes that obscure the intraday shape).
MIN_TRADES_LOW = 500
trade_col = next(
(c for c in ("trade_count", "n_trades", "total_trades") if c in trade_summary.columns),
None,
)
if trade_col is None:
# Fallback: use top half by total_value if no trade-count column.
active_summary = trade_summary.head(len(trade_summary) // 2)
else:
active_summary = trade_summary.filter(pl.col(trade_col) >= MIN_TRADES_LOW)
num_syms = len(active_summary)
high_sym = active_summary["ticker"][0] # highest value, still traded
mid_sym = active_summary["ticker"][num_syms // 2] # middle of active band
low_sym = active_summary["ticker"][-1] # lowest value above min-activity floor
print(
f"Loaded trade summary: {len(trade_summary)} tickers; {num_syms} above min-activity floor"
)
else:
# Fallback to common symbols if no summary available
high_sym, mid_sym, low_sym = "AAPL", "INTC", "UGA"
print(f"Trade summary not found at {TRADE_SUMMARY_PATH}")
print("Using fallback symbols (run 05_itch_trading_activity first for best results)")
# Load canonical trades (single source of truth for trade extraction)
if TRADES_PATH.exists():
all_trades = pl.read_parquet(TRADES_PATH)
data_available = True
print(f"Loaded canonical trades: {len(all_trades):,} trades")
if "msg_type" in all_trades.columns:
msg_breakdown = all_trades.group_by("msg_type").len().sort("msg_type")
print(" Message type breakdown:")
for row in msg_breakdown.iter_rows():
print(f" {row[0]}: {row[1]:>12,}")
else:
all_trades = None
data_available = MESSAGE_DIR.exists()
print(f"Canonical trades not found at {TRADES_PATH}")
print("Run 05_itch_trading_activity first to generate trades.parquet")
print("\nSelected tickers for analysis:")
print(f" High liquidity: {high_sym}")
print(f" Medium liquidity: {mid_sym}")
print(f" Low liquidity: {low_sym}")
# %% [markdown]
# ### 5.1. Intraday Volume and Price Analysis
# %%
def intraday_resample(trades_df: pl.DataFrame, ticker: str, freq: str = "5m") -> pl.DataFrame:
"""
Filter trades for a single ticker and resample to intraday bars.
Args:
trades_df: Canonical trades DataFrame from notebook 05 (trades.parquet)
ticker: Stock symbol to filter
freq: Bar frequency (e.g., "5m", "15m", "30m")
Returns:
DataFrame with columns: timestamp, shares, value, price, vwap, trade_count
"""
if trades_df is None or len(trades_df) == 0:
return pl.DataFrame()
# Filter to ticker
df = trades_df.filter(pl.col("ticker") == ticker)
if len(df) == 0:
return pl.DataFrame()
# Ensure we have required columns (compute value if missing)
required = ["timestamp", "shares", "price"]
if not all(c in df.columns for c in required):
return pl.DataFrame()
if "value" not in df.columns:
df = df.with_columns((pl.col("shares") * pl.col("price")).alias("value"))
df = df.select(["timestamp", "shares", "price", "value"]).sort("timestamp")
# Resample to bars using group_by_dynamic
bars = df.group_by_dynamic("timestamp", every=freq).agg(
[
pl.col("shares").sum().alias("shares"),
pl.col("value").sum().alias("value"),
pl.col("price").last().alias("price"),
pl.len().alias("trade_count"),
]
)
# Calculate VWAP
bars = bars.with_columns(
pl.when(pl.col("shares") > 0)
.then(pl.col("value") / pl.col("shares"))
.otherwise(None)
.alias("vwap")
)
bars = bars.drop_nulls(subset=["price"])
return bars
# %% [markdown]
# ### Plot Intraday Bars
# Resample trades for a ticker and visualize volume and price patterns side by side.
# %%
def plot_intraday_bars(trades_df: pl.DataFrame, ticker: str, freq: str = "5m") -> None:
"""Resample trades for ticker and plot volume and price patterns."""
bars = intraday_resample(trades_df, ticker, freq)
if len(bars) == 0:
print(f"No trade data found for {ticker}.")
return
# Convert to pandas for matplotlib
bars_pd = bars.to_pandas()
fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True)
fig.suptitle(f"{ticker} - Intraday Patterns ({freq} bars)", fontsize=14)
# Volume & Trade Count
ax1 = axes[0]
ax1.bar(bars_pd["timestamp"], bars_pd["shares"], alpha=0.7, label="Volume (Shares)")
ax1.set_ylabel("Shares Traded")
ax1.legend(loc="upper left")
ax1_2 = ax1.twinx()
ax1_2.plot(bars_pd["timestamp"], bars_pd["trade_count"], color="tab:orange", label="Trades")
ax1_2.set_ylabel("Trade Count")
ax1_2.legend(loc="upper right")
# Price & VWAP
ax2 = axes[1]
ax2.plot(bars_pd["timestamp"], bars_pd["price"], label="Last Price", color="tab:green")
ax2.plot(bars_pd["timestamp"], bars_pd["vwap"], label="VWAP", color="tab:red", linestyle="--")
ax2.set_ylabel("Price ($)")
ax2.set_xlabel("Time")
ax2.legend()
plt.tight_layout()
plt.show()
# %%
if data_available and all_trades is not None:
print(f"High-Volume Ticker: {high_sym}")
plot_intraday_bars(all_trades, high_sym, freq="5m")
# %%
if data_available and all_trades is not None:
print(f"Medium-Volume Ticker: {mid_sym}")
plot_intraday_bars(all_trades, mid_sym, freq="5m")
# %%
if data_available and all_trades is not None:
print(f"Low-Volume Ticker: {low_sym}")
plot_intraday_bars(all_trades, low_sym, freq="15m") # Longer bars for sparse data
# ## 7. LOB Stylized Facts: Empirical Patterns
#
# This section demonstrates key empirical regularities in order flow data that motivate
# the microstructure features we construct in Chapter 8. These "stylized facts" are
# robust patterns observed across markets and time periods.
# %% [markdown]
# ### 7.1 The Intraday U-Shape
#
# Trading activity follows a characteristic U-shaped pattern throughout the day:
# - **High at open**: Price discovery, overnight information incorporation
# - **Low at midday**: "Lunch lull" with reduced institutional activity
# - **High at close**: Portfolio rebalancing, index arbitrage, MOC orders
#
# This pattern affects feature construction: time-of-day features capture these regularities.
# %%
def compute_intraday_pattern(
trades_df: pl.DataFrame, tickers: list[str], freq: str = "30m"
) -> pl.DataFrame:
"""
Compute average intraday patterns across multiple tickers.
Args:
trades_df: Canonical trades DataFrame from notebook 05
tickers: List of stock symbols to analyze
freq: Bar frequency (e.g., "30m")
Returns:
DataFrame with time_slot, vol_pct, trade_count, ticker
"""
all_patterns = []
for ticker in tickers:
bars = intraday_resample(trades_df, ticker, freq)
if len(bars) == 0:
continue
# Extract hour and compute relative metrics
bars = bars.with_columns(
pl.col("timestamp").dt.hour().alias("hour"),
pl.col("timestamp").dt.minute().alias("minute"),
)
# Compute time-of-day slot (e.g., 9:30 -> 9.5)
bars = bars.with_columns((pl.col("hour") + pl.col("minute") / 60).alias("time_slot"))
# Normalize volume by total (to compare across tickers)
# Note: For multi-day data, this normalizes by total across all days,
# which gives average intraday pattern. For single-day data, this is daily total.
total_vol = bars["shares"].sum()
if total_vol > 0:
bars = bars.with_columns(
(pl.col("shares") / total_vol).alias("vol_pct"),
pl.lit(ticker).alias("ticker"),
)
all_patterns.append(bars.select(["time_slot", "vol_pct", "trade_count", "ticker"]))
if not all_patterns:
return pl.DataFrame()
return pl.concat(all_patterns)
# %%
if data_available and all_trades is not None:
# Get top 20 liquid tickers for pattern analysis
if TRADE_SUMMARY_PATH.exists():
top_tickers = trade_summary.head(20)["ticker"].to_list()
else:
# Fallback to well-known liquid tickers
top_tickers = ["AAPL", "MSFT", "AMZN", "GOOGL", "META", "NVDA", "TSLA", "AMD"]
pattern_df = compute_intraday_pattern(all_trades, top_tickers, freq="30m")
if len(pattern_df) > 0:
# Aggregate across tickers
hourly_pattern = (
pattern_df.group_by("time_slot")
.agg(
[
pl.col("vol_pct").mean().alias("avg_vol_pct"),
pl.col("vol_pct").std().alias("std_vol_pct"),
pl.col("trade_count").mean().alias("avg_trades"),
]
)
.sort("time_slot")
)
# Filter to regular trading hours (9:30-16:00)
hourly_pattern = hourly_pattern.filter(
(pl.col("time_slot") >= 9.5) & (pl.col("time_slot") <= 16)
)
# %%
if data_available and all_trades is not None:
# Plot U-shape
times = hourly_pattern["time_slot"].to_numpy()
vol_pct = hourly_pattern["avg_vol_pct"].to_numpy() * 100
vol_std = hourly_pattern["std_vol_pct"].to_numpy() * 100
trades = hourly_pattern["avg_trades"].to_numpy()
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Volume U-shape
axes[0].fill_between(times, vol_pct - vol_std, vol_pct + vol_std, alpha=0.3)
axes[0].plot(times, vol_pct, "b-", linewidth=2, marker="o")
axes[0].set_xlabel("Time of Day")
axes[0].set_ylabel("Share of Daily Volume (%)")
axes[0].set_title("Intraday Volume Pattern (U-Shape)")
axes[0].axhline(100 / len(times), color="red", linestyle="--", label="Uniform distribution")
axes[0].legend()
axes[0].set_xlim(9.5, 16)
# Trade count pattern
axes[1].bar(times, trades, width=0.4, alpha=0.7)
axes[1].set_xlabel("Time of Day")
axes[1].set_ylabel("Average Trade Count (per 30min)")
axes[1].set_title("Intraday Trade Frequency")
axes[1].set_xlim(9.5, 16)
plt.tight_layout()
plt.show()
# Print statistics
print("Intraday Volume Distribution:")
print(f" First 30min (9:30-10:00): {vol_pct[0]:.1f}% of daily volume")
print(f" Midday (12:00-12:30): {vol_pct[len(vol_pct) // 2]:.1f}% of daily volume")
print(f" Last 30min (15:30-16:00): {vol_pct[-1]:.1f}% of daily volume")
print(
f" Open/Close ratio to midday: {(vol_pct[0] + vol_pct[-1]) / (2 * vol_pct[len(vol_pct) // 2]):.1f}x"
)
# %% [markdown]
# ## Key Takeaways
#
# 1. **Intraday U-shape**: Volume and trade frequency peak at the open and the
# close on the top-20 most active NASDAQ tickers in this sample.
# 2. **Scope**: The pattern is computed across the top-20 tickers and shown
# separately for one high-, one medium-, and one low-liquidity ticker; this
# notebook does not test how the shape varies systematically with liquidity.
# 3. **ML implications**: Time-of-day features capture this intraday variation.
#
# **Next**: See `07_itch_stylized_facts` for bid-ask bounce and liquidity analysis.
#
# ---
#
# ## Reference
#
# Bouchaud, J.-P., Bonart, J., Donier, J., & Gould, M. (2018).
# *Trades, Quotes and Prices: Financial Markets Under the Microscope*.
# Cambridge University Press.
# [https://doi.org/10.1017/9781009028943](https://doi.org/10.1017/9781009028943)
03_market_microstructure/07_itch_stylized_facts.ipynb
+1322
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/07_itch_stylized_facts.py
+808
View File
@@ -0,0 +1,808 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Microstructure Stylized Facts: Bid-Ask Bounce and Liquidity
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Demonstrate two of §3.3's stylized facts using NASDAQ ITCH-derived trade
# data: (i) bid-ask bounce as negative first-order autocorrelation in
# tick-level returns, and (ii) the liquidity spectrum that motivates Chapter
# 19's price-impact model.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Compute lag-1 autocorrelation of tick-level trade returns and explain why
# it is negative for typical equities (bid-ask bounce).
# - Compare liquidity metrics (volume, dollar value, average trade size,
# intraday volatility) across high-, medium-, and low-liquidity tickers.
# - Recognize when trade-price returns must be replaced with mid-price
# returns to avoid microstructure noise.
#
# ## Book reference
#
# Section §3.3, *From Raw Messages to the Limit Order Book* — stylized-facts
# subsection on bid-ask bounce.
#
# ## Prerequisites
#
# - The canonical enriched-trade parquet at
# `output/ch03/nasdaq_itch/trading_activity/trades.parquet` and the matching
# `trade_summary.parquet` (used for ticker selection and the liquidity-spectrum
# section); both are produced by `05_itch_trading_activity`.
# - For the order-arrival panel, parsed ITCH `A`/`F`/`X` parquets at the
# canonical `data/equities/market/microstructure/nasdaq_itch/messages/`
# path (output of `01_itch_parser`).
#
# ---
# %% [markdown]
# ## Setup
# %%
"""Microstructure Stylized Facts — bid-ask bounce, order flow dynamics, and the liquidity spectrum."""
import warnings
from pathlib import Path
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
import pyarrow.compute as pc
import pyarrow.dataset as ds
import seaborn as sns
from IPython.display import display # noqa: F401
from utils.paths import get_output_dir
sns.set_style("whitegrid")
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
MAX_SYMBOLS = 0 # 0 = all
# %%
# Configuration - Unified output directory structure
# All ITCH-related outputs under a single chapter directory
from data.equities.loader import load_nasdaq_itch
NASDAQ_ITCH_OUTPUT = get_output_dir(3, "nasdaq_itch")
# Input: Parsed messages live under the canonical loader path (not under output/).
MESSAGE_DIR = load_nasdaq_itch(get_base_path=True)
# Input: Trade summary from notebook 05 (trading_activity_overview)
TRADING_ACTIVITY_DIR = NASDAQ_ITCH_OUTPUT / "trading_activity"
print(f"Input directory (messages): {MESSAGE_DIR}")
print(f"Input directory (trade summary): {TRADING_ACTIVITY_DIR}")
if not MESSAGE_DIR.exists():
print(f"\nWARNING: Message directory not found: {MESSAGE_DIR}")
print(" Run 01_itch_parser first.")
# %% [markdown]
# ## Load Trade Data
#
# Load outputs from `05_itch_trading_activity`:
# - `trade_summary.parquet`: Aggregated stats by ticker (for symbol selection)
# - `trades.parquet`: Canonical tick-level trades (for analysis)
#
# Using the canonical trades file ensures we include enriched E/C data
# (executions with stock attribution added by notebook 01).
# %%
# Load trade summary and canonical trades from notebook 05
TRADE_SUMMARY_PATH = TRADING_ACTIVITY_DIR / "trade_summary.parquet"
TRADES_PATH = TRADING_ACTIVITY_DIR / "trades.parquet"
# Load trade summary for ticker selection
if TRADE_SUMMARY_PATH.exists():
trade_summary = pl.read_parquet(TRADE_SUMMARY_PATH)
# Sort explicitly by value to ensure correct selection
trade_summary = trade_summary.sort("total_value", descending=True)
num_syms = len(trade_summary)
high_sym = trade_summary["ticker"][0] # highest value
mid_sym = trade_summary["ticker"][num_syms // 2] # middle
low_sym = trade_summary["ticker"][-1] # lowest value
print(f"Loaded trade summary: {num_syms} tickers")
else:
# Fallback to common symbols if no summary available
high_sym, mid_sym, low_sym = "AAPL", "INTC", "UGA"
trade_summary = None
print(f"Trade summary not found at {TRADE_SUMMARY_PATH}")
print("Using fallback symbols (run 05_itch_trading_activity first for best results)")
# Load canonical trades (single source of truth for trade extraction)
if TRADES_PATH.exists():
all_trades = pl.read_parquet(TRADES_PATH)
data_available = True
print(f"Loaded canonical trades: {len(all_trades):,} trades")
if "msg_type" in all_trades.columns:
msg_breakdown = all_trades.group_by("msg_type").len().sort("msg_type")
print(" Message type breakdown:")
for row in msg_breakdown.iter_rows():
print(f" {row[0]}: {row[1]:>12,}")
else:
all_trades = None
data_available = MESSAGE_DIR.exists()
print(f"Canonical trades not found at {TRADES_PATH}")
print("Run 05_itch_trading_activity first to generate trades.parquet")
print("\nSelected tickers for analysis:")
print(f" High liquidity: {high_sym}")
print(f" Medium liquidity: {mid_sym}")
print(f" Low liquidity: {low_sym}")
# %% [markdown]
# ### Intraday Resampling
# Resample tick-level trades to fixed-frequency bars for intraday pattern analysis.
# %%
def intraday_resample(trades_df: pl.DataFrame, ticker: str, freq: str = "5m") -> pl.DataFrame:
"""
Filter trades for a single ticker and resample to intraday bars.
Args:
trades_df: Canonical trades DataFrame from notebook 05 (trades.parquet)
ticker: Stock symbol to filter
freq: Bar frequency (e.g., "5m", "15m", "30m", "1s")
Returns:
DataFrame with columns: timestamp, shares, value, price, vwap, trade_count
"""
if trades_df is None or len(trades_df) == 0:
return pl.DataFrame()
# Filter to ticker
df = trades_df.filter(pl.col("ticker") == ticker)
if len(df) == 0:
return pl.DataFrame()
# Ensure we have required columns (compute value if missing)
required = ["timestamp", "shares", "price"]
if not all(c in df.columns for c in required):
return pl.DataFrame()
if "value" not in df.columns:
df = df.with_columns((pl.col("shares") * pl.col("price")).alias("value"))
df = df.select(["timestamp", "shares", "price", "value"]).sort("timestamp")
# Resample to bars using group_by_dynamic
bars = df.group_by_dynamic("timestamp", every=freq).agg(
[
pl.col("shares").sum().alias("shares"),
pl.col("value").sum().alias("value"),
pl.col("price").last().alias("price"),
pl.len().alias("trade_count"),
]
)
# Calculate VWAP
bars = bars.with_columns(
pl.when(pl.col("shares") > 0)
.then(pl.col("value") / pl.col("shares"))
.otherwise(None)
.alias("vwap")
)
bars = bars.drop_nulls(subset=["price"])
return bars
# %% [markdown]
# ## 6. Order Arrivals & Cancellations
#
# We analyze the **Add** and **Cancel** message types to understand:
# - How many orders are submitted vs. canceled
# - Typical order sizes
# - The cancellation rate (key microstructure metric)
# %%
def load_add_cancel_for_ticker(base_dir: Path, ticker: str) -> tuple[pl.DataFrame, pl.DataFrame]:
"""
Load 'A' (Add), 'F' (Add with Mpid), and 'X' (Cancel) for a single ticker.
For X (cancel) messages, prefers enriched files (from notebook 01's enrichment)
which include the stock column. Raw X messages only have stock_locate.
Returns tuple of (adds_df, cancels_df)
"""
add_msgs = []
for mt in ["A", "F"]:
folder = base_dir / mt
if not folder.is_dir():
continue
try:
dset = ds.dataset(folder.as_posix(), format="parquet")
# Try filtering by ticker or stock
try:
subset = dset.to_table(filter=pc.equal(pc.field("ticker"), ticker))
except Exception:
try:
subset = dset.to_table(filter=pc.equal(pc.field("stock"), ticker))
except Exception:
continue
if subset.num_rows > 0:
add_msgs.append(pl.from_arrow(subset))
except Exception:
pass
add_df = pl.concat(add_msgs, how="diagonal_relaxed") if add_msgs else pl.DataFrame()
# Load cancels - prefer enriched X which has stock column
cancel_df = pl.DataFrame()
enriched_x = base_dir / "enriched" / "X.parquet"
x_folder = base_dir / "X"
if enriched_x.exists():
# Use enriched X (has stock column from notebook 01 enrichment)
try:
df = pl.read_parquet(enriched_x)
cancel_df = df.filter(pl.col("stock") == ticker)
except Exception:
pass
elif x_folder.is_dir():
# Fall back to raw X - but note: raw X lacks ticker column
# This will likely return empty since filtering by ticker/stock won't work
try:
dset = ds.dataset(x_folder.as_posix(), format="parquet")
try:
sub_x = dset.to_table(filter=pc.equal(pc.field("ticker"), ticker))
except Exception:
try:
sub_x = dset.to_table(filter=pc.equal(pc.field("stock"), ticker))
except Exception:
sub_x = None
if sub_x and sub_x.num_rows > 0:
cancel_df = pl.from_arrow(sub_x)
except Exception:
pass
return add_df, cancel_df
# %% [markdown]
# ### Analyze Order Flow for a Ticker
# Compute order submission, cancellation, and size statistics for a given stock.
# %%
def analyze_order_flow_for_ticker(base_dir: Path, ticker: str) -> tuple[dict, pl.DataFrame]:
"""
Analyze order flow patterns for a given ticker.
Returns tuple of (results dict, standardized add orders DataFrame).
"""
add_df, cancel_df = load_add_cancel_for_ticker(base_dir, ticker)
if len(add_df) == 0:
print(f"{ticker}: No 'Add Order' messages found.")
return {}, pl.DataFrame()
# Track whether the cancel-rate proxy can actually be computed. Raw X
# messages lack a 'stock' column, so cancels can only be attributed to a
# ticker via the enriched X file produced by notebook 01. If that file is
# absent, cancel_df comes back empty and the proxy collapses to 0 — a
# spurious zero, not a real "no cancels" result. The summary printer
# surfaces this so the reader doesn't read the zero as a finding.
cancel_data_available = (base_dir / "enriched" / "X.parquet").exists()
results = {"ticker": ticker, "cancel_data_available": cancel_data_available}
# Standardize columns using Polars (vectorized)
# Cast to string first to handle bytes/string mix
if "buy_sell_indicator" in add_df.columns:
add_df = add_df.with_columns(pl.col("buy_sell_indicator").cast(pl.Utf8).alias("_side_str"))
add_df = add_df.with_columns(
pl.when(pl.col("_side_str") == "B")
.then(1)
.when(pl.col("_side_str") == "S")
.then(-1)
.otherwise(0)
.alias("side")
).drop("_side_str")
if "shares" in add_df.columns:
add_df = add_df.with_columns(pl.col("shares").cast(pl.Float64))
# Calculate cancellation rate
if len(cancel_df) > 0 and "cancelled_shares" in cancel_df.columns:
total_canceled = cancel_df.select(pl.col("cancelled_shares").sum()).item()
elif len(cancel_df) > 0 and "canceled_shares" in cancel_df.columns:
total_canceled = cancel_df.select(pl.col("canceled_shares").sum()).item()
else:
total_canceled = 0
total_shares = add_df.select(pl.col("shares").sum()).item() if "shares" in add_df.columns else 0
# PROXY for cancellation rate: canceled_shares / submitted_shares
# Can exceed 1.0 due to replaces (U) or multiple partial cancels on same shares
cancel_rate_proxy = total_canceled / total_shares if total_shares > 0 else 0
results["total_orders"] = len(add_df)
results["total_shares_submitted"] = total_shares
results["total_shares_canceled"] = total_canceled
results["cancel_rate_proxy"] = cancel_rate_proxy
return results, add_df
# %% [markdown]
# ### Print Order Flow Summary
# Display order flow statistics including counts, cancel rates, and size distribution.
# %%
def print_order_flow_summary(results: dict, add_df: pl.DataFrame) -> None:
"""Print order flow summary statistics."""
if not results:
return
# Order size distribution
if "shares" in add_df.columns:
size_stats = add_df.select(
[
pl.col("shares").mean().alias("mean"),
pl.col("shares").median().alias("median"),
]
).row(0)
results["avg_order_size"] = size_stats[0]
results["median_order_size"] = size_stats[1]
ticker = results["ticker"]
print(f"\n{'=' * 50}")
print(f"Order Flow Analysis: {ticker}")
print(f"{'=' * 50}")
print(f"Total Orders: {results['total_orders']:>12,}")
print(f"Shares Submitted: {results['total_shares_submitted']:>12,.0f}")
if results.get("cancel_data_available", True):
print(f"Shares Canceled: {results['total_shares_canceled']:>12,.0f}")
print(f"Cancel Rate (proxy):{results['cancel_rate_proxy']:>12.1%}")
if results["cancel_rate_proxy"] > 1.0:
print(" (>100% can occur due to replaces or multiple partial cancels)")
else:
# The cancel proxy needs the enriched X file produced by notebook 01.
# When it is missing, raw X parquet lacks a 'stock' column so the
# filtered cancel_df is empty and total_canceled trivially equals 0.
# Print this loudly so the reader does not mistake the zero for a
# microstructure finding.
print(f"Shares Canceled: {'N/A':>12}")
print(f"Cancel Rate (proxy):{'N/A':>12}")
print(
" (enriched/X.parquet not found — re-run notebook 01 to enable the cancel-rate proxy)"
)
if "avg_order_size" in results:
print("\nOrder Size Statistics:")
print(f" Mean: {results['avg_order_size']:>10,.0f} shares")
print(f" Median: {results['median_order_size']:>10,.0f} shares")
# %% [markdown]
# ### Plot Order Flow
# Visualize hourly order arrivals and print top order sizes for a ticker.
# %%
def plot_order_flow(add_df: pl.DataFrame, ticker: str) -> None:
"""Plot hourly order arrivals and print top order sizes."""
if len(add_df) == 0:
return
# Plot order arrivals by time
if "timestamp" in add_df.columns:
add_with_ts = add_df.filter(pl.col("timestamp").is_not_null())
if len(add_with_ts) > 0:
arrivals = (
add_with_ts.sort("timestamp")
.group_by_dynamic("timestamp", every="1h")
.agg(pl.len().alias("count"))
)
arrivals_pd = arrivals.to_pandas()
# Use the actual clock hour (HH:MM ET) for the x-axis labels rather than
# 0..N positional indices. ITCH timestamps are nanoseconds since session
# midnight ET (the exchange's local clock); strftime renders them directly.
hour_labels = [ts.strftime("%H:%M") for ts in arrivals_pd["timestamp"]]
fig, ax = plt.subplots(figsize=(10, 4))
ax.bar(range(len(arrivals_pd)), arrivals_pd["count"], color="steelblue")
ax.set_xticks(range(len(arrivals_pd)))
ax.set_xticklabels(hour_labels, rotation=45, ha="right")
ax.set_title(f"{ticker} - Order Arrivals by Hour")
ax.set_xlabel("Hour of session (ET)")
ax.set_ylabel("Number of Orders")
plt.tight_layout()
plt.show()
# Top order sizes
if "shares" in add_df.columns:
size_counts = (
add_df.group_by("shares")
.agg(pl.len().alias("count"))
.sort("count", descending=True)
.head(10)
)
print("\nTop 10 Order Sizes:")
for row in size_counts.iter_rows():
print(f" {row[0]:>8,.0f} shares: {row[1]:>6,} orders")
# %%
if MESSAGE_DIR.exists() and data_available:
flow_results, flow_add_df = analyze_order_flow_for_ticker(MESSAGE_DIR, high_sym)
print_order_flow_summary(flow_results, flow_add_df)
plot_order_flow(flow_add_df, high_sym)
# %%
if MESSAGE_DIR.exists() and data_available:
flow_results, flow_add_df = analyze_order_flow_for_ticker(MESSAGE_DIR, mid_sym)
print_order_flow_summary(flow_results, flow_add_df)
plot_order_flow(flow_add_df, mid_sym)
# %% [markdown]
#
# %% [markdown]
# ### 7.2 The Bid-Ask Bounce
#
# A fundamental microstructure phenomenon: trade prices bounce between bid and ask,
# creating **negative autocorrelation** in tick-level returns. This is why:
# - Mid-price returns are preferred over trade-price returns
# - Tick-level "momentum" signals fail
# - The Roll (1984) spread estimator works
#
# **Implication for Chapter 8**: Always compute returns from mid-prices, not trade prices.
# %%
def compute_tick_autocorrelation(trades_df: pl.DataFrame, ticker: str, max_lags: int = 20) -> dict:
"""
Compute autocorrelation of tick-level returns to demonstrate bid-ask bounce.
Note: We use 1-second bars as a proxy for tick data. True tick-by-tick analysis
would use individual trade prices, but the bounce effect is still visible at
1-second resolution for liquid stocks.
Args:
trades_df: Canonical trades DataFrame from notebook 05
ticker: Stock symbol to analyze
max_lags: Maximum number of lags to compute
Returns:
Dict with autocorrelation results
"""
# Use 1-second bars as proxy for ticks (true tick data would be even noisier)
bars = intraday_resample(trades_df, ticker, freq="1s")
if len(bars) < 100:
return {}
# Compute returns in basis points (1bp = 0.01% = 0.0001)
# pct_change() returns decimal (e.g., 0.01 = 1%), so multiply by 10000 for bps
bars = bars.with_columns(
(pl.col("price").pct_change() * 10000).alias("return_bps"),
).drop_nulls()
returns = bars["return_bps"].to_numpy()
# Compute autocorrelations
n = len(returns)
mean_r = np.mean(returns)
var_r = np.var(returns)
autocorrs = []
for lag in range(1, max_lags + 1):
if n - lag < 10:
break
cov = np.mean((returns[lag:] - mean_r) * (returns[:-lag] - mean_r))
autocorrs.append(cov / var_r if var_r > 0 else 0)
return {
"ticker": ticker,
"autocorrs": autocorrs,
"n_obs": n,
"lag1_autocorr": autocorrs[0] if autocorrs else 0,
}
# %%
if data_available and all_trades is not None:
# Compute autocorrelation for high-liquidity ticker
bounce_result = compute_tick_autocorrelation(all_trades, high_sym, max_lags=10)
if bounce_result and bounce_result["autocorrs"]:
lags = list(range(1, len(bounce_result["autocorrs"]) + 1))
autocorrs = bounce_result["autocorrs"]
fig, ax = plt.subplots(figsize=(10, 5))
colors = ["red" if ac < 0 else "blue" for ac in autocorrs]
ax.bar(lags, autocorrs, color=colors, alpha=0.7)
ax.axhline(0, color="black", linewidth=0.5)
ax.axhline(-0.1, color="gray", linestyle="--", alpha=0.5)
ax.axhline(0.1, color="gray", linestyle="--", alpha=0.5)
ax.set_xlabel("Lag (seconds)")
ax.set_ylabel("Autocorrelation")
ax.set_title(
f"Bid-Ask Bounce: Return Autocorrelation for {high_sym}\n"
f"(Negative lag-1 = bounce between bid and ask)"
)
ax.set_xticks(lags)
# Add annotation
ax.annotate(
f"Lag-1: {autocorrs[0]:.3f}",
xy=(1, autocorrs[0]),
xytext=(3, autocorrs[0] - 0.1),
arrowprops=dict(arrowstyle="->", color="red"),
fontsize=12,
color="red",
)
plt.tight_layout()
plt.show()
print(f"\nBid-Ask Bounce Analysis for {high_sym}:")
print(f" Lag-1 autocorrelation: {autocorrs[0]:.4f}")
print(" (Negative value confirms bounce between bid and ask)")
print("\n Implication: Use mid-price returns, not trade-price returns!")
# %% [markdown]
# ### 7.3 The Liquidity Spectrum: From Blue Chips to Small Caps
#
# One of the most important microstructure insights: **liquidity varies enormously**
# across stocks. A 1000-share order has near-zero impact on AAPL but can move
# an illiquid stock by 50+ basis points.
#
# This comparison foreshadows the price impact discussion in Chapter 19.
# %%
def compare_liquidity_metrics(trades_df: pl.DataFrame, tickers: list[str]) -> pl.DataFrame:
"""
Compare key liquidity metrics across tickers.
Args:
trades_df: Canonical trades DataFrame from notebook 05
tickers: List of stock symbols to compare
Returns:
DataFrame with liquidity metrics per ticker
"""
results = []
for ticker in tickers:
bars = intraday_resample(trades_df, ticker, freq="1m")
if len(bars) < 10:
continue
# Compute metrics
total_volume = bars["shares"].sum()
total_value = bars["value"].sum()
avg_trade_size = (
total_volume / bars["trade_count"].sum() if bars["trade_count"].sum() > 0 else 0
)
# Compute volatility (1-minute return std), winsorized at the 1st/99th
# percentile so a single mis-printed tick in a thinly traded name does not
# dominate the estimate — raw ITCH prints occasionally carry bad prices.
returns_bps = (
bars.with_columns((pl.col("price").pct_change() * 10000).alias("return_bps"))[
"return_bps"
]
.drop_nulls()
.to_numpy()
)
if len(returns_bps) >= 5:
lo, hi = np.percentile(returns_bps, [1, 99])
volatility = float(np.std(np.clip(returns_bps, lo, hi)))
else:
volatility = float("nan")
# Price range as the 5th-95th percentile spread relative to the median bar
# price. The raw max-min range is corrupted by the same single bad prints.
prices = bars["price"].to_numpy()
price_range = (
(np.percentile(prices, 95) - np.percentile(prices, 5)) / np.median(prices) * 100
)
results.append(
{
"ticker": ticker,
"total_volume": total_volume,
"total_value": total_value,
"avg_trade_size": avg_trade_size,
"volatility_bps": volatility,
"price_range_pct": price_range,
"n_trades": bars["trade_count"].sum(),
}
)
return pl.DataFrame(results)
# %%
if data_available and all_trades is not None:
# Select stocks spanning the liquidity spectrum. We draw the tiers from the
# *tradeable* universe — names with enough activity to form intraday bars —
# because the long tail of the ITCH session (thousands of tickers with only a
# handful of prints) cannot be resampled to one-minute bars and would drop out.
MIN_TRADES = 500 # floor that guarantees each tier resamples to many 1-min bars
if trade_summary is not None:
tradeable = trade_summary.filter(pl.col("trade_count") >= MIN_TRADES)
n_pool = len(tradeable)
if n_pool >= 5:
# `tradeable` is sorted by total_value descending; span it evenly.
tier_idx = [0, n_pool // 4, n_pool // 2, 3 * n_pool // 4, n_pool - 1]
spectrum_tickers = [tradeable["ticker"][i] for i in tier_idx]
elif n_pool > 0:
spectrum_tickers = tradeable["ticker"].to_list()
else:
spectrum_tickers = trade_summary["ticker"].to_list()
else:
# Fallback: well-known tickers across the liquidity spectrum
spectrum_tickers = ["AAPL", "MSFT", "INTC", "AMD", "UGA"]
liquidity_comparison = compare_liquidity_metrics(all_trades, spectrum_tickers)
kept = liquidity_comparison["ticker"].to_list() if len(liquidity_comparison) else []
dropped = [t for t in spectrum_tickers if t not in kept]
if dropped:
print(f"Note: dropped {', '.join(dropped)} (too few one-minute bars to compare).\n")
if len(liquidity_comparison) > 0:
# Sort by volume
liquidity_comparison = liquidity_comparison.sort("total_volume", descending=True)
print("=" * 80)
print("LIQUIDITY SPECTRUM: From Blue Chips to Small Caps")
print("=" * 80)
print(
f"{'Ticker':<8} {'Volume':>12} {'Value ($M)':>12} {'Avg Trade':>10} {'Volatility':>12} {'Price Range':>12}"
)
print("-" * 80)
for row in liquidity_comparison.iter_rows(named=True):
print(
f"{row['ticker']:<8} {row['total_volume']:>12,.0f} {row['total_value'] / 1e6:>12.1f} "
f"{row['avg_trade_size']:>10.0f} {row['volatility_bps']:>11.1f}bp {row['price_range_pct']:>11.2f}%"
)
# %%
# Liquidity spectrum visualization
if data_available and all_trades is not None and len(liquidity_comparison) > 0:
tickers = liquidity_comparison["ticker"].to_list()
volumes = liquidity_comparison["total_volume"].to_numpy()
volatilities = liquidity_comparison["volatility_bps"].to_numpy()
trade_sizes = liquidity_comparison["avg_trade_size"].to_numpy()
price_ranges = liquidity_comparison["price_range_pct"].to_numpy()
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
axes[0, 0].bar(tickers, volumes, color="steelblue", alpha=0.7)
axes[0, 0].set_yscale("log")
axes[0, 0].set_ylabel("Total Volume (shares, log scale)")
axes[0, 0].set_title("Daily Volume by Liquidity Tier")
axes[0, 0].tick_params(axis="x", rotation=45)
axes[0, 1].bar(tickers, volatilities, color="indianred", alpha=0.7)
axes[0, 1].set_ylabel("1-minute Return Volatility (bps)")
axes[0, 1].set_title("1-Minute Return Volatility by Tier")
axes[0, 1].tick_params(axis="x", rotation=45)
axes[1, 0].bar(tickers, trade_sizes, color="green", alpha=0.7)
axes[1, 0].set_ylabel("Average Trade Size (shares)")
axes[1, 0].set_title("Trade Size by Liquidity Tier")
axes[1, 0].tick_params(axis="x", rotation=45)
axes[1, 1].bar(tickers, price_ranges, color="orange", alpha=0.7)
axes[1, 1].set_ylabel("Daily Price Range (%)")
axes[1, 1].set_title("Price Range (Impact Proxy)")
axes[1, 1].tick_params(axis="x", rotation=45)
plt.suptitle("The Liquidity Spectrum: How Microstructure Varies Across Stocks", fontsize=14)
plt.tight_layout()
plt.show()
# %%
# Liquidity ratio summary
if data_available and all_trades is not None and len(liquidity_comparison) >= 2:
share_ratio = volumes[0] / volumes[-1] if volumes[-1] > 0 else float("inf")
values = liquidity_comparison["total_value"].to_numpy()
value_ratio = values[0] / values[-1] if values[-1] > 0 else float("inf")
print("Key Insight:")
print(f" Share-volume ratio (most vs least active): {share_ratio:,.0f}x")
print(f" Dollar-value ratio (most vs least active): {value_ratio:,.0f}x")
print("\n → Turnover spans orders of magnitude across the spectrum, but 1-minute")
print(" return volatility does not track liquidity monotonically — it is driven")
print(" by name-specific factors and spikes only at the micro-cap extreme.")
print(" → Execution-cost magnitudes motivate the price impact analysis in Chapter 19.")
# %% [markdown]
#
# ## 8. Key Takeaways
#
# ### Market Microstructure Insights
#
# 1. **Volume Concentration**: A small number of tickers account for most trading activity.
# The top 50 stocks account for ~46% of total dollar volume (see `05_itch_trading_activity`).
#
# 2. **Intraday U-Shape**: Volume and trading activity follow a characteristic pattern—high
# at open/close, low at midday. This affects optimal execution timing.
#
# 3. **Bid-Ask Bounce**: Trade prices bounce between bid and ask, creating negative
# autocorrelation at tick level. Use mid-price returns for ML features.
#
# 4. **Liquidity Spectrum**: Turnover spans orders of magnitude across the tiers in
# this sample — dollar value runs into the thousands-fold range from the most to
# least active name (see the comparison above). Intraday return volatility, by
# contrast, does not track liquidity monotonically: the most active name is more
# volatile than the mid-tier, and only the micro-cap extreme stands out. Volatility
# reflects name-specific factors, not liquidity alone. Execution-cost magnitudes
# across liquidity tiers are not estimated here; price-impact modelling is the
# subject of Chapter 19.
#
# 5. **Cancellation Rates**: High cancellation rates (>90%) are normal for modern markets,
# reflecting continuous quote adjustment by market makers.
#
# ### Bridge to Later Chapters
#
# | Stylized Fact | Chapter 8 Application | Chapter 19 Application |
# |---------------|----------------------|------------------------|
# | **Intraday U-shape** | Time-of-day features | Execution timing |
# | **Bid-ask bounce** | Mid-price return targets | Transaction cost models |
# | **Liquidity spectrum** | Liquidity features | Price impact estimation |
# | **Order flow imbalance** | OFI features | - |
#
# ### Next Steps
#
# - **`04_itch_order_lifecycle_analysis`**: Deep dive into why ~96% of orders cancel
# - **`02_itch_lob_reconstruction`**: Build the LOB from message events
# - **`16_itch_information_bars`**: Convert ticks to ML-ready bars
#
# ---
#
# **References**:
# - Harris, L. (2003). *Trading and Exchanges: Market Microstructure for Practitioners*.
# - Roll, R. (1984). "A Simple Implicit Measure of the Effective Bid-Ask Spread."
# - Cont, R., Kukanov, A., & Stoikov, S. (2014). "The Price Impact of Order Book Events."
#
# ---
#
# ## Reference
#
# Bouchaud, J.-P., Bonart, J., Donier, J., & Gould, M. (2018).
# *Trades, Quotes and Prices: Financial Markets Under the Microscope*.
# Cambridge University Press.
# [https://doi.org/10.1017/9781009028943](https://doi.org/10.1017/9781009028943)
03_market_microstructure/08_databento_lob_reconstruction.ipynb
+2103
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/08_databento_lob_reconstruction.py
+1037
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/09_databento_mbo_analysis.ipynb
+1711
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/09_databento_mbo_analysis.py
+888
View File
@@ -0,0 +1,888 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # DataBento MBO: Order Flow Predictability
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Quantify whether one-minute order-flow imbalance carries predictive
# information for next-minute returns on NVDA, and contrast midprice
# markouts with executable (bid/ask-at-decision-time) markouts to expose
# the gap §3.3 warns about between statistical signal and trading P&L.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Build minute-resolution OFI from the LOB-reconstruction parquet that
# `08_databento_lob_reconstruction` writes.
# - Measure predictability at multiple horizons (1, 5, 10, 30 min) and
# recognize that the correlations are tiny and noisy on a sample of this
# length.
# - Distinguish midprice markouts from executable markouts and explain why
# spread and queue dynamics dominate P&L at sub-minute horizons.
#
# ## Book reference
#
# Section §3.3, *From Raw Messages to the Limit Order Book* — tradability
# caveat under "LOB Stylized Facts: Predictive Patterns".
#
# ## Prerequisites
#
# - DataBento XNAS-ITCH MBO parquets at
# `data/equities/market/microstructure/market_by_order/NVDA/` (10 trading
# days, November 2024).
#
# ## Data Source
#
# DataBento provides institutional-quality **Market-by-Order (MBO)** data:
# - Every order add, cancel, modify, and fill
# - Nanosecond timestamps (both exchange and receipt time)
# - Pre-filtered by symbol (no ITCH stock_locate mapping needed)
#
# ---
# %% [markdown]
# ## Setup
# %%
"""DataBento MBO: Order Flow Predictability — analyzing OFI signals in tick data."""
import warnings
from pathlib import Path
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import polars as pl
import seaborn as sns
from tqdm.auto import tqdm
from data import load_mbo_data
from utils.paths import get_output_dir
from utils.reproducibility import set_global_seeds
# %% tags=["parameters"]
MAX_SYMBOLS = 0 # 0 = all
SEED = 42
# %%
set_global_seeds(SEED)
sns.set_style("whitegrid")
# ML4T Blue Period palette
COLORS = {
"blue": "#1E3A5F",
"accent": "#4A90A4",
"warm": "#8B4513",
"neutral": "#5D5D5D",
}
# %%
OUTPUT_DIR = get_output_dir(3, "databento")
SYMBOL = "NVDA"
# Get file paths from canonical loader
data_files = load_mbo_data(symbols=[SYMBOL], list_files=True)
SYMBOL_DIR = data_files[0].parent if data_files else None
# Production: use all files; testing: limit
MAX_FILES = None
MAX_ROWS = None
if SYMBOL_DIR and SYMBOL_DIR.exists():
data_files = sorted(SYMBOL_DIR.glob("*.parquet"))
if MAX_FILES:
data_files = data_files[:MAX_FILES]
print(f"Symbol: {SYMBOL}")
print(f"Data files: {len(data_files)} days")
if data_files:
dates = [f.stem.split("-")[-1].split(".")[0] for f in data_files]
print(f"Date range: {dates[0]} to {dates[-1]}")
else:
print(f"DataBento data not found at {SYMBOL_DIR}")
data_files = []
# %% [markdown]
# ## 1. Understanding MBO Data
#
# Market-by-Order data captures every change to the order book:
#
# | Action | Meaning | Order Flow Signal |
# |--------|---------|-------------------|
# | **Add (A)** | New limit order placed | Passive liquidity supply |
# | **Cancel (C)** | Order withdrawn | Liquidity withdrawal |
# | **Fill (F)** | Order executed | Active demand/supply |
# | **Trade (T)** | Trade report | Price discovery |
#
# The **side** field tells us whether the order was on the bid (buy) or ask (sell).
# Combining action and side reveals the order flow dynamics.
# %%
ACTION_MAP = {"A": "Add", "C": "Cancel", "F": "Fill", "M": "Modify", "R": "Clear", "T": "Trade"}
SIDE_MAP = {"A": "Ask", "B": "Bid", "N": "None"}
def load_databento_day(file_path: Path, max_rows: int | None = None) -> pl.DataFrame:
"""Load one day of DataBento MBO data."""
df = pl.read_parquet(file_path)
if max_rows and len(df) > max_rows:
df = df.head(max_rows)
# Handle fixed-point prices if needed
if "price" in df.columns and df["price"].max() > 1_000_000:
df = df.with_columns((pl.col("price") / 1e9).alias("price"))
return df.with_columns(pl.col("timestamp").cast(pl.Datetime("ns")))
# %%
sample_df = None
multi_day = None
if data_files:
sample_df = load_databento_day(data_files[0], max_rows=MAX_ROWS)
date_str = data_files[0].stem.split("-")[-1].split(".")[0]
print(f"=== {SYMBOL} on {date_str} ===")
print(f"Total messages: {len(sample_df):,}")
# Message composition
print("\nMessage composition:")
action_counts = sample_df.group_by("action").len().sort("len", descending=True)
total = len(sample_df)
for row in action_counts.iter_rows():
pct = 100 * row[1] / total
print(f" {ACTION_MAP.get(row[0], row[0]):8s} {row[1]:>12,} ({pct:5.1f}%)")
# %% [markdown]
# **What this tells us**: Adds and cancels dominate (~48% each), with trades
# comprising only ~2% of messages. This ~40:1 ratio of quote changes to trades
# is typical for liquid stocks—the order book is constantly reshaping itself
# around each trade.
# %% [markdown]
# ## 2. Book Pressure: Measuring Order Flow Momentum
#
# The core idea is simple: when buyers are more aggressive than sellers,
# prices should rise. We measure this by tracking the net flow of orders:
#
# **Book Pressure** = Σ (sign × weight × size × decay)
#
# Where:
# - **sign**: +1 for bids, -1 for asks
# - **weight**: +1 for adds, -1 for cancels, +0.5 for fills
# - **decay**: exp(-λ × distance_from_mid) — orders near the spread matter more
#
# This creates a momentum indicator: positive pressure suggests buying interest,
# negative suggests selling.
# %%
def compute_book_pressure(
df: pl.DataFrame,
decay_lambda: float = 0.01,
ema_halflife: int = 100,
) -> pl.DataFrame:
"""Compute book pressure from MBO messages with EMA smoothing."""
book_actions = df.filter(pl.col("action").is_in(["A", "C", "F"]))
if len(book_actions) == 0:
return df.with_columns(pl.lit(0.0).alias("book_pressure"))
# Rolling midprice from recent trades (avoids lookahead)
trades = df.filter(pl.col("action") == "T").select(["timestamp", "price"])
if len(trades) > 0:
rolling_mid = trades.with_columns(
pl.col("price").rolling_mean(window_size=100, min_periods=1).alias("mid_price")
).select(["timestamp", "mid_price"])
df = df.sort("timestamp").join_asof(
rolling_mid.sort("timestamp"), on="timestamp", strategy="backward"
)
df = df.with_columns(pl.col("mid_price").fill_null(rolling_mid["mid_price"][0]))
else:
df = df.with_columns(pl.lit(book_actions["price"].median()).alias("mid_price"))
# Compute pressure components
result = (
df.with_columns(
[
pl.when(pl.col("side") == "B")
.then(1.0)
.when(pl.col("side") == "A")
.then(-1.0)
.otherwise(0.0)
.alias("side_sign"),
pl.when(pl.col("action") == "A")
.then(1.0)
.when(pl.col("action") == "C")
.then(-1.0)
.when(pl.col("action") == "F")
.then(0.5)
.otherwise(0.0)
.alias("action_weight"),
(pl.col("price") - pl.col("mid_price")).abs().alias("dist_from_mid"),
]
)
.with_columns((-decay_lambda * pl.col("dist_from_mid")).exp().alias("decay_weight"))
.with_columns(
(
pl.col("side_sign")
* pl.col("action_weight")
* pl.col("size")
* pl.col("decay_weight")
).alias("raw_pressure")
)
.with_columns(
pl.col("raw_pressure").ewm_mean(half_life=ema_halflife).alias("book_pressure")
)
)
return result.select(
["timestamp", "action", "side", "price", "size", "order_id", "book_pressure"]
)
# %%
# Single cell: build the two-panel figure (price + book pressure) end-to-end
# so papermill emits both panels populated. Splitting price and pressure into
# separate cells leaves the second panel empty in the rendered .ipynb.
pressure_df = None
if sample_df is not None and len(sample_df) > 0:
# One hour of data (10:00-11:00 is typically liquid)
sample_hour = sample_df.filter(
(pl.col("timestamp").dt.hour() >= 10) & (pl.col("timestamp").dt.hour() < 11)
)
if len(sample_hour) > 1000:
pressure_df = compute_book_pressure(sample_hour)
fig, axes = plt.subplots(2, 1, figsize=(14, 8), sharex=True)
# Top: trade prices
trades = pressure_df.filter(pl.col("action") == "T").to_pandas()
if len(trades) > 0:
axes[0].plot(
trades["timestamp"],
trades["price"],
".",
markersize=1,
alpha=0.5,
color=COLORS["blue"],
)
axes[0].set_ylabel("Trade Price ($)")
axes[0].set_title(f"Book Pressure vs Price - {SYMBOL} ({date_str})")
# Bottom: book pressure (clipped for readability)
pressure_pd = pressure_df.to_pandas()
pressure_clipped = pressure_pd["book_pressure"].clip(
lower=pressure_pd["book_pressure"].quantile(0.01),
upper=pressure_pd["book_pressure"].quantile(0.99),
)
axes[1].plot(
pressure_pd["timestamp"], pressure_clipped, linewidth=0.5, color=COLORS["accent"]
)
axes[1].axhline(0, color="black", linestyle="--", linewidth=0.5)
axes[1].set_ylabel("Book Pressure (clipped at 1%/99%)")
axes[1].set_xlabel("Time (ET)")
axes[1].fill_between(
pressure_pd["timestamp"],
pressure_clipped,
0,
where=pressure_clipped > 0,
alpha=0.3,
color="green",
label="Buy pressure",
)
axes[1].fill_between(
pressure_pd["timestamp"],
pressure_clipped,
0,
where=pressure_clipped < 0,
alpha=0.3,
color="red",
label="Sell pressure",
)
axes[1].legend(loc="upper right")
plt.tight_layout()
plt.show()
# Statistics
print("=== Book Pressure Statistics ===")
print(f"Mean: {pressure_pd['book_pressure'].mean():>10.2f}")
print(f"Std: {pressure_pd['book_pressure'].std():>10.2f}")
print(f"Skewness: {pressure_pd['book_pressure'].skew():>10.2f}")
# %% [markdown]
# **What the chart reveals**:
#
# The book pressure oscillates around zero, with occasional sustained moves
# that often precede price changes. Notice how:
#
# - **Green zones** (positive pressure) tend to occur before price upticks
# - **Red zones** (negative pressure) tend to precede price drops
# - The signal is noisy—individual spikes aren't reliable, but trends are
#
# This is the fundamental insight: order flow contains information about
# short-term price direction, but the signal-to-noise ratio is low.
# %% [markdown]
# ## 3. Building Minute Bars with Microstructure Features
#
# Tick data is too noisy for most analysis. We aggregate to minute bars while
# preserving the microstructure information that matters for prediction.
# %%
def reconstruct_bbo_bars(df: pl.DataFrame, bar_freq: str = "1m") -> pl.DataFrame:
"""Reconstruct the best bid/offer from the MBO stream and snapshot it per bar.
A markout that reflects *tradable* P&L needs the quotes a marketable order
would actually hit — the best bid and best ask at decision time — not a
trade-price proxy. We replay the order stream to maintain top of book
(resting size by price on each side), then take the last quote in each bar.
Add (A) inserts resting size; Cancel (C) removes it; Fill (F) reduces it;
Modify (M) re-prices an order; Clear (R) wipes the side. We key on
``order_id`` so cancels, fills, and modifies adjust the level the order
actually rested on.
"""
orders: dict[int, tuple[str, float, int]] = {} # order_id -> (side, price, size)
bids: dict[float, int] = {} # price -> resting size
asks: dict[float, int] = {} # price -> resting size
# Track the inside quote incrementally so we never scan the whole book per
# message. Recomputing max(bids)/min(asks) on every one of tens of millions
# of MBO messages is the bottleneck on multi-day streams; here the O(levels)
# scan only fires when the prevailing best level itself empties.
best_bid: float | None = None
best_ask: float | None = None
def _book(side: str) -> dict[float, int]:
return bids if side == "B" else asks
def _add(side: str, price: float, size: int) -> None:
nonlocal best_bid, best_ask
book = _book(side)
book[price] = book.get(price, 0) + size
if side == "B":
if best_bid is None or price > best_bid:
best_bid = price
elif best_ask is None or price < best_ask:
best_ask = price
def _reduce(side: str, price: float, size: int) -> None:
nonlocal best_bid, best_ask
book = _book(side)
if price in book:
book[price] -= size
if book[price] <= 0:
del book[price]
# Only when the inside level vacates do we rescan that side.
if side == "B" and price == best_bid:
best_bid = max(bids) if bids else None
elif side == "A" and price == best_ask:
best_ask = min(asks) if asks else None
ts_out: list = []
bid_out: list[float | None] = []
ask_out: list[float | None] = []
for ts, action, side, price, size, oid in (
df.sort("timestamp")
.select(["timestamp", "action", "side", "price", "size", "order_id"])
.iter_rows()
):
if action == "A" and side in ("B", "A"):
orders[oid] = (side, price, size)
_add(side, price, size)
elif action == "C":
if oid in orders:
s, p, sz = orders.pop(oid)
_reduce(s, p, sz)
elif side in ("B", "A"):
_reduce(side, price, size)
elif action == "F":
if oid in orders:
s, p, sz = orders[oid]
_reduce(s, p, size)
if sz - size > 0:
orders[oid] = (s, p, sz - size)
else:
orders.pop(oid, None)
elif side in ("B", "A"):
_reduce(side, price, size)
elif action == "M":
if oid in orders:
s, p, sz = orders.pop(oid)
_reduce(s, p, sz)
if side in ("B", "A"):
orders[oid] = (side, price, size)
_add(side, price, size)
elif action == "R":
orders.clear()
bids.clear()
asks.clear()
best_bid = best_ask = None
# Trade (T) prints do not change the book; the matching Fill (F) does.
ts_out.append(ts)
bid_out.append(best_bid)
ask_out.append(best_ask)
bbo = pl.DataFrame(
{"timestamp": ts_out, "best_bid": bid_out, "best_ask": ask_out},
schema={"timestamp": pl.Datetime("ns"), "best_bid": pl.Float64, "best_ask": pl.Float64},
)
# Snapshot the quote prevailing at each bar close (last update within the bar).
return bbo.group_by_dynamic("timestamp", every=bar_freq, period=bar_freq).agg(
pl.col("best_bid").last(), pl.col("best_ask").last()
)
# %%
def process_day_to_bars(
file_path: Path, bar_freq: str = "1m", max_rows: int | None = None
) -> pl.DataFrame:
"""Process one day to minute bars with OHLCV and order flow metrics."""
df = load_databento_day(file_path, max_rows=max_rows)
# Reconstruct top of book from the full stream (orders resting before the
# open carry into the session), then restrict bars to regular trading hours.
bbo_bars = reconstruct_bbo_bars(df, bar_freq=bar_freq)
# Regular trading hours (UTC: 13:30-21:00 covers both EST and EDT)
df = df.filter(
(
(pl.col("timestamp").dt.hour() > 13)
| ((pl.col("timestamp").dt.hour() == 13) & (pl.col("timestamp").dt.minute() >= 30))
)
& (pl.col("timestamp").dt.hour() < 21)
)
if len(df) == 0:
return pl.DataFrame()
trades = df.filter(pl.col("action") == "T")
if len(trades) == 0:
return pl.DataFrame()
# Aggregate to bars
bars = trades.group_by_dynamic("timestamp", every=bar_freq, period=bar_freq).agg(
[
pl.col("price").first().alias("open"),
pl.col("price").max().alias("high"),
pl.col("price").min().alias("low"),
pl.col("price").last().alias("close"),
pl.col("size").sum().alias("volume"),
pl.len().alias("trade_count"),
(pl.when(pl.col("side") == "B").then(pl.col("size")).otherwise(0))
.sum()
.alias("buy_volume"),
(pl.when(pl.col("side") == "A").then(pl.col("size")).otherwise(0))
.sum()
.alias("sell_volume"),
(pl.col("price") * pl.col("size")).sum().alias("notional"),
]
)
# Derived features
bars = bars.with_columns(
[
((pl.col("high") + pl.col("low")) / 2).alias("mid_price"),
(pl.col("close") / pl.col("open") - 1).alias("return"),
pl.when(pl.col("buy_volume") + pl.col("sell_volume") > 0)
.then(
(pl.col("buy_volume") - pl.col("sell_volume"))
/ (pl.col("buy_volume") + pl.col("sell_volume"))
)
.otherwise(0.0)
.alias("ofi"),
(pl.col("notional") / pl.col("volume")).alias("vwap"),
]
)
# Attach the reconstructed quote prevailing at each bar close. ``mid_quote``
# is the true (bid+ask)/2 midpoint; the trade-derived ``mid_price`` above is
# only a proxy. Keeping both lets us separate the price-response signal
# (midpoint markout) from the spread a real order pays (executable markout).
bars = bars.join(bbo_bars, on="timestamp", how="left")
bars = bars.with_columns(((pl.col("best_bid") + pl.col("best_ask")) / 2).alias("mid_quote"))
return bars
# %%
if data_files:
all_bars = []
n_days = MAX_FILES or len(data_files)
for file in tqdm(data_files[:n_days], desc="Processing days"):
bars = process_day_to_bars(file, max_rows=MAX_ROWS)
if len(bars) > 0:
all_bars.append(bars)
if all_bars:
multi_day = pl.concat(all_bars).sort("timestamp")
print("\n=== Multi-Day Dataset ===")
print(f"Minute bars: {len(multi_day):,}")
print(f"Trading days: {multi_day['timestamp'].dt.date().n_unique()}")
print(
f"Date range: {multi_day['timestamp'].min().date()} to {multi_day['timestamp'].max().date()}"
)
# %% [markdown]
# ## 4. The Key Question: Does OFI Predict Returns?
#
# Order Flow Imbalance (OFI) measures the balance between buyer and seller
# aggression in each minute:
#
# **OFI = (Buy Volume - Sell Volume) / Total Volume**
#
# Values range from -1 (all selling) to +1 (all buying). If markets are
# efficient, OFI should predict short-term returns—aggressive buyers push
# prices up, aggressive sellers push prices down.
#
# But how strong is this relationship, and how long does it last?
# %%
# Latency in bars between signal time and earliest fillable execution.
# Defined once so the markout call and the downstream "h == LATENCY_BARS"
# guard in the plotting cell cannot drift apart.
LATENCY_BARS = 1
def compute_markouts(
bars: pl.DataFrame, horizons: list | None = None, latency_bars: int = LATENCY_BARS
) -> pl.DataFrame:
"""Compute forward returns at multiple horizons within each trading session."""
if horizons is None:
horizons = [1, 5, 10, 30]
result = bars.clone().with_columns(pl.col("timestamp").dt.date().alias("session_date"))
for h in horizons:
# Standard markout: price change from now to h minutes ahead
result = result.with_columns(
(pl.col("mid_price").shift(-h).over("session_date") / pl.col("mid_price") - 1).alias(
f"markout_{h}"
)
)
# Latency-adjusted: assumes 1-bar execution delay
p1 = pl.col("mid_price").shift(-latency_bars).over("session_date")
p2 = pl.col("mid_price").shift(-h).over("session_date")
result = result.with_columns((p2 / p1 - 1).alias(f"markout_{h}_adj"))
# Executable markout: a long pays the ask at decision time and exits at
# the bid h bars later — the round trip a marketable order realizes.
# Its gap to the midpoint markout is exactly the spread the book charges.
ask_now = pl.col("best_ask")
bid_future = pl.col("best_bid").shift(-h).over("session_date")
result = result.with_columns((bid_future / ask_now - 1).alias(f"markout_{h}_exec"))
return result
# %%
if multi_day is not None and len(multi_day) > 0:
multi_day = compute_markouts(multi_day, horizons=[1, 5, 10, 30], latency_bars=LATENCY_BARS)
# Add lagged OFI within sessions
multi_day = multi_day.with_columns(
[
pl.col("ofi").shift(1).over("session_date").alias("ofi_lag1"),
pl.col("ofi")
.rolling_mean(window_size=5)
.shift(1)
.over("session_date")
.alias("ofi_ma5"),
]
)
# Correlation analysis
pdf = multi_day.drop_nulls(
["ofi", "ofi_lag1", "markout_1", "markout_5", "markout_10"]
).to_pandas()
print("=== OFI Predictive Power ===")
print("Correlation of OFI(t-1) with future returns:\n")
print(f"{'Horizon':<12} {'Correlation':>12} {'Interpretation':<30}")
print("-" * 55)
interpretations = {
1: "Very short-term momentum",
5: "Short-term persistence",
10: "Medium-term (weak)",
30: "Longer-term (negligible)",
}
for h in [1, 5, 10, 30]:
if f"markout_{h}" in pdf.columns:
corr = pdf["ofi_lag1"].corr(pdf[f"markout_{h}"])
interp = interpretations.get(h, "")
print(f"{h:>3} min {corr:>12.4f} {interp}")
# %% [markdown]
# Pearson correlation between OFI(t-1) and forward markouts is ≈ 0.002–0.004
# at 1–5 minute horizons and ≈ 0.02 at 30 minutes. These correlations are
# computed on the **minute-bar** panel printed above (rows in the low thousands
# across the slice run here), not on the underlying tick stream. At that bar
# count $1/\sqrt{n}$ is of order $10^{-2}$, so the 1–5 minute estimates sit
# inside one SE of zero — they are within sampling noise — and the 30-minute
# point estimate is roughly one to two SE above zero, i.e. marginal rather than
# decisive. Statistical detectability is therefore part of the question; the
# economic question — whether the implied magnitude survives transaction costs —
# is the binding one, and the next panel addresses it directly.
# %%
# Single cell so both panels (raw scatter + binned decile bars) render in
# one .ipynb output. Split-cell variants left the right panel empty under
# papermill's auto-display.
if multi_day is not None and len(multi_day) > 0:
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Sample for scatter plot (too many points otherwise)
sample = pdf.sample(min(10000, len(pdf)), random_state=42)
# Left panel: scatter OFI vs 5-min return + trend
axes[0].scatter(
sample["ofi_lag1"],
sample["markout_5"] * 10000,
alpha=0.1,
s=2,
color=COLORS["blue"],
)
axes[0].set_xlabel("OFI (t-1)")
axes[0].set_ylabel("5-Minute Return (bps)")
axes[0].set_title("OFI vs 5-Minute Forward Returns")
axes[0].axhline(0, color="black", linestyle="--", linewidth=0.5)
axes[0].axvline(0, color="black", linestyle="--", linewidth=0.5)
z = np.polyfit(sample["ofi_lag1"].dropna(), sample["markout_5"].dropna() * 10000, 1)
p = np.poly1d(z)
x_line = np.linspace(-1, 1, 100)
axes[0].plot(
x_line, p(x_line), color=COLORS["warm"], linewidth=2, label=f"Trend (slope={z[0]:.2f})"
)
axes[0].legend()
# Right panel: binned decile analysis
pdf["ofi_bin"] = pd.qcut(pdf["ofi_lag1"], q=10, labels=False, duplicates="drop")
binned = pdf.groupby("ofi_bin")["markout_5"].mean() * 10000
colors = [COLORS["warm"] if v < 0 else COLORS["accent"] for v in binned.values]
axes[1].bar(range(len(binned)), binned.values, color=colors, edgecolor="black", linewidth=0.5)
axes[1].set_xlabel("OFI Decile (1=Most Selling, 10=Most Buying)")
axes[1].set_ylabel("Mean 5-Min Return (bps)")
axes[1].set_title("Mean 5-Min Return by OFI Decile (Within Noise)")
axes[1].axhline(0, color="black", linestyle="--", linewidth=0.5)
spread = binned.iloc[-1] - binned.iloc[0]
axes[1].annotate(
f"Spread: {spread:.1f} bps",
xy=(8.5, binned.iloc[-1]),
fontsize=10,
ha="center",
)
plt.suptitle(f"OFI Predictive Power - {SYMBOL} ({len(pdf):,} observations)", fontsize=12)
plt.tight_layout()
plt.show()
# %% [markdown]
# **The binned analysis shows no usable directional structure**:
#
# - The decile means alternate sign, and the largest-magnitude bins are interior
# rather than at the extremes — there is no monotone ramp and no clean tail signal.
# - The bottom (heavy-selling) decile mean is positive and the top (heavy-buying)
# decile mean is negative — the opposite of a momentum read.
# - The top-minus-bottom spread is about −0.7 bps: slightly negative and, like the
# near-zero correlations above, within sampling noise of zero.
#
# A decile spread that is within noise and on the order of round-trip execution
# costs (~1–2 bps for liquid US equities) is the realistic baseline for a raw
# microstructure signal at this horizon: there is no edge to harvest before costs,
# let alone after them. The next panel makes the cost frictions explicit and shows
# why even a marginally positive correlation would not convert to tradable P&L.
# %% [markdown]
# ## 5. From Price Response to Tradable P&L: Three Markout Types
#
# The correlation above is a *price-response* signal measured on the midpoint.
# What a desk actually keeps is smaller, because two frictions sit between the
# signal and the fill. We separate them with three markouts at each horizon:
#
# - **Midpoint (p₂ − p₀)**: the raw price response — return of the quote
# midpoint from signal time to *h* bars ahead. This is the number the OFI
# correlation is built on, and the most generous reading of the edge.
# - **Latency-adjusted (p₂ − p₁)**: the same midpoint return after a one-bar
# execution delay. The gap to the midpoint markout is the alpha that decays
# while the order is in flight.
# - **Executable (bid/ask)**: a long pays the **ask** at decision time and
# exits at the **bid** *h* bars later. The gap to the midpoint markout is the
# spread the book charges — the friction §3.3 warns turns a statistical
# signal into negative trading P&L at sub-minute horizons.
#
# Reading the three together shows where the edge goes: latency erodes it, and
# the spread can erase it outright.
# %%
# Plot midpoint vs executable markout distributions across horizons, and
# summarize all three types (midpoint / latency-adjusted / executable).
if multi_day is not None and len(multi_day) > 0:
summary_rows = []
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
for i, h in enumerate([1, 5, 10, 30]):
ax = axes[i // 2, i % 2]
mid_col, adj_col, exec_col = f"markout_{h}", f"markout_{h}_adj", f"markout_{h}_exec"
pdf_clean = multi_day.drop_nulls([mid_col, exec_col]).to_pandas()
def _clip_bps(series):
return series.clip(lower=series.quantile(0.01), upper=series.quantile(0.99)) * 10000
ax.hist(
_clip_bps(pdf_clean[mid_col]),
bins=50,
alpha=0.5,
label="Midpoint (p₂-p₀)",
color=COLORS["blue"],
density=True,
)
ax.hist(
_clip_bps(pdf_clean[exec_col]),
bins=50,
alpha=0.5,
label="Executable (bid/ask)",
color=COLORS["warm"],
density=True,
)
ax.axvline(0, color="black", linestyle="--", linewidth=0.5)
ax.set_xlabel("Markout (bps)")
ax.set_ylabel("Density")
ax.set_title(f"{h}-Minute Horizon")
ax.legend()
mid_mean = pdf_clean[mid_col].mean() * 10000
exec_mean = pdf_clean[exec_col].mean() * 10000
ax.annotate(
f"mean: {mid_mean:+.1f} vs {exec_mean:+.1f} bps\nspread cost: {mid_mean - exec_mean:.1f} bps",
xy=(0.95, 0.95),
xycoords="axes fraction",
ha="right",
va="top",
fontsize=9,
)
# Latency-adjusted markout is identically zero at h == LATENCY_BARS
# (p₂ == p₁ by construction), so report its mean only past that horizon.
adj_mean = (
multi_day.drop_nulls([adj_col])[adj_col].mean() * 10000 if h > LATENCY_BARS else None
)
summary_rows.append((h, mid_mean, adj_mean, exec_mean))
plt.suptitle(f"Midpoint vs Executable Markouts by Horizon - {SYMBOL}", fontsize=12)
plt.tight_layout()
plt.show()
print("=== Mean markout by type (bps) ===")
print(f"{'Horizon':<10}{'Midpoint':>12}{'Latency-adj':>14}{'Executable':>14}")
print("-" * 50)
for h, mid_mean, adj_mean, exec_mean in summary_rows:
adj_str = f"{adj_mean:>14.1f}" if adj_mean is not None else f"{'≡ 0':>14}"
print(f"{h:>3} min {mid_mean:>12.1f}{adj_str}{exec_mean:>14.1f}")
# %% [markdown]
# **Key insights from the markout analysis**:
#
# 1. **Midpoint markout is the generous view**: it credits the signal with the
# full price response and ignores both execution delay and the spread.
#
# 2. **Latency erodes the edge**: a single bar of delay (p₂ − p₁) removes the
# fraction of the move that lands in the first minute — largest at short
# horizons, negligible by 30 minutes once total movement dominates.
#
# 3. **The spread can erase it**: paying the ask and exiting at the bid shifts
# the executable markout left of the midpoint markout by roughly one spread.
# At sub-minute horizons that gap routinely exceeds the signal itself, which
# is why a positive midpoint correlation is necessary but not sufficient for
# tradable P&L.
# %% [markdown]
# ## 6. Save Results
#
# Export the processed minute bars for use in downstream analysis.
# %%
if multi_day is not None and len(multi_day) > 0:
output_file = OUTPUT_DIR / f"{SYMBOL}_minute_bars.parquet"
multi_day.write_parquet(output_file)
print(f"Saved: {output_file}")
print(f"Shape: {multi_day.shape}")
# %% [markdown]
# ## Key Takeaways
#
# ### 1. OFI ↔ Forward-Return Correlation
#
# OFI has a positive correlation with forward returns ranging from ~0.002 at
# the 1-minute horizon to ~0.02 at the 30-minute horizon. The sign is the
# expected one — aggressive buying co-moves with subsequent positive returns
# — but the per-observation magnitudes are small relative to the dispersion
# of returns at the same horizons.
#
# ### 2. Horizon Dependence
#
# The 1-5 minute correlations sit around 0.002-0.004 and the 30-minute
# correlation around 0.02. This notebook does not separate that horizon
# dependence into a permanent vs transient component; it reports the
# unconditional correlation at each horizon.
#
# ### 3. Latency Matters
#
# The difference between standard and latency-adjusted markouts shows that
# execution speed is critical. Traders who can't act within 1 minute lose
# a meaningful fraction of the available alpha.
#
# ### 4. Practical Alpha Is Limited
#
# The top-minus-bottom OFI decile spread is about −0.7 bps on this slice — within
# noise and on the order of round-trip transaction costs. There is no standalone
# edge here: the decile means alternate sign with no tail structure, matching the
# near-zero correlations. OFI is more useful as a **filter** (avoid trading against
# strong flow) than as a primary alpha source, and any directional use would demand
# far stronger, cost-aware evidence than this horizon provides.
#
# ### DataBento vs ITCH: When to Use Each
#
# | Aspect | ITCH (Free) | DataBento (~$10/symbol/mo) |
# |--------|-------------|----------------------------|
# | Format | Binary, all NASDAQ stocks | Parquet, per-symbol |
# | Preprocessing | Significant | Minimal |
# | Multi-exchange | No | Yes |
# | Best for | Learning, backtesting | Production, research |
#
# ---
#
# **Next**: The notebooks in Chapter 8 build on these microstructure insights
# to engineer alpha factors for machine learning models.
03_market_microstructure/10_iex_lob_reconstruction.ipynb
+52578
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/10_iex_lob_reconstruction.py
+752
View File
@@ -0,0 +1,752 @@
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # IEX LOB Reconstruction: Free Market Data Alternative
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Reconstruct a price-level (L2) limit order book from IEX DEEP — the only
# free, redistribution-friendly L2 feed at this granularity — and contrast
# the resulting book with the order-level (L3) NASDAQ ITCH reconstruction in
# `02_itch_lob_reconstruction`.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Parse IEX DEEP pcap binary into structured messages and explain the
# feed's 11 message types versus ITCH's ~20.
# - Maintain a per-price-level book from `PriceLevelUpdate` messages and
# recognize what depth-only feeds cannot tell you (no individual order
# tracking, no per-order cancellation rates).
# - Validate the reconstruction (positive spreads, sane depth) and quantify
# what's lost relative to ITCH (queue position, order lifecycles).
#
# ## Book reference
#
# Section §3.2 (data-feed taxonomy entry for IEX HIST/DEEP) and §3.3
# (notebook is named alongside ITCH and DataBento as a third
# reconstruction implementation).
#
# ## Prerequisites
#
# - IEX DEEP pcap at
# `data/equities/market/microstructure/iex/deep/*.pcap.gz`
# (downloaded via `data/equities/market/microstructure/iex_download.py --deep --smallest`).
#
# > **Data note (important):** `--smallest` selects IEX's lowest-volume DEEP file
# > to keep the download tiny. On the shipped run that file is a non-trading day
# > whose only instruments are IEX's internal **test symbols** (ZIEXT, ZEXIT,
# > ZXIET) at flat placeholder prices. So the reconstruction, spread, and depth
# > statistics below validate the *parser and L2 algorithm* against IEX's
# > synthetic test instruments — they are **not** the microstructure of a traded
# > security. To reproduce these analyses on a real name, download a full
# > trading-day DEEP file (drop `--smallest`) and filter to the symbol of
# > interest; the code path is identical.
#
# ## IEX vs NASDAQ: Key Differences
#
# | Aspect | NASDAQ ITCH | IEX DEEP |
# |--------|-------------|----------|
# | LOB granularity | Order-level (L3) | Price-level aggregated (L2) |
# | Market share | ~20% | ~3% |
# | Speed bump | None | 350μs delay |
# | Message types | ~20 | 11 |
# | Data access | Licensed | Free public download |
#
# > **Attribution (Required):** Data provided for free by IEX. By accessing or
# > using IEX Historical Data, you agree to the
# > [IEX Historical Data Terms of Use](https://www.iexexchange.io/legal/hist-data-terms).
# >
# > **Note on DEEP**: full depth at each price level but without individual
# > order IDs — you see total size at each price, but not how many orders
# > comprise it.
# %%
"""IEX LOB Reconstruction — free market data alternative for limit order book analysis."""
import warnings
from pathlib import Path
warnings.filterwarnings("ignore")
import polars as pl
from data import load_iex_hist
from utils.paths import get_output_dir
# %% tags=["parameters"]
MAX_MESSAGES = 0 # 0 = all messages
SAVE_PARSED = True # Save parsed data to output directory
# %%
# Paths
OUTPUT_DIR = get_output_dir(3, "iex_deep")
# Normalize MAX_MESSAGES: 0 means no limit
if MAX_MESSAGES == 0:
MAX_MESSAGES = None
# %% [markdown]
# ## 1. Load Raw IEX HIST Data
#
# IEX provides HIST data on a T+1 basis, with 12 months of rolling history.
# We use the canonical loader to find downloaded pcap files.
#
# **Available feeds:**
# - **TOPS** (Top of Book) - Best bid/ask quotes and trades
# - **DEEP** (Depth of Book) - Full price-level depth (required for LOB reconstruction)
#
# To download data, run:
# ```bash
# python data/equities/market/microstructure/iex_download.py --deep --smallest
# ```
# %%
# Get raw pcap files from canonical location
deep_files = load_iex_hist(feed="deep", get_raw_files=True)
assert deep_files, (
"No IEX DEEP data found. Download first:\n"
" uv run python data/equities/market/microstructure/iex_download.py --deep --smallest"
)
print(f"Found {len(deep_files)} DEEP file(s):")
for f in deep_files:
size_mb = f.stat().st_size / 1e6
print(f" {f.name} ({size_mb:.1f} MB)")
try:
tops_files = load_iex_hist(feed="tops", get_raw_files=True)
print(f"\nFound {len(tops_files)} TOPS file(s):")
for f in tops_files:
size_mb = f.stat().st_size / 1e6
print(f" {f.name} ({size_mb:.1f} MB)")
except Exception:
tops_files = [] # TOPS is optional
# %% [markdown]
# ## 2. Parse IEX-TP Protocol
#
# IEX uses their own binary protocol (IEX-TP) which wraps message payloads.
# The `iex_parser` library handles the low-level decoding.
#
# **DEEP Message Types:**
# - `price_level_update` - LOB depth changes at each price level
# - `quote_update` - Best bid/ask changes
# - `trade_report` - Executed trades
# %%
from iex_parser import DEEP_1_0, TOPS_1_6, Parser
print("iex_parser library available")
# %% [markdown]
# ### Extract Symbol from IEX Message
#
# Decode the symbol field from raw bytes or string in an IEX message.
# %%
def _extract_symbol(msg: dict) -> str:
"""Decode symbol from raw IEX message (handles bytes and str)."""
symbol_raw = msg.get("symbol", b"")
if isinstance(symbol_raw, bytes):
return symbol_raw.decode().strip()
return str(symbol_raw).strip()
# %% [markdown]
# ### Normalize IEX Side Field
#
# IEX encodes the order side in several formats; normalize to "bid" or "ask".
# %%
def _normalize_side(msg: dict, side_debug: set) -> str:
"""Normalize the IEX side field to 'bid' or 'ask', tracking raw values for debugging."""
# Side can be "B"/"S", b"B"/b"S", 0/1, "buy"/"sell", or bytes
side_raw_orig = msg.get("side", msg.get("msg_flag", ""))
side_debug.add((type(side_raw_orig).__name__, repr(side_raw_orig)))
side_raw = side_raw_orig
if isinstance(side_raw, bytes):
side_raw = side_raw.decode()
side_raw = str(side_raw).upper().strip()
# IEX spec: 0x42='B' (buy side), 0x53='S' (sell side)
is_bid = side_raw in ("B", "BUY", "0", "66") # 66 = ord('B')
return "bid" if is_bid else "ask"
# %% [markdown]
# ### Dispatch a Single DEEP Message
#
# Route each parsed message to the appropriate buffer based on its type.
# %%
def _process_deep_message(msg, symbol_str, trades, quotes, price_levels, side_debug):
"""Dispatch a single DEEP message into the appropriate accumulator list."""
msg_type = msg.get("type")
if msg_type == "price_level_update":
price_levels.append(
{
"timestamp": msg["timestamp"],
"symbol": symbol_str,
"side": _normalize_side(msg, side_debug),
"price": float(msg["price"]),
"size": msg["size"],
}
)
elif msg_type == "quote_update":
bid = float(msg["bid_price"])
ask = float(msg["ask_price"])
if bid > 0 and ask > 0:
quotes.append(
{
"timestamp": msg["timestamp"],
"symbol": symbol_str,
"bid_price": bid,
"bid_size": msg["bid_size"],
"ask_price": ask,
"ask_size": msg["ask_size"],
}
)
elif msg_type == "trade_report":
price = float(msg["price"])
if price > 0:
trades.append(
{
"timestamp": msg["timestamp"],
"symbol": symbol_str,
"price": price,
"size": msg["size"],
}
)
# %% [markdown]
# ### Build Result DataFrames
#
# Convert accumulated message buffers into Polars DataFrames for downstream analysis.
# %%
def _buffers_to_dataframes(
trades: list, quotes: list, price_levels: list, total: int, side_debug: set
) -> dict:
"""Package accumulated message buffers into a result dict of DataFrames."""
return {
"trades": pl.DataFrame(trades) if trades else pl.DataFrame(),
"quotes": pl.DataFrame(quotes) if quotes else pl.DataFrame(),
"price_levels": pl.DataFrame(price_levels) if price_levels else pl.DataFrame(),
"total_messages": total,
"side_debug": side_debug,
}
# %% [markdown]
# ### Parse IEX DEEP Messages
#
# Extract trades, quotes, and price level updates from raw IEX DEEP pcap files.
# %%
def parse_iex_deep(
pcap_path: Path,
symbols: list[str] | None = None,
max_messages: int | None = None,
) -> dict:
"""
Parse IEX DEEP pcap file into structured DataFrames.
DEEP provides full depth of book via price_level_update messages,
plus all TOPS message types (quotes, trades).
Parameters
----------
pcap_path : Path
Path to .pcap.gz file (must be DEEP format, not TOPS)
symbols : list[str], optional
Filter to specific symbols (e.g., ['AAPL', 'MSFT'])
max_messages : int, optional
Stop after N messages (for testing large files)
Returns
-------
dict with keys: 'trades', 'quotes', 'price_levels', 'total_messages'
"""
trades = []
quotes = []
price_levels = []
total = 0
side_debug = set()
symbols_upper = None
if symbols:
symbols_upper = {s.upper().strip() for s in symbols}
with Parser(str(pcap_path), DEEP_1_0) as reader:
try:
msg_iter = iter(reader)
except Exception:
msg_iter = reader
while True:
try:
msg = next(msg_iter)
except StopIteration:
break
except (ValueError, OSError, EOFError) as e:
# scapy pcap reader can raise ValueError on Python 3.13+
# or at EOF in gzipped pcap files — treat as end of stream
print(f"\n [Parser] Stopped after {total:,} messages ({type(e).__name__})")
break
total += 1
symbol_str = _extract_symbol(msg)
if symbols_upper and symbol_str not in symbols_upper:
continue
_process_deep_message(msg, symbol_str, trades, quotes, price_levels, side_debug)
if max_messages and total >= max_messages:
break
return _buffers_to_dataframes(trades, quotes, price_levels, total, side_debug)
# %% [markdown]
# ## 3. Parse DEEP Data
#
# Parse the downloaded DEEP file to extract quotes, trades, and price level updates.
# %%
if deep_files:
pcap_file = deep_files[0]
print(f"Parsing: {pcap_file.name}")
data = parse_iex_deep(pcap_file, max_messages=MAX_MESSAGES)
print(f"\nParsed {data['total_messages']:,} messages")
print(f" Quotes: {len(data['quotes']):,}")
print(f" Trades: {len(data['trades']):,}")
print(f" Price Levels: {len(data['price_levels']):,}")
# Debug: show what raw side values we saw
if data.get("side_debug"):
print(f"\nRaw side values seen: {data['side_debug']}")
# Show sample of each
if not data["quotes"].is_empty():
print("\nSample quotes:")
print(data["quotes"].head(5))
if not data["trades"].is_empty():
print("\nSample trades:")
print(data["trades"].head(5))
if not data["price_levels"].is_empty():
print("\nSample price levels:")
print(data["price_levels"].head(5))
# Diagnostic: side distribution
side_counts = data["price_levels"].group_by("side").agg(pl.len().alias("count"))
print("\nSide distribution:")
print(side_counts)
else:
print("No DEEP files to parse")
data = None
# %% [markdown]
# ## 4. LOB Reconstruction from Price Level Updates
#
# IEX DEEP provides **price-level aggregated** updates, simplifying reconstruction:
# - Each `price_level_update` tells you the new total size at a price
# - Size of 0 means that price level is removed
# - No need to track individual order IDs (unlike ITCH)
# %%
class IEXOrderBook:
"""
Simple limit order book reconstructed from IEX DEEP price level updates.
Unlike ITCH reconstruction (which tracks individual orders), IEX provides
aggregated sizes at each price level, simplifying the state machine.
"""
def __init__(self, symbol: str):
self.symbol = symbol
self.bids: dict[float, int] = {} # price -> size
self.asks: dict[float, int] = {} # price -> size
self.last_update = None
def update(self, timestamp: int, side: str, price: float, size: int):
"""Apply a price level update."""
book = self.bids if side == "bid" else self.asks
if size == 0:
# Remove price level
book.pop(price, None)
else:
# Update size at price
book[price] = size
self.last_update = timestamp
def best_bid(self) -> tuple[float, int] | None:
"""Return (price, size) of best bid."""
if not self.bids:
return None
price = max(self.bids.keys())
return (price, self.bids[price])
def best_ask(self) -> tuple[float, int] | None:
"""Return (price, size) of best ask."""
if not self.asks:
return None
price = min(self.asks.keys())
return (price, self.asks[price])
def spread(self) -> float | None:
"""Return bid-ask spread."""
bid = self.best_bid()
ask = self.best_ask()
if bid and ask:
return ask[0] - bid[0]
return None
def midpoint(self) -> float | None:
"""Return midpoint price."""
bid = self.best_bid()
ask = self.best_ask()
if bid and ask:
return (bid[0] + ask[0]) / 2
return None
def depth(self, levels: int = 5) -> dict:
"""Return top N levels on each side."""
bid_prices = sorted(self.bids.keys(), reverse=True)[:levels]
ask_prices = sorted(self.asks.keys())[:levels]
return {
"bids": [(p, self.bids[p]) for p in bid_prices],
"asks": [(p, self.asks[p]) for p in ask_prices],
}
# %% [markdown]
# ### Capture a Single LOB Snapshot
#
# Record the current book state (best bid/ask, spread, midpoint) at a given timestamp.
# %%
def _capture_snapshot(book: IEXOrderBook, timestamp) -> dict:
"""Return a snapshot dict of the current order book state."""
bid = book.best_bid()
ask = book.best_ask()
return {
"timestamp": timestamp,
"bid_price": bid[0] if bid else None,
"bid_size": bid[1] if bid else None,
"ask_price": ask[0] if ask else None,
"ask_size": ask[1] if ask else None,
"spread": book.spread(),
"midpoint": book.midpoint(),
}
# %% [markdown]
# ### Reconstruct LOB Snapshots
#
# Build periodic snapshots from price level updates using the IEXOrderBook state machine.
# %%
def reconstruct_lob_snapshots(
price_levels: pl.DataFrame,
symbol: str,
snapshot_interval_sec: float = 1.0, # 1 second default
) -> pl.DataFrame:
"""
Reconstruct periodic LOB snapshots from price level updates.
Parameters
----------
price_levels : pl.DataFrame
Price level updates for a single symbol
symbol : str
Symbol to reconstruct
snapshot_interval_sec : float
Seconds between snapshots (default: 1.0)
Returns
-------
DataFrame with columns: timestamp, bid_price, bid_size, ask_price, ask_size,
spread, midpoint
"""
from datetime import timedelta
# Filter to symbol and sort by time
df = price_levels.filter(pl.col("symbol") == symbol).sort("timestamp")
if df.is_empty():
return pl.DataFrame()
book = IEXOrderBook(symbol)
snapshots = []
# Get first timestamp - could be datetime or nanoseconds
start_time = df["timestamp"][0]
snapshot_delta = timedelta(seconds=snapshot_interval_sec)
# Handle both datetime and integer nanoseconds
if isinstance(start_time, int):
# Convert ns integer to datetime
import datetime as dt
start_time = dt.datetime.fromtimestamp(start_time / 1e9)
# Convert column too
df = df.with_columns(pl.col("timestamp").cast(pl.Datetime("ns")).alias("timestamp"))
next_snapshot = start_time + snapshot_delta
for row in df.iter_rows(named=True):
# Take snapshot if interval elapsed
while row["timestamp"] >= next_snapshot:
snapshots.append(_capture_snapshot(book, next_snapshot))
next_snapshot += snapshot_delta
# Apply update
book.update(row["timestamp"], row["side"], row["price"], row["size"])
return pl.DataFrame(snapshots)
# %% [markdown]
# ## 5. Reconstruct LOB for Active Symbols
# %%
if data and not data["price_levels"].is_empty():
# Find most active symbols
symbol_counts = (
data["price_levels"]
.group_by("symbol")
.agg(pl.len().alias("count"))
.sort("count", descending=True)
)
print("Most active symbols (by price level updates):")
print(symbol_counts.head(10))
# Reconstruct LOB for top symbol
top_symbol = symbol_counts["symbol"][0]
print(f"\nReconstructing LOB for: {top_symbol}")
snapshots = reconstruct_lob_snapshots(
data["price_levels"],
top_symbol,
snapshot_interval_sec=1.0, # 1 second
)
print(f"Generated {len(snapshots):,} snapshots")
if not snapshots.is_empty():
print("\nSample snapshots:")
print(snapshots.head(10))
else:
print("No price level data to reconstruct")
snapshots = pl.DataFrame()
top_symbol = None
# %%
# Check data completeness
if not snapshots.is_empty():
valid_spreads = snapshots.filter(pl.col("spread").is_not_null())
valid_midpoints = snapshots.filter(pl.col("midpoint").is_not_null())
print("Data completeness:")
print(
f" Snapshots with spread: {len(valid_spreads):,} ({100 * len(valid_spreads) / len(snapshots):.1f}%)"
)
print(
f" Snapshots with midpoint: {len(valid_midpoints):,} ({100 * len(valid_midpoints) / len(snapshots):.1f}%)"
)
if not valid_spreads.is_empty():
print("\nSpread statistics:")
print(f" Min: ${valid_spreads['spread'].min():.4f}")
print(f" Mean: ${valid_spreads['spread'].mean():.4f}")
print(f" Max: ${valid_spreads['spread'].max():.4f}")
else:
print("\nWARNING: No valid spread data - order book may be one-sided")
# %% [markdown]
# ## 6. Save Parsed Data to Canonical Location
#
# Save the parsed data so it can be loaded directly via `load_iex_hist()`.
# %%
if data and SAVE_PARSED:
# Extract date from filename (e.g., 20180908_IEXTP1_DEEP1.0.pcap.gz)
date_str = pcap_file.name.split("_")[0]
# Save each data type to OUTPUT_DIR
for dtype, parsed_df in [
("quotes", data["quotes"]),
("trades", data["trades"]),
("price_levels", data["price_levels"]),
]:
if not parsed_df.is_empty():
type_dir = OUTPUT_DIR / dtype
type_dir.mkdir(parents=True, exist_ok=True)
output_path = type_dir / f"{date_str}.parquet"
parsed_df.write_parquet(output_path)
print(f"Saved {len(parsed_df):,} rows to {output_path}")
print(f"\nData saved to: {OUTPUT_DIR}")
else:
print("Skipping save (SAVE_PARSED=False or no data)")
# %% [markdown]
# ## 7. Visualization: LOB Evolution
# %%
import plotly.graph_objects as go
from plotly.subplots import make_subplots
if not snapshots.is_empty() and top_symbol:
# Filter to rows with valid midpoint and spread (need both bid and ask for these)
valid_snapshots = snapshots.filter(
pl.col("midpoint").is_not_null() & pl.col("spread").is_not_null()
)
print(f"Total snapshots: {len(snapshots):,}")
print(f"Valid snapshots (with midpoint/spread): {len(valid_snapshots):,}")
if valid_snapshots.is_empty():
print("\nWARNING: No valid snapshots to visualize.")
print("This can happen when the order book only has one side (bids or asks, not both).")
print("Test symbols like ZIEXT/ZEXIT may not have two-sided markets.")
# Show what we do have
has_bid = snapshots.filter(pl.col("bid_price").is_not_null())
has_ask = snapshots.filter(pl.col("ask_price").is_not_null())
print(f"\nSnapshots with bids: {len(has_bid):,}")
print(f"Snapshots with asks: {len(has_ask):,}")
else:
print("No snapshots to visualize")
# %%
if not snapshots.is_empty() and top_symbol:
valid_snapshots = snapshots.filter(
pl.col("midpoint").is_not_null() & pl.col("spread").is_not_null()
)
if not valid_snapshots.is_empty():
# Timestamps are already datetime
snapshots_plot = valid_snapshots.with_columns(pl.col("timestamp").alias("time"))
else:
snapshots_plot = pl.DataFrame()
# %%
if not snapshots.is_empty() and top_symbol and not snapshots_plot.is_empty():
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
subplot_titles=(f"{top_symbol} Midpoint", "Bid-Ask Spread"),
vertical_spacing=0.1,
)
# Midpoint
fig.add_trace(
go.Scatter(
x=snapshots_plot["time"].to_list(),
y=snapshots_plot["midpoint"].to_list(),
mode="lines",
name="Midpoint",
line=dict(color="#1a4b6e", width=1),
),
row=1,
col=1,
)
# Spread
fig.add_trace(
go.Scatter(
x=snapshots_plot["time"].to_list(),
y=snapshots_plot["spread"].to_list(),
mode="lines",
name="Spread",
line=dict(color="#2d5a87", width=1),
fill="tozeroy",
fillcolor="rgba(45, 90, 135, 0.2)",
),
row=2,
col=1,
)
# %%
if not snapshots.is_empty() and top_symbol and not snapshots_plot.is_empty():
fig.update_layout(
title=f"IEX LOB Evolution - {top_symbol}",
height=500,
showlegend=True,
template="plotly_white",
)
fig.update_yaxes(title_text="Price ($)", row=1, col=1)
fig.update_yaxes(title_text="Spread ($)", row=2, col=1)
fig.show()
# %% [markdown]
# ## 8. Key Takeaways: IEX vs ITCH
#
# | Concept | ITCH Approach | IEX DEEP Approach |
# |---------|---------------|-------------------|
# | **LOB State** | Track individual orders (Add/Modify/Delete) | Update aggregated price levels |
# | **Trade Attribution** | Match execute messages to orders | Direct trade reports |
# | **Message Volume** | Very high (~millions/day) | Lower (~100k-1M/day) |
# | **Complexity** | Higher (order lifecycle tracking) | Lower (level updates) |
# | **Use Case** | HFT research, order flow analysis | General microstructure, spread analysis |
#
# **Bottom line:** IEX HIST is a free, redistribution-friendly L2 feed that
# supports the LOB-dynamics, spread, and basic-microstructure analyses in this
# notebook. It does not provide order-level (L3) granularity, so per-order
# cancellation rates and queue-position questions require ITCH or DataBento.
# (Recall the data note above: the numbers here come from IEX test symbols on the
# `--smallest` file, so they exercise the pipeline rather than describe a traded
# security — rerun on a full trading-day file for real spread/depth figures.)
#
# ---
#
# > **Attribution:** Data provided for free by IEX. By accessing or using IEX Historical
# > Data, you agree to the [IEX Historical Data Terms of Use](https://www.iexexchange.io/legal/hist-data-terms).
# %% [markdown]
# ## References
#
# - [IEX HIST Data Download](https://iextrading.com/trading/market-data/)
# - [IEX DEEP Specification](https://iextrading.com/docs/IEX%20DEEP%20Specification.pdf)
# - [iex_parser Python Library](https://github.com/rob-blackbourn/iex_parser)
# - [IEX Historical Data Terms](https://www.iexexchange.io/legal/hist-data-terms)
03_market_microstructure/11_algoseek_taq_eda.ipynb
+97657
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/11_algoseek_taq_eda.py
+635
View File
@@ -0,0 +1,635 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # AlgoSeek TAQ: Anatomy of a Market Crash
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Walk through tick-level AAPL TAQ data on March 16, 2020 (the S&P fell 12%,
# its worst single-day drop since 1987) to see how market-microstructure
# observables — quote activity, spread, fragmentation, trade flow, price —
# behave under extreme stress.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Load nanosecond-precision AlgoSeek TAQ event streams via
# `load_nasdaq100_taq` and filter to regular trading hours.
# - Quantify how quote update frequency, NBBO spread, and exchange-of-record
# distribution shift between the calm pre-crisis baseline and the March-16
# panic session.
# - Distinguish trade events from NBBO updates within a single TAQ stream
# and explain why TAQ does not provide depth.
#
# ## Book reference
#
# Section §3.2, *The Anatomy of Modern Market Data Feeds* — AlgoSeek TAQ
# bullet points; §3.3 references this notebook (with `12_algoseek_taq_lob_reconstruction`)
# for tick-level patterns during the March 2020 crash.
#
# ## Prerequisites
#
# - AlgoSeek TAQ parquets under
# `data/equities/market/microstructure/algoseek_taq/` (download via the
# AlgoSeek loader; AAPL on 2020-03-16 is the focus day).
# %%
"""AlgoSeek TAQ: Anatomy of a Market Crash — tick-level microstructure exploration of AAPL on March 16, 2020."""
from datetime import time
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_nasdaq100_taq
# ML4T Blue Period palette
COLORS = {
"blue": "#1E3A5F",
"accent": "#4A90A4",
"warm": "#8B4513",
"neutral": "#5D5D5D",
}
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ## 1. Loading the Data
#
# We load AAPL for March 16, 2020 and immediately filter to regular trading
# hours. Pre-market and after-hours data has thin liquidity and extreme spreads
# that would distort our analysis.
# %%
SYMBOL = "AAPL"
DATE = "20200316"
DATE_ISO = f"{DATE[:4]}-{DATE[4:6]}-{DATE[6:]}"
# Regular trading hours (ET)
MARKET_OPEN = time(9, 30)
MARKET_CLOSE = time(16, 0)
# Load and filter to regular hours
taq_raw = load_nasdaq100_taq(symbols=[SYMBOL], start_date=DATE_ISO, end_date=DATE_ISO)
taq = taq_raw.filter(
(pl.col("timestamp").dt.time() >= MARKET_OPEN) & (pl.col("timestamp").dt.time() <= MARKET_CLOSE)
)
print(f"=== {SYMBOL} on March 16, 2020 ===")
print(f"Raw events: {len(taq_raw):,}")
print(f"Regular hours: {len(taq):,} ({len(taq) / len(taq_raw) * 100:.1f}%)")
# %% [markdown]
# Most activity occurs during regular hours, but the pre/post-market events
# we filtered out would create misleading outliers in our spread analysis.
# %%
# Event type composition
event_counts = (
taq.group_by("event_type")
.len()
.with_columns((pl.col("len") / pl.sum("len") * 100).alias("pct"))
.sort("len", descending=True)
)
print("\nEvent composition:")
for row in event_counts.iter_rows(named=True):
print(f" {row['event_type']:15} {row['len']:>10,} ({row['pct']:5.1f}%)")
# %% [markdown]
# **Key observation**: Quote updates outnumber trades by ~10:1. This reflects
# how market makers continuously adjust their prices in response to order flow
# and information - the quote stream is where price discovery really happens.
# Trades are just the tip of the iceberg.
# %% [markdown]
# ## 2. The Opening Chaos
#
# March 16 opened with a 7% gap down, immediately triggering Level 1 circuit
# breakers. Let's see how trading activity evolved through the day.
# %%
# Extract trades, filtering out erroneous prints
# - Condition 80000002 = late-reported trades with incorrect prices (all at $277.97)
# - Price bounds: AAPL traded $240-260 that day; anything outside is an error
trades = taq.filter(
(pl.col("event_type") == "TRADE")
& (pl.col("conditions") != "80000002") # Exclude erroneous late-reported trades
& (pl.col("price") >= 235) # Exclude unreasonably low prices
& (pl.col("price") <= 265) # Exclude erroneous high prices ($277.97 prints)
)
# Aggregate by minute
minute_activity = (
trades.with_columns(pl.col("timestamp").dt.truncate("1m").alias("minute"))
.group_by("minute")
.agg(
pl.len().alias("trade_count"),
pl.col("quantity").sum().alias("volume"),
pl.col("price").mean().alias("vwap"),
)
.sort("minute")
)
# %%
# Build intraday activity figure
fig = make_subplots(
rows=2,
cols=1,
row_heights=[0.6, 0.4],
shared_xaxes=True,
vertical_spacing=0.08,
)
fig.add_trace(
go.Scatter(
x=minute_activity["minute"].to_list(),
y=minute_activity["trade_count"].to_list(),
name="Trades/min",
line=dict(color=COLORS["blue"], width=1),
fill="tozeroy",
fillcolor="rgba(30, 58, 95, 0.3)",
),
row=1,
col=1,
)
fig.add_trace(
go.Bar(
x=minute_activity["minute"].to_list(),
y=minute_activity["volume"].to_list(),
name="Volume",
marker_color=COLORS["warm"],
opacity=0.7,
),
row=2,
col=1,
)
fig.update_layout(
title=f"Intraday Trading Activity - {SYMBOL} (March 16, 2020)",
height=500,
showlegend=False,
)
fig.update_yaxes(title_text="Trades per Minute", row=1, col=1)
fig.update_yaxes(title_text="Volume (shares)", tickformat=",", row=2, col=1)
fig.update_xaxes(title_text="Time (ET)", row=2, col=1)
fig.show()
# %%
# Quantify the pattern
open_hour = minute_activity.filter(pl.col("minute").dt.hour() == 9)
midday = minute_activity.filter(pl.col("minute").dt.hour().is_between(11, 14))
close_hour = minute_activity.filter(pl.col("minute").dt.hour() == 15)
print("=== Trading Intensity by Period ===")
print(f"Opening hour (9:30-10:30): {open_hour['trade_count'].mean():,.0f} trades/min avg")
print(f"Midday (11:00-14:00): {midday['trade_count'].mean():,.0f} trades/min avg")
print(f"Closing hour (15:00-16:00): {close_hour['trade_count'].mean():,.0f} trades/min avg")
# %% [markdown]
# **The U-shaped pattern is amplified**: On a normal day, we'd see 2-3x more
# activity at open/close vs midday. On March 16, the opening spike is extreme -
# pent-up overnight selling hitting the market all at once. The circuit breaker
# halt (9:34-9:49 AM) is visible as the brief dip after the initial surge.
# %% [markdown]
# ## 3. Spread Dynamics: The Cost of Panic
#
# The bid-ask spread is the price of immediacy. During calm markets, AAPL
# trades with a 1-2 cent spread (~1-2 bps). What happened on March 16?
# %%
# Extract NBBO (National Best Bid/Offer) quotes only
# Use "QUOTE BID NB" and "QUOTE ASK NB" - these are the consolidated best prices
# Filter out zero-price quotes (stale/empty from some exchanges)
nbbo_quotes = (
taq.filter(
pl.col("event_type").str.contains("NB") # Only NBBO quotes
& (pl.col("price") > 0) # Exclude zero-price quotes
)
.select(["timestamp", "event_type", "price"])
.sort("timestamp")
)
# Pivot to get bid and ask columns, then forward-fill
nbbo_raw = (
nbbo_quotes.with_columns(
pl.when(pl.col("event_type") == "QUOTE BID NB").then(pl.col("price")).alias("bid"),
pl.when(pl.col("event_type") == "QUOTE ASK NB").then(pl.col("price")).alias("ask"),
)
.select(["timestamp", "bid", "ask"])
.with_columns(
pl.col("bid").forward_fill(),
pl.col("ask").forward_fill(),
)
.drop_nulls() # Drop rows before we have both bid and ask
)
# %%
# Sample at 1-second intervals for visualization (take last value per second)
nbbo = (
nbbo_raw.group_by_dynamic("timestamp", every="1s")
.agg(
pl.col("bid").last(),
pl.col("ask").last(),
)
.with_columns(
(pl.col("ask") - pl.col("bid")).alias("spread"),
((pl.col("ask") - pl.col("bid")) / ((pl.col("ask") + pl.col("bid")) / 2) * 10000).alias(
"spread_bps"
),
((pl.col("bid") + pl.col("ask")) / 2).alias("midpoint"),
)
.filter(pl.col("spread") > 0) # Remove crossed/locked quotes
.filter(pl.col("spread_bps") < 500) # Remove outliers (>5% spread is data error)
)
# %%
# Spread statistics
spread_stats = nbbo.select(
pl.col("spread_bps").mean().alias("mean"),
pl.col("spread_bps").median().alias("median"),
pl.col("spread_bps").quantile(0.95).alias("p95"),
pl.col("spread_bps").max().alias("max"),
)
print("=== Spread Statistics (Regular Hours) ===")
print(f" Mean: {spread_stats['mean'][0]:>6.1f} bps")
print(f" Median: {spread_stats['median'][0]:>6.1f} bps")
print(f" 95th: {spread_stats['p95'][0]:>6.1f} bps")
print(f" Max: {spread_stats['max'][0]:>6.1f} bps")
# Normal day comparison
print("\n (Normal day median for AAPL: ~1-2 bps)")
# %%
# Spread evolution figure
fig = make_subplots(
rows=2,
cols=1,
row_heights=[0.6, 0.4],
shared_xaxes=True,
vertical_spacing=0.08,
subplot_titles=("Bid-Ask Spread", "Midpoint Price"),
)
spread_cap = nbbo["spread_bps"].quantile(0.99)
fig.add_trace(
go.Scatter(
x=nbbo["timestamp"].to_list(),
y=nbbo["spread_bps"].clip(upper_bound=spread_cap).to_list(),
name="Spread",
line=dict(color=COLORS["warm"], width=1),
fill="tozeroy",
fillcolor="rgba(139, 69, 19, 0.2)",
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=nbbo["timestamp"].to_list(),
y=nbbo["midpoint"].to_list(),
name="Midpoint",
line=dict(color=COLORS["blue"], width=1),
),
row=2,
col=1,
)
fig.update_layout(
title=f"Spread Dynamics During Market Stress - {SYMBOL} (March 16, 2020)",
height=500,
showlegend=False,
)
fig.update_yaxes(title_text=f"Spread (bps, capped at {spread_cap:.0f})", row=1, col=1)
fig.update_yaxes(title_text="Price ($)", row=2, col=1)
fig.update_xaxes(title_text="Time (ET)", row=2, col=1)
fig.show()
# %% [markdown]
# **What the spread tells us**:
#
# - At the open, spreads spiked as market makers widened quotes to protect
# against adverse selection - they couldn't tell if the next trade was
# informed or noise
# - The spread narrows through midday as volatility subsided and market makers
# regained confidence
# - Even the median spread (~2.4 bps) is 2-3x wider than a normal day
# - The spread-price relationship is clear: when price drops sharply, spreads
# widen as uncertainty increases
# %% [markdown]
# ## 4. Exchange Fragmentation: Where Did Liquidity Go?
#
# Modern equity markets are fragmented across 16+ exchanges and dozens of
# dark pools. During stress, does liquidity concentrate or disperse?
# %%
# Exchange distribution
exchange_dist = (
trades.group_by("exchange")
.agg(
pl.len().alias("trades"),
pl.col("quantity").sum().alias("volume"),
)
.with_columns((pl.col("volume") / pl.sum("volume") * 100).alias("share"))
.sort("volume", descending=True)
)
# Show top venues
print("=== Exchange Market Share (by volume) ===")
for row in exchange_dist.head(10).iter_rows(named=True):
print(f" {row['exchange']:4} {row['share']:5.1f}% ({row['trades']:,} trades)")
# %%
# Visualize top 10 exchanges
top_exchanges = exchange_dist.head(10)
fig = px.bar(
top_exchanges.to_pandas(),
y="exchange",
x="share",
orientation="h",
text="share",
color_discrete_sequence=[COLORS["blue"]],
)
fig.update_traces(texttemplate="%{text:.1f}%", textposition="outside")
fig.update_layout(
title=f"Exchange Market Share - {SYMBOL} (March 16, 2020)",
xaxis_title="Volume Share (%)",
yaxis_title="Exchange",
yaxis=dict(categoryorder="total ascending"),
height=400,
showlegend=False,
)
fig.show()
# %% [markdown]
# **Reading the venue breakdown** (the chart labels venues by name):
# - **NASDAQ** (43.2%): the primary listing exchange for AAPL
# - **FINRA** (21.6%): the off-exchange TRF, where dark-pool and internalized
# prints are reported — the second-largest share on this day
# - **Cboe/BATS family**: BATS (9.9%), EDGX (6.7%), EDGA (1.0%), BATS Y (0.8%)
# - **CSE** (6.5%), **NYSE Arca** (6.1%), **NASDAQ BX** (1.9%), **NYSE** (0.8%)
#
# **Key insight**: Even during extreme stress, liquidity remains fragmented.
# No single exchange dominates - algorithmic traders must aggregate across
# venues to get a complete picture of available liquidity.
# %% [markdown]
# ## 5. Trade Size Distribution: Retail vs Institutional
#
# Trade size reveals who's in the market. Small odd-lot trades (<100 shares)
# often indicate retail; larger trades suggest institutional activity.
# %%
# Categorize trade sizes
size_categories = (
trades.with_columns(
pl.when(pl.col("quantity") < 100)
.then(pl.lit("Odd lot (<100)"))
.when(pl.col("quantity") <= 500)
.then(pl.lit("Small (100-500)"))
.when(pl.col("quantity") <= 2000)
.then(pl.lit("Medium (501-2000)"))
.otherwise(pl.lit("Large (>2000)"))
.alias("category")
)
.group_by("category")
.agg(
pl.len().alias("count"),
pl.col("quantity").sum().alias("volume"),
)
.with_columns(
(pl.col("count") / pl.sum("count") * 100).alias("count_pct"),
(pl.col("volume") / pl.sum("volume") * 100).alias("volume_pct"),
)
)
# Order for display
order = ["Odd lot (<100)", "Small (100-500)", "Medium (501-2000)", "Large (>2000)"]
size_categories = size_categories.with_columns(pl.col("category").cast(pl.Enum(order))).sort(
"category"
)
print("=== Trade Size Distribution ===")
print(f"{'Category':<20} {'Trades':>10} {'Volume':>10}")
print("-" * 42)
for row in size_categories.iter_rows(named=True):
print(f"{row['category']:<20} {row['count_pct']:>9.1f}% {row['volume_pct']:>9.1f}%")
# %%
# Visualize the disconnect between trade count and volume
fig = go.Figure()
fig.add_trace(
go.Bar(
name="% of Trades",
x=order,
y=[size_categories.filter(pl.col("category") == c)["count_pct"][0] for c in order],
marker_color=COLORS["blue"],
)
)
fig.add_trace(
go.Bar(
name="% of Volume",
x=order,
y=[size_categories.filter(pl.col("category") == c)["volume_pct"][0] for c in order],
marker_color=COLORS["warm"],
)
)
fig.update_layout(
title=f"Trade Size Distribution - {SYMBOL} (March 16, 2020)",
xaxis_title="Trade Size Category",
yaxis_title="Percentage",
barmode="group",
height=400,
legend=dict(orientation="h", yanchor="bottom", y=1.02, xanchor="right", x=1),
)
fig.show()
# %% [markdown]
# **The odd-lot era**: Odd-lot trades (<100 shares) dominate both by count
# (~99.9%) and by volume (~81%). At AAPL's pre-split price of ~$242, even
# moderate dollar amounts translate to fewer than 100 shares. The high odd-lot
# share of volume reflects both retail participation and institutional
# algorithms that slice orders into small lots to minimize market impact.
#
# The few large trades (>2,000 shares) account for ~18% of volume despite
# being a negligible fraction of trades—these are the block-sized institutional
# prints that move the market.
# %% [markdown]
# ## 6. The Day's Journey: Price Action
#
# Finally, let's see how the price evolved throughout this historic day.
#
# > **Note on prices**: AAPL had a 4:1 stock split on August 31, 2020.
# > The tick data shows pre-split prices (~$240-260). Multiply by 0.25
# > to compare with split-adjusted historical data (~$60-65).
# %%
# Build 5-minute OHLCV bars from cleaned trade data
ohlcv = (
trades.group_by_dynamic("timestamp", every="5m")
.agg(
pl.col("price").first().alias("open"),
pl.col("price").max().alias("high"),
pl.col("price").min().alias("low"),
pl.col("price").last().alias("close"),
pl.col("quantity").sum().alias("volume"),
pl.len().alias("trade_count"), # For sanity checking
)
.sort("timestamp")
.filter(pl.col("trade_count") > 0) # Remove empty bars
)
# Daily summary
day_open = ohlcv["open"][0]
day_high = ohlcv["high"].max()
day_low = ohlcv["low"].min()
day_close = ohlcv["close"][-1]
day_volume = ohlcv["volume"].sum()
print(f"=== {SYMBOL} - March 16, 2020 Summary ===")
print(f" Open: ${day_open:.2f}")
print(f" High: ${day_high:.2f}")
print(f" Low: ${day_low:.2f}")
print(f" Close: ${day_close:.2f}")
print(f" Volume: {day_volume:,.0f} shares")
print(
f"\n Intraday range: ${day_high - day_low:.2f} ({(day_high - day_low) / day_open * 100:.1f}%)"
)
print(f" Daily return: {(day_close / day_open - 1) * 100:+.1f}%")
# %%
# Candlestick chart with volume
# Build the candlestick + volume figure in a SINGLE cell so the inline
# backend doesn't flush an intermediate (candle-only, no labels) render.
fig = make_subplots(
rows=2,
cols=1,
row_heights=[0.7, 0.3],
shared_xaxes=True,
vertical_spacing=0.05,
)
fig.add_trace(
go.Candlestick(
x=ohlcv["timestamp"].to_list(),
open=ohlcv["open"].to_list(),
high=ohlcv["high"].to_list(),
low=ohlcv["low"].to_list(),
close=ohlcv["close"].to_list(),
increasing_line_color=COLORS["accent"],
decreasing_line_color=COLORS["warm"],
name="OHLC",
),
row=1,
col=1,
)
colors = [
COLORS["accent"] if c >= o else COLORS["warm"]
for o, c in zip(ohlcv["open"].to_list(), ohlcv["close"].to_list(), strict=False)
]
fig.add_trace(
go.Bar(
x=ohlcv["timestamp"].to_list(),
y=ohlcv["volume"].to_list(),
marker_color=colors,
opacity=0.7,
name="Volume",
),
row=2,
col=1,
)
fig.update_layout(
title=f"Intraday Price Action - {SYMBOL} (March 16, 2020)",
xaxis_rangeslider_visible=False,
height=550,
showlegend=False,
)
fig.update_yaxes(title_text="Price ($)", row=1, col=1)
fig.update_yaxes(title_text="Volume", tickformat=",", row=2, col=1)
fig.update_xaxes(title_text="Time (ET)", row=2, col=1)
fig.show()
# %% [markdown]
# **Reading the chart**:
#
# - **9:30 AM**: Opens at ~$242 (pre-split), about 13% below Friday's close of ~$278
# - **9:34-9:49 AM**: Circuit breaker halt (S&P 500 fell 7%)
# - **9:49-11:00 AM**: Resumed trading, continued volatility
# - **11:00 AM-2:00 PM**: Price consolidates in the $248-253 range
# - **2:00-4:00 PM**: Sells off into close, finishing near the open
#
# The ~$24 intraday range (~10% of price) is extraordinary — AAPL typically
# moves 1-2% in a day. This kind of volatility creates both opportunity and
# risk for algorithmic traders: spreads widen, but so do potential profits
# from correct directional bets.
# %% [markdown]
# ## Key Takeaways
#
# **1. Quote activity dominates**: 10x more quote updates than trades - this is
# where price discovery happens. Analyzing only trades misses most of the story.
#
# **2. Spreads reveal stress**: The 2-3 bps median spread (vs ~1 bps normally)
# shows market makers demanding compensation for uncertainty. Execution costs
# on March 16 were 2-3x higher than normal.
#
# **3. Fragmentation persists**: Even during panic, no single exchange captures
# majority flow. Algorithmic traders must aggregate liquidity across venues.
#
# **4. Odd lots dominate**: At AAPL's pre-split price (~$242), odd-lot trades
# dominate both by count (~99.9%) and volume (~81%), reflecting algorithmic
# order slicing and the retail trading boom during COVID.
#
# **5. The U-shape amplifies**: The normal open/close activity concentration
# becomes extreme during stress as participants rush to adjust positions.
#
# ## Next Steps
#
# - **LOB Reconstruction**: [`12_algoseek_taq_lob_reconstruction`](12_algoseek_taq_lob_reconstruction.ipynb) - Build
# NBBO at each trade for Lee-Ready classification
# - **Minute Bars**: [`13_algoseek_minute_bars_eda`](13_algoseek_minute_bars_eda.ipynb) - Pre-aggregated data
# for longer-horizon analysis
03_market_microstructure/12_algoseek_taq_lob_reconstruction.ipynb
+8403
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/12_algoseek_taq_lob_reconstruction.py
+599
View File
@@ -0,0 +1,599 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # TAQ LOB Reconstruction: Measuring Trade Aggression
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Build a forward-filled NBBO timeline from AlgoSeek TAQ events, classify
# each AAPL trade on 2020-03-16 with the Lee-Ready algorithm, and use the
# resulting buy/sell stream to compute order-imbalance and trade-aggression
# metrics that characterize the crash session.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Interleave trade and quote events on a nanosecond timeline and use
# forward-fill to attach the prevailing NBBO to each trade.
# - Apply the Lee-Ready quote-test + tick-test cascade and read out
# buy/sell ratios across the trading day.
# - Generate cumulative-order-imbalance and effective-spread visualizations
# that quantify "the cost of immediacy during panic".
#
# ## Book reference
#
# Section §3.2 (`Notebooks 15-16 analyze tick-level patterns during the
# March 2020 crash`); §3.3 references the wider stylized-facts pattern.
#
# ## Prerequisites
#
# - AlgoSeek TAQ parquets (AAPL, 2020-03-16) accessible via `load_nasdaq100_taq`.
#
# ## The Lee-Ready Algorithm
#
# Lee and Ready (1991) proposed a simple rule:
#
# 1. **Quote test**: If trade price > midpoint → buyer initiated; < midpoint → seller
# 2. **Tick test**: If at midpoint, use price change: uptick → buy, downtick → sell
#
# Validated to ~94-95% accuracy on modern markets in
# [`15_itch_lee_ready`](15_itch_lee_ready.ipynb) using DataBento's
# ground-truth aggressor labels.
# %%
"""TAQ LOB Reconstruction — measuring trade aggression with Lee-Ready classification."""
from datetime import time
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_nasdaq100_taq
COLORS = {
"blue": "#1E3A5F",
"accent": "#4A90A4",
"warm": "#8B4513",
"buy": "#228B22",
"sell": "#B22222",
"neutral": "#5D5D5D",
}
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ## 1. Load and Filter Data
#
# We filter to regular trading hours (9:30 AM - 4:00 PM ET) to avoid pre-market
# artifacts. During pre-market, thin liquidity creates artificially wide spreads
# that would distort our analysis.
# %%
SYMBOL = "AAPL"
DATE = "20200316"
DATE_ISO = f"{DATE[:4]}-{DATE[4:6]}-{DATE[6:]}"
MARKET_OPEN = time(9, 30)
MARKET_CLOSE = time(16, 0)
taq_raw = load_nasdaq100_taq(symbols=[SYMBOL], start_date=DATE_ISO, end_date=DATE_ISO)
taq = taq_raw.filter(
(pl.col("timestamp").dt.time() >= MARKET_OPEN) & (pl.col("timestamp").dt.time() <= MARKET_CLOSE)
)
print(f"=== {SYMBOL} on March 16, 2020 ===")
print(f"Raw events: {len(taq_raw):,}")
print(f"Regular hours: {len(taq):,}")
# %% [markdown]
# ## 2. Build the NBBO Timeline
#
# For each trade, we need the prevailing NBBO. The challenge: quotes and trades
# are interleaved in time. We use forward-fill to carry the last known bid/ask
# to each trade timestamp.
# %%
# Extract quote and trade events
bids = (
taq.filter(pl.col("event_type") == "QUOTE BID")
.select(["timestamp", pl.col("price").alias("bid"), pl.col("quantity").alias("bid_size")])
.sort("timestamp")
)
asks = (
taq.filter(pl.col("event_type") == "QUOTE ASK")
.select(["timestamp", pl.col("price").alias("ask"), pl.col("quantity").alias("ask_size")])
.sort("timestamp")
)
trades = (
taq.filter(pl.col("event_type") == "TRADE")
.select(
["timestamp", pl.col("price").alias("trade_price"), pl.col("quantity").alias("trade_size")]
)
.sort("timestamp")
)
print(f"Bid quotes: {len(bids):,}")
print(f"Ask quotes: {len(asks):,}")
print(f"Trades: {len(trades):,}")
# %%
# Combine all events chronologically
bids_marked = bids.with_columns(pl.lit("bid").alias("event"))
asks_marked = asks.with_columns(pl.lit("ask").alias("event"))
trades_marked = trades.with_columns(pl.lit("trade").alias("event"))
all_events = pl.concat(
[
bids_marked.select(["timestamp", "event", "bid", "bid_size"]),
asks_marked.select(["timestamp", "event", "ask", "ask_size"]),
trades_marked.select(["timestamp", "event", "trade_price", "trade_size"]),
],
how="diagonal",
).sort("timestamp")
# Forward-fill bid/ask to get NBBO at each point
nbbo_at_trades = (
all_events.with_columns(
pl.col("bid").forward_fill(),
pl.col("bid_size").forward_fill(),
pl.col("ask").forward_fill(),
pl.col("ask_size").forward_fill(),
)
.filter(pl.col("event") == "trade")
.drop_nulls(subset=["bid", "ask"])
.with_columns(
(pl.col("ask") - pl.col("bid")).alias("spread"),
((pl.col("ask") - pl.col("bid")) / ((pl.col("ask") + pl.col("bid")) / 2) * 10000).alias(
"spread_bps"
),
((pl.col("bid") + pl.col("ask")) / 2).alias("midpoint"),
)
.filter(pl.col("spread") > 0) # Remove crossed/locked markets
)
print(f"\nTrades with valid NBBO: {len(nbbo_at_trades):,}")
# %% [markdown]
# ## 3. Spread at Trade Time
#
# Before classifying trades, let's understand the spread environment they
# executed in. The spread is the "toll" for crossing from passive to aggressive.
# %%
# Spread statistics at trade times
spread_stats = nbbo_at_trades.select(
pl.col("spread_bps").mean().alias("mean"),
pl.col("spread_bps").median().alias("median"),
pl.col("spread_bps").quantile(0.95).alias("p95"),
pl.col("spread_bps").max().alias("max"),
)
print("=== Spread at Trade Time ===")
print(f" Mean: {spread_stats['mean'][0]:.1f} bps")
print(f" Median: {spread_stats['median'][0]:.1f} bps")
print(f" 95th: {spread_stats['p95'][0]:.1f} bps")
print(f" Max: {spread_stats['max'][0]:.1f} bps")
print("\n (Normal day: ~1-2 bps median)")
# %%
# Spread distribution
fig = px.histogram(
nbbo_at_trades.filter(pl.col("spread_bps") < 50).to_pandas(), # Cap for visibility
x="spread_bps",
nbins=100,
color_discrete_sequence=[COLORS["blue"]],
)
fig.add_vline(
x=spread_stats["median"][0],
line_dash="dash",
line_color=COLORS["warm"],
annotation_text=f"Median: {spread_stats['median'][0]:.1f} bps",
)
fig.update_layout(
title=f"Spread Distribution at Trade Time - {SYMBOL} (March 16, 2020)",
xaxis_title="Spread (bps)",
yaxis_title="Count",
height=400,
)
fig.show()
# %% [markdown]
# **Interpretation**: Most trades execute in a 2-5 bps spread environment, but
# the long tail shows moments of stress where spreads widened to 20+ bps. These
# are expensive trades - the aggressor pays a significant premium for immediacy.
# %% [markdown]
# ## 4. Lee-Ready Classification
#
# Now we apply Lee-Ready to classify each trade. The quote test handles ~90%
# of trades; the tick test fills in when prices land exactly at midpoint.
# %%
# Apply Lee-Ready
trades_classified = (
nbbo_at_trades.with_columns(
# Quote test: compare to midpoint
pl.when(pl.col("trade_price") > pl.col("midpoint"))
.then(pl.lit(1))
.when(pl.col("trade_price") < pl.col("midpoint"))
.then(pl.lit(-1))
.otherwise(pl.lit(0))
.alias("quote_rule"),
# Tick test: direction of price change
pl.col("trade_price").diff().sign().fill_null(0).alias("tick_rule"),
)
.with_columns(
# Final classification
pl.when(pl.col("quote_rule") != 0)
.then(pl.col("quote_rule"))
.otherwise(pl.col("tick_rule"))
.alias("trade_sign")
)
.with_columns(
pl.when(pl.col("trade_sign") == 1)
.then(pl.lit("BUY"))
.when(pl.col("trade_sign") == -1)
.then(pl.lit("SELL"))
.otherwise(pl.lit("UNKNOWN"))
.alias("direction")
)
)
# %%
# Classification breakdown
classification = (
trades_classified.group_by("direction")
.agg(
pl.len().alias("count"),
pl.col("trade_size").sum().alias("volume"),
)
.with_columns(
(pl.col("count") / pl.sum("count") * 100).alias("count_pct"),
(pl.col("volume") / pl.sum("volume") * 100).alias("volume_pct"),
)
.sort("volume", descending=True)
)
print("=== Lee-Ready Classification ===")
for row in classification.iter_rows(named=True):
print(
f" {row['direction']:7} {row['count']:>10,} trades ({row['count_pct']:5.1f}%) "
f"{row['volume']:>15,} shares ({row['volume_pct']:5.1f}%)"
)
# %%
# Visualize classification
colors_map = {"BUY": COLORS["buy"], "SELL": COLORS["sell"], "UNKNOWN": COLORS["neutral"]}
fig = make_subplots(
rows=1,
cols=2,
specs=[[{"type": "pie"}, {"type": "pie"}]],
subplot_titles=("By Trade Count", "By Volume"),
)
fig.add_trace(
go.Pie(
labels=classification["direction"].to_list(),
values=classification["count"].to_list(),
marker=dict(colors=[colors_map[d] for d in classification["direction"].to_list()]),
textinfo="label+percent",
hole=0.4,
),
row=1,
col=1,
)
fig.add_trace(
go.Pie(
labels=classification["direction"].to_list(),
values=classification["volume"].to_list(),
marker=dict(colors=[colors_map[d] for d in classification["direction"].to_list()]),
textinfo="label+percent",
hole=0.4,
),
row=1,
col=2,
)
fig.update_layout(
title=f"Trade Direction (Lee-Ready) - {SYMBOL} (March 16, 2020)",
height=400,
showlegend=False,
)
fig.show()
# %% [markdown]
# **What we see**: On this crash day, seller-initiated trades slightly dominate
# both by count and volume. This confirms the intuition that March 16 was a
# day of panic selling - the aggressive side was overwhelmingly sellers
# demanding immediacy.
# %% [markdown]
# ## 5. Order Imbalance Over Time
#
# Order imbalance = (Buy Volume - Sell Volume) / Total Volume
#
# This signal captures the net direction of aggressive trading. Strong positive
# imbalance indicates buying pressure; negative indicates selling.
# %%
# Compute minute-level order imbalance
minute_stats = (
trades_classified.with_columns(
(pl.col("trade_size") * pl.col("trade_sign")).alias("signed_volume"),
)
.group_by_dynamic("timestamp", every="1m")
.agg(
pl.col("trade_price").first().alias("open"),
pl.col("trade_price").last().alias("close"),
pl.col("trade_size").sum().alias("volume"),
pl.col("signed_volume").sum().alias("signed_volume"),
pl.col("spread_bps").mean().alias("avg_spread"),
pl.len().alias("trades"),
)
.with_columns(
(pl.col("close") / pl.col("open") - 1).alias("return"),
(pl.col("signed_volume") / pl.col("volume")).alias("imbalance"),
)
.drop_nulls()
)
print(f"Minute bars: {len(minute_stats)}")
# %%
# Correlation between imbalance and returns
corr = minute_stats.select(pl.corr("imbalance", "return"))
print(f"\nImbalance ↔ Return correlation: {corr[0, 0]:.3f}")
# %% [markdown]
# The Pearson correlation between minute-level imbalance and minute returns is
# ~0.08 on this single AAPL day. The relationship is contemporaneous;
# converting it into a tradeable signal requires predicting future imbalance
# rather than reading the realized contemporaneous value.
# %%
# Build three-panel figure: Price, Imbalance, Spread (single cell so the
# figure is emitted once, fully populated — split-cell variants trigger
# papermill's intermediate auto-display and leave the third panel empty).
fig = make_subplots(
rows=3,
cols=1,
row_heights=[0.4, 0.3, 0.3],
shared_xaxes=True,
vertical_spacing=0.06,
subplot_titles=("Price", "Order Imbalance", "Spread"),
)
fig.add_trace(
go.Scatter(
x=minute_stats["timestamp"].to_list(),
y=minute_stats["close"].to_list(),
name="Price",
line=dict(color=COLORS["blue"], width=1),
),
row=1,
col=1,
)
imbalance_colors = [
COLORS["buy"] if x > 0 else COLORS["sell"] for x in minute_stats["imbalance"].to_list()
]
fig.add_trace(
go.Bar(
x=minute_stats["timestamp"].to_list(),
y=minute_stats["imbalance"].to_list(),
name="Imbalance",
marker_color=imbalance_colors,
),
row=2,
col=1,
)
fig.add_trace(
go.Scatter(
x=minute_stats["timestamp"].to_list(),
y=minute_stats["avg_spread"].to_list(),
name="Spread",
line=dict(color=COLORS["warm"], width=1),
fill="tozeroy",
fillcolor="rgba(139, 69, 19, 0.2)",
),
row=3,
col=1,
)
fig.update_layout(
title=f"Price, Imbalance, and Spread - {SYMBOL} (March 16, 2020)",
height=600,
showlegend=False,
)
fig.update_yaxes(title_text="Price ($)", row=1, col=1)
fig.update_yaxes(title_text="Imbalance", row=2, col=1)
fig.update_yaxes(title_text="Spread (bps)", row=3, col=1)
fig.update_xaxes(title_text="Time (ET)", row=3, col=1)
fig.show()
# %% [markdown]
# **Reading the panel**:
#
# - **Top (Price)**: The crash unfolds - gap down at open, circuit breaker halt,
# continued selling, then stabilization
# - **Middle (Imbalance)**: Red bars dominate early (sell pressure), more mixed later
# - **Bottom (Spread)**: Spikes during price dislocations, narrows when calm
#
# The three series are connected: when imbalance is strongly negative (selling),
# price drops, and spreads widen as market makers retreat.
# %%
# Scatter: imbalance vs return
fig = px.scatter(
minute_stats.to_pandas(),
x="imbalance",
y="return",
color="avg_spread",
color_continuous_scale="RdYlBu_r",
opacity=0.6,
)
# Regression line
x = minute_stats["imbalance"].to_numpy()
y = minute_stats["return"].to_numpy()
mask = ~(np.isnan(x) | np.isnan(y))
if mask.sum() > 2:
z = np.polyfit(x[mask], y[mask], 1)
p = np.poly1d(z)
x_line = np.linspace(x[mask].min(), x[mask].max(), 100)
fig.add_trace(
go.Scatter(
x=x_line,
y=p(x_line),
mode="lines",
name="Trend",
line=dict(color=COLORS["warm"], width=2, dash="dash"),
)
)
fig.update_layout(
title=f"Imbalance-Return Relationship (r={corr[0, 0]:.2f})",
xaxis_title="Order Imbalance",
yaxis_title="Minute Return",
yaxis=dict(tickformat=".1%"),
coloraxis_colorbar_title="Spread (bps)",
height=450,
)
fig.show()
# %% [markdown]
# **The scatter shows a weak positive tilt**: consistent with the ~0.08
# correlation, the cloud is diffuse rather than tight, but buy-dominated minutes
# (imbalance > 0) lean toward positive returns and sell-dominated minutes toward
# negative ones. The color shows that high-spread moments (yellow/red) are often
# extreme imbalance/return events.
# %% [markdown]
# ## 6. Intraday Imbalance Pattern
#
# Does imbalance vary systematically through the day? Let's aggregate by hour.
# %%
hourly_imbalance = (
minute_stats.with_columns(pl.col("timestamp").dt.hour().alias("hour"))
.group_by("hour")
.agg(
pl.col("imbalance").mean().alias("avg_imbalance"),
pl.col("volume").sum().alias("total_volume"),
pl.col("avg_spread").mean().alias("avg_spread"),
)
.sort("hour")
)
print("=== Hourly Pattern ===")
print(hourly_imbalance)
# %%
fig = make_subplots(specs=[[{"secondary_y": True}]])
fig.add_trace(
go.Bar(
x=hourly_imbalance["hour"].to_list(),
y=hourly_imbalance["avg_imbalance"].to_list(),
name="Avg Imbalance",
marker_color=[
COLORS["buy"] if x > 0 else COLORS["sell"]
for x in hourly_imbalance["avg_imbalance"].to_list()
],
),
secondary_y=False,
)
fig.add_trace(
go.Scatter(
x=hourly_imbalance["hour"].to_list(),
y=hourly_imbalance["avg_spread"].to_list(),
name="Avg Spread",
mode="lines+markers",
line=dict(color=COLORS["warm"], width=2),
marker=dict(size=8),
),
secondary_y=True,
)
fig.update_layout(
title=f"Hourly Imbalance and Spread - {SYMBOL} (March 16, 2020)",
xaxis_title="Hour (ET)",
height=400,
legend=dict(orientation="h", yanchor="bottom", y=1.02),
)
fig.update_yaxes(title_text="Avg Order Imbalance", secondary_y=False)
fig.update_yaxes(title_text="Avg Spread (bps)", secondary_y=True)
fig.show()
# %% [markdown]
# **Selling pressure deepens into the afternoon**: the open (Hour 9) already
# carries negative imbalance (-0.12) and wide spreads (~132 bps), but the
# imbalance becomes *most* negative midday-to-afternoon, reaching -0.16 by
# Hour 14, while the widest spread sits at Hour 12 (~137 bps). Spreads are
# tightest mid-morning and late afternoon (~75 bps in Hours 10 and 14), so the
# pattern is a worsening midday imbalance rather than a clean open-to-close
# moderation.
# %% [markdown]
# ## Key Takeaways
#
# **1. NBBO reconstruction is straightforward**: Forward-fill bid/ask to each
# trade timestamp. This is the foundation for all trade classification.
#
# **2. Lee-Ready classification**: The quote test handles trades away from the
# midpoint; the tick test fills the gap at the midpoint. On March 16, sellers
# accounted for ~58% of classified volume.
#
# **3. Order imbalance co-moves with returns**: contemporaneous correlation of
# ~0.08 at minute frequency for this single-day AAPL sample.
#
# **4. Stress amplifies patterns**: The open was dominated by sell imbalance
# and wide spreads; both moderated through the day.
#
# **5. The three metrics are connected**: Price, imbalance, and spread move
# together - understanding one requires understanding all three.
#
# ## Next Steps
#
# - **Minute Bars**: [`13_algoseek_minute_bars_eda`](13_algoseek_minute_bars_eda.ipynb) - Pre-aggregated data
# for longer-horizon analysis
# - **Feature Engineering (Ch8)**: Build microstructure features from signed
# trades for ML models
# - **VPIN (Ch8)**: Volume-synchronized probability of informed trading
03_market_microstructure/13_algoseek_minute_bars_eda.ipynb
+2468
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/13_algoseek_minute_bars_eda.py
+1063
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/14_itch_bar_sampling.ipynb
+1770
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/14_itch_bar_sampling.py
+788
View File
@@ -0,0 +1,788 @@
# ---
# jupyter:
# jupytext:
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Information-Driven Bars: Beyond Time Sampling
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Build the four bar families §3.4 walks through (time, tick, volume, dollar
# plus imbalance/run information bars) from a single AAPL ITCH trading day,
# compare their statistical properties (normality, autocorrelation), and
# generate the Figure 3.4 sampling-comparison panel.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Construct tick, volume, dollar, and imbalance bars from raw trades using
# `ml4t.engineer.bars` (vectorized polars + numba).
# - Quantify the gain in normality (lower JB stat, lower kurtosis) when moving
# from time bars to volume/dollar bars.
# - Apply Lee-Ready aggressor-side classification on ITCH (which lacks the
# aggressor field) and contrast tick-test vs midpoint-based imbalance bars.
#
# ## Book reference
#
# Section §3.4, *The Art of Sampling: From Ticks to Bars*. Notebook generates
# Figure 3.4 (2-hour window, 10:00-12:00 ET).
#
# ## Prerequisites
#
# - Parsed ITCH `P`/`A`/`F`/`D`/`X`/`E`/`C`/`U` parquets at the canonical
# loader path (output of `01_itch_parser`). The Lee-Ready section
# reconstructs the LOB on the fly from these messages.
#
# ---
# %% [markdown]
# ## Setup
# %%
"""Information-Driven Bars — constructing tick, volume, dollar, and imbalance bars from raw trades."""
import warnings
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import polars as pl
import seaborn as sns
from limit_orderbook import classify_trades_lee_ready
# ML4T Engineer - Bar samplers
from ml4t.engineer.bars import (
DollarBarSampler,
FixedTickImbalanceBarSampler,
TickBarSampler,
VolumeBarSampler,
)
from ml4t.engineer.bars.run import TickRunBarSampler
from scipy import stats
from data.equities.loader import load_nasdaq_itch
from utils.paths import get_output_dir
# %% tags=["parameters"]
SYMBOL = "AAPL"
TRADING_DATE = "2020-01-30"
MAX_TRADES = 0 # 0 = all trades
# %%
sns.set_style("whitegrid")
# Normalize MAX_TRADES: 0 means no limit
if MAX_TRADES == 0:
MAX_TRADES = None
# %%
# Input: Pre-parsed ITCH messages from canonical data location
MESSAGE_DIR = load_nasdaq_itch(get_base_path=True)
# Output: This notebook's bar outputs
OUTPUT_DIR = get_output_dir(3, "nasdaq_itch") / "bars"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
print(f"Input directory (messages): {MESSAGE_DIR}")
print(f"Output directory (bars): {OUTPUT_DIR}")
# Validate parsed ITCH data — produced by 01_itch_parser or Rust parser
assert MESSAGE_DIR.exists(), (
f"Parsed ITCH data not found at {MESSAGE_DIR}.\n"
"Run the ITCH pipeline first:\n"
" 1. Download: uv run python data/equities/market/microstructure/nasdaq_itch_download.py\n"
" 2. Parse: Run 01_itch_parser.py (Section 4) or Rust parser (Section 6)"
)
msg_types = sorted([d.name for d in MESSAGE_DIR.iterdir() if d.is_dir()])
assert "P" in msg_types, f"Trade messages (P) not found in {MESSAGE_DIR}. Available: {msg_types}"
print(f"\nAvailable message types: {msg_types}")
# %% [markdown]
# ## 1. Load Trade Data from ITCH
#
# We use trade messages (type 'P') from ITCH data. These represent actual
# executions on the exchange, providing the raw tick stream for bar construction.
# %%
def load_itch_trades(symbol: str, max_trades: int | None = None) -> pl.DataFrame:
"""Load trade messages from ITCH parquet files, tick-test classified.
Args:
symbol: Stock symbol to filter
max_trades: Maximum (chronological) trades to return
The ml4t.engineer.bars module expects:
- timestamp: datetime column
- price: numeric price
- volume: trade size
- side: 1 for buyer-initiated, -1 for seller-initiated
ITCH Trade (P) messages carry ``buy_sell_indicator='B'`` for *every*
execution — the field reflects the resting (non-displayed) order's side,
not the aggressor, so it cannot signal direction. We therefore infer the
aggressor with the **tick test** (Lee & Ready 1991): an uptick is
buyer-initiated (+1), a downtick seller-initiated (-1), and a zero-tick
inherits the previous non-zero direction. This is the cheap ~78%-accurate
classifier; §7 reconstructs the LOB for the ~94%-accurate Lee-Ready
quote-midpoint version and contrasts the two.
"""
trade_dir = MESSAGE_DIR / "P"
assert trade_dir.exists(), f"No trade data at {trade_dir}"
# Use lazy scan with predicate pushdown for efficiency
# Filter is pushed down to parquet read, reducing memory
df = (
pl.scan_parquet(trade_dir / "*.parquet")
.filter(pl.col("stock") == symbol)
.collect()
# The tick test is path-dependent, so classify on the chronological
# stream (and take the first N chronological trades, not an arbitrary
# parquet-order slice, when capped).
.sort("timestamp")
.with_columns((pl.col("price") / 10000).alias("price"))
)
if max_trades is not None:
df = df.head(max_trades)
# Tick test: sign of the price change, with zero-ticks carried forward from
# the last directional trade. The very first trade has no predecessor, so it
# defaults to buyer-initiated.
trades = (
df.with_columns(
pl.when(pl.col("price").diff() > 0)
.then(1)
.when(pl.col("price").diff() < 0)
.then(-1)
.otherwise(None)
.alias("side")
)
.with_columns(pl.col("side").forward_fill().fill_null(1).cast(pl.Int64))
.select(
[
pl.col("timestamp"),
pl.col("price"),
pl.col("shares").alias("volume"),
pl.col("side"),
]
)
)
n_buy = trades.filter(pl.col("side") == 1).height
n_sell = trades.filter(pl.col("side") == -1).height
print(
f"Tick-test classification: {n_buy:,} buyer-initiated, {n_sell:,} seller-initiated "
f"({100 * n_sell / max(len(trades), 1):.1f}% sells)"
)
return trades
# %% [markdown]
# **Note**: The `classify_trades_lee_ready` function is imported from `utils.limit_orderbook` where
# it shares the correct LOB reconstruction logic with `02_itch_lob_reconstruction`.
# This ensures consistent order pool tracking and proper handling of Replace (U) chains.
# %%
# Load trades for our symbol
# Initialize variables for graceful handling when data missing
all_trades = None
trades = None
time_1m = None
tick_bars = None
volume_bars = None
dollar_bars = None
imbalance_bars = None
run_bars = None
if MESSAGE_DIR.exists():
all_trades = load_itch_trades(SYMBOL, max_trades=MAX_TRADES)
print(f"Symbol: {SYMBOL}")
print(f"Trading Date: {TRADING_DATE}")
print(f"Total trades: {len(all_trades):,}")
if len(all_trades) > 0:
print("\nSample:")
print(all_trades.head(5))
print("\nStats:")
print(f" Price range: ${all_trades['price'].min():.2f} - ${all_trades['price'].max():.2f}")
print(f" Total volume: {all_trades['volume'].sum():,.0f} shares")
print(f" Dollar volume: ${(all_trades['price'] * all_trades['volume']).sum():,.0f}")
# %%
# Filter to regular trading hours (9:30 AM - 4:00 PM ET)
# Note: ITCH timestamps are in US/Eastern (exchange local time), timezone-naive.
# This filter assumes the data is already in ET. For DST-aware filtering,
# you would need to localize timestamps first.
if all_trades is not None and len(all_trades) > 0:
start_time = pd.Timestamp(f"{TRADING_DATE} 09:30:00")
end_time = pd.Timestamp(f"{TRADING_DATE} 16:00:00")
trades = all_trades.filter(
(pl.col("timestamp") >= start_time) & (pl.col("timestamp") <= end_time)
).sort("timestamp") # Must be sorted for group_by_dynamic
print(f"\nTrades in regular hours: {len(trades):,}")
print(f"Pre-market trades excluded: {len(all_trades) - len(trades):,}")
# %% [markdown]
# ## 2. Create All Bar Types Using ml4t.engineer
#
# The `ml4t.engineer.bars` module provides vectorized bar construction:
# - **TickBarSampler**: Fixed number of trades per bar
# - **VolumeBarSampler**: Fixed total volume per bar
# - **DollarBarSampler**: Fixed dollar volume per bar
# - **FixedTickImbalanceBarSampler**: Bars close when the signed tick imbalance
# exceeds a fixed threshold (stable; avoids the adaptive feedback loop)
# - **TickRunBarSampler**: Bars close when a "run" of same-side trades exceeds expected length
#
# The imbalance and run families are signed: they need a buyer/seller label per
# trade. Here that label is the **tick test** computed at load time (§1); §7
# rebuilds the imbalance family with the more accurate Lee-Ready quote-midpoint
# classifier and contrasts the two.
# %%
if trades is not None and len(trades) > 0:
print("Creating bars using ml4t.engineer...")
# Time bars using Polars resample (standard approach)
time_1m = (
trades.group_by_dynamic("timestamp", every="1m")
.agg(
[
pl.col("price").first().alias("open"),
pl.col("price").max().alias("high"),
pl.col("price").min().alias("low"),
pl.col("price").last().alias("close"),
pl.col("volume").sum().alias("volume"),
pl.len().alias("tick_count"),
]
)
.drop_nulls()
)
print(f"Time bars (1-min): {len(time_1m):,}")
# Tick bars - 100 trades per bar (AAPL has fewer trades than NVDA)
tick_sampler = TickBarSampler(ticks_per_bar=100)
tick_bars = tick_sampler.sample(trades)
print(f"Tick bars (100): {len(tick_bars):,}")
# Volume bars - 10K shares per bar (AAPL pre-split had ~$320 price)
volume_sampler = VolumeBarSampler(volume_per_bar=10_000)
volume_bars = volume_sampler.sample(trades)
print(f"Volume bars (10K): {len(volume_bars):,}")
# Dollar bars - $3M per bar (adjusted for ~$320 price)
dollar_sampler = DollarBarSampler(dollars_per_bar=3_000_000)
dollar_bars = dollar_sampler.sample(trades)
print(f"Dollar bars ($3M): {len(dollar_bars):,}")
# %%
# Imbalance bars - close when the signed tick imbalance exceeds a fixed threshold
# AFML tick-imbalance statistic: θ = Σ b_t (fixed threshold, not adaptive E[θ_T])
if trades is not None and len(trades) > 0:
# A meaningful imbalance/run bar needs genuine two-sided flow. Require BOTH
# directions to be present — an all-one-side stream (e.g. the raw ITCH
# buy_sell_indicator before tick-test classification) would make signed
# imbalance monotonic and degenerate the bars into near-tick bars.
n_buy = trades.filter(pl.col("side") == 1).height
n_sell = trades.filter(pl.col("side") == -1).height
has_valid_sides = n_buy > 0 and n_sell > 0
# Use a FIXED imbalance threshold rather than the adaptive (EWMA) sampler.
# The adaptive E[θ_T] = E[T] × |2P[b=1] − 1| feeds back on its own bar
# lengths and threshold-spirals on persistently one-sided flow — the
# instability this section warns about. The fixed-threshold sampler the
# library recommends for production is stable and reproducible. threshold=20
# yields ~130 imbalance bars on this ~15k-trade AAPL day (comparable to the
# volume/dollar bars above).
if has_valid_sides:
imbalance_sampler = FixedTickImbalanceBarSampler(threshold=20)
imbalance_bars = imbalance_sampler.sample(trades)
print(f"Tick imbalance bars: {len(imbalance_bars):,}")
else:
print("Imbalance bars skipped (insufficient side attribution in trade data)")
imbalance_bars = pl.DataFrame()
# %%
# Run bars - close when cumulative run count exceeds expected threshold
# AFML formula: θ = max(cumulative_buys, cumulative_sells)
# Bars form when one side dominates, signaling sustained directional flow
if trades is not None and len(trades) > 0 and has_valid_sides:
try:
run_sampler = TickRunBarSampler(
expected_ticks_per_bar=50, # Initial E[T]
alpha=0.001, # small EWMA decay: keeps the adaptive threshold stable
)
run_bars = run_sampler.sample(trades)
print(f"Tick run bars: {len(run_bars):,}")
except Exception as e:
print(f"Run bars skipped (requires trade direction): {e}")
run_bars = pl.DataFrame()
elif trades is not None and len(trades) > 0:
print("Run bars skipped (insufficient side attribution in trade data)")
run_bars = pl.DataFrame()
# %% [markdown]
# ## 3. Compare Statistical Properties
#
# We compare key properties that affect ML model performance:
# - **Return distribution**: Closer to normal is better
# - **Autocorrelation**: Lower is better (more IID)
# - **Heteroskedasticity**: More stable variance is better
# %%
def analyze_returns(bars: pl.DataFrame, bar_type: str) -> dict | None:
"""Analyze return statistics for a bar type."""
if "close" not in bars.columns or len(bars) < 10:
return None
# Convert to pandas for scipy stats
returns = bars["close"].pct_change().drop_nulls().to_numpy()
if len(returns) < 10:
return None
# Normality test (Jarque-Bera)
jb_stat, jb_pval = stats.jarque_bera(returns)
# Autocorrelation at lag 1
autocorr = np.corrcoef(returns[:-1], returns[1:])[0, 1] if len(returns) > 1 else 0
return {
"bar_type": bar_type,
"n_bars": len(bars),
"mean_return": np.mean(returns) * 100,
"std_return": np.std(returns) * 100,
"skewness": stats.skew(returns),
"kurtosis": stats.kurtosis(returns),
"jb_stat": jb_stat,
"jb_pval": jb_pval,
"autocorr_1": autocorr,
}
# %%
if time_1m is not None:
# Analyze all bar types
results = []
bar_list = [
(time_1m, "Time (1-min)"),
(tick_bars, "Tick (100)"),
(volume_bars, "Volume (10K)"),
(dollar_bars, "Dollar ($3M)"),
(imbalance_bars, "Imbalance"),
]
# Add run bars if available
if run_bars is not None and len(run_bars) > 0:
bar_list.append((run_bars, "Run"))
for bars, name in bar_list:
result = analyze_returns(bars, name)
if result:
results.append(result)
stats_df = pd.DataFrame(results).set_index("bar_type")
print("\n=== Statistical Properties Comparison ===\n")
print(stats_df.round(4).to_string())
# %% [markdown]
# ### Interpretation
#
# | Metric | Meaning | Better Value |
# |--------|---------|--------------|
# | JB Stat | Distance from normality | Lower |
# | Autocorr | Serial dependence | Closer to 0 |
# | Kurtosis | Fat tails (excess over 3) | Lower |
# %% [markdown]
# ## 4. Visualize Return Distributions
# %%
if time_1m is not None:
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
axes = axes.flatten()
bar_types = [
(time_1m, "Time (1-min)"),
(tick_bars, "Tick (100)"),
(volume_bars, "Volume (10K)"),
(dollar_bars, "Dollar ($3M)"),
(imbalance_bars, "Imbalance"),
]
for i, (bars, name) in enumerate(bar_types):
ax = axes[i]
if "close" in bars.columns and len(bars) > 10:
returns = bars["close"].pct_change().drop_nulls().to_numpy() * 100
returns = returns[(returns > -2) & (returns < 2)] # Clip outliers
if len(returns) > 5:
ax.hist(returns, bins=50, density=True, alpha=0.7, color="steelblue")
# Overlay normal distribution
x = np.linspace(returns.min(), returns.max(), 100)
ax.plot(
x,
stats.norm.pdf(x, returns.mean(), returns.std()),
"r-",
linewidth=2,
label="Normal",
)
ax.set_title(f"{name}\n(n={len(bars):,})")
ax.set_xlabel("Return (%)")
ax.set_ylabel("Density")
ax.legend()
# Hide empty subplot
axes[5].axis("off")
plt.suptitle(f"{SYMBOL} - Return Distributions by Bar Type ({TRADING_DATE})", fontsize=14)
plt.tight_layout()
plt.show()
# %% [markdown]
# ## 5. Bar Duration Analysis
#
# Information-driven bars have variable duration - they are faster during
# high activity periods and slower during quiet periods.
# %%
if tick_bars is not None:
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
bar_types = [
(tick_bars, "Tick (100)"),
(volume_bars, "Volume (10K)"),
(dollar_bars, "Dollar ($3M)"),
(imbalance_bars, "Imbalance"),
]
for i, (bars, name) in enumerate(bar_types):
ax = axes[i // 2, i % 2]
# Calculate duration between bars
timestamps = bars["timestamp"].to_numpy()
if len(timestamps) > 1:
durations = np.diff(timestamps).astype("timedelta64[s]").astype(float)
ax.plot(range(len(durations)), durations, alpha=0.5, linewidth=0.5)
ax.axhline(
np.mean(durations),
color="red",
linestyle="--",
label=f"Mean: {np.mean(durations):.1f}s",
)
ax.set_title(f"{name} Bar Duration")
ax.set_xlabel("Bar Index")
ax.set_ylabel("Duration (seconds)")
ax.legend()
plt.suptitle(f"{SYMBOL} - Bar Duration Over Time ({TRADING_DATE})", fontsize=14)
plt.tight_layout()
plt.show()
# %% [markdown]
# ## 6. Buy/Sell Volume Decomposition
#
# The ml4t.engineer bar samplers track buy and sell volume separately from the
# per-trade `side` label. Because **ITCH Trade (P) messages do not carry true
# aggressor direction** — the `buy_sell_indicator` field is uniformly 'B' for
# every trade in this dataset — the decomposition below rests on the **tick-test**
# classification from §1, not an exchange-provided aggressor field. Treat it as a
# ~78%-accurate proxy.
#
# More accurate alternatives:
# - **Lee-Ready algorithm**: compare trade price to quote midpoint (§7 below)
# - **Order matching**: track which book side was reduced by execution
# - **DataBento MBO+MBP-1**: harmonized data with a true trade-direction field
# %%
if volume_bars is not None and "buy_volume" in volume_bars.columns:
# Check if we have actual buy/sell variance
volume_bars_pd = volume_bars.to_pandas()
has_sell_volume = volume_bars_pd["sell_volume"].sum() > 0
if has_sell_volume:
print("=== Order Flow Analysis ===\n")
# Compute signed imbalance in [-1, 1]. buy_volume/sell_volume are unsigned
# (uint32), so cast to float BEFORE subtracting — an unsigned difference
# underflows to ~2^32 on every sell-dominated bar.
buy_v = volume_bars_pd["buy_volume"].astype("float64")
sell_v = volume_bars_pd["sell_volume"].astype("float64")
volume_bars_pd["imbalance"] = (buy_v - sell_v) / (buy_v + sell_v)
print(f"Mean imbalance: {volume_bars_pd['imbalance'].mean():.4f}")
print(f"Std imbalance: {volume_bars_pd['imbalance'].std():.4f}")
else:
print("=== Order Flow Analysis ===\n")
print("WARNING: ITCH Trade (P) messages do not provide aggressor direction.")
print(" All trades show buy_sell_indicator='B' uniformly.")
print("\nSee Section 7 below for Lee-Ready classification using LOB midpoint.")
# %%
if volume_bars is not None and "buy_volume" in volume_bars.columns:
if volume_bars_pd["sell_volume"].sum() > 0:
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Imbalance over time
ax = axes[0]
colors = ["green" if x > 0 else "red" for x in volume_bars_pd["imbalance"]]
ax.bar(
range(len(volume_bars_pd)),
volume_bars_pd["imbalance"],
color=colors,
alpha=0.6,
width=1.0,
)
ax.axhline(0, color="black", linewidth=0.5)
ax.set_title("Volume Imbalance Over Time")
ax.set_xlabel("Bar Index")
ax.set_ylabel("(Buy - Sell) / Total")
# Imbalance vs returns
ax = axes[1]
volume_bars_pd["return"] = volume_bars_pd["close"].pct_change()
ax.scatter(
volume_bars_pd["imbalance"].iloc[:-1],
volume_bars_pd["return"].iloc[1:] * 100,
alpha=0.3,
s=5,
)
ax.axhline(0, color="black", linewidth=0.5)
ax.axvline(0, color="black", linewidth=0.5)
ax.set_title("Imbalance vs Next Bar Return")
ax.set_xlabel("Imbalance (t)")
ax.set_ylabel("Return (t+1, %)")
plt.tight_layout()
plt.show()
# %% [markdown]
# ## 7. Lee-Ready Trade Classification
#
# Since ITCH Trade (P) messages don't provide aggressor direction, we implement
# the **Lee-Ready algorithm** to classify trades. **Validated accuracy: ~94%**
# (compared to ~78% for tick-test alone; see `15_itch_lee_ready` for details).
#
# **Quote test** (primary):
# - Compare trade price to midpoint of best bid/ask
# - Price > midpoint → buy-initiated
# - Price < midpoint → sell-initiated
#
# **Tick test** (fallback for trades at midpoint):
# - Uptick from previous trade → buy
# - Downtick → sell
# - Zero tick → use last tick direction
#
# **Why tick-test alone fails**: 83% of consecutive trades have the same price.
# The continuous tick test carries forward the previous direction, but errors
# propagate and decorrelate from actual aggressor intent.
#
# This requires reconstructing the LOB state at each trade timestamp—see
# `02_itch_lob_reconstruction` for the complete LOB state machine.
# %%
# Lee-Ready classification (slower but accurate)
# Only run in full mode due to computational cost
lee_ready_trades = None
lee_ready_imbalance_bars = None
if MESSAGE_DIR.exists():
print("=== Lee-Ready Classification ===\n")
print("Reconstructing LOB state to classify trades by quote midpoint...")
print("(Using shared utils.limit_orderbook with correct order pool tracking)\n")
try:
# Use shared function from utils.limit_orderbook with proper LOB reconstruction
# Filter to regular trading hours (9:30 AM - 4:00 PM)
from datetime import datetime
start_time = datetime.strptime(f"{TRADING_DATE} 09:30:00", "%Y-%m-%d %H:%M:%S")
end_time = datetime.strptime(f"{TRADING_DATE} 16:00:00", "%Y-%m-%d %H:%M:%S")
lee_ready_trades = classify_trades_lee_ready(
itch_dir=MESSAGE_DIR,
symbol=SYMBOL,
start_time=start_time,
end_time=end_time,
)
# Rename 'shares' to 'volume' for ImbalanceBarSampler compatibility
if lee_ready_trades is not None and len(lee_ready_trades) > 0:
lee_ready_trades = lee_ready_trades.rename({"shares": "volume"})
except Exception as e:
print(f"Lee-Ready classification failed: {e}")
print("This requires complete ITCH data (A, D, E, X, P messages).")
# %%
if lee_ready_trades is not None and len(lee_ready_trades) > 0:
# Build imbalance bars with Lee-Ready classification
usable = lee_ready_trades.filter(pl.col("side") != 0)
if len(usable) > len(lee_ready_trades) * 0.5:
print("\nBuilding tick imbalance bars with Lee-Ready classification...")
# Same sampler (fixed threshold=20) as the §2 tick-test imbalance bars so
# the comparison below isolates the classification method, not the binning.
imb_sampler = FixedTickImbalanceBarSampler(threshold=20)
lee_ready_imbalance_bars = imb_sampler.sample(lee_ready_trades)
print(f"Lee-Ready Imbalance bars: {len(lee_ready_imbalance_bars):,}")
# Compare to tick-test imbalance bars
if imbalance_bars is not None:
print("\n=== Classification Method Comparison ===")
print(f"Tick-test imbalance bars: {len(imbalance_bars):,}")
print(f"Lee-Ready imbalance bars: {len(lee_ready_imbalance_bars):,}")
# Compare JB stats
from scipy import stats as sp_stats
def jb_stat(bars):
closes = bars["close"].to_numpy()
returns = np.diff(np.log(closes))
return sp_stats.jarque_bera(returns)[0]
tick_jb = jb_stat(imbalance_bars) if len(imbalance_bars) > 30 else None
lr_jb = (
jb_stat(lee_ready_imbalance_bars) if len(lee_ready_imbalance_bars) > 30 else None
)
if tick_jb and lr_jb:
# Lower Jarque-Bera = closer to normal. Report both and let
# the difference speak for itself.
diff = abs(lr_jb - tick_jb)
closer = "Lee-Ready" if lr_jb < tick_jb else "Tick-test"
print("\nJarque-Bera (lower = closer to normal):")
print(f" Tick-test: {tick_jb:.1f}")
print(f" Lee-Ready: {lr_jb:.1f}")
print(f" Difference: {diff:.1f} (closer to normal: {closer})")
# %% [markdown]
# ## 8. Intraday Pattern Analysis
#
# ITCH data captures the full trading day, allowing us to analyze
# intraday patterns in bar formation.
# %%
if tick_bars is not None and len(tick_bars) > 0:
# Add hour column
tick_bars_pd = tick_bars.to_pandas()
tick_bars_pd["hour"] = tick_bars_pd["timestamp"].dt.hour
# Count bars per hour
bars_per_hour = tick_bars_pd.groupby("hour").size()
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Bars per hour
ax = axes[0]
bars_per_hour.plot(kind="bar", ax=ax, color="steelblue", alpha=0.7)
ax.set_title(f"{SYMBOL} - Tick Bars per Hour")
ax.set_xlabel("Hour (ET)")
ax.set_ylabel("Number of Bars")
ax.set_xticklabels([f"{h}:00" for h in bars_per_hour.index], rotation=45)
# Volume per hour
ax = axes[1]
volume_per_hour = tick_bars_pd.groupby("hour")["volume"].sum()
volume_per_hour.plot(kind="bar", ax=ax, color="green", alpha=0.7)
ax.set_title(f"{SYMBOL} - Volume per Hour")
ax.set_xlabel("Hour (ET)")
ax.set_ylabel("Volume (shares)")
ax.set_xticklabels([f"{h}:00" for h in volume_per_hour.index], rotation=45)
plt.tight_layout()
plt.show()
print("\nIntraday Pattern (U-shape expected):")
print("Opening hour typically has highest activity due to overnight information processing.")
print("Closing hour has high activity due to portfolio rebalancing and MOC orders.")
# %% [markdown]
# ## 9. Save Results
# %%
if time_1m is not None:
# Save bar data
bar_outputs = [
(time_1m, "time_1m"),
(tick_bars, "tick_100"),
(volume_bars, "volume_10k"),
(dollar_bars, "dollar_3m"),
(imbalance_bars, "imbalance_tick"),
(lee_ready_imbalance_bars, "imbalance_lee_ready"),
]
for bars, name in bar_outputs:
if bars is not None and len(bars) > 0:
output_file = OUTPUT_DIR / f"{SYMBOL}_{name}_bars.parquet"
bars.write_parquet(output_file)
print(f"Saved: {output_file.name}")
# %% [markdown]
# ## Key Takeaways
#
# ### Bar Type Selection Guidelines
#
# | Bar Type | Best For | Pros | Cons |
# |----------|----------|------|------|
# | **Time** | Calendar alignment | Simple, universal | Unequal information |
# | **Tick** | Order flow analysis | Equal trades | Ignores size |
# | **Volume** | Activity sampling | Size-weighted | Irregular timing |
# | **Dollar** | Economic activity | Price-adjusted | Complex to compute |
# | **Imbalance** | Informed trading | Captures signals | Parameter-sensitive |
#
# ### Trade Classification Methods (Validated Accuracy)
#
# | Method | Accuracy | Coverage | Data Required | Use Case |
# |--------|----------|----------|---------------|----------|
# | **Direct aggressor** | 100% | 100% | DataBento MBO, CME FIX | Production systems |
# | **Lee-Ready** | **~94%** | 100% | LOB + trades | ITCH reconstruction |
# | **Tick test only** | **~78%** | 100% | Trade prices only | Weak fallback |
#
# **Critical finding**: The tick test alone (~78% accuracy) substantially underperforms
# Lee-Ready (~94%). Zero-tick trades (same price as previous) propagate classification
# errors. Do not use tick-test alone for imbalance bars.
#
# Lee-Ready achieves ~94% accuracy by using LOB midpoint for classification,
# falling back to tick-test only for the minority of trades at the midpoint.
# Direct aggressor field (DataBento, CME) is always preferred when available.
# See `15_itch_lee_ready` for full multi-day validation results.
#
# ### ITCH Data Advantages
#
# 1. **Complete day coverage**: All stocks, all messages for entire trading day
# 2. **Message-level granularity**: See every order add, cancel, execute
# 3. **Free sample data**: NASDAQ provides sample files for research
# 4. **Lee-Ready possible**: Can reconstruct LOB for trade classification
#
# ### ml4t.engineer Bar Library
#
# 1. **Vectorized**: 10,000+ rows/sec with Polars/Numba on the comparison run above
# 2. **Buy/Sell tracking**: Order-flow decomposition built into the bar output
# 3. **Consistent API**: Same interface for tick, volume, dollar, and imbalance bars
# 4. **Typed and tested**: type hints throughout; covered by the library's test suite
#
# ---
#
# **Reference**: López de Prado, M. (2018). *Advances in Financial Machine Learning*. Wiley.
03_market_microstructure/15_itch_lee_ready.ipynb
+1361
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/15_itch_lee_ready.py
+681
View File
@@ -0,0 +1,681 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Lee-Ready Trade Classification Validation
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Quantify how well the Lee-Ready algorithm recovers the aggressor side of
# trades, using DataBento XNAS-ITCH MBO data (which carries the ground-truth
# aggressor flag) as the benchmark.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Reconstruct the LOB from DataBento MBO messages and align trades to the
# contemporaneous quote midpoint.
# - Apply the Lee-Ready quote-test + tick-test cascade and compare its
# accuracy against the tick test alone.
# - Read the per-day breakdown (NVDA, 5 trading days) and reproduce the ~16pp
# gap §3.4 reports between tick-only and Lee-Ready classification.
#
# ## Book reference
#
# Section §3.4, *The Art of Sampling* — Lee-Ready subsection (Table 3.3).
#
# ## Prerequisites
#
# - DataBento XNAS-ITCH MBO parquets at
# `data/equities/market/microstructure/market_by_order/{SYMBOL}/`
# (downloaded via `data/equities/market/microstructure/databento_mbo_download.py`).
#
# ## Decision Time vs Exchange Time
#
# This validation uses **exchange timestamps** for trade-quote alignment, which
# matches the ground truth's timestamping. In live trading, observation delay
# means the quote you could actually have seen at decision time lags the
# exchange state — apply a conservative lag (~1ms co-located, ~10ms retail)
# in any backtest that relies on this kind of classification.
#
# ---
# %% [markdown]
# ## Setup
# %%
"""Lee-Ready Trade Classification Validation — validate Lee-Ready against DataBento ground truth aggressor labels."""
import warnings
from collections import Counter
from pathlib import Path
warnings.filterwarnings("ignore")
import polars as pl
# Import loader for MBO data
from data import load_mbo_data
# ML4T imports - path resolution
from utils.paths import get_output_dir
# %% tags=["parameters"]
SYMBOL = "NVDA"
MAX_ROWS = 0 # 0 = all rows per file
MAX_VALIDATION_DAYS = 5 # Number of days for multi-day validation
# %%
# Normalize MAX_ROWS: 0 means no limit
if MAX_ROWS == 0:
MAX_ROWS = None
# Get file paths from the canonical loader (handles legacy/new path resolution)
data_files = load_mbo_data(symbols=[SYMBOL], list_files=True)
SYMBOL_DIR = data_files[0].parent if data_files else None
OUTPUT_DIR = get_output_dir(3, "algoseek")
# Check data availability (files already loaded via load_mbo_data)
print(f"Symbol: {SYMBOL}")
print(f"Data files: {len(data_files)}")
if data_files:
print(f"First file: {data_files[0].name}")
# %% [markdown]
# ## 1. Load DataBento MBO Data
#
# DataBento MBO format:
# - `action`: A(Add), C(Cancel), F(Fill), M(Modify), T(Trade), R(Clear)
# - `side`: B(Bid/Buy), A(Ask/Sell), N(None)
# - `price`: Integer fixed-point (nanodollars, divide by 1e9)
# - `size`: Order/trade size
# - `order_id`: Unique order reference
# - `timestamp`: Exchange timestamp
# %%
def load_databento_mbo(file_path: Path, max_rows: int | None = None) -> pl.DataFrame:
"""Load and normalize DataBento MBO data."""
df = pl.read_parquet(file_path)
# Apply row limit
if max_rows is not None:
df = df.head(max_rows)
# Normalize timestamps
df = df.with_columns(pl.col("timestamp").cast(pl.Datetime("ns")))
# Convert fixed-point prices to dollars if needed
if "price" in df.columns and df["price"].max() > 1_000_000:
df = df.with_columns((pl.col("price") / 1e9).alias("price"))
# Filter to trading hours (9:30-16:00 ET)
# Note: This uses 14:30-20:00 UTC which is correct for EDT (summer).
# For EST (winter), market closes at 21:00 UTC. A robust solution would:
# 1. Convert to America/New_York first
# 2. Filter on local time
# 3. Convert back to UTC
# For this validation, we use the conservative window that works for EDT.
df = df.filter(
(
(pl.col("timestamp").dt.hour() > 14)
| ((pl.col("timestamp").dt.hour() == 14) & (pl.col("timestamp").dt.minute() >= 30))
)
& (pl.col("timestamp").dt.hour() < 20)
)
# Sort by timestamp. Note: Messages with identical timestamps may not have
# a guaranteed order; DataBento sequence numbers could be used if available.
return df.sort("timestamp")
# %%
# Load one day for validation
sample_df = None
if data_files:
sample_df = load_databento_mbo(data_files[0], max_rows=MAX_ROWS)
print(f"Loaded {len(sample_df):,} messages")
# Action distribution
action_counts = sample_df.group_by("action").len().sort("len", descending=True)
print("\nAction distribution:")
for row in action_counts.iter_rows():
print(f" {row[0]}: {row[1]:,}")
# %% [markdown]
# ## 2. Lee-Ready Classification with LOB Reconstruction
#
# Lee-Ready algorithm (1991):
# 1. **Quote test**: Compare trade price to midpoint
# - Above midpoint → buy-initiated
# - Below midpoint → sell-initiated
# 2. **Tick test** (fallback when at midpoint):
# - Higher than previous trade → buy
# - Lower than previous trade → sell
#
# Key implementation detail: We must maintain LOB state as we process trades
# to get the correct midpoint at each trade time.
# %%
def _update_book(
action: str,
side: str,
price: float,
size: int,
order_id: int,
book: dict[str, Counter],
order_registry: dict[int, dict],
) -> dict[str, Counter]:
"""Apply a book-affecting action (R/A/M/C/F) to the LOB state."""
if action == "R": # Clear book
book = {"B": Counter(), "A": Counter()}
order_registry.clear()
elif action == "A": # Add order
order_registry[order_id] = {"side": side, "price": price, "size": size}
book[side][price] += size
elif action == "M": # Modify order
if order_id in order_registry:
old = order_registry[order_id]
book[old["side"]][old["price"]] -= old["size"]
if book[old["side"]][old["price"]] <= 0:
del book[old["side"]][old["price"]]
order_registry[order_id] = {"side": side, "price": price, "size": size}
book[side][price] += size
elif action in ("C", "F"): # Cancel or Fill
if order_id in order_registry:
reg = order_registry[order_id]
book[reg["side"]][reg["price"]] -= size
if book[reg["side"]][reg["price"]] <= 0:
del book[reg["side"]][reg["price"]]
reg["size"] -= size
if reg["size"] <= 0:
del order_registry[order_id]
return book
# %% [markdown]
# ### Apply Lee-Ready Classification
# Quote test with tick test fallback for trade direction inference.
# %%
def _apply_lee_ready(
price: float,
book: dict[str, Counter],
last_price: float | None,
last_tick_dir: int,
) -> tuple[int, int]:
"""Apply Lee-Ready quote test + tick test fallback. Returns (classification, updated_tick_dir)."""
if book["B"] and book["A"]:
best_bid = max(book["B"].keys())
best_ask = min(book["A"].keys())
midpoint = (best_bid + best_ask) / 2
# Quote test
if price > midpoint:
lee_ready = 1 # Buy
elif price < midpoint:
lee_ready = -1 # Sell
else:
# At midpoint - use tick test
if last_price is not None:
if price > last_price:
last_tick_dir = 1
elif price < last_price:
last_tick_dir = -1
lee_ready = last_tick_dir
else:
# No book - use tick test only
if last_price is not None:
if price > last_price:
lee_ready = 1
elif price < last_price:
lee_ready = -1
else:
lee_ready = last_tick_dir
else:
lee_ready = 0
return lee_ready, last_tick_dir
# %% [markdown]
# ### Classify Trades via Lee-Ready
# Walk through MBO messages, maintain book state, and classify each trade.
# %%
def classify_trades_lee_ready_databento(
df: pl.DataFrame, show_progress: bool = True
) -> pl.DataFrame:
"""
Classify trade direction using Lee-Ready on DataBento MBO data.
Maintains LOB state while processing to get accurate midpoint at trade time.
Parameters
----------
df : pl.DataFrame
DataBento MBO data with action, side, price, size, order_id, timestamp
show_progress : bool
Whether to show progress bar
Returns
-------
pl.DataFrame
Trades with columns: timestamp, price, size, ground_truth, lee_ready, correct
"""
order_registry: dict[int, dict] = {}
book: dict[str, Counter] = {"B": Counter(), "A": Counter()}
classified_trades = []
last_price = None
last_tick_dir = 0
cols_df = df.select(["timestamp", "action", "side", "price", "size", "order_id"])
if show_progress:
print(f"Processing {len(cols_df):,} messages...")
for row in cols_df.iter_rows(named=True):
action, side, price, size = row["action"], row["side"], row["price"], row["size"]
order_id, ts = row["order_id"], row["timestamp"]
if side == "N" and action != "T":
continue
if action in ("R", "A", "M", "C", "F"):
book = _update_book(action, side, price, size, order_id, book, order_registry)
elif action == "T":
# Get ground truth (DataBento provides aggressor side)
ground_truth = 1 if side == "B" else (-1 if side == "A" else 0)
if ground_truth == 0:
continue
lee_ready, last_tick_dir = _apply_lee_ready(price, book, last_price, last_tick_dir)
classified_trades.append(
{
"timestamp": ts,
"price": price,
"size": size,
"ground_truth": ground_truth,
"lee_ready": lee_ready,
"correct": int(ground_truth == lee_ready),
}
)
last_price = price
print(f"Classified {len(classified_trades):,} trades")
return pl.DataFrame(classified_trades)
# %% [markdown]
# ## 3. Run Validation
# %%
results = None
if sample_df is not None and len(sample_df) > 1000:
results = classify_trades_lee_ready_databento(sample_df)
if len(results) > 0:
# Compute accuracy
accuracy = results["correct"].mean() * 100
# Breakdown by ground truth
buy_trades = results.filter(pl.col("ground_truth") == 1)
sell_trades = results.filter(pl.col("ground_truth") == -1)
buy_accuracy = buy_trades["correct"].mean() * 100 if len(buy_trades) > 0 else 0
sell_accuracy = sell_trades["correct"].mean() * 100 if len(sell_trades) > 0 else 0
print("\n" + "=" * 50)
print("LEE-READY VALIDATION RESULTS")
print("=" * 50)
print(f"\nTotal trades classified: {len(results):,}")
print(f"Overall accuracy: {accuracy:.2f}%")
print("\nBy ground truth:")
print(f" Buy-initiated: {buy_accuracy:.2f}% ({len(buy_trades):,} trades)")
print(f" Sell-initiated: {sell_accuracy:.2f}% ({len(sell_trades):,} trades)")
# Confusion matrix
true_pos_buy = results.filter(
(pl.col("ground_truth") == 1) & (pl.col("lee_ready") == 1)
).height
false_neg_buy = results.filter(
(pl.col("ground_truth") == 1) & (pl.col("lee_ready") != 1)
).height
true_pos_sell = results.filter(
(pl.col("ground_truth") == -1) & (pl.col("lee_ready") == -1)
).height
false_neg_sell = results.filter(
(pl.col("ground_truth") == -1) & (pl.col("lee_ready") != -1)
).height
print("\nConfusion matrix:")
print(f" True Buy (GT=B, LR=B): {true_pos_buy:,}")
print(f" False Buy (GT=B, LR≠B): {false_neg_buy:,}")
print(f" True Sell (GT=A, LR=A): {true_pos_sell:,}")
print(f" False Sell(GT=A, LR≠A): {false_neg_sell:,}")
# %% [markdown]
# ## 4. Multi-Day Validation
#
# Run validation across multiple days to get robust statistics.
# %%
def validate_multiple_days(
data_files: list,
max_files: int | None = None,
show_progress: bool = True,
max_rows_per_file: int | None = None,
) -> pl.DataFrame:
"""Run Lee-Ready validation across multiple days."""
all_results = []
files_to_process = data_files[:max_files] if max_files else data_files
rows_limit = max_rows_per_file
for i, file in enumerate(files_to_process):
print(f"Processing day {i + 1}/{len(files_to_process)}: {file.name}")
try:
df = load_databento_mbo(file, max_rows=rows_limit)
if len(df) > 1000:
day_results = classify_trades_lee_ready_databento(df, show_progress=False)
if len(day_results) > 0:
date_str = file.stem.split("-")[-1].split(".")[0]
day_results = day_results.with_columns(pl.lit(date_str).alias("timestamp"))
all_results.append(day_results)
except Exception as e:
print(f"Error processing {file.name}: {e}")
if all_results:
return pl.concat(all_results)
return pl.DataFrame()
# %%
multi_day_results = None
if data_files:
multi_day_results = validate_multiple_days(
data_files, max_files=MAX_VALIDATION_DAYS, max_rows_per_file=MAX_ROWS
)
if multi_day_results is not None and len(multi_day_results) > 0:
print("\n" + "=" * 50)
print("MULTI-DAY VALIDATION SUMMARY")
print("=" * 50)
# Overall accuracy
overall_accuracy = multi_day_results["correct"].mean() * 100
print(f"\nTotal trades: {len(multi_day_results):,}")
print(f"Overall accuracy: {overall_accuracy:.2f}%")
# By day
daily_stats = (
multi_day_results.group_by("timestamp")
.agg(
[
pl.len().alias("trades"),
(pl.col("correct").sum() / pl.len() * 100).alias("accuracy"),
]
)
.sort("timestamp")
)
print("\nDaily breakdown:")
for row in daily_stats.iter_rows():
print(f" {row[0]}: {row[2]:.2f}% ({row[1]:,} trades)")
# %% [markdown]
# ## 5. Compare to Tick Test Only
#
# Compare Lee-Ready (quote + tick test) vs tick test alone.
# %%
def classify_tick_test_only(df: pl.DataFrame, show_progress: bool = True) -> pl.DataFrame:
"""Classify trades using tick test only (no quote test)."""
trades = df.filter(pl.col("action") == "T").filter(pl.col("side").is_in(["B", "A"]))
if len(trades) == 0:
return pl.DataFrame()
# Add ground truth
trades = trades.with_columns(
pl.when(pl.col("side") == "B")
.then(1)
.when(pl.col("side") == "A")
.then(-1)
.otherwise(0)
.alias("ground_truth")
)
# Tick test: compare to previous price
trades = trades.with_columns(
pl.when(pl.col("price") > pl.col("price").shift(1))
.then(1)
.when(pl.col("price") < pl.col("price").shift(1))
.then(-1)
.otherwise(0)
.alias("tick_test")
)
# Zero-tick handling: use last non-zero direction
classified = []
last_dir = 0
for row in trades.iter_rows(named=True):
if row["tick_test"] != 0:
last_dir = row["tick_test"]
pred = last_dir if row["tick_test"] == 0 else row["tick_test"]
correct = 1 if pred == row["ground_truth"] else 0
classified.append(
{
"timestamp": row["timestamp"],
"ground_truth": row["ground_truth"],
"tick_test": pred,
"correct": correct,
}
)
return pl.DataFrame(classified)
# %%
if sample_df is not None and len(sample_df) > 1000:
tick_results = classify_tick_test_only(sample_df)
if len(tick_results) > 0:
tick_accuracy = tick_results["correct"].mean() * 100
print("\n" + "=" * 50)
print("TICK TEST ONLY RESULTS")
print("=" * 50)
print(f"Trades: {len(tick_results):,}")
print(f"Accuracy: {tick_accuracy:.2f}%")
# Compare to Lee-Ready
if results is not None and len(results) > 0:
lr_accuracy = results["correct"].mean() * 100
print("\nComparison:")
print(f" Lee-Ready: {lr_accuracy:.2f}%")
print(f" Tick only: {tick_accuracy:.2f}%")
print(f" Improvement: {lr_accuracy - tick_accuracy:.2f}%")
# %% [markdown]
# ## 6. Multi-Day Classification Accuracy (Table 3.3)
#
# Aggregate Lee-Ready and tick-test classification accuracy across the same
# 5-day window used in §3.4 Table 3.3. Tick-test reports two cohorts:
# **continuous** (zero-tick trades carry forward the last non-zero direction —
# 100% coverage) and **non-zero** (only classify trades whose price changed —
# coverage equals the share of non-zero-tick trades). Persists a summary
# parquet read by `book/03_market_microstructure/figures/scripts/generate_table_3_3.py`.
# %%
def tick_test_cohorts(df: pl.DataFrame) -> dict:
"""Return continuous and non-zero tick-test accuracy/coverage for one day."""
trades = df.filter(pl.col("action") == "T").filter(pl.col("side").is_in(["B", "A"]))
if len(trades) == 0:
return {"n_trades": 0, "continuous_correct": 0, "nonzero_n": 0, "nonzero_correct": 0}
# Enforce per-day chronological order locally so the price.shift(1)
# tick-test does not rely on upstream sort invariants.
trades = trades.sort("timestamp")
trades = trades.with_columns(
pl.when(pl.col("side") == "B")
.then(1)
.when(pl.col("side") == "A")
.then(-1)
.otherwise(0)
.alias("ground_truth"),
pl.when(pl.col("price") > pl.col("price").shift(1))
.then(1)
.when(pl.col("price") < pl.col("price").shift(1))
.then(-1)
.otherwise(0)
.alias("raw_tick"),
)
# Continuous: zero-tick rows carry forward last non-zero direction.
trades = trades.with_columns(
pl.when(pl.col("raw_tick") == 0)
.then(None)
.otherwise(pl.col("raw_tick"))
.forward_fill()
.fill_null(0)
.alias("continuous_tick")
)
n_trades = len(trades)
continuous_correct = (trades["continuous_tick"] == trades["ground_truth"]).sum()
nonzero = trades.filter(pl.col("raw_tick") != 0)
nonzero_n = len(nonzero)
nonzero_correct = (nonzero["raw_tick"] == nonzero["ground_truth"]).sum() if nonzero_n > 0 else 0
return {
"n_trades": n_trades,
"continuous_correct": int(continuous_correct),
"nonzero_n": int(nonzero_n),
"nonzero_correct": int(nonzero_correct),
}
# %%
multi_day_tick_summary = None
if data_files:
per_day_rows = []
for i, file in enumerate(data_files[:MAX_VALIDATION_DAYS]):
print(f"Tick-test day {i + 1}/{MAX_VALIDATION_DAYS}: {file.name}")
day_df = load_databento_mbo(file, max_rows=MAX_ROWS)
date_str = file.stem.split("-")[-1].split(".")[0]
agg = tick_test_cohorts(day_df)
agg["date"] = date_str
per_day_rows.append(agg)
per_day = pl.DataFrame(per_day_rows)
total_trades = int(per_day["n_trades"].sum())
total_cont_correct = int(per_day["continuous_correct"].sum())
total_nonzero_n = int(per_day["nonzero_n"].sum())
total_nonzero_correct = int(per_day["nonzero_correct"].sum())
continuous_accuracy = 100.0 * total_cont_correct / max(total_trades, 1)
nonzero_coverage = 100.0 * total_nonzero_n / max(total_trades, 1)
nonzero_accuracy = 100.0 * total_nonzero_correct / max(total_nonzero_n, 1)
lr_total_accuracy = (
100.0 * float(multi_day_results["correct"].mean())
if multi_day_results is not None and len(multi_day_results) > 0
else float("nan")
)
lr_total_trades = (
int(len(multi_day_results))
if multi_day_results is not None and len(multi_day_results) > 0
else 0
)
print("\n" + "=" * 60)
print("TABLE 3.3 — CLASSIFICATION ACCURACY (5-day NVDA aggregate)")
print("=" * 60)
print(f"Total trades classified: {total_trades:,}")
print(f"\n{'Method':<24} {'Coverage':>10} {'Accuracy':>10}")
print(f"{'Lee-Ready (quote+tick)':<24} {'100%':>10} {f'{lr_total_accuracy:.1f}%':>10}")
print(f"{'Tick test (continuous)':<24} {'100%':>10} {f'{continuous_accuracy:.1f}%':>10}")
print(
f"{'Tick test (non-zero)':<24} {f'{nonzero_coverage:.0f}%':>10} {f'{nonzero_accuracy:.1f}%':>10}"
)
# Persist Table 3.3 summary for book-side script regeneration.
output_dir = get_output_dir(3, "databento")
output_dir.mkdir(parents=True, exist_ok=True)
summary_path = output_dir / "table_3_3_classification_accuracy.parquet"
multi_day_tick_summary = pl.DataFrame(
[
{
"method": "Lee-Ready (quote+tick)",
"coverage_pct": 100.0,
"accuracy_pct": lr_total_accuracy,
"n_trades": lr_total_trades,
},
{
"method": "Tick test (continuous)",
"coverage_pct": 100.0,
"accuracy_pct": continuous_accuracy,
"n_trades": total_trades,
},
{
"method": "Tick test (non-zero)",
"coverage_pct": nonzero_coverage,
"accuracy_pct": nonzero_accuracy,
"n_trades": total_nonzero_n,
},
]
)
multi_day_tick_summary.write_parquet(summary_path)
print(f"\nSaved: {summary_path}")
# %% [markdown]
# ## Key Takeaways
#
# ### Lee-Ready Validation Results
#
# | Method | Accuracy | Notes |
# |--------|----------|-------|
# | Lee-Ready (quote + tick) | ~94-95% | Uses LOB midpoint when available |
# | Tick test only | ~78% | Significantly worse without quote test |
# | Ground truth | 100% | DataBento provides actual aggressor |
#
# ### Why Lee-Ready Works
#
# The quote test (comparing trade price to midpoint) captures the fundamental
# market microstructure: buyer-initiated trades tend to occur at or above the ask
# (above midpoint), while seller-initiated trades occur at or below the bid.
#
# ### Implementation Notes
#
# 1. **LOB state matters**: Must maintain accurate book state at each trade
# 2. **Tick test fallback**: Only used when trade exactly at midpoint
# 3. **Zero-tick handling**: Preserve last direction on unchanged price
#
# ---
#
# **Reference**: Lee & Ready (1991), "Inferring Trade Direction from Intraday Data"
03_market_microstructure/16_itch_information_bars.ipynb
+2557
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/16_itch_information_bars.py
+634
View File
@@ -0,0 +1,634 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Information-Driven Bars: Formulas and Parameter Study
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Two-part validation of imbalance-bar construction: (1) verify the AFML
# tick-imbalance formula matches the `ml4t.engineer.bars` library exactly on
# DataBento NVDA trades, and (2) sweep $\alpha$ and the target $E[T]$ across
# three families (alpha-based EWMA, fixed threshold, rolling window) to expose
# the parameter-instability issue that §3.4 warns about.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - State the tick-imbalance threshold formula
# $\theta = \sum b_t$, $E[\theta_T] = E[T] \cdot |2P[b=1] - 1|$
# and the volume-imbalance variant.
# - Compare alpha-based, fixed-threshold, and rolling-window imbalance bars,
# and recognize when the alpha-based scheme drifts (e.g., $\alpha=0.1$
# produces ~17x E[T] inflation on liquid equities).
# - Choose imbalance-bar parameters that yield well-behaved
# Jarque-Bera / variance-ratio diagnostics.
#
# ## Book reference
#
# Section §3.4, *The Art of Sampling* — information-driven-bars subsection.
#
# ## Prerequisites
#
# - DataBento XNAS-ITCH MBO parquets at
# `data/equities/market/microstructure/market_by_order/NVDA/`.
# %%
"""Information-Driven Bars: Formulas and Parameter Study — verifying AFML imbalance bar formulas and exploring parameter sensitivity."""
import warnings
from pathlib import Path
warnings.filterwarnings("ignore")
import numpy as np
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from scipy import stats
# Import loader for MBO data
from data import load_mbo_data
# Polars display configuration
# %% tags=["parameters"]
MAX_DAYS = 3
# %%
# Get file paths from the canonical loader (handles legacy/new path resolution)
data_files = load_mbo_data(symbols=["NVDA"], list_files=True)
DATABENTO_DIR = data_files[0].parent if data_files else None
# %% [markdown]
# ### Load Trade Data
# Extract and filter trade records from multi-day MBO parquet files.
# %%
def load_trades(data_dir: Path, max_days: int = 10) -> tuple[pl.DataFrame | None, list[str]]:
"""Load trade data from multiple days."""
data_files = sorted(data_dir.glob("*.parquet"))[:max_days]
if not data_files:
return None, []
all_trades = []
dates = []
for file_path in data_files:
date_str = file_path.name.split("-")[2].split(".")[0]
trade_date = f"{date_str[:4]}-{date_str[4:6]}-{date_str[6:8]}"
dates.append(trade_date)
df = pl.read_parquet(file_path)
ts_col = "ts_event" if "ts_event" in df.columns else "ts_recv"
df = df.select([ts_col, "action", "side", "price", "size"])
df = df.with_columns(pl.col(ts_col).cast(pl.Datetime("ns")).alias("timestamp"))
df = df.filter(pl.col("action") == "T")
# Filter to trading hours (9:30-16:00 ET)
# Note: This uses 13:30-21:00 UTC which covers both EDT (summer) and EST (winter).
# The conservative filter may include 15-30 min of pre/post market data depending
# on season. A robust solution would convert to America/New_York first.
df = df.filter(
(
(pl.col("timestamp").dt.hour() > 13)
| ((pl.col("timestamp").dt.hour() == 13) & (pl.col("timestamp").dt.minute() >= 30))
)
& (pl.col("timestamp").dt.hour() < 21)
)
df = df.with_columns(
pl.when(pl.col("side") == "A")
.then(1)
.when(pl.col("side") == "B")
.then(-1)
.otherwise(0)
.alias("side_num")
)
df = df.select(
[
"timestamp",
pl.col("price"),
pl.col("size").alias("volume"),
pl.col("side_num").alias("side"),
]
).sort("timestamp")
all_trades.append(df)
return pl.concat(all_trades), dates
# %%
# Load data
trades, dates = load_trades(DATABENTO_DIR, max_days=MAX_DAYS)
if trades is None or len(trades) == 0:
raise FileNotFoundError(
"Missing DataBento MBO trade data for NVDA. "
"Expected parquet files under data/equities/market_by_order/NVDA."
)
trades = trades.filter(pl.col("side") != 0)
print(f"Loaded {len(trades):,} trades from {len(dates)} days")
print(f"Date range: {dates[0]} to {dates[-1]}")
print(f"Buy fraction: {(trades['side'] > 0).mean():.2%}")
# %% [markdown]
# ## 2. Formula Verification: Manual vs Library
#
# We implement tick imbalance bars manually to verify the library is correct.
# %%
def calculate_tick_imbalance_bars_manual(
sides: np.ndarray,
expected_t: float = 1000.0,
alpha: float = 0.1,
min_bars_warmup: int = 10,
) -> tuple[list[int], list[dict]]:
"""
Manual AFML tick imbalance bars.
θ = Σ b_t (cumulative signed ticks)
E[θ_T] = E[T] × |2P[b=1] - 1|
"""
n = len(sides)
# Initialize from warmup (matches library)
warmup_size = min(1000, n)
p_buy = float(np.mean(sides[:warmup_size] > 0))
bar_indices = []
bar_info = []
cumulative_theta = 0.0
bar_tick_count = 0
bar_buy_count = 0
n_bars = 0
for i in range(n):
side = sides[i]
is_buy = side > 0
cumulative_theta += side
bar_tick_count += 1
if is_buy:
bar_buy_count += 1
# AFML threshold
threshold = expected_t * abs(2 * p_buy - 1)
if abs(cumulative_theta) >= threshold:
bar_indices.append(i)
bar_info.append(
{
"bar": n_bars,
"ticks": bar_tick_count,
"theta": cumulative_theta,
"threshold": threshold,
"E[T]": expected_t,
"P[b=1]": p_buy,
}
)
n_bars += 1
# Update EWMA after warmup
if n_bars > min_bars_warmup:
expected_t = alpha * bar_tick_count + (1 - alpha) * expected_t
bar_p_buy = bar_buy_count / bar_tick_count
p_buy = alpha * bar_p_buy + (1 - alpha) * p_buy
# Reset
cumulative_theta = 0.0
bar_tick_count = 0
bar_buy_count = 0
return bar_indices, bar_info
# %%
# Run manual calculation
# Use slow adaptation to prevent threshold spiral
sides_arr = trades["side"].to_numpy()
VERIFY_ET = 1000
VERIFY_ALPHA = 0.001
VERIFY_WARMUP = 100
manual_indices, manual_info = calculate_tick_imbalance_bars_manual(
sides_arr, expected_t=VERIFY_ET, alpha=VERIFY_ALPHA, min_bars_warmup=VERIFY_WARMUP
)
print(f"Manual TIB calculation: {len(manual_indices)} bars")
# %%
# Compare with library
from ml4t.engineer.bars import TickImbalanceBarSampler
sampler = TickImbalanceBarSampler(
expected_ticks_per_bar=VERIFY_ET,
alpha=VERIFY_ALPHA,
min_bars_warmup=VERIFY_WARMUP,
)
library_bars = sampler.sample(trades)
print(f"Library TIB calculation: {len(library_bars)} bars")
# Verify match
if len(manual_indices) == len(library_bars):
manual_thresholds = [d["threshold"] for d in manual_info]
library_thresholds = library_bars["expected_imbalance"].to_list()
max_diff = max(
abs(m - lib) for m, lib in zip(manual_thresholds, library_thresholds, strict=False)
)
print(f"Max threshold difference: {max_diff:.6f}")
print("[OK] Manual and library match!")
else:
print("[FAIL] Bar counts differ")
# Show first few bars
print("\nFirst 5 bars:")
pl.DataFrame(manual_info[:5])
# %% [markdown]
# ## 3. Parameter Study Using Library
#
# Now we use the faster library implementation to study how E[T] affects properties.
#
# **Statistical Metrics Explained**:
# - **Jarque-Bera (JB)**: Tests normality. Lower = more normal (JB=0 is perfectly normal).
# High JB indicates fat tails/skewness.
# - **Autocorrelation(1)**: Correlation of returns with 1-bar-lagged returns.
# Should be ~0 for efficient markets.
# - **Variance Ratio(5)**: Var(5-bar returns) / (5 × Var(1-bar returns)).
# Should be ~1 for random walk. >1 = momentum, <1 = mean reversion.
# %%
from ml4t.engineer.bars import ImbalanceBarSampler, TickImbalanceBarSampler
def compute_stats(bars: pl.DataFrame) -> dict:
"""Compute statistical properties of bar returns."""
if len(bars) < 30:
return {
"n_bars": len(bars),
"jarque_bera": np.nan,
"autocorr_1": np.nan,
"variance_ratio_5": np.nan,
}
returns = bars["close"].pct_change().drop_nulls().to_numpy()
returns = returns[np.isfinite(returns)]
if len(returns) < 10:
return {
"n_bars": len(bars),
"jarque_bera": np.nan,
"autocorr_1": np.nan,
"variance_ratio_5": np.nan,
}
jb, _ = stats.jarque_bera(returns)
ac = np.corrcoef(returns[:-1], returns[1:])[0, 1] if len(returns) > 1 else np.nan
if len(returns) > 5:
var_1 = np.var(returns)
summed = np.array([np.sum(returns[i : i + 5]) for i in range(len(returns) - 4)])
var_5 = np.var(summed) / 5
vr = var_5 / var_1 if var_1 > 0 else np.nan
else:
vr = np.nan
return {"n_bars": len(bars), "jarque_bera": jb, "autocorr_1": ac, "variance_ratio_5": vr}
# %%
# Parameter grids
# Key insight: with persistent order flow imbalance, need SLOW adaptation (α=0.001)
# to prevent threshold spiral. Also use longer warmup.
TIB_ET = [500, 700, 1000, 1500, 2000, 3000]
VIB_ET = [2000, 5000, 10000, 20000, 50000]
ALPHA = 0.001 # Very slow adaptation - critical for stability
WARMUP = 100 # Longer warmup
print("=" * 70)
print("TICK IMBALANCE BARS (TIBs) - alpha=0.001, warmup=100")
print("=" * 70)
tib_results = []
for et in TIB_ET:
bars = TickImbalanceBarSampler(
expected_ticks_per_bar=et, alpha=ALPHA, min_bars_warmup=WARMUP
).sample(trades)
s = compute_stats(bars)
s["expected_t"] = et
tib_results.append(s)
print(
f"E[T]={et:>6}: {s['n_bars']:>5} bars, JB={s['jarque_bera']:>8.1f}, "
f"AC(1)={s['autocorr_1']:>6.3f}, VR(5)={s['variance_ratio_5']:>5.2f}"
)
print("\n" + "=" * 70)
print("VOLUME IMBALANCE BARS (VIBs) - alpha=0.001, warmup=100")
print("=" * 70)
vib_results = []
for et in VIB_ET:
bars = ImbalanceBarSampler(
expected_ticks_per_bar=et, alpha=ALPHA, min_bars_warmup=WARMUP
).sample(trades)
s = compute_stats(bars)
s["expected_t"] = et
vib_results.append(s)
print(
f"E[T]={et:>6}: {s['n_bars']:>5} bars, JB={s['jarque_bera']:>8.1f}, "
f"AC(1)={s['autocorr_1']:>6.3f}, VR(5)={s['variance_ratio_5']:>5.2f}"
)
# %% [markdown]
# ## 4. Visualize Results
# %%
tib_df = pl.DataFrame(tib_results)
vib_df = pl.DataFrame(vib_results)
# %%
# Construct all four panels and show the figure exactly once to avoid
# matplotlib/plotly intermediate-cell renders of an incomplete figure.
fig = make_subplots(
rows=2,
cols=2,
subplot_titles=[
"Bar Count vs E[T]",
"Jarque-Bera vs Bar Count",
"Autocorrelation(1) vs Bar Count",
"Variance Ratio(5) vs Bar Count",
],
vertical_spacing=0.15,
horizontal_spacing=0.12,
)
tib_color, vib_color = "#1e3a5f", "#c74b16"
# Panel 1: bar count vs E[T]
for name, color, df in [("TIB", tib_color, tib_df), ("VIB", vib_color, vib_df)]:
fig.add_trace(
go.Scatter(
x=df["expected_t"].to_list(),
y=df["n_bars"].to_list(),
mode="lines+markers",
name=name,
line=dict(color=color),
),
row=1,
col=1,
)
fig.update_xaxes(type="log", title_text="E[T]", row=1, col=1)
fig.update_yaxes(type="log", title_text="Number of Bars", row=1, col=1)
# Panel 2: Jarque-Bera vs bar count
for name, color, df in [("TIB", tib_color, tib_df), ("VIB", vib_color, vib_df)]:
fig.add_trace(
go.Scatter(
x=df["n_bars"].to_list(),
y=df["jarque_bera"].to_list(),
mode="markers",
marker=dict(color=color, size=10),
showlegend=False,
),
row=1,
col=2,
)
fig.update_xaxes(type="log", title_text="Number of Bars", row=1, col=2)
fig.update_yaxes(type="log", title_text="Jarque-Bera", row=1, col=2)
# Panel 3: autocorrelation(1) vs bar count
for name, color, df in [("TIB", tib_color, tib_df), ("VIB", vib_color, vib_df)]:
fig.add_trace(
go.Scatter(
x=df["n_bars"].to_list(),
y=df["autocorr_1"].to_list(),
mode="markers",
marker=dict(color=color, size=10),
showlegend=False,
),
row=2,
col=1,
)
fig.add_hline(y=0, line_dash="dash", line_color="gray", row=2, col=1)
fig.update_xaxes(type="log", title_text="Number of Bars", row=2, col=1)
fig.update_yaxes(title_text="Autocorrelation(1)", row=2, col=1)
# Panel 4: variance ratio(5) vs bar count
for name, color, df in [("TIB", tib_color, tib_df), ("VIB", vib_color, vib_df)]:
fig.add_trace(
go.Scatter(
x=df["n_bars"].to_list(),
y=df["variance_ratio_5"].to_list(),
mode="markers",
marker=dict(color=color, size=10),
showlegend=False,
),
row=2,
col=2,
)
fig.add_hline(y=1, line_dash="dash", line_color="gray", row=2, col=2)
fig.update_xaxes(type="log", title_text="Number of Bars", row=2, col=2)
fig.update_yaxes(title_text="Variance Ratio(5)", row=2, col=2)
fig.update_layout(
title="TIB vs VIB Statistical Properties",
height=650,
legend=dict(x=0.5, y=1.02, xanchor="center", orientation="h"),
)
fig.show()
# %% [markdown]
# ## 5. Key Takeaways
#
# | Property | TIBs | VIBs |
# |----------|------|------|
# | Accumulates | Trade signs (+1/-1) | Signed volume |
# | Threshold scale | ~0.2 × E[T] | ~800 × E[T] |
# | Bars for same E[T] | Many more | Far fewer |
#
# ### Critical Calibration Insight
#
# With persistent order flow imbalance (common in real data), the adaptive EWMA
# can cause **threshold spiral** - bars get progressively larger as the algorithm
# adapts E[T] and P[b=1] upward.
#
# **Solution**: Use very slow adaptation:
# - α = 0.001 (not 0.1)
# - warmup = 100+ bars (not 10)
#
# ### AFML Insight
#
# There's no "optimal" E[T]. Choose based on:
# - Desired bar frequency (more bars = better normality, but more noise)
# - Trading horizon (intraday needs more bars than swing)
# - Signal strength vs statistical properties tradeoff
# %% [markdown]
# ## 6. Comparing Three Approaches: α-Based, Fixed, and Window-Based
#
# The ml4t-engineer library provides three different implementations:
#
# 1. **α-Based (AFML)**: Exponential decay for E[T] and P[b=1] - requires careful α tuning
# 2. **Fixed Threshold**: No adaptation - simplest and most predictable
# 3. **Window-Based**: Rolling window adaptation - bounded drift
#
# Let's compare them on the same data.
# %%
from ml4t.engineer.bars import (
FixedTickImbalanceBarSampler, # Fixed threshold
TickImbalanceBarSampler, # α-based
WindowTickImbalanceBarSampler, # Window-based
)
# Test parameters - match the handoff comparison
COMPARE_ET = 1000
COMPARE_THRESHOLD = 100 # For fixed
# Track how E[T] drifts for each method
def measure_et_drift(bars: pl.DataFrame) -> float:
"""Measure E[T] drift (last / first expected_t)."""
if "expected_t" not in bars.columns or len(bars) == 0:
return 1.0 # No drift for fixed or empty bars
first = bars["expected_t"][0]
last = bars["expected_t"][-1]
return last / first if first > 0 else 1.0
# %%
print("=" * 70)
print("COMPARING THREE TICK IMBALANCE BAR APPROACHES")
print("=" * 70)
comparison = []
# 1. α-based with different alphas
for alpha in [0.001, 0.01, 0.1]:
bars = TickImbalanceBarSampler(
expected_ticks_per_bar=COMPARE_ET,
alpha=alpha,
min_bars_warmup=100,
).sample(trades)
drift = measure_et_drift(bars)
bar_stats = compute_stats(bars)
comparison.append(
{
"method": f"α={alpha}",
"n_bars": len(bars),
"avg_ticks": len(trades) / len(bars) if len(bars) > 0 else 0,
"et_drift": drift,
"jb": bar_stats["jarque_bera"],
"ac1": bar_stats["autocorr_1"],
}
)
print(
f"α-based α={alpha}: {len(bars):>4} bars, "
f"avg_ticks={len(trades) / len(bars) if len(bars) > 0 else 0:>7.0f}, E[T] drift={drift:.2f}x"
)
# %%
# 2. Fixed threshold
for thresh in [50, 100, 200]:
bars = FixedTickImbalanceBarSampler(threshold=thresh).sample(trades)
bar_stats = compute_stats(bars)
comparison.append(
{
"method": f"fixed={thresh}",
"n_bars": len(bars),
"avg_ticks": len(trades) / len(bars) if len(bars) > 0 else 0,
"et_drift": 1.0,
"jb": bar_stats["jarque_bera"],
"ac1": bar_stats["autocorr_1"],
}
)
print(
f"Fixed thresh={thresh}: {len(bars):>4} bars, "
f"avg_ticks={len(trades) / len(bars) if len(bars) > 0 else 0:>7.0f}, E[T] drift=N/A"
)
# %%
# 3. Window-based with different tick windows
for tick_win in [2000, 5000, 10000]:
bars = WindowTickImbalanceBarSampler(
initial_expected_t=COMPARE_ET,
bar_window=10,
tick_window=tick_win,
).sample(trades)
drift = measure_et_drift(bars)
bar_stats = compute_stats(bars)
comparison.append(
{
"method": f"window={tick_win}",
"n_bars": len(bars),
"avg_ticks": len(trades) / len(bars) if len(bars) > 0 else 0,
"et_drift": drift,
"jb": bar_stats["jarque_bera"],
"ac1": bar_stats["autocorr_1"],
}
)
print(
f"Window tick_win={tick_win}: {len(bars):>4} bars, "
f"avg_ticks={len(trades) / len(bars) if len(bars) > 0 else 0:>7.0f}, E[T] drift={drift:.2f}x"
)
# %%
# Summary table
compare_df = pl.DataFrame(comparison)
print("\n" + "=" * 70)
print("COMPARISON SUMMARY")
print("=" * 70)
compare_df = compare_df.with_columns(
[
pl.col("avg_ticks").round(0).cast(pl.Int64),
pl.col("et_drift").round(2),
pl.col("jb").round(1),
pl.col("ac1").round(3),
]
)
print(compare_df)
# %% [markdown]
# ## 7. Recommendations
#
# | Use Case | Recommended Method | Why |
# |----------|-------------------|-----|
# | Production | `FixedTickImbalanceBarSampler` | Simplest, no drift, predictable |
# | Research | `TickImbalanceBarSampler(α=0.001)` | Closest to textbook, slow adaptation |
#
# **Key Findings**:
# - The adaptive α-scheme fails in two opposite ways here: α=0.01 collapses the
# threshold (399,507 bars, avg 1 tick/bar), while α=0.1 inflates it (169 bars,
# avg 3,401 ticks/bar, ~17.5x E[T] drift). Only α=0.001 stays near target
# (594 bars, avg 968 ticks/bar, ~1x drift).
# - The window-based sampler also produces degenerate bars in this test — likely a
# parameterization or implementation issue to investigate further
# - α=0.001 with warmup=100 and fixed thresholds are the only reliable approaches here
# %%
print("\n" + "=" * 70)
print("NOTEBOOK SUMMARY")
print("=" * 70)
print("\nTIBs:")
print(tib_df.select(["expected_t", "n_bars", "jarque_bera", "autocorr_1", "variance_ratio_5"]))
print("\nVIBs:")
print(vib_df.select(["expected_t", "n_bars", "jarque_bera", "autocorr_1", "variance_ratio_5"]))
print("\nThree-Approach Comparison:")
print(compare_df)
print("\nNotebook completed.")
03_market_microstructure/17_databento_bar_sampling.ipynb
+2922
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/17_databento_bar_sampling.py
+1693
View File
File diff suppressed because it is too large Load Diff
03_market_microstructure/18_algoseek_jump_detection.ipynb
+1218
View File
File diff suppressed because one or more lines are too long
03_market_microstructure/18_algoseek_jump_detection.py
+492
View File
@@ -0,0 +1,492 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Detecting Price Jumps in Intraday Returns
#
# **Chapter 3: Market Microstructure**
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Realized intraday variance mixes two distinct processes: a continuous
# diffusive component driven by the steady arrival of small order-flow
# innovations, and a discrete jump component driven by news, scheduled
# announcements, and large block trades. This notebook separates the two
# on AlgoSeek NASDAQ-100 minute bars using bipower variation
# (Barndorff-Nielsen and Shephard, 2004) for the continuous part and the
# Lee–Mykland (2008) nonparametric test to time individual jumps. The
# resulting daily features — jump count, signed jump variance, jump share
# of realized variance — feed directly into the label-conditioning logic
# of Chapter 7 and the volatility-regime features of Chapter 8.
#
# ## Learning Objectives
#
# - Decompose daily realized variance into continuous and jump components
# via bipower variation
# - Apply the Lee–Mykland test to flag individual jumps with a
# leakage-free, multiple-testing-aware critical value
# - Quantify how concentrated jumps are in time of day and across the
# trading year
# - Show why a naive |z|>k threshold over-detects in low-vol regimes and
# under-detects in high-vol regimes
# - Materialize a per-symbol, per-day jump-feature panel ready for
# downstream feature engineering
#
# ## Book reference
#
# Section §3.5, *Detecting Price Jumps in Intraday Returns*.
#
# ## Prerequisites
#
# - AlgoSeek TAQ minute bars under
# `data/equities/market/nasdaq100/minute_bars/year=YYYY/month=MM.parquet`
# (Hive-partitioned), loaded via `load_nasdaq100_bars`.
# - Companion: `13_algoseek_minute_bars_eda` for raw-bar microstructure
# schema.
# %%
"""Lee–Mykland jump detection on AlgoSeek minute bars."""
from __future__ import annotations
import warnings
warnings.filterwarnings("ignore")
import math
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
from data import load_nasdaq100_bars
from utils.paths import get_output_dir
# %% tags=["parameters"]
# AlgoSeek preserves historical tickers — for 2020 the Meta entry is "FB"
# (the symbol change to "META" landed in mid-2022). load_nasdaq100_bars will
# return zero rows for "META" on 2020 dates.
SYMBOLS = ["AMD", "AMZN", "FB"]
START_DATE = "2020-01-01"
END_DATE = "2020-12-31"
BAR_MINUTES = 5 # Bar frequency in minutes
LEE_MYKLAND_K = 12 # Within-session local-volatility window (~60 min at 5-min)
JUMP_ALPHA = 0.01 # Family-wise significance level for the daily jump test
NAIVE_Z_THRESHOLD = 4.0 # Naive comparison rule: flag |return / rolling_std| > NAIVE_Z
# %%
OUTPUT_DIR = get_output_dir(3, "jump_features")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
# Bipower variation theoretical constant: E[|Z|] = sqrt(2/pi) for Z ~ N(0,1)
MU1 = math.sqrt(2.0 / math.pi)
BARS_PER_DAY = (16 - 9.5) * 60 // BAR_MINUTES # Regular hours, 78 at 5-min
# %% [markdown]
# ## 1. Load minute bars and compute log returns
#
# We restrict to regular hours (09:30–16:00 ET) so that the diffusion
# assumption underlying both bipower variation and the Lee–Mykland test
# is not contaminated by extended-session stale quotes. Log returns are
# computed per symbol-day to avoid bridging the overnight gap, which
# behaves as a jump but is not the object of intraday inference.
# %%
bars = (
load_nasdaq100_bars(
frequency=f"{BAR_MINUTES}m",
symbols=SYMBOLS,
start_date=START_DATE,
end_date=END_DATE,
regular_hours=True,
)
.sort(["symbol", "timestamp"])
.with_columns(date=pl.col("timestamp").dt.date())
)
per_symbol_days = bars.group_by("symbol").agg(n_days=pl.col("date").n_unique())
missing = [s for s in SYMBOLS if s not in set(per_symbol_days["symbol"].to_list())]
if missing:
raise ValueError(
f"load_nasdaq100_bars returned zero rows for {missing}; check the ticker "
f"history (e.g. FB→META in mid-2022) against the date window."
)
max_days = per_symbol_days["n_days"].max()
short = per_symbol_days.filter(pl.col("n_days") < 0.5 * max_days)
if short.height:
raise ValueError(
f"Coverage gap: {short.to_dicts()} cover <50% of the longest-symbol session "
f"count ({max_days}); a window-shift bug may have left a ticker partially "
f"unavailable in the requested range."
)
returns = bars.with_columns(
log_return=(pl.col("close").log() - pl.col("close").log().shift(1)).over(["symbol", "date"]),
).drop_nulls("log_return")
print(f"Bars loaded: {bars.height:,} symbols: {returns['symbol'].n_unique()}")
print(f"Trading days: {returns['date'].n_unique():,} bars/day target: {int(BARS_PER_DAY)}")
# %% [markdown]
# ## 2. Realized variance and bipower variation
#
# For day $d$ with $n$ intraday log returns $r_1, \dots, r_n$, the
# realized variance is
# $$\mathrm{RV}_d = \sum_{i=1}^{n} r_i^2,$$
# which is a consistent estimator of total quadratic variation —
# continuous *plus* squared jumps. The bipower variation,
# $$\mathrm{BV}_d = \mu_1^{-2} \sum_{i=2}^{n} |r_i| \, |r_{i-1}|, \qquad \mu_1 = \sqrt{2/\pi},$$
# is jump-robust: a single large $|r_i|$ is multiplied by its small
# neighbor and contributes negligibly. The gap
# $\max(\mathrm{RV}_d - \mathrm{BV}_d, 0)$ is therefore an estimate of
# the jump-attributable variance on day $d$.
# %%
daily = (
returns.group_by(["symbol", "date"])
.agg(
rv=(pl.col("log_return") ** 2).sum(),
bv=(pl.col("log_return").abs() * pl.col("log_return").abs().shift(1)).sum() / (MU1**2),
n_bars=pl.len(),
)
.with_columns(
jump_var=(pl.col("rv") - pl.col("bv")).clip(lower_bound=0.0),
jump_share_rv=((pl.col("rv") - pl.col("bv")).clip(lower_bound=0.0) / pl.col("rv")).clip(
0, 1
),
)
)
daily.describe()
# %% [markdown]
# Annualized, BV gives the continuous-volatility floor of each name and
# the jump-variance share quantifies how much of the total annual
# realized variance is attributable to discrete jumps rather than
# diffusion.
# %%
annual_summary = (
daily.group_by("symbol")
.agg(
rv_annual=pl.col("rv").sum(),
bv_annual=pl.col("bv").sum(),
jump_var_annual=pl.col("jump_var").sum(),
)
.with_columns(
jump_share_annual=pl.col("jump_var_annual") / pl.col("rv_annual"),
cont_vol_annual=(pl.col("bv_annual").sqrt()),
rv_vol_annual=(pl.col("rv_annual").sqrt()),
)
)
annual_summary
# %% [markdown]
# ## 3. The Lee–Mykland jump test
#
# Lee and Mykland (2008) construct a test that times jumps to the bar.
# For bar $i$ within a session of $n$ bars they form a standardized statistic
# $$L_i = \frac{r_i}{\hat\sigma_i}, \qquad
# \hat\sigma_i^2 = \frac{1}{K-2}\sum_{j=i-K+2}^{i-1} |r_j|\,|r_{j-1}|,$$
# where $\hat\sigma_i$ is a local bipower volatility built from a window
# of $K$ prior returns *within the same session* — it adapts to slow-moving
# intraday volatility regimes and is robust to the jumps it is trying to
# detect. Building the window per session avoids bridging the overnight
# gap, which behaves as a jump and would contaminate the local-volatility
# denominator for the first $K$ bars of each day. The trade-off is that
# the first $K-1$ bars of every session are not directly testable.
#
# Under the no-jump-at-$i$ null and a continuous-diffusion alternative,
# $\max_i |L_i|$ over the daily session converges to a Gumbel distribution.
# The rejection rule at level $\alpha$ is
# $$|L_i| > S_n \beta^*(\alpha) + C_n, \qquad
# C_n = \frac{(2\log n)^{1/2}}{\mu_1} - \frac{\log\pi + \log\log n}{2\mu_1 (2\log n)^{1/2}}, \;
# S_n = \frac{1}{\mu_1 (2\log n)^{1/2}},$$
# and $\beta^*(\alpha) = -\log(-\log(1-\alpha))$. The multiple-testing
# burden is absorbed by the $n$-dependent constants, not by an ad-hoc
# Bonferroni adjustment. Lee and Mykland recommend $K \approx \sqrt n$
# (≈9 for 78 5-minute bars per day); we use $K = 12$ as a slightly more
# conservative window.
# %%
def gumbel_threshold(n: int, alpha: float) -> float:
"""Lee–Mykland critical value for n bars per session at level alpha."""
c_n = (np.sqrt(2 * np.log(n)) / MU1) - (np.log(np.pi) + np.log(np.log(n))) / (
2 * MU1 * np.sqrt(2 * np.log(n))
)
s_n = 1.0 / (MU1 * np.sqrt(2 * np.log(n)))
beta_star = -np.log(-np.log(1 - alpha))
return s_n * beta_star + c_n
def lee_mykland_jumps_session(rets: pl.DataFrame, k: int, threshold: float) -> pl.DataFrame:
"""Apply Lee–Mykland (2008) per (symbol, date) so the rolling window
of K-2 pair products is built only from same-session bars.
Returns the input frame with three columns appended: L_stat,
sigma_local, is_jump. The first K-1 bars of each session have NaN
sigma_local (no within-session history yet) and is_jump=False.
"""
sessions: list[pl.DataFrame] = []
for (_sym, _date), session in rets.group_by(["symbol", "date"], maintain_order=True):
r = session["log_return"].to_numpy()
n_obs = r.shape[0]
sigma_hat = np.full(n_obs, np.nan)
if n_obs >= k:
abs_r = np.abs(r)
# pair_prod[j] = |r_{j}| * |r_{j-1}| for j = 1 .. n_obs-1, same-session
pair_prod = abs_r[1:] * abs_r[:-1]
ps_cum = np.concatenate(([0.0], np.cumsum(pair_prod)))
# For bar i (i >= k-1, 0-indexed), window uses pair_prod[i-k+1 .. i-2]
# (k-2 terms ending at bar i-1); first testable bar is index k-1, so
# exactly K-1 leading bars per session carry NaN sigma_local.
idx = np.arange(k - 1, n_obs)
sigma_sq = (ps_cum[idx - 1] - ps_cum[idx - k + 1]) / (k - 2)
sigma_hat[idx] = np.sqrt(np.maximum(sigma_sq, 0.0))
# NaN sigma_local → NaN L → comparison yields False; no explicit guard needed.
L = r / sigma_hat
sessions.append(
session.with_columns(
L_stat=pl.Series(L),
sigma_local=pl.Series(sigma_hat),
is_jump=pl.Series(np.abs(L) > threshold),
)
)
return pl.concat(sessions)
# %%
# Multiple-testing constants depend on bars per session; computed once across
# all (symbol, date) groups so every bar is tested against the same threshold.
n_per_session = returns.group_by(["symbol", "date"]).len()["len"].to_numpy()
N_PER_SESSION = int(np.median(n_per_session))
threshold_used = gumbel_threshold(N_PER_SESSION, JUMP_ALPHA)
flagged = lee_mykland_jumps_session(returns, k=LEE_MYKLAND_K, threshold=threshold_used)
jumps = flagged.filter(pl.col("is_jump"))
print(f"Bars per session (median): {N_PER_SESSION}")
print(f"Lee–Mykland critical value at α={JUMP_ALPHA}: {threshold_used:.3f}")
print(f"Jumps flagged: {jumps.height:,} across {flagged['symbol'].n_unique()} symbols")
print(
f"Jumps per symbol-day: {jumps.height / max(1, flagged.group_by(['symbol', 'date']).len().height):.2f}"
)
# %% [markdown]
# ## 4. How often do jumps fire and how big are they?
#
# A symbol-day typically sees a handful of test rejections; days with
# scheduled news cluster materially higher. The signed magnitude of
# the underlying log return distinguishes upside surprises from
# downside.
# %%
daily_jump_counts = (
flagged.group_by(["symbol", "date"]).agg(
jump_count=pl.col("is_jump").sum(),
signed_jump_var=pl.when(pl.col("is_jump"))
.then(pl.col("log_return") ** 2 * pl.col("log_return").sign())
.otherwise(0.0)
.sum(),
)
).join(
daily.select("symbol", "date", "rv", "bv", "jump_var", "jump_share_rv"), on=["symbol", "date"]
)
print(daily_jump_counts.describe())
# %%
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for sym in SYMBOLS:
counts = daily_jump_counts.filter(pl.col("symbol") == sym)["jump_count"].to_numpy()
axes[0].hist(counts, bins=range(0, int(counts.max()) + 2), alpha=0.5, label=sym)
axes[0].set_xlabel("Lee–Mykland jumps per day")
axes[0].set_ylabel("Trading days")
axes[0].set_title("Daily jump-count distribution")
axes[0].legend()
# Q-Q of standardized returns: all vs jump-filtered. sigma_local is NaN for
# the first LEE_MYKLAND_K-1 bars of each session (no within-session window
# yet), so division yields NaN — filter explicitly.
all_z_raw = (flagged["log_return"] / flagged["sigma_local"]).to_numpy()
all_z = all_z_raw[np.isfinite(all_z_raw)]
filt = flagged.filter(~pl.col("is_jump"))
filt_z_raw = (filt["log_return"] / filt["sigma_local"]).to_numpy()
filt_z = filt_z_raw[np.isfinite(filt_z_raw)]
qs = np.linspace(0.001, 0.999, 200)
from scipy.stats import norm # noqa: PLC0415
norm_q = norm.ppf(qs)
axes[1].scatter(norm_q, np.quantile(all_z, qs), s=10, alpha=0.7, label="All returns")
axes[1].scatter(norm_q, np.quantile(filt_z, qs), s=10, alpha=0.7, label="Jump-filtered")
axes[1].plot([-4, 4], [-4, 4], "k--", lw=0.8)
axes[1].set_xlim(-4, 4)
axes[1].set_ylim(-8, 8)
axes[1].set_xlabel("Theoretical quantile (N(0,1))")
axes[1].set_ylabel("Empirical quantile")
axes[1].set_title("Standardized-return Q-Q: jumps cause the tail")
axes[1].legend()
fig.tight_layout()
fig.show()
# %% [markdown]
# ## 5. Jumps cluster in time of day
#
# The opening minutes and the close concentrate price discovery: news
# released overnight is impounded at the auction, and end-of-day
# rebalancing flows produce sharp moves. The session-boundary fix means
# the first `LEE_MYKLAND_K - 1` bars of each day (≈55 min) are not
# testable, so any "first 30 min" detection here is limited; we report
# the share within the testable window and look for the close-of-day
# uptick that the test can see in full.
# %%
tod = flagged.filter(pl.col("is_jump")).with_columns(
minute_of_day=pl.col("timestamp").dt.hour().cast(pl.Int32) * 60
+ pl.col("timestamp").dt.minute()
)
tod_hist = tod.group_by("minute_of_day").len().sort("minute_of_day")
all_minutes = (
flagged.with_columns(
minute_of_day=pl.col("timestamp").dt.hour().cast(pl.Int32) * 60
+ pl.col("timestamp").dt.minute()
)
.group_by("minute_of_day")
.len()
.rename({"len": "all_bars"})
.sort("minute_of_day")
)
tod_rate = tod_hist.join(all_minutes, on="minute_of_day", how="right").with_columns(
rate_per_1k=pl.col("len").fill_null(0) / pl.col("all_bars") * 1000
)
share_first_30 = tod.filter(pl.col("minute_of_day") < 9 * 60 + 30 + 30).height / tod.height
share_last_30 = tod.filter(pl.col("minute_of_day") >= 16 * 60 - 30).height / tod.height
print(f"Share of jumps in first 30 min: {share_first_30:.1%}")
print(f"Share of jumps in last 30 min: {share_last_30:.1%}")
# %%
fig, ax = plt.subplots(figsize=(9, 3.5))
ax.bar(
tod_rate["minute_of_day"].to_numpy() / 60,
tod_rate["rate_per_1k"].to_numpy(),
width=BAR_MINUTES / 60.0,
edgecolor="none",
)
ax.set_xlabel("Hour of day (ET)")
ax.set_ylabel("Jumps per 1,000 bars")
ax.set_title("Lee–Mykland jump rate by time of day")
ax.axvspan(9.5, 10.0, alpha=0.1, color="grey")
ax.axvspan(15.5, 16.0, alpha=0.1, color="grey")
fig.tight_layout()
fig.show()
# %% [markdown]
# ## 6. Variance decomposition over the year
#
# Stacking the continuous (bipower) and jump variance components over
# time shows that jump contribution is episodic: most days have a small
# or zero jump variance, punctuated by clusters around earnings,
# macroeconomic releases, and stress periods.
# %%
fig, ax = plt.subplots(figsize=(11, 4))
focus = daily.filter(pl.col("symbol") == SYMBOLS[0]).sort("date").to_pandas()
ax.fill_between(focus["date"], 0, focus["bv"], alpha=0.6, label="Continuous (BV)")
ax.fill_between(
focus["date"],
focus["bv"],
focus["bv"] + focus["jump_var"],
alpha=0.7,
label="Jump variance (RV − BV)",
)
ax.set_xlabel("Date")
ax.set_ylabel("Daily realized variance (log returns)")
ax.set_title(f"{SYMBOLS[0]}: continuous and jump components of daily RV")
ax.legend()
fig.tight_layout()
fig.show()
# %% [markdown]
# ## 7. Why the local-volatility adjustment matters
#
# A naive rule that flags any bar with $|r_i / \hat\sigma_d| > 4$, where
# $\hat\sigma_d$ is a single daily volatility, ignores time-of-day
# volatility shape and slow regime changes. Compared with Lee–Mykland
# it over-detects on calm days (the daily $\hat\sigma_d$ is small, so
# even modestly elevated bars exceed the fixed $4\hat\sigma_d$
# threshold) and under-detects on volatile days (the daily $\hat\sigma_d$
# is inflated by the very moves we are trying to flag, so large bars
# fail to clear $4\hat\sigma_d$).
# %%
daily_sigma = returns.group_by(["symbol", "date"]).agg(daily_std=pl.col("log_return").std())
naive = (
returns.join(daily_sigma, on=["symbol", "date"])
.with_columns(naive_z=pl.col("log_return") / pl.col("daily_std"))
.with_columns(naive_jump=pl.col("naive_z").abs() > NAIVE_Z_THRESHOLD)
)
comparison = (
flagged.join(naive.select("symbol", "timestamp", "naive_jump"), on=["symbol", "timestamp"])
.group_by("symbol")
.agg(
lm_jumps=pl.col("is_jump").sum(),
naive_jumps=pl.col("naive_jump").sum(),
both=(pl.col("is_jump") & pl.col("naive_jump")).sum(),
lm_only=(pl.col("is_jump") & ~pl.col("naive_jump")).sum(),
naive_only=(~pl.col("is_jump") & pl.col("naive_jump")).sum(),
)
)
comparison
# %% [markdown]
# ## 8. Materialize the jump-feature panel
#
# The output is a per-symbol, per-day frame with the features that
# downstream chapters consume directly: total jump count, signed jump
# variance, and the jump share of total realized variance. These join
# cleanly to the bar-level feature panels of Chapters 7–8.
# %%
features = daily_jump_counts.select(
"symbol",
"date",
"jump_count",
"signed_jump_var",
pl.col("jump_var").alias("jump_variance"),
pl.col("jump_share_rv").alias("jump_share"),
pl.col("rv").alias("rv_total"),
pl.col("bv").alias("continuous_variance"),
).sort(["symbol", "date"])
features.write_parquet(OUTPUT_DIR / "daily_jump_features.parquet")
print(f"Wrote {features.height:,} rows to {OUTPUT_DIR / 'daily_jump_features.parquet'}")
features.head(10)
# %% [markdown]
# ## Takeaways
#
# - Realized variance over-states diffusive risk because it absorbs
# jumps; bipower variation gives a jump-robust continuous-variance
# estimate and the difference times jump contribution.
# - The Lee–Mykland test fires the multiple-testing-correct number of
# jumps per day in expectation and adapts to time-varying volatility,
# so it is comparable across regimes in a way that fixed-threshold
# rules are not.
# - The Lee–Mykland jump rate by time of day shows a clear close-of-day
# uptick within the testable window (the first `LEE_MYKLAND_K - 1` bars
# of each session are not directly testable). See §5 for the printed
# per-window shares. The notebook does not condition on scheduled news
# days; the daily variance decomposition in §6 shows episodic jump
# contribution but is not aligned to an external news calendar.
# - The daily features written above plug into Chapter 7 (jump-conditional
# labels and event-time sampling) and Chapter 8 (volatility-regime
# features and feature stratification).
03_market_microstructure/README.md
+112
View File
@@ -0,0 +1,112 @@
# Chapter 3: Market Microstructure
The chapter explains why market data cannot be treated as a neutral price series. It shows how spreads, depth, resiliency, order types, and intraday trading regimes shape both execution quality and the meaning of observed trades and quotes. For readers building trading systems, this is the section that turns "price data" into an economic object with frictions, incentives, and timing effects.
## Learning Objectives
* Explain how liquidity, order types, market design, and intraday trading regimes shape observed market data and execution quality.
* Distinguish among major market data products, including L1, L2, L3, TAQ, and enriched bar datasets, and choose data that matches a research or trading objective.
* Parse message-based exchange data and reconstruct a venue-local limit order book while enforcing core lifecycle and accounting invariants.
* Interpret key order-book measures and empirical microstructure patterns, while recognizing the limits of visible single-venue data.
* Build and compare time-, activity-, and information-driven bars, including when trade-direction classification and Lee-Ready alignment are required.
* Apply intraday data-quality and sessionization checks that prevent sequencing, timestamp, and calendar errors from contaminating downstream analysis.
## Sections
### 3.1 Microstructure: The DNA of Price Formation
This section explains why market data cannot be treated as a neutral price series. It shows how spreads, depth, resiliency, order types, and intraday trading regimes shape both execution quality and the meaning of observed trades and quotes. For readers building trading systems, this is the section that turns "price data" into an economic object with frictions, incentives, and timing effects.
### 3.2 The Anatomy of Modern Market Data Feeds
This section maps the market data hierarchy from top-of-book quotes to full order-level feeds and makes clear that data choice is a strategy choice. It helps readers understand what L1, L2, L3, TAQ, proprietary feeds, and enriched vendor products can and cannot reveal, and why timestamp semantics, latency, and fragmentation affect both research validity and live feasibility.
### 3.3 From Raw Messages to the Limit Order Book
This section turns raw exchange messages into a concrete engineering workflow: parse, normalize, replay, validate, and analyze a venue-local order book. It matters because it connects message traffic to state, shows how to maintain book invariants, and clarifies both the power and the limits of reconstructed visible liquidity. It also gives the chapter empirical weight by tying reconstruction to concrete stylized facts and practical research outputs.
- [`01_itch_parser`](01_itch_parser.ipynb) — This notebook demonstrates how to parse NASDAQ's TotalView-ITCH binary protocol. Understanding MBO (message-by-order) data is foundational for microstructure-based ML features. (_runtime: ~22 minutes; ~8 GB RAM_)
- [`02_itch_lob_reconstruction`](02_itch_lob_reconstruction.ipynb) — This notebook reconstructs the limit order book (LOB) from ITCH messages. Uses itch_messages data.
- [`03_itch_lob_analysis`](03_itch_lob_analysis.ipynb) — This notebook analyzes reconstructed order book data to demonstrate empirically-grounded patterns with predictive power. These patterns motivate feature engineering in Chapter 7 and execution cost modeling in Chapter 19. (_~13 GB RAM_)
- [`04_itch_order_lifecycle_analysis`](04_itch_order_lifecycle_analysis.ipynb) — This notebook analyzes the complete lifecycle of limit orders using NASDAQ ITCH data, revealing critical insights about cancellation rates, time dynamics, and the role of high-frequency market making in modern markets. Uses message_type, nasdaq_itch data. (_~33 GB RAM_)
- [`05_itch_trading_activity`](05_itch_trading_activity.ipynb) — This notebook provides a high-level view of NASDAQ trading activity using TotalView-ITCH data. We examine message composition and volume concentration across securities. (_~31 GB RAM_)
- [`06_itch_intraday_patterns`](06_itch_intraday_patterns.ipynb) — This notebook analyzes intraday volume patterns from NASDAQ ITCH data, demonstrating the well-documented U-shape. Volatility patterns are covered in 07_itch_stylized_facts.py (spread dynamics, bid-ask bounce).
- [`07_itch_stylized_facts`](07_itch_stylized_facts.ipynb) — This notebook demonstrates fundamental microstructure patterns: order arrival/cancellation dynamics, the bid-ask bounce phenomenon, and the liquidity spectrum across stocks. Uses add_cancel_for_ticker data.
- [`08_databento_lob_reconstruction`](08_databento_lob_reconstruction.ipynb) — Part 1 of 3: LOB Reconstruction → Order Flow Analysis → Bar Calibration Uses mbo_data data.
- [`09_databento_mbo_analysis`](09_databento_mbo_analysis.ipynb) — Part 2 of 3: LOB Reconstruction → Order Flow Analysis → Bar Calibration Uses databento_day, mbo_data data.
- [`10_iex_lob_reconstruction`](10_iex_lob_reconstruction.ipynb) — This notebook demonstrates LOB reconstruction using IEX's free historical market data, providing a reader-runnable alternative to the NASDAQ ITCH examples in earlier notebooks. Uses iex_hist data.
- [`11_algoseek_taq_eda`](11_algoseek_taq_eda.ipynb) — On March 16, 2020, the S&P 500 fell 12% - its worst single-day drop since 1987. Circuit breakers halted trading for 15 minutes when the index fell 7% at open. (_~22 GB RAM_)
- [`12_algoseek_taq_lob_reconstruction`](12_algoseek_taq_lob_reconstruction.ipynb) — Every trade has two sides: a buyer and a seller. But one side is aggressive - they crossed the spread to execute immediately. (_~22 GB RAM_)
- [`13_algoseek_minute_bars_eda`](13_algoseek_minute_bars_eda.ipynb) — This notebook provides a comprehensive exploration of the AlgoSeek TAQ minute bar dataset. Unlike simple OHLCV data, TAQ bars contain 61 pre-computed columns that capture the full microstructure of each trading minute: quote dynamics, trade execution, aggressor behavior, and liquidity conditions.
### 3.4 The Art of Sampling: From Ticks to Bars
This section argues that bar construction is not bookkeeping but measurement design. It compares time, tick, volume, dollar, and information-driven bars, explains trade classification and Lee-Ready, and gives readers a grounded basis for choosing simpler activity-time bars versus more demanding imbalance-based methods. The payoff is direct relevance for machine learning: the sampling rule changes the statistical properties of the training data.
- [`14_itch_bar_sampling`](14_itch_bar_sampling.ipynb) — Traditional OHLCV bars sample data at fixed time intervals. But markets don't generate information at constant rates.
- [`15_itch_lee_ready`](15_itch_lee_ready.ipynb) — This notebook validates our Lee-Ready implementation against DataBento's ground truth aggressor labels. Uses databento_mbo, mbo_data data.
- [`16_itch_information_bars`](16_itch_information_bars.ipynb) — This notebook has two parts: Uses mbo_data, trades data.
- [`17_databento_bar_sampling`](17_databento_bar_sampling.ipynb) — Part 3 of 3: LOB Reconstruction → Order Flow Analysis → Bar Calibration Uses mbo_data, multiday_trades data.
### 3.5 Detecting Price Jumps in Intraday Returns
This section separates the continuous and jump components of intraday returns using bipower variation and the Lee–Mykland (2008) test. The decomposition matters because realized variance, tail-heavy returns, and label distributions all change shape once jumps are removed, and the resulting jump features feed label conditioning in Chapter 7 and volatility-regime features in Chapter 8.
- [`18_algoseek_jump_detection`](18_algoseek_jump_detection.ipynb) — Decomposes daily realized variance into continuous (bipower) and jump components on AlgoSeek minute bars; applies the Lee–Mykland nonparametric test to time individual jumps with a multiple-testing-correct critical value; compares with a naive |z|>4 rule that misses most jumps because a single daily volatility ignores time-of-day shape; emits a per-symbol-day jump-feature panel.
### 3.6 Data Quality and Sessionization
This section focuses on intraday failure modes that quietly corrupt research: sequencing errors, timestamp confusion, stale quotes, invalid book transitions, bad qualifiers, and session-boundary mistakes. It is especially useful because it frames data quality as invariants and auditability rather than generic cleaning, giving readers a practical QA mindset for microstructure work.
## Running the Notebooks
```bash
# From the repository root
uv run python 03_market_microstructure/<notebook>.py
# Test mode (reduced data via Papermill)
uv run pytest tests/test_notebooks.py -v -k "03_market_microstructure"
```
### Memory requirements
The ITCH and AlgoSeek-TAQ notebooks materialize multi-day, multi-symbol message and tick partitions into memory. We recommend at least 32 GB of system RAM (64 GB ideal) when running the chapter end-to-end, since the five high-RAM notebooks below cumulatively exceed 100 GB if executed without releasing memory between runs. Peak resident-set sizes observed on a 64 GB workstation:
- `04_itch_order_lifecycle_analysis` — ~33 GB peak
- `05_itch_trading_activity` — ~31 GB peak
- `11_algoseek_taq_eda` — ~22 GB peak
- `12_algoseek_taq_lob_reconstruction` — ~22 GB peak
- `03_itch_lob_analysis` — ~13 GB peak
The remaining numbered notebooks fit comfortably in 8 GB. If your workstation has less than 32 GB, run the high-memory notebooks one at a time with no other heavy processes; restart the kernel between them so Polars releases its thread-local arenas.
### Runtime requirements
Most notebooks complete in under a minute. The only long-running notebook is the raw ITCH parse:
- `01_itch_parser` — ~22 min wall-clock, ~8 GB peak RSS (single-pass parse of one 423 M-message trading day)
## References
- **Matteo Aquilina et al.** (2021). [Quantifying the High-Frequency Trading “Arms Race”](https://doi.org/10.3386/w29011).
- **Rama Cont et al.** (2014). [The Price Impact of Order Book Events](https://doi.org/10.1093/jjfinec/nbt003). *Journal of Financial Econometrics*.
- **David Easley et al.** (2012). [The Volume Clock: Insights into the High Frequency Paradigm](https://doi.org/10.2139/ssrn.2034858).
- **David Easley et al.** (2021). [Microstructure in the Machine Age](https://doi.org/10.1093/rfs/hhaa078). *The Review of Financial Studies*.
- **Lawrence R. Glosten and Paul R. Milgrom** (1985). [Bid, ask and transaction prices in a specialist market with heterogeneously informed traders](https://doi.org/10.1016/0304-405X(85)90044-3). *Journal of Financial Economics*.
- **Martin D. Gould et al.** (2013). [Limit Order Books](http://arxiv.org/abs/1012.0349). *arXiv:1012.0349 [physics, q-fin]*.
- **Nikolaus Hautsch and Ruihong Huang** (2012). [The market impact of a limit order](https://doi.org/10.1016/j.jedc.2011.09.012). *Journal of Economic Dynamics and Control*.
- **Craig W. Holden et al.** (2014). [The Empirical Analysis of Liquidity](https://doi.org/10.1561/0500000044). *Foundations and Trends® in Finance*.
- **Craig W. Holden et al.** (2023). [In the Blink of an Eye: Exchange-to-SIP Latency and Trade Classification Accuracy](https://doi.org/10.2139/ssrn.4441422).
- **Albert S. Kyle** (1985). [Continuous Auctions and Insider Trading](https://doi.org/10.2307/1913210). *Econometrica*.
- **Charles M. C. Lee and Mark J. Ready** (1991). [Inferring Trade Direction from Intraday Data](https://doi.org/10.1111/j.1540-6261.1991.tb02683.x). *The Journal of Finance*.
- **Ananth Madhavan et al.** (1997). [Why Do Security Prices Change? A Transaction-Level Analysis of NYSE Stocks](https://www.jstor.org/stable/2962338). *The Review of Financial Studies*.
- **Ananth Madhavan** (2002). [Market Microstructure: A Practitioner's Guide](https://www.jstor.org/stable/4480415). *Financial Analysts Journal*.
- **Andy Novocin and Bruce Weber** (2022). [Emerging Technologies and the Transformation of Exchange Trading Platforms](https://doi.org/10.3905/jpm.2022.1.390). *The Journal of Portfolio Management*.
- **Maureen O'Hara** (2011). Market Microstructure Theory. *Wiley*.
- **Maureen O’Hara** (2015). [High frequency market microstructure](https://doi.org/10.1016/j.jfineco.2015.01.003). *Journal of Financial Economics*.
- **Marcos Lopez de Prado** (2018). Advances in Financial Machine Learning. *John Wiley & Sons*.
- **Martin Reck** (2022). [Market Design—A Practitioner’s Perspective](https://doi.org/10.3905/jpm.2022.1.383). *The Journal of Portfolio Management*.
- **SEC** (2020). [Staff Report on Algorithmic Trading in U.S. Capital Markets](https://www.sec.gov/file/staff-report-algorithmic-trading-us-capital-markets).
- **Robert A. Schwartz et al.** (2022). [Equity Market Structure and the Persistence of Unsolved Problems: A Microstructure Perspective](https://doi.org/10.3905/jpm.2022.1.384). *The Journal of Portfolio Management*.
- **Zihao Zhang et al.** (2019). [DeepLOB: Deep Convolutional Neural Networks for Limit Order Books](https://doi.org/10.1109/TSP.2019.2907260). *IEEE Transactions on Signal Processing*.
- **Yichi Zhang et al.** (2025). [ClusterLOB: Enhancing Trading Strategies by Clustering Orders in Limit Order Books](https://doi.org/10.48550/arXiv.2504.20349).
03_market_microstructure/itch_message_specs.py
+429
View File
@@ -0,0 +1,429 @@
"""NASDAQ TotalView-ITCH 5.0 Message Specifications.
This module defines the binary message formats for the ITCH protocol.
Used by both Python (educational) and Rust (production) parsers.
Reference: https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf
"""
import struct
from collections import namedtuple
from datetime import datetime
from pathlib import Path
import polars as pl
# ITCH 5.0 Message Specifications
# Each message has a fixed binary structure defined by field name and struct format code.
# Format codes: H=uint16, I=uint32, Q=uint64, s=char, Ns=N chars
MESSAGE_SPECS = {
"S": { # System Event
"name": "System Event",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("event_code", "s"),
],
},
"R": { # Stock Directory
"name": "Stock Directory",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("stock", "8s"),
("market_category", "s"),
("financial_status", "s"),
("round_lot_size", "I"),
("round_lots_only", "s"),
("issue_classification", "s"),
("issue_subtype", "2s"),
("authenticity", "s"),
("short_sale_threshold", "s"),
("ipo_flag", "s"),
("luld_reference_price_tier", "s"),
("etp_flag", "s"),
("etp_leverage_factor", "I"),
("inverse_indicator", "s"),
],
},
"A": { # Add Order (No MPID)
"name": "Add Order",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
("buy_sell_indicator", "s"),
("shares", "I"),
("stock", "8s"),
("price", "I"),
],
},
"F": { # Add Order (With MPID)
"name": "Add Order MPID",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
("buy_sell_indicator", "s"),
("shares", "I"),
("stock", "8s"),
("price", "I"),
("attribution", "4s"),
],
},
"E": { # Order Executed
"name": "Order Executed",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
("executed_shares", "I"),
("match_number", "Q"),
],
},
"C": { # Order Executed with Price
"name": "Order Executed with Price",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
("executed_shares", "I"),
("match_number", "Q"),
("printable", "s"),
("execution_price", "I"),
],
},
"X": { # Order Cancel
"name": "Order Cancel",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
("cancelled_shares", "I"),
],
},
"D": { # Order Delete
"name": "Order Delete",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
],
},
"U": { # Order Replace
"name": "Order Replace",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("original_order_reference_number", "Q"),
("new_order_reference_number", "Q"),
("shares", "I"),
("price", "I"),
],
},
"P": { # Trade (Non-Cross)
"name": "Trade",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("order_reference_number", "Q"),
("buy_sell_indicator", "s"),
("shares", "I"),
("stock", "8s"),
("price", "I"),
("match_number", "Q"),
],
},
"Q": { # Cross Trade
"name": "Cross Trade",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("shares", "Q"),
("stock", "8s"),
("cross_price", "I"),
("match_number", "Q"),
("cross_type", "s"),
],
},
"H": { # Stock Trading Action
"name": "Trading Action",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("stock", "8s"),
("trading_state", "s"),
("reserved", "s"),
("reason", "4s"),
],
},
"Y": { # Reg SHO Short Sale Price Test Restricted Indicator
"name": "Reg SHO Restriction",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("stock", "8s"),
("reg_sho_action", "s"),
],
},
"L": { # Market Participant Position
"name": "Market Participant Position",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("mpid", "4s"),
("stock", "8s"),
("primary_market_maker", "s"),
("market_maker_mode", "s"),
("market_participant_state", "s"),
],
},
"V": { # MWCB Decline Level
"name": "MWCB Decline Level",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("level_1", "Q"),
("level_2", "Q"),
("level_3", "Q"),
],
},
"W": { # MWCB Status
"name": "MWCB Status",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("breached_level", "s"),
],
},
"J": { # LULD Auction Collar
"name": "LULD Auction Collar",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("stock", "8s"),
("auction_collar_reference_price", "I"),
("upper_auction_collar_price", "I"),
("lower_auction_collar_price", "I"),
("auction_collar_extension", "I"),
],
},
"K": { # IPO Quoting Period Update
"name": "IPO Quoting Period",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("stock", "8s"),
("ipo_quotation_release_time", "I"),
("ipo_quotation_release_qualifier", "s"),
("ipo_price", "I"),
],
},
"B": { # Broken Trade
"name": "Broken Trade",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("match_number", "Q"),
],
},
"I": { # NOII (Net Order Imbalance Indicator)
"name": "NOII",
"fields": [
("stock_locate", "H"),
("tracking_number", "H"),
("timestamp", "6s"),
("paired_shares", "Q"),
("imbalance_shares", "Q"),
("imbalance_direction", "s"),
("stock", "8s"),
("far_price", "I"),
("near_price", "I"),
("current_reference_price", "I"),
("cross_type", "s"),
("price_variation_indicator", "s"),
],
},
}
# System event codes (for S messages)
EVENT_CODES = {
"O": "Start of Messages",
"S": "Start of System Hours",
"Q": "Start of Market Hours",
"M": "End of Market Hours",
"E": "End of System Hours",
"C": "End of Messages",
}
# Mapping from struct format codes to Polars data types
# Used for explicit casting in Parquet output
_STRUCT_TO_POLARS = {
"H": pl.UInt16, # unsigned short (2 bytes)
"I": pl.UInt32, # unsigned int (4 bytes)
"Q": pl.UInt64, # unsigned long long (8 bytes)
}
def build_parsers(specs: dict | None = None) -> tuple[dict, dict]:
"""Build namedtuples and struct format strings for each message type.
Args:
specs: Message specifications dict. Defaults to MESSAGE_SPECS.
Returns:
Tuple of (namedtuples_dict, format_strings_dict)
"""
if specs is None:
specs = MESSAGE_SPECS
namedtuples = {}
formats = {}
for msg_type, spec in specs.items():
fields = [f[0] for f in spec["fields"]]
fmt_codes = [f[1] for f in spec["fields"]]
namedtuples[msg_type] = namedtuple(msg_type, fields)
formats[msg_type] = ">" + "".join(fmt_codes) # Big-endian
return namedtuples, formats
def build_parquet_schemas(specs: dict | None = None) -> dict[str, dict[str, pl.DataType]]:
"""Generate Parquet schema definitions from message specifications.
Only includes fields that need explicit casting (numeric types).
String fields and timestamp are handled specially during parsing.
Args:
specs: Message specifications dict. Defaults to MESSAGE_SPECS.
Returns:
Dict mapping message type to column->dtype mappings.
"""
if specs is None:
specs = MESSAGE_SPECS
schemas = {}
for msg_type, spec in specs.items():
schema = {}
for field_name, fmt_code in spec["fields"]:
# Skip timestamp (handled specially) and string fields
if fmt_code == "6s" or fmt_code.endswith("s"):
continue
# Map struct format to Polars type
if fmt_code in _STRUCT_TO_POLARS:
schema[field_name] = _STRUCT_TO_POLARS[fmt_code]
schemas[msg_type] = schema
return schemas
# Pre-built parsers and schemas for convenience
NT_DICT, FMT_DICT = build_parsers()
PARQUET_SCHEMAS = build_parquet_schemas()
def parse_timestamp(ts_bytes: bytes) -> int:
"""Convert 6-byte ITCH timestamp to nanoseconds since midnight."""
return int.from_bytes(ts_bytes, byteorder="big")
def parse_price4(price_int: int) -> float:
"""Convert ITCH price4 format (4 implied decimals) to float."""
return price_int / 10000
def get_message_size(msg_type: str) -> int:
"""Get the byte size of a message type (excluding length prefix)."""
if msg_type not in FMT_DICT:
raise ValueError(f"Unknown message type: {msg_type}")
return struct.calcsize(FMT_DICT[msg_type])
def print_message_formats() -> None:
"""Print all message formats with their sizes."""
print("ITCH Message Formats:\n")
for msg_type, spec in MESSAGE_SPECS.items():
fmt = FMT_DICT[msg_type]
size = struct.calcsize(fmt)
print(f" {msg_type} ({spec['name']:25}): {size:2} bytes - {fmt}")
def flush_to_parquet(
buffers: dict[str, list[dict]],
output_dir: Path,
base_ts: datetime,
file_counters: dict[str, int],
schemas: dict[str, dict[str, pl.DataType]] | None = None,
) -> None:
"""Write message buffers to Parquet files with unique names.
Args:
buffers: Dict mapping message type to list of parsed message dicts.
output_dir: Directory for Parquet output (one subdir per message type).
base_ts: Base timestamp (midnight) for datetime conversion.
file_counters: Mutable dict tracking file indices per message type.
schemas: Optional schema overrides. Defaults to PARQUET_SCHEMAS.
"""
if schemas is None:
schemas = PARQUET_SCHEMAS
for msg_type, records in buffers.items():
if not records:
continue
df = pl.DataFrame(records)
# Cast to correct types to match Rust parser schema
if msg_type in schemas:
cast_exprs = [
pl.col(col).cast(dtype)
for col, dtype in schemas[msg_type].items()
if col in df.columns
]
if cast_exprs:
df = df.with_columns(cast_exprs)
# Convert timestamp from nanoseconds-since-midnight to datetime
# Uses Polars native datetime with nanosecond precision
if "timestamp" in df.columns:
df = df.with_columns(
(
pl.lit(base_ts).cast(pl.Datetime("ns"))
+ pl.duration(nanoseconds=pl.col("timestamp"))
).alias("timestamp")
)
out_dir = output_dir / msg_type
out_dir.mkdir(parents=True, exist_ok=True)
# Write with unique filename per flush to avoid overwriting
file_idx = file_counters[msg_type]
df.write_parquet(out_dir / f"part-{file_idx:06d}.parquet")
file_counters[msg_type] += 1
buffers.clear()
03_market_microstructure/limit_orderbook.py
+1271
View File
File diff suppressed because it is too large Load Diff
04_fundamental_alternative_data/01_academic_characteristics.ipynb
+5770
View File
File diff suppressed because one or more lines are too long
04_fundamental_alternative_data/01_academic_characteristics.py
+518
View File
@@ -0,0 +1,518 @@
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Chen-Pelger-Zhu Academic Asset Pricing Dataset
#
# **Chapter 4: Fundamental and Alternative Data**
# **Docker image**: `ml4t`
# **Section Reference**: Section 4.1 (The Point-in-Time Pipeline)
#
# ## Purpose
#
# This notebook introduces the Chen-Pelger-Zhu (2020) academic dataset, which provides
# a standardized benchmark for comparing ML models in asset pricing. With ~1.2M stock-month
# observations and 46 firm characteristics, this anonymized dataset enables reproducible
# research without requiring WRDS access.
#
# ## Learning Objectives
#
# After completing this notebook, you will be able to:
# - Load and explore the Chen-Pelger-Zhu academic dataset
# - Understand the 46 firm characteristics and their categories
# - Recognize the cross-sectional rank normalization applied to features
# - Understand the train/valid/test split methodology
# - Know the limitations (no asset IDs = no backtesting possible)
#
# ## Cross-References
#
# - **Downstream**: Ch11 `case_studies/us_firm_characteristics/05_linear.py`, Ch12 `06_gbm.py`
# - **Related**: Ch14 `case_studies/us_firm_characteristics/08_latent_factors.py`
# - **Synthesis**: `case_studies/us_firm_characteristics/10_model_analysis.py`
#
# ## Prerequisites
#
# Download the dataset before running this notebook:
# ```bash
# python data/equities/firm_characteristics/download.py
# ```
#
# ## Data Source
#
# | Attribute | Value |
# |-----------|-------|
# | **Paper** | "Deep Learning in Asset Pricing" (Chen, Pelger, Zhu, 2021) |
# | **Repository** | https://github.com/jasonzy121/Deep_Learning_Asset_Pricing |
# | **Observations** | ~1.2M stock-months |
# | **Features** | 46 firm characteristics (rank-normalized) |
# | **Returns** | Next-month excess returns (raw, not normalized) |
# | **Period** | 1967-2016 (50 years) |
# | **Identifiers** | **None** (fully anonymized) |
#
# ## Data Construction (from paper Section III.A)
#
# ### Stock Universe
# - Source: All securities on CRSP (~31,000 stocks)
# - Only stocks with all 46 characteristics available are included
# - This removes predominantly small-cap stocks with missing data
# - The released dataset contains ~2,000 stocks per month on average
#
# ### 46 Firm Characteristics
# Characteristics are sourced from:
# 1. Kenneth French Data Library
# 2. Freyberger, Neuhierl, and Weber (2020)
#
# **Categories** (cf. Table A.II in paper; this notebook groups them as):
# - **Valuation**: BEME, E2P, S2P, CF2P, D2P, A2ME, Q
# - **Profitability**: ROA, ROE, PROF, OP, PM, PCM, NI, RNA
# - **Investment**: Investment, NOA, OA, AC, AT, D2A
# - **Momentum / past returns**: r2_1, r12_2, r12_7, r36_13, ST_REV, LT_Rev, Rel2High
# - **Risk**: Beta, MktBeta, IdioVol, Variance, Resid_Var
# - **Liquidity / Size**: LME, LTurnover, Spread, SUV
# - **Leverage**: Lev, OL, FC2Y, C, CF, DPI2A
# - **Other**: ATO, CTO, SGA2S
#
# **Construction**:
# - Yearly variables: Updated end of June (Fama-French convention)
# - Monthly variables: Updated end of month for use in next month
# - All from CRSP/Compustat accounting data or CRSP past returns
#
# ### 178 Macroeconomic Time Series
# 1. 124 from FRED-MD database (McCracken and Ng, 2016)
# 2. 46 cross-sectional medians of firm characteristics
# 3. 8 equity premium predictors from Welch and Goyal (2007)
#
# ### Cross-Sectional Rank Normalization
# Following Kelly, Pruitt, Su (2019), Kozak, Nagel, Santosh (2020):
# - Each characteristic ranked cross-sectionally each month
# - Converted to quantiles in [-0.5, +0.5] range
# - Handles different scales and reduces outlier impact
#
# ## Key Concepts
#
# - **Cross-sectional rank normalization**: Features scaled to [-0.5, +0.5] each month
# - **Next-month return prediction**: `ret` is the raw excess return to be predicted (not normalized)
# - **No asset identifiers**: Individual stocks cannot be tracked over time
#
# ---
# %%
"""Chen-Pelger-Zhu Academic Asset Pricing Dataset — explore anonymized firm characteristics for ML benchmarking."""
import warnings
warnings.filterwarnings("ignore")
from datetime import datetime
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
import statsmodels.api as sm
from data import load_firm_characteristics
from utils.reproducibility import set_global_seeds
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
SEED = 42
# %%
set_global_seeds(SEED)
# %% [markdown]
# ---
#
# ## Section 1: Data Overview
#
# The Chen-Pelger-Zhu dataset provides a clean, standardized benchmark for comparing
# ML approaches to return prediction. All features are cross-sectionally rank-normalized
# each month, eliminating scale differences and reducing outlier impact.
# %%
# Load full dataset with split labels
df = load_firm_characteristics(split="all")
print(f"Total observations: {len(df):,}")
print(f"Columns: {len(df.columns)}")
df.columns
# %%
# Check date range and split distribution
date_stats = df.group_by("split").agg(
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.len().alias("n_obs"),
)
date_stats.sort("start")
# %%
# Observations per month
monthly_counts = (
df.group_by("timestamp")
.len()
.sort("timestamp")
.with_columns(
pl.col("timestamp").dt.year().alias("year"),
)
)
print("\nObservations per month:")
print(f" Min: {monthly_counts['len'].min()}")
print(f" Max: {monthly_counts['len'].max()}")
print(f" Mean: {monthly_counts['len'].mean():.0f}")
# Plot observations over time
fig = px.line(
monthly_counts.to_pandas(),
x="timestamp",
y="len",
title="Coverage Grew from ~430 Stocks in 1967 to ~2,800 in 2016",
labels={"len": "Stock Count", "timestamp": "Date"},
)
# Add vertical lines for split boundaries (without annotation_text to avoid datetime bug)
fig.add_vline(x=datetime(1990, 1, 1), line_dash="dash", line_color="red")
fig.add_vline(x=datetime(2000, 1, 1), line_dash="dash", line_color="green")
# Add annotations separately
fig.add_annotation(
x=datetime(1990, 1, 1), y=2800, text="Valid", showarrow=False, font=dict(color="red")
)
fig.add_annotation(
x=datetime(2000, 1, 1), y=2800, text="Test", showarrow=False, font=dict(color="green")
)
fig.show()
# %% [markdown]
# ---
#
# ## Section 2: Characteristic Taxonomy
#
# The 46 firm characteristics span multiple categories that capture different dimensions
# of stock behavior: valuation, profitability, investment, momentum, risk, and market metrics.
#
# All characteristics are cross-sectionally rank-normalized each month to values in [-0.5, +0.5].
# %%
# Define characteristic categories
CHARACTERISTIC_CATEGORIES = {
"Valuation": ["BEME", "E2P", "S2P", "CF2P", "D2P", "A2ME", "Q"],
"Profitability": ["ROA", "ROE", "PROF", "OP", "PM", "PCM", "NI", "RNA"],
"Investment": ["Investment", "NOA", "OA", "AC", "AT", "D2A"],
"Momentum": ["r2_1", "r12_2", "r12_7", "r36_13", "ST_REV", "LT_Rev", "Rel2High"],
"Risk": ["Beta", "MktBeta", "IdioVol", "Variance", "Resid_Var"],
"Liquidity/Size": ["LME", "LTurnover", "Spread", "SUV"],
"Leverage": ["Lev", "OL", "FC2Y", "C", "CF", "DPI2A"],
"Other": ["ATO", "CTO", "SGA2S"],
}
# Print category summary
print("Characteristic Categories:")
print("=" * 60)
total = 0
for category, features in CHARACTERISTIC_CATEGORIES.items():
print(f"\n{category} ({len(features)} features):")
print(f" {', '.join(features)}")
total += len(features)
print(f"\nTotal: {total} characteristics")
# %%
# Distribution of each characteristic (confirming rank normalization)
feature_cols = [c for c in df.columns if c not in ["timestamp", "ret", "split"]]
# Compute statistics efficiently using Polars - no loops needed
sample_features = feature_cols[:10]
stats_df = df.select(sample_features).describe()
stats_df.filter(pl.col("statistic").is_in(["min", "max", "mean", "std"]))
# %% [markdown]
# ---
#
# ## Section 3: Correlation Structure
#
# Understanding correlations between characteristics helps identify redundant features
# and understand the factor structure in the data.
# %%
# Compute correlation matrix
corr_data = df.select(feature_cols).to_numpy()
corr_matrix = np.corrcoef(corr_data, rowvar=False)
# Find highly correlated pairs
high_corr_pairs = []
for i in range(len(feature_cols)):
for j in range(i + 1, len(feature_cols)):
if abs(corr_matrix[i, j]) > 0.5:
high_corr_pairs.append(
{
"feature_1": feature_cols[i],
"feature_2": feature_cols[j],
"correlation": corr_matrix[i, j],
}
)
high_corr_df = pl.DataFrame(high_corr_pairs).sort("correlation", descending=True)
high_corr_df
# %%
# Correlation heatmap
fig = go.Figure(
data=go.Heatmap(
z=corr_matrix,
x=feature_cols,
y=feature_cols,
colorscale="RdBu",
zmid=0,
zmin=-1,
zmax=1,
)
)
fig.update_layout(
title="Valuation and Volatility Form the Densest Correlation Clusters",
width=1000,
height=900,
xaxis_tickangle=-45,
)
fig.show()
# %% [markdown]
# ---
#
# ## Section 4: Predictive Relationships
#
# The key question: how do characteristics relate to next-month returns?
# These Information Coefficients (ICs) measure the predictive power of each feature.
# %% [markdown]
# We compute monthly cross-sectional ICs (one rank correlation per month) and
# average across months — the Fama-MacBeth template. The naive t-statistic on
# this series assumes monthly ICs are i.i.d., but adjacent months share
# slow-moving common factors that induce autocorrelation. The reported
# significance test uses Newey-West (HAC) standard errors on the time series
# of monthly ICs; for comparison we also keep the i.i.d. t-statistic.
# %%
# Step 1: Compute IC (correlation with returns) for each characteristic, per month
monthly_ics = (
df.group_by("timestamp")
.agg([pl.corr(col, "ret").alias(col) for col in feature_cols])
.drop("timestamp")
)
# Step 2: Compute mean IC, i.i.d. t-stat, and Newey-West HAC t-stat across months
n_months = len(monthly_ics)
# Newey-West lag selection: floor(4 * (T/100)^(2/9)); ~6 for monthly series of
# this length. Use 12 to be conservative against the annual cycle in factor returns.
nw_maxlags = 12
ic_stats = []
for col in feature_cols:
ic_values = monthly_ics[col].to_numpy()
ic_values = ic_values[~np.isnan(ic_values)] # constant-feature months
mean_ic = float(np.mean(ic_values))
std_ic = float(np.std(ic_values, ddof=1))
t_stat_iid = mean_ic / (std_ic / np.sqrt(len(ic_values)))
# HAC t-stat: regress IC_t on a constant with Newey-West covariance.
nw = sm.OLS(ic_values, np.ones(len(ic_values))).fit(
cov_type="HAC", cov_kwds={"maxlags": nw_maxlags}
)
t_stat_nw = float(nw.tvalues[0])
ic_stats.append(
{
"feature": col,
"IC": mean_ic,
"IC_std": std_ic,
"t_stat_iid": t_stat_iid,
"t_stat_NW": t_stat_nw,
"abs_IC": abs(mean_ic),
}
)
ic_df = pl.DataFrame(ic_stats).sort("IC", descending=True)
print(
f"Cross-sectional IC across {n_months} monthly cross-sections "
f"(|t_NW| > 2 ≈ significant; Newey-West with {nw_maxlags} lags):"
)
ic_df
# %%
# Visualize ICs by category
ic_with_category = []
for row in ic_df.iter_rows(named=True):
category = "Other"
for cat, features in CHARACTERISTIC_CATEGORIES.items():
if row["feature"] in features:
category = cat
break
ic_with_category.append({**row, "category": category})
ic_cat_df = pl.DataFrame(ic_with_category)
fig = px.bar(
ic_cat_df.to_pandas(),
x="feature",
y="IC",
color="category",
title="Short-Term Reversal and 12-Month Momentum Carry the Largest Single-Characteristic ICs",
labels={"IC": "Information Coefficient", "feature": "Characteristic"},
)
fig.update_layout(xaxis_tickangle=-45, height=500)
fig.add_hline(y=0, line_dash="dash", line_color="gray")
fig.show()
# %% [markdown]
# ---
#
# ## Section 5: Return Distribution
#
# The target variable `ret` represents next-month excess returns. Unlike the
# characteristics, returns are **not** rank-normalized — they remain in their
# original scale, as required for economic interpretation of predictions.
# %%
# Return statistics by split
return_stats = df.group_by("split").agg(
pl.col("ret").mean().alias("mean"),
pl.col("ret").std().alias("std"),
pl.col("ret").min().alias("min"),
pl.col("ret").max().alias("max"),
pl.col("ret").quantile(0.25).alias("q25"),
pl.col("ret").quantile(0.75).alias("q75"),
)
return_stats
# %%
# Return distribution
sample_size = min(100_000, len(df))
fig = px.histogram(
df.sample(n=sample_size, seed=SEED).to_pandas(), # Convert for Plotly compatibility
x="ret",
color="split",
nbins=100,
title="Return Distributions Widen After 1990 as the Cross-Section Expands",
labels={"ret": "Next-Month Return", "count": "Frequency"},
opacity=0.7,
barmode="overlay",
)
fig.show()
# %% [markdown]
# ---
#
# ## Section 6: Macro Indicators (Optional Companion Data)
#
# The Chen-Pelger-Zhu paper pairs the firm characteristics with 178 macroeconomic
# time series — 124 from FRED-MD (McCracken and Ng, 2016), 46 cross-sectional
# medians of firm characteristics, and 8 equity-premium predictors from Welch and
# Goyal (2007). Conditional models such as SDF-GAN (Ch14) consume them as state
# variables.
#
# The shipped academic parquet contains characteristics + returns only; macro
# columns ship separately. Calling `include_macro=True` is a no-op against the
# shipped file. To assemble the macro panel for conditional models, follow the
# instructions in the original paper repository
# (https://github.com/jasonzy121/Deep_Learning_Asset_Pricing) and merge on
# `timestamp`.
# %%
df_with_macro = load_firm_characteristics(split="all", include_macro=True)
macro_cols = [c for c in df_with_macro.columns if c.startswith("macro_")]
print(f"macro_* columns in shipped dataset: {len(macro_cols)}")
# %% [markdown]
# ---
#
# ## Section 7: Limitations and Use Cases
#
# ### What This Dataset IS Good For
#
# 1. **Benchmarking ML models**: Compare linear, tree, and neural network approaches
# 2. **Hyperparameter optimization studies**: Large sample size enables thorough HPO
# 3. **Reproducible research**: No WRDS access required
# 4. **Educational purposes**: Clean, standardized dataset for learning
#
# ### What This Dataset IS NOT Good For
#
# 1. **Backtesting strategies**: No asset identifiers = cannot track stocks over time
# 2. **Portfolio construction**: Cannot form portfolios without knowing which returns belong together
# 3. **Transaction cost analysis**: No price levels, only normalized returns
# 4. **Fundamental analysis**: Characteristics are anonymized and normalized
#
# ### Key Insight
#
# This dataset answers: **"Can ML predict cross-sectional return variation from characteristics?"**
#
# It does NOT answer: **"Can this prediction be monetized in practice?"**
#
# For practical trading strategies, see the ETF, Crypto, and Futures case studies
# throughout this book.
# %% [markdown]
# ## Key Takeaways
#
# 1. The Chen-Pelger-Zhu dataset provides a clean benchmark for comparing ML models in asset pricing.
# 2. Cross-sectional rank normalization places every characteristic in $[-0.5, +0.5]$, which removes scale differences and tames outliers.
# 3. Single-characteristic ICs are small ($|\bar{\rho}| \le 0.04$) but several remain significant after Newey-West correction for autocorrelation in the monthly IC series ($|t_{\text{NW}}| > 5$ for the leading signals — SUV, ST_REV, NI, BEME, r12_2).
# 4. Strong within-category correlations (e.g., $|\rho| > 0.8$ between BEME / Q / A2ME) motivate regularised linear models and tree ensembles that handle collinearity natively.
# 5. **Limitation**: no asset identifiers — this dataset supports prediction-accuracy studies, not backtesting.
#
# **Next**: See `case_studies/us_firm_characteristics/` for ML models applied to this data.
# %%
# Compute summary statistics for quantitative insights
top_ic = ic_df.head(5)
bottom_ic = ic_df.sort("IC").head(5)
high_corr_count = len(high_corr_df)
print("=" * 70)
print("KEY INSIGHTS")
print("=" * 70)
print(f"""
Chen-Pelger-Zhu (2020) Academic Asset Pricing Dataset
## Dataset Structure
- Observations: {len(df):,} stock-months
- Features: 46 firm characteristics (rank-normalized to [-0.5, +0.5])
- Returns: Next-month excess returns (raw)
- Period: 1967-2016 (50 years)
- Splits: Train (1967-1989), Valid (1990-1999), Test (2000-2016)
## Quantitative Findings
### Predictive Power (Information Coefficients)
Top 3 positive ICs: {", ".join([f"{r['feature']} (IC={r['IC']:.3f}, t_NW={r['t_stat_NW']:.1f})" for r in top_ic.head(3).iter_rows(named=True)])}
Top 3 negative ICs: {", ".join([f"{r['feature']} (IC={r['IC']:.3f}, t_NW={r['t_stat_NW']:.1f})" for r in bottom_ic.head(3).iter_rows(named=True)])}
### Correlation Structure
- Highly correlated pairs (|r| > 0.5): {high_corr_count}
- Valuation metrics form tight clusters (high collinearity)
- Momentum features relatively orthogonal to value/profitability
### Implications for ML Models
1. Lasso/Ridge may be needed to handle correlated features
2. Tree models naturally handle correlated features via splits
3. Neural networks benefit from the [-0.5, +0.5] normalization
4. The lack of asset IDs limits this to prediction accuracy studies only
## Usage
from data import load_firm_characteristics
train = load_firm_characteristics(split="train")
X = train.drop(["timestamp", "ret"]).to_numpy()
y = train["ret"].to_numpy()
## Reference
Chen, Pelger, and Zhu (2020). "Deep Learning in Asset Pricing"
https://github.com/jasonzy121/Deep_Learning_Asset_Pricing
""")
04_fundamental_alternative_data/02_sec_filing_explorer.ipynb
+2285
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More