wehub/stefan-jansen--machine-learning-for-trading
1
0
Fork 0
Code Issues Pull Requests Actions 3 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:26:28 +08:00
commit 0249a267e1
495 changed files with 1277094 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/README.md
+340
View File
@@ -0,0 +1,340 @@
# ML4T Data Infrastructure
Central data management for *Machine Learning for Trading, 3rd Edition*.
Each dataset has its own directory with a download script, loader, config, and exploration notebook. All loaders return Polars DataFrames with a consistent API.
---
## Quick Start
```bash
# 1. Set data path in repository root .env file
ML4T_DATA_PATH=/path/to/your/data
# 2. Download free datasets (no API keys needed)
uv run python data/download_all.py --free-only
# 3. Use in notebooks
from data import load_etfs
df = load_etfs()
```
---
## Dataset Catalog
Organized by asset class and data type. "Type" column maps each dataset
to its place in the Ch2/Ch4 taxonomy: market (OHLCV / microstructure /
options), fundamentals (accounting + regulatory filings), positioning
(positions / insider activity), onchain (crypto-native fundamentals), or
cross-asset (factors, macro, prediction markets, news, text).
| Dataset | Asset Class | Type | Frequency | Symbols | Coverage | Source | Access |
| ------------------------ | ------------ | ------------- | ------------ | ------- | --------- | ---------------- | ------ |
| ETF Universe | Equity | Market | Daily | 100 | 2006-2025 | Yahoo Finance | No |
| US Equities | Equity | Market | Daily | 3,199 | 1962-2018 | NASDAQ DL | Free |
| S&P 500 Bars | Equity | Market | Daily | ~638 | 2017-2021 | AlgoSeek | Soon |
| S&P 500 Options | Equity | Market | Daily | ~500 | 2017-2021 | AlgoSeek | Soon |
| NASDAQ-100 Bars | Equity | Market | Minute | ~100 | 2020-2021 | AlgoSeek | Soon |
| TAQ Tick | Equity | Market | Tick | 1 | Mar 2020 | AlgoSeek | Soon |
| MBO Tick | Equity | Market | Tick | 1 | Nov 2024 | Databento | Manual |
| NASDAQ ITCH | Equity | Market | Tick | all | varies | NASDAQ FTP | No |
| IEX DEEP/TOPS | Equity | Market | Tick | all | varies | IEX public | No |
| SEC XBRL Fundamentals | Equity | Fundamentals | Quarterly | 20 | 2022-2024 | SEC EDGAR | No |
| SEC 10-K (SP100) | Equity | Fundamentals | Annual | ~100 | 2020-2025 | SEC EDGAR | No |
| SEC 10-Q MD&A (SP500) | Equity | Fundamentals | Quarterly | ~600 | 2017-2021 | SEC EDGAR | No |
| SEC 8-K (SP100) | Equity | Fundamentals | Event | ~100 | 2024-2025 | SEC EDGAR | No |
| 13F Institutional | Equity | Positioning | Quarterly | 10 inst | rolling | SEC EDGAR | No |
| Form 4 Insider | Equity | Positioning | Event | varies | varies | SEC EDGAR | No |
| Firm Characteristics | Equity | Packaged | Monthly | anon | 1967-2016 | GitHub | No |
| CME Futures | Futures | Market | Daily/Hourly | 30 | 2011-2025 | Databento | Paid |
| CFTC Commitment of Traders | Futures | Positioning | Weekly | 25+ | 2020-2025 | CFTC public | No |
| Crypto Perps | Crypto | Market | 1h | 19 | 2020-2025 | Binance Public | No |
| Crypto Premium | Crypto | Market | 8h | 19 | 2020-2025 | Binance Public | No |
| DefiLlama TVL | Crypto | Onchain | Daily | chains | varies | DefiLlama | No |
| CoinGecko OHLCV | Crypto | Onchain | Daily | varies | 365 days | CoinGecko | No |
| FX Pairs | Currency | Market | 4h/Daily | 20 | 2011-2025 | OANDA | Free |
| FF Factors | Cross-asset | Factors | Monthly | 5 | 1926-now | Ken French | No |
| AQR Factors | Cross-asset | Factors | Monthly | 8 | varies | AQR | No |
| FRED Macro | Cross-asset | Macro | Various | 40 | 2000-2025 | FRED | Free |
| Kalshi events | Cross-asset | Prediction | Daily | varies | 2021-2025 | Kalshi public | No |
| Polymarket events | Cross-asset | Prediction | Daily | varies | 2020-2025 | Polymarket public| No |
| FNSPID news | Cross-asset | News | Daily | 4,775 | 1999-2023 | HuggingFace | No |
| Bloomberg news archive | Cross-asset | News | Daily | mixed | 2006-2013 | HuggingFace | No |
| Financial Phrasebank | Cross-asset | Text | Static | — | n/a | HuggingFace | No |
**Access legend.** `No` — included with the repo or fetched by an
unauthenticated script. `Free` — script-download, free API key required.
`Paid` — script-download, billed API (see per-dataset estimates).
`Manual` — reader downloads from a hosted URL or provider portal and
places the files under `$ML4T_DATA_PATH` (no script); the DataBento MBO
one-off has step-by-step instructions below. `Soon` — the reduced
reader-facing AlgoSeek datasets are being prepared for hosting; the
download URL and instructions will be published before launch.
---
## Download Commands
> **Times below are rough, indicative estimates only.** Actual duration
> depends on your bandwidth, the providers' current rate limits, and disk
> speed — treat them as ballpark, not guarantees.
### Free Datasets (No API Keys)
```bash
# All free datasets at once (includes the ~1.5 GB firm-characteristics
# dataset; add --skip-firm-characteristics to leave it out)
uv run python data/download_all.py --free-only
# Individual datasets (from repo root)
uv run python data/etfs/market/download.py # ~30s
uv run python data/crypto/market/download.py # ~10-15 min (see note)
uv run python data/factors/ff_download.py # ~5s
uv run python data/factors/aqr_download.py # ~5s
uv run python data/equities/firm_characteristics/download.py # ~1.5 GB, largest free dataset; downloads + converts (minutes, bandwidth-dependent)
uv run python data/futures/positioning/cot_download.py # ~2-3 min (CFTC CoT)
```
**Note on crypto download time**: The Binance public API returns max 1,500 rows per request with ~1s server response time. Downloading 5 years of hourly data for 19 symbols requires ~700 API calls. Downloads run in parallel (5 concurrent), but the total still takes 10-15 minutes. This is a Binance server-side rate limit, not a bug.
### Free API Key Required
```bash
# FRED macro indicators
uv run python data/macro/download.py
# US Equities (NASDAQ Data Link — frozen, ends 2018)
uv run python data/equities/market/us_equities/download.py
# FX pairs (OANDA)
uv run python data/fx/market/download.py # 4-hourly (default)
uv run python data/fx/market/download.py --daily # Daily
```
### Paid API Key (Databento)
```bash
# CME Futures — ALWAYS estimate cost first!
uv run python data/futures/market/download.py --estimate-only
uv run python data/futures/market/download.py
```
### Manual Download (Databento Download Center)
The Chapter 3 MBO slice (NVDA, 10 trading days in November 2024) is best
obtained as a one-off download from the Databento Download Center —
total cost is under $10 and the files stay available for 30 days.
See `data/equities/market/microstructure/MBO_DOWNLOAD.md` for step-by-step
instructions. An API-based alternative (`mbo_download.py`) is available
for users who already have a `DATABENTO_API_KEY`.
### Update Existing Data
Extend datasets beyond the default end date:
```bash
uv run python data/download_all.py --update
```
---
## Using Loaders
All loaders are importable from `data` and return Polars DataFrames:
```python
from data import (
load_etfs,
load_crypto_perps,
load_crypto_premium,
load_cme_futures,
load_cot,
load_fx_pairs,
load_macro,
load_us_equities,
load_ff_factors,
load_aqr_factors,
load_firm_characteristics,
load_nasdaq100_bars,
load_sp500_daily_bars,
load_sp500_options,
load_sp500_options_eda,
load_sp500_options_straddles_raw,
load_sp500_options_surface,
load_sp500_options_straddles,
load_nasdaq100_taq,
load_mbo_data,
load_nasdaq_itch,
load_iex_hist,
)
# All loaders support filtering
df = load_etfs(symbols=["SPY", "QQQ"], start_date="2020-01-01")
# Futures use 'products' instead of 'symbols'
futures = load_cme_futures(products=["ES", "NQ"], start_date="2020-01-01")
# Test mode: limit to N random symbols (seed-deterministic)
df = load_etfs(max_symbols=15)
```
When data is missing, loaders raise `DataNotFoundError` with download instructions.
---
## API Keys
### Free API Keys
| Provider | Variable | Sign Up |
| ---------------- | ---------------- | ------------------------------------------------- |
| FRED | `FRED_API_KEY` | https://fred.stlouisfed.org/docs/api/api_key.html |
| NASDAQ Data Link | `QUANDL_API_KEY` | https://data.nasdaq.com/sign-up |
| OANDA | `OANDA_API_KEY` | https://www.oanda.com/ |
### Paid API Keys
| Provider | Variable | Cost |
| --------- | ------------------- | ---------------- |
| Databento | `DATABENTO_API_KEY` | $125 free credit |
### Configuration
Create `.env` in repository root:
```bash
ML4T_DATA_PATH=/path/to/your/data
# Free API keys
FRED_API_KEY=your-fred-key
QUANDL_API_KEY=your-nasdaq-key
OANDA_API_KEY=your-oanda-key
# Paid
DATABENTO_API_KEY=db-your-key
```
---
## Directory Structure
Every dataset directory is self-contained: a download script, a loader
(or re-export from a parent loader), a README with the full instructions
that `DataNotFoundError` points readers to, and optionally a config
and exploration notebook.
Data is organized by **asset class × data type**, matching the Ch2 /
Ch4 taxonomy. Each asset class has `market/` for OHLCV-style data, and
optionally `fundamentals/`, `positioning/`, or other type-specific
subdirectories. Cross-asset datasets (`factors/`, `macro/`,
`prediction_markets/`, `alternative/`) sit at the top level.
```
data/
├── __init__.py # Single import point for all loaders
├── exceptions.py # DataNotFoundError, DownloadError, MissingDependencyError
├── download_all.py # Download orchestrator
├── README.md # (this file)
│
├── equities/ # US equities
│ ├── market/ # us_equities, sp500 (daily + options), nasdaq100, microstructure
│ ├── fundamentals/ # 10-K / 10-Q / 8-K filings, XBRL financials
│ ├── positioning/ # 13F institutional holdings, Form 4 insider
│ ├── firm_characteristics/ # Chen-Pelger-Zhu panel (standalone packaged dataset)
│ └── loader.py # All equities loaders in one module
│
├── futures/ # CME futures
│ ├── market/ # Databento continuous + individual contracts
│ ├── positioning/ # CFTC Commitment of Traders (CoT)
│ └── loader.py
│
├── crypto/ # Crypto
│ ├── market/ # Binance perps OHLCV + premium index
│ ├── onchain/ # DefiLlama TVL + CoinGecko OHLCV
│ └── loader.py
│
├── fx/market/ # FX pairs (OANDA)
├── etfs/market/ # ETF universe (Yahoo)
│
├── factors/ # Fama-French, AQR (cross-asset, academic)
├── macro/ # FRED macro indicators (cross-asset)
├── prediction_markets/ # Kalshi + Polymarket events
│
└── alternative/ # Cross-asset third-party alt data
├── news/ # Bloomberg, FNSPID
└── text/ # Financial Phrasebook sentiment benchmark
```
Every subdirectory owns its data's lifecycle — a reader can open any
leaf README and find the download command and file layout without
consulting the top-level doc.
### Equities Loaders (all in `equities/loader.py`)
**Market (OHLCV, microstructure, options):**
| Loader | Dataset | Source |
| ------ | ------- | ------ |
| `load_sp500_index()` | S&P 500 index OHLCV | Bundled |
| `load_us_equities()` | 3,199 US stocks (1962-2018) | NASDAQ DL |
| `load_sp500_daily_bars()` | S&P 500 daily OHLCV | AlgoSeek |
| `load_sp500_options()` | Raw options chains (legacy) | AlgoSeek |
| `load_sp500_options_eda()` | Options EDA slice (8 symbols, 2019-2020) | AlgoSeek (slim) |
| `load_sp500_options_straddles_raw()` | ATM-band raw chains, lifecycle-preserving (2017-2021) | AlgoSeek (slim) |
| `load_sp500_options_surface()` | Daily IV surface summary | Materialized |
| `load_sp500_options_straddles()` | Daily ATM straddles | Materialized |
| `load_nasdaq100_bars()` | NASDAQ-100 bars (minute default; resampling, quotes, full microstructure) | AlgoSeek |
| `load_nasdaq100_taq()` | TAQ tick data (AAPL, 2020-03-13 / 2020-03-16) | AlgoSeek (slim) |
| `load_mbo_data()` | MBO order book data | Databento |
| `load_nasdaq_itch()` | NASDAQ ITCH messages | NASDAQ FTP |
| `load_iex_hist()` | IEX DEEP/TOPS data | IEX (free) |
**Fundamentals (SEC filings + XBRL):**
| Loader | Dataset | Source |
| ------ | ------- | ------ |
| `load_sp500_10q_mda()` | S&P 500 10-Q MD&A text (2017-2021) | SEC EDGAR |
| `load_sec_filings(form_type)` | 10-K / 10-Q / 8-K aggregate text | SEC EDGAR |
| `resolve_sec_filings_dir()` | Per-ticker filings directory (Ch22 RAG) | SEC EDGAR |
| `load_sec_xbrl_fundamentals()` | XBRL financial facts (CIK × quarter × concept) | SEC XBRL Frames |
**Positioning (13F):**
| Loader | Dataset | Source |
| ------ | ------- | ------ |
| `load_institutional_holdings_13f()` | 13F holdings (per-cik, 10 curated managers) | SEC EDGAR |
| `load_13f_bulk_holdings(quarter)` | 13F full universe (~3M rows per quarter) | SEC bulk |
| `load_13f_stock_features()` | Stock-level features (breadth, concentration) | Derived |
| `load_13f_edges()` | Institution → stock edge list (graph) | Derived |
**Firm characteristics (packaged dataset):**
| Loader | Dataset | Source |
| ------ | ------- | ------ |
| `load_firm_characteristics()` | Chen-Pelger-Zhu panel (~180 features, returns + accounting) | GitHub |
---
## Storage Requirements
| Tier | Datasets | Size |
| ------------- | ------------------------------------- | ------- |
| Minimum | ETFs, Crypto, Factors | ~70 MB |
| Standard | + Macro, FX | ~75 MB |
| With Equities | + US Equities | ~740 MB |
| With Futures | + CME Futures | ~825 MB |
| Full | + AlgoSeek slim package, ITCH, MBO | ~7 GB |
---
## Canonical Schema
All loaders return data with consistent column names:
- **Entity column**: `symbol` (exception: CME futures use `product`)
- **Time column**: `timestamp` (for all frequencies — daily, hourly, minute, tick)
Notebooks should always use these canonical names. If older data files use legacy names like `asset`, `date`, `ticker`, or `pair`, the loaders normalize them automatically.