wehub-resource-sync
341 lines
15 KiB
Markdown
341 lines
15 KiB
Markdown
# ML4T Data Infrastructure
|
||
|
||
Central data management for *Machine Learning for Trading, 3rd Edition*.
|
||
|
||
Each dataset has its own directory with a download script, loader, config, and exploration notebook. All loaders return Polars DataFrames with a consistent API.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
# 1. Set data path in repository root .env file
|
||
ML4T_DATA_PATH=/path/to/your/data
|
||
|
||
# 2. Download free datasets (no API keys needed)
|
||
uv run python data/download_all.py --free-only
|
||
|
||
# 3. Use in notebooks
|
||
from data import load_etfs
|
||
df = load_etfs()
|
||
```
|
||
|
||
---
|
||
|
||
## Dataset Catalog
|
||
|
||
Organized by asset class and data type. "Type" column maps each dataset
|
||
to its place in the Ch2/Ch4 taxonomy: market (OHLCV / microstructure /
|
||
options), fundamentals (accounting + regulatory filings), positioning
|
||
(positions / insider activity), onchain (crypto-native fundamentals), or
|
||
cross-asset (factors, macro, prediction markets, news, text).
|
||
|
||
| Dataset | Asset Class | Type | Frequency | Symbols | Coverage | Source | Access |
|
||
| ------------------------ | ------------ | ------------- | ------------ | ------- | --------- | ---------------- | ------ |
|
||
| ETF Universe | Equity | Market | Daily | 100 | 2006-2025 | Yahoo Finance | No |
|
||
| US Equities | Equity | Market | Daily | 3,199 | 1962-2018 | NASDAQ DL | Free |
|
||
| S&P 500 Bars | Equity | Market | Daily | ~638 | 2017-2021 | AlgoSeek | Soon |
|
||
| S&P 500 Options | Equity | Market | Daily | ~500 | 2017-2021 | AlgoSeek | Soon |
|
||
| NASDAQ-100 Bars | Equity | Market | Minute | ~100 | 2020-2021 | AlgoSeek | Soon |
|
||
| TAQ Tick | Equity | Market | Tick | 1 | Mar 2020 | AlgoSeek | Soon |
|
||
| MBO Tick | Equity | Market | Tick | 1 | Nov 2024 | Databento | Manual |
|
||
| NASDAQ ITCH | Equity | Market | Tick | all | varies | NASDAQ FTP | No |
|
||
| IEX DEEP/TOPS | Equity | Market | Tick | all | varies | IEX public | No |
|
||
| SEC XBRL Fundamentals | Equity | Fundamentals | Quarterly | 20 | 2022-2024 | SEC EDGAR | No |
|
||
| SEC 10-K (SP100) | Equity | Fundamentals | Annual | ~100 | 2020-2025 | SEC EDGAR | No |
|
||
| SEC 10-Q MD&A (SP500) | Equity | Fundamentals | Quarterly | ~600 | 2017-2021 | SEC EDGAR | No |
|
||
| SEC 8-K (SP100) | Equity | Fundamentals | Event | ~100 | 2024-2025 | SEC EDGAR | No |
|
||
| 13F Institutional | Equity | Positioning | Quarterly | 10 inst | rolling | SEC EDGAR | No |
|
||
| Form 4 Insider | Equity | Positioning | Event | varies | varies | SEC EDGAR | No |
|
||
| Firm Characteristics | Equity | Packaged | Monthly | anon | 1967-2016 | GitHub | No |
|
||
| CME Futures | Futures | Market | Daily/Hourly | 30 | 2011-2025 | Databento | Paid |
|
||
| CFTC Commitment of Traders | Futures | Positioning | Weekly | 25+ | 2020-2025 | CFTC public | No |
|
||
| Crypto Perps | Crypto | Market | 1h | 19 | 2020-2025 | Binance Public | No |
|
||
| Crypto Premium | Crypto | Market | 8h | 19 | 2020-2025 | Binance Public | No |
|
||
| DefiLlama TVL | Crypto | Onchain | Daily | chains | varies | DefiLlama | No |
|
||
| CoinGecko OHLCV | Crypto | Onchain | Daily | varies | 365 days | CoinGecko | No |
|
||
| FX Pairs | Currency | Market | 4h/Daily | 20 | 2011-2025 | OANDA | Free |
|
||
| FF Factors | Cross-asset | Factors | Monthly | 5 | 1926-now | Ken French | No |
|
||
| AQR Factors | Cross-asset | Factors | Monthly | 8 | varies | AQR | No |
|
||
| FRED Macro | Cross-asset | Macro | Various | 40 | 2000-2025 | FRED | Free |
|
||
| Kalshi events | Cross-asset | Prediction | Daily | varies | 2021-2025 | Kalshi public | No |
|
||
| Polymarket events | Cross-asset | Prediction | Daily | varies | 2020-2025 | Polymarket public| No |
|
||
| FNSPID news | Cross-asset | News | Daily | 4,775 | 1999-2023 | HuggingFace | No |
|
||
| Bloomberg news archive | Cross-asset | News | Daily | mixed | 2006-2013 | HuggingFace | No |
|
||
| Financial Phrasebank | Cross-asset | Text | Static | — | n/a | HuggingFace | No |
|
||
|
||
**Access legend.** `No` — included with the repo or fetched by an
|
||
unauthenticated script. `Free` — script-download, free API key required.
|
||
`Paid` — script-download, billed API (see per-dataset estimates).
|
||
`Manual` — reader downloads from a hosted URL or provider portal and
|
||
places the files under `$ML4T_DATA_PATH` (no script); the DataBento MBO
|
||
one-off has step-by-step instructions below. `Soon` — the reduced
|
||
reader-facing AlgoSeek datasets are being prepared for hosting; the
|
||
download URL and instructions will be published before launch.
|
||
|
||
---
|
||
|
||
## Download Commands
|
||
|
||
> **Times below are rough, indicative estimates only.** Actual duration
|
||
> depends on your bandwidth, the providers' current rate limits, and disk
|
||
> speed — treat them as ballpark, not guarantees.
|
||
|
||
### Free Datasets (No API Keys)
|
||
|
||
```bash
|
||
# All free datasets at once (includes the ~1.5 GB firm-characteristics
|
||
# dataset; add --skip-firm-characteristics to leave it out)
|
||
uv run python data/download_all.py --free-only
|
||
|
||
# Individual datasets (from repo root)
|
||
uv run python data/etfs/market/download.py # ~30s
|
||
uv run python data/crypto/market/download.py # ~10-15 min (see note)
|
||
uv run python data/factors/ff_download.py # ~5s
|
||
uv run python data/factors/aqr_download.py # ~5s
|
||
uv run python data/equities/firm_characteristics/download.py # ~1.5 GB, largest free dataset; downloads + converts (minutes, bandwidth-dependent)
|
||
uv run python data/futures/positioning/cot_download.py # ~2-3 min (CFTC CoT)
|
||
```
|
||
|
||
**Note on crypto download time**: The Binance public API returns max 1,500 rows per request with ~1s server response time. Downloading 5 years of hourly data for 19 symbols requires ~700 API calls. Downloads run in parallel (5 concurrent), but the total still takes 10-15 minutes. This is a Binance server-side rate limit, not a bug.
|
||
|
||
### Free API Key Required
|
||
|
||
```bash
|
||
# FRED macro indicators
|
||
uv run python data/macro/download.py
|
||
|
||
# US Equities (NASDAQ Data Link — frozen, ends 2018)
|
||
uv run python data/equities/market/us_equities/download.py
|
||
|
||
# FX pairs (OANDA)
|
||
uv run python data/fx/market/download.py # 4-hourly (default)
|
||
uv run python data/fx/market/download.py --daily # Daily
|
||
```
|
||
|
||
### Paid API Key (Databento)
|
||
|
||
```bash
|
||
# CME Futures — ALWAYS estimate cost first!
|
||
uv run python data/futures/market/download.py --estimate-only
|
||
uv run python data/futures/market/download.py
|
||
```
|
||
|
||
### Manual Download (Databento Download Center)
|
||
|
||
The Chapter 3 MBO slice (NVDA, 10 trading days in November 2024) is best
|
||
obtained as a one-off download from the Databento Download Center —
|
||
total cost is under $10 and the files stay available for 30 days.
|
||
|
||
See `data/equities/market/microstructure/MBO_DOWNLOAD.md` for step-by-step
|
||
instructions. An API-based alternative (`mbo_download.py`) is available
|
||
for users who already have a `DATABENTO_API_KEY`.
|
||
|
||
### Update Existing Data
|
||
|
||
Extend datasets beyond the default end date:
|
||
|
||
```bash
|
||
uv run python data/download_all.py --update
|
||
```
|
||
|
||
---
|
||
|
||
## Using Loaders
|
||
|
||
All loaders are importable from `data` and return Polars DataFrames:
|
||
|
||
```python
|
||
from data import (
|
||
load_etfs,
|
||
load_crypto_perps,
|
||
load_crypto_premium,
|
||
load_cme_futures,
|
||
load_cot,
|
||
load_fx_pairs,
|
||
load_macro,
|
||
load_us_equities,
|
||
load_ff_factors,
|
||
load_aqr_factors,
|
||
load_firm_characteristics,
|
||
load_nasdaq100_bars,
|
||
load_sp500_daily_bars,
|
||
load_sp500_options,
|
||
load_sp500_options_eda,
|
||
load_sp500_options_straddles_raw,
|
||
load_sp500_options_surface,
|
||
load_sp500_options_straddles,
|
||
load_nasdaq100_taq,
|
||
load_mbo_data,
|
||
load_nasdaq_itch,
|
||
load_iex_hist,
|
||
)
|
||
|
||
# All loaders support filtering
|
||
df = load_etfs(symbols=["SPY", "QQQ"], start_date="2020-01-01")
|
||
|
||
# Futures use 'products' instead of 'symbols'
|
||
futures = load_cme_futures(products=["ES", "NQ"], start_date="2020-01-01")
|
||
|
||
# Test mode: limit to N random symbols (seed-deterministic)
|
||
df = load_etfs(max_symbols=15)
|
||
```
|
||
|
||
When data is missing, loaders raise `DataNotFoundError` with download instructions.
|
||
|
||
---
|
||
|
||
## API Keys
|
||
|
||
### Free API Keys
|
||
|
||
| Provider | Variable | Sign Up |
|
||
| ---------------- | ---------------- | ------------------------------------------------- |
|
||
| FRED | `FRED_API_KEY` | https://fred.stlouisfed.org/docs/api/api_key.html |
|
||
| NASDAQ Data Link | `QUANDL_API_KEY` | https://data.nasdaq.com/sign-up |
|
||
| OANDA | `OANDA_API_KEY` | https://www.oanda.com/ |
|
||
|
||
### Paid API Keys
|
||
|
||
| Provider | Variable | Cost |
|
||
| --------- | ------------------- | ---------------- |
|
||
| Databento | `DATABENTO_API_KEY` | $125 free credit |
|
||
|
||
### Configuration
|
||
|
||
Create `.env` in repository root:
|
||
|
||
```bash
|
||
ML4T_DATA_PATH=/path/to/your/data
|
||
|
||
# Free API keys
|
||
FRED_API_KEY=your-fred-key
|
||
QUANDL_API_KEY=your-nasdaq-key
|
||
OANDA_API_KEY=your-oanda-key
|
||
|
||
# Paid
|
||
DATABENTO_API_KEY=db-your-key
|
||
```
|
||
|
||
---
|
||
|
||
## Directory Structure
|
||
|
||
Every dataset directory is self-contained: a download script, a loader
|
||
(or re-export from a parent loader), a README with the full instructions
|
||
that `DataNotFoundError` points readers to, and optionally a config
|
||
and exploration notebook.
|
||
|
||
Data is organized by **asset class × data type**, matching the Ch2 /
|
||
Ch4 taxonomy. Each asset class has `market/` for OHLCV-style data, and
|
||
optionally `fundamentals/`, `positioning/`, or other type-specific
|
||
subdirectories. Cross-asset datasets (`factors/`, `macro/`,
|
||
`prediction_markets/`, `alternative/`) sit at the top level.
|
||
|
||
```
|
||
data/
|
||
├── __init__.py # Single import point for all loaders
|
||
├── exceptions.py # DataNotFoundError, DownloadError, MissingDependencyError
|
||
├── download_all.py # Download orchestrator
|
||
├── README.md # (this file)
|
||
│
|
||
├── equities/ # US equities
|
||
│ ├── market/ # us_equities, sp500 (daily + options), nasdaq100, microstructure
|
||
│ ├── fundamentals/ # 10-K / 10-Q / 8-K filings, XBRL financials
|
||
│ ├── positioning/ # 13F institutional holdings, Form 4 insider
|
||
│ ├── firm_characteristics/ # Chen-Pelger-Zhu panel (standalone packaged dataset)
|
||
│ └── loader.py # All equities loaders in one module
|
||
│
|
||
├── futures/ # CME futures
|
||
│ ├── market/ # Databento continuous + individual contracts
|
||
│ ├── positioning/ # CFTC Commitment of Traders (CoT)
|
||
│ └── loader.py
|
||
│
|
||
├── crypto/ # Crypto
|
||
│ ├── market/ # Binance perps OHLCV + premium index
|
||
│ ├── onchain/ # DefiLlama TVL + CoinGecko OHLCV
|
||
│ └── loader.py
|
||
│
|
||
├── fx/market/ # FX pairs (OANDA)
|
||
├── etfs/market/ # ETF universe (Yahoo)
|
||
│
|
||
├── factors/ # Fama-French, AQR (cross-asset, academic)
|
||
├── macro/ # FRED macro indicators (cross-asset)
|
||
├── prediction_markets/ # Kalshi + Polymarket events
|
||
│
|
||
└── alternative/ # Cross-asset third-party alt data
|
||
├── news/ # Bloomberg, FNSPID
|
||
└── text/ # Financial Phrasebook sentiment benchmark
|
||
```
|
||
|
||
Every subdirectory owns its data's lifecycle — a reader can open any
|
||
leaf README and find the download command and file layout without
|
||
consulting the top-level doc.
|
||
|
||
### Equities Loaders (all in `equities/loader.py`)
|
||
|
||
**Market (OHLCV, microstructure, options):**
|
||
|
||
| Loader | Dataset | Source |
|
||
| ------ | ------- | ------ |
|
||
| `load_sp500_index()` | S&P 500 index OHLCV | Bundled |
|
||
| `load_us_equities()` | 3,199 US stocks (1962-2018) | NASDAQ DL |
|
||
| `load_sp500_daily_bars()` | S&P 500 daily OHLCV | AlgoSeek |
|
||
| `load_sp500_options()` | Raw options chains (legacy) | AlgoSeek |
|
||
| `load_sp500_options_eda()` | Options EDA slice (8 symbols, 2019-2020) | AlgoSeek (slim) |
|
||
| `load_sp500_options_straddles_raw()` | ATM-band raw chains, lifecycle-preserving (2017-2021) | AlgoSeek (slim) |
|
||
| `load_sp500_options_surface()` | Daily IV surface summary | Materialized |
|
||
| `load_sp500_options_straddles()` | Daily ATM straddles | Materialized |
|
||
| `load_nasdaq100_bars()` | NASDAQ-100 bars (minute default; resampling, quotes, full microstructure) | AlgoSeek |
|
||
| `load_nasdaq100_taq()` | TAQ tick data (AAPL, 2020-03-13 / 2020-03-16) | AlgoSeek (slim) |
|
||
| `load_mbo_data()` | MBO order book data | Databento |
|
||
| `load_nasdaq_itch()` | NASDAQ ITCH messages | NASDAQ FTP |
|
||
| `load_iex_hist()` | IEX DEEP/TOPS data | IEX (free) |
|
||
|
||
**Fundamentals (SEC filings + XBRL):**
|
||
|
||
| Loader | Dataset | Source |
|
||
| ------ | ------- | ------ |
|
||
| `load_sp500_10q_mda()` | S&P 500 10-Q MD&A text (2017-2021) | SEC EDGAR |
|
||
| `load_sec_filings(form_type)` | 10-K / 10-Q / 8-K aggregate text | SEC EDGAR |
|
||
| `resolve_sec_filings_dir()` | Per-ticker filings directory (Ch22 RAG) | SEC EDGAR |
|
||
| `load_sec_xbrl_fundamentals()` | XBRL financial facts (CIK × quarter × concept) | SEC XBRL Frames |
|
||
|
||
**Positioning (13F):**
|
||
|
||
| Loader | Dataset | Source |
|
||
| ------ | ------- | ------ |
|
||
| `load_institutional_holdings_13f()` | 13F holdings (per-cik, 10 curated managers) | SEC EDGAR |
|
||
| `load_13f_bulk_holdings(quarter)` | 13F full universe (~3M rows per quarter) | SEC bulk |
|
||
| `load_13f_stock_features()` | Stock-level features (breadth, concentration) | Derived |
|
||
| `load_13f_edges()` | Institution → stock edge list (graph) | Derived |
|
||
|
||
**Firm characteristics (packaged dataset):**
|
||
|
||
| Loader | Dataset | Source |
|
||
| ------ | ------- | ------ |
|
||
| `load_firm_characteristics()` | Chen-Pelger-Zhu panel (~180 features, returns + accounting) | GitHub |
|
||
|
||
---
|
||
|
||
## Storage Requirements
|
||
|
||
| Tier | Datasets | Size |
|
||
| ------------- | ------------------------------------- | ------- |
|
||
| Minimum | ETFs, Crypto, Factors | ~70 MB |
|
||
| Standard | + Macro, FX | ~75 MB |
|
||
| With Equities | + US Equities | ~740 MB |
|
||
| With Futures | + CME Futures | ~825 MB |
|
||
| Full | + AlgoSeek slim package, ITCH, MBO | ~7 GB |
|
||
|
||
---
|
||
|
||
## Canonical Schema
|
||
|
||
All loaders return data with consistent column names:
|
||
|
||
- **Entity column**: `symbol` (exception: CME futures use `product`)
|
||
- **Time column**: `timestamp` (for all frequencies — daily, hourly, minute, tick)
|
||
|
||
Notebooks should always use these canonical names. If older data files use legacy names like `asset`, `date`, `ticker`, or `pair`, the loaders normalize them automatically.
|