wehub-resource-sync
1532 lines
57 KiB
Plaintext
1532 lines
57 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6f4222c3",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.00199,
|
|
"end_time": "2026-06-13T02:34:22.545834+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:22.543844+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"# Data Management: From Download to Production Pipeline\n",
|
|
"\n",
|
|
"**Docker image**: `ml4t`\n",
|
|
"\n",
|
|
"**Chapter 2: The Financial Data Universe**\n",
|
|
"\n",
|
|
"Previous notebooks fetched and validated data. This notebook shows how to\n",
|
|
"**manage** it at scale using ml4t-data's production features:\n",
|
|
"\n",
|
|
"- **DataManager**: Unified entry point for fetching, storing, and updating\n",
|
|
"- **Universe**: Predefined symbol lists (S&P 500, NASDAQ 100, etc.)\n",
|
|
"- **HiveStorage**: Partitioned Parquet for fast queries and incremental writes\n",
|
|
"- **Incremental Updates**: Keep data fresh without re-downloading history\n",
|
|
"- **CLI**: Command-line interface for scripted workflows\n",
|
|
"\n",
|
|
"## Learning Objectives\n",
|
|
"\n",
|
|
"By completing this notebook, you will:\n",
|
|
"1. Use `DataManager` as a single entry point for all data operations\n",
|
|
"2. Load predefined universes with the `Universe` class\n",
|
|
"3. Store and query data with Hive-partitioned Parquet\n",
|
|
"4. Perform incremental updates and detect gaps\n",
|
|
"5. Use the `ml4t-data` CLI for scripted workflows\n",
|
|
"\n",
|
|
"## Why This Matters\n",
|
|
"\n",
|
|
"A one-time download is fine for a tutorial. A trading system needs:\n",
|
|
"- **Daily updates** that only fetch new data (10x faster than full refresh)\n",
|
|
"- **Partitioned storage** that supports fast date-range queries\n",
|
|
"- **Gap detection** to ensure completeness before backtesting\n",
|
|
"- **Metadata tracking** so you know what you have and when it was updated\n",
|
|
"\n",
|
|
"> **ml4t-data docs**: See the [Incremental Updates Guide](https://ml4trading.io/docs/data/user-guide/incremental-updates/)\n",
|
|
"> and [Storage Guide](https://ml4trading.io/docs/data/user-guide/storage/) for full reference.\n",
|
|
"\n",
|
|
"**Prerequisites**: ml4t-data installed; live network access for Yahoo Finance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "693f5b6f",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.00132,
|
|
"end_time": "2026-06-13T02:34:22.548651+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:22.547331+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## Setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "b8d5cf35",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:21.743935Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:21.743764Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:24.218350Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:24.217856Z"
|
|
},
|
|
"lines_to_next_cell": 2,
|
|
"papermill": {
|
|
"duration": 2.75515,
|
|
"end_time": "2026-06-13T02:34:25.305106+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:22.549956+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:32: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.default_format is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.default_format instead.\n",
|
|
"\n",
|
|
" self.default_format = \"png\"\n",
|
|
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:33: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.default_width is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.default_width instead.\n",
|
|
"\n",
|
|
" self.default_width = 700\n",
|
|
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:34: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.default_height is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.default_height instead.\n",
|
|
"\n",
|
|
" self.default_height = 500\n",
|
|
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:35: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.default_scale is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.default_scale instead.\n",
|
|
"\n",
|
|
" self.default_scale = 1\n",
|
|
".venv/lib/python3.14/site-packages/plotly/io/_kaleido.py:133: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.mathjax is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.mathjax instead.\n",
|
|
"\n",
|
|
" setattr(self._scope, name, value)\n",
|
|
".venv/lib/python3.14/site-packages/plotly/io/_kaleido.py:141: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.plotlyjs is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.plotlyjs instead.\n",
|
|
"\n",
|
|
" scope.plotlyjs = os.path.join(package_dir, \"plotly.min.js\")\n",
|
|
".venv/lib/python3.14/site-packages/plotly/io/_kaleido.py:147: DeprecationWarning: \n",
|
|
"Use of plotly.io.kaleido.scope.mathjax is deprecated and support will be removed after September 2025.\n",
|
|
"Please use plotly.io.defaults.mathjax instead.\n",
|
|
"\n",
|
|
" scope.mathjax = (\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Storage directory: 02_financial_data_universe/output/data_management\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"\"\"\"Data Management — DataManager, Universe, HiveStorage, and incremental updates.\"\"\"\n",
|
|
"\n",
|
|
"import logging\n",
|
|
"import shutil\n",
|
|
"from datetime import datetime\n",
|
|
"from pathlib import Path\n",
|
|
"\n",
|
|
"import polars as pl\n",
|
|
"import structlog\n",
|
|
"\n",
|
|
"# ml4t-data emits structured debug logs on every fetch/store; route them\n",
|
|
"# through stdlib logging at WARNING so the notebook output stays focused on\n",
|
|
"# the demonstration.\n",
|
|
"structlog.configure(\n",
|
|
" wrapper_class=structlog.make_filtering_bound_logger(logging.WARNING),\n",
|
|
")\n",
|
|
"\n",
|
|
"# ml4t-data core imports\n",
|
|
"from ml4t.data import DataManager\n",
|
|
"from ml4t.data.storage import HiveStorage\n",
|
|
"from ml4t.data.storage.backend import StorageConfig\n",
|
|
"from ml4t.data.universe import Universe\n",
|
|
"\n",
|
|
"from utils.paths import get_output_dir\n",
|
|
"\n",
|
|
"# Working directory for this notebook's storage examples. Wipe any artifacts\n",
|
|
"# from a previous run so the demo is fully reproducible.\n",
|
|
"STORAGE_DIR = get_output_dir(2, \"data_management\")\n",
|
|
"if STORAGE_DIR.exists():\n",
|
|
" shutil.rmtree(STORAGE_DIR)\n",
|
|
"STORAGE_DIR.mkdir(parents=True, exist_ok=True)\n",
|
|
"\n",
|
|
"print(f\"Storage directory: {STORAGE_DIR}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "1102ed41",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:24.219656Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:24.219572Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:24.221409Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:24.220932Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004003,
|
|
"end_time": "2026-06-13T02:34:25.310638+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.306635+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": [
|
|
"parameters"
|
|
]
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Production defaults — Papermill injects overrides for CI"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a3cc8ecb",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001332,
|
|
"end_time": "2026-06-13T02:34:25.313369+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.312037+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## 1. DataManager: The Unified Entry Point\n",
|
|
"\n",
|
|
"`DataManager` abstracts away provider selection, storage, and updates\n",
|
|
"behind a single interface. Compare:\n",
|
|
"\n",
|
|
"```python\n",
|
|
"# Without DataManager (manual)\n",
|
|
"provider = YahooFinanceProvider()\n",
|
|
"df = provider.fetch_ohlcv(\"AAPL\", \"2024-01-01\", \"2024-12-31\", \"daily\")\n",
|
|
"\n",
|
|
"# With DataManager (unified)\n",
|
|
"dm = DataManager()\n",
|
|
"df = dm.fetch(\"AAPL\", \"2024-01-01\", \"2024-12-31\")\n",
|
|
"```\n",
|
|
"\n",
|
|
"The real power shows with batch operations, storage integration, and updates."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bd45acdd",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.00133,
|
|
"end_time": "2026-06-13T02:34:25.316025+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.314695+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Fetch: Single Symbol"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "8596f551",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:24.222250Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:24.222188Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:24.722840Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:24.721742Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.256419,
|
|
"end_time": "2026-06-13T02:34:25.573802+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.317383+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"AAPL: 252 rows, 7 columns\n",
|
|
"Date range: 2024-01-02 to 2024-12-31\n",
|
|
"Columns: ['timestamp', 'symbol', 'open', 'high', 'low', 'close', 'volume']\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div><style>\n",
|
|
".dataframe > thead > tr,\n",
|
|
".dataframe > tbody > tr {\n",
|
|
" text-align: right;\n",
|
|
" white-space: pre-wrap;\n",
|
|
"}\n",
|
|
"</style>\n",
|
|
"<small>shape: (3, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>symbol</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>datetime[μs]</td><td>str</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>2024-01-02 00:00:00</td><td>"AAPL"</td><td>185.055288</td><td>186.330858</td><td>181.831782</td><td>183.562195</td><td>8.24887e7</td></tr><tr><td>2024-01-03 00:00:00</td><td>"AAPL"</td><td>182.158112</td><td>183.799536</td><td>181.376945</td><td>182.187775</td><td>5.84145e7</td></tr><tr><td>2024-01-04 00:00:00</td><td>"AAPL"</td><td>180.111221</td><td>181.040702</td><td>178.855446</td><td>179.873917</td><td>7.19836e7</td></tr></tbody></table></div>"
|
|
],
|
|
"text/plain": [
|
|
"shape: (3, 7)\n",
|
|
"┌─────────────────────┬────────┬────────────┬────────────┬────────────┬────────────┬───────────┐\n",
|
|
"│ timestamp ┆ symbol ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
|
|
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
|
"│ datetime[μs] ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │\n",
|
|
"╞═════════════════════╪════════╪════════════╪════════════╪════════════╪════════════╪═══════════╡\n",
|
|
"│ 2024-01-02 00:00:00 ┆ AAPL ┆ 185.055288 ┆ 186.330858 ┆ 181.831782 ┆ 183.562195 ┆ 8.24887e7 │\n",
|
|
"│ 2024-01-03 00:00:00 ┆ AAPL ┆ 182.158112 ┆ 183.799536 ┆ 181.376945 ┆ 182.187775 ┆ 5.84145e7 │\n",
|
|
"│ 2024-01-04 00:00:00 ┆ AAPL ┆ 180.111221 ┆ 181.040702 ┆ 178.855446 ┆ 179.873917 ┆ 7.19836e7 │\n",
|
|
"└─────────────────────┴────────┴────────────┴────────────┴────────────┴────────────┴───────────┘"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# DataManager without storage — pure fetch mode\n",
|
|
"dm = DataManager()\n",
|
|
"\n",
|
|
"# Fetch a single symbol (defaults to Yahoo Finance for equities)\n",
|
|
"aapl = dm.fetch(\"AAPL\", \"2024-01-01\", \"2024-12-31\", provider=\"yahoo\")\n",
|
|
"\n",
|
|
"print(f\"AAPL: {aapl.shape[0]} rows, {aapl.shape[1]} columns\")\n",
|
|
"print(f\"Date range: {aapl['timestamp'].min().date()} to {aapl['timestamp'].max().date()}\")\n",
|
|
"print(f\"Columns: {aapl.columns}\")\n",
|
|
"aapl.head(3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "afbd7467",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.003784,
|
|
"end_time": "2026-06-13T02:34:25.580744+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.576960+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Batch Fetch: Multiple Symbols\n",
|
|
"\n",
|
|
"`batch_load` fetches multiple symbols in parallel and returns a single\n",
|
|
"stacked DataFrame — the standard multi-asset format used throughout the book."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "52f4499c",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:24.725532Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:24.725316Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:24.983341Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:24.982994Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.213252,
|
|
"end_time": "2026-06-13T02:34:25.797979+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.584727+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Combined: 1,260 rows across 5 symbols\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div><style>\n",
|
|
".dataframe > thead > tr,\n",
|
|
".dataframe > tbody > tr {\n",
|
|
" text-align: right;\n",
|
|
" white-space: pre-wrap;\n",
|
|
"}\n",
|
|
"</style>\n",
|
|
"<small>shape: (5, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>"GLD"</td><td>252</td></tr><tr><td>"IWM"</td><td>252</td></tr><tr><td>"QQQ"</td><td>252</td></tr><tr><td>"SPY"</td><td>252</td></tr><tr><td>"TLT"</td><td>252</td></tr></tbody></table></div>"
|
|
],
|
|
"text/plain": [
|
|
"shape: (5, 2)\n",
|
|
"┌────────┬─────┐\n",
|
|
"│ symbol ┆ len │\n",
|
|
"│ --- ┆ --- │\n",
|
|
"│ str ┆ u32 │\n",
|
|
"╞════════╪═════╡\n",
|
|
"│ GLD ┆ 252 │\n",
|
|
"│ IWM ┆ 252 │\n",
|
|
"│ QQQ ┆ 252 │\n",
|
|
"│ SPY ┆ 252 │\n",
|
|
"│ TLT ┆ 252 │\n",
|
|
"└────────┴─────┘"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Fetch 5 ETFs in parallel\n",
|
|
"etf_symbols = [\"SPY\", \"QQQ\", \"IWM\", \"TLT\", \"GLD\"]\n",
|
|
"etf_data = dm.batch_load(\n",
|
|
" symbols=etf_symbols,\n",
|
|
" start=\"2024-01-01\",\n",
|
|
" end=\"2024-12-31\",\n",
|
|
" provider=\"yahoo\",\n",
|
|
" max_workers=4,\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Combined: {etf_data.shape[0]:,} rows across {etf_data['symbol'].n_unique()} symbols\")\n",
|
|
"\n",
|
|
"etf_data.group_by(\"symbol\").len().sort(\"symbol\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854cd89b",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001615,
|
|
"end_time": "2026-06-13T02:34:25.801510+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.799895+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## 2. Universe: Predefined Symbol Lists\n",
|
|
"\n",
|
|
"Instead of maintaining symbol lists in YAML or hardcoding them, ml4t-data\n",
|
|
"ships curated universes that stay current with index rebalances."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "1f5eeff1",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:24.984290Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:24.984205Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:24.986043Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:24.985699Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.00422,
|
|
"end_time": "2026-06-13T02:34:25.807236+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.803016+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Available universes:\n",
|
|
" CRYPTO_TOP_100: 100 symbols\n",
|
|
" FOREX_MAJORS: 28 symbols\n",
|
|
" NASDAQ100: 100 symbols\n",
|
|
" SP500: 503 symbols\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# List available universes\n",
|
|
"print(\"Available universes:\")\n",
|
|
"for name in Universe.list_universes():\n",
|
|
" symbols = Universe.get(name)\n",
|
|
" print(f\" {name}: {len(symbols)} symbols\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "b6e95f87",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:24.986744Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:24.986673Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:24.988278Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:24.988076Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004131,
|
|
"end_time": "2026-06-13T02:34:25.812869+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.808738+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"S&P 500: 503 symbols\n",
|
|
"First 10: ['AAPL', 'MSFT', 'NVDA', 'GOOGL', 'GOOG', 'AMZN', 'META', 'TSLA', 'AVGO', 'ORCL']\n",
|
|
"Last 10: ['PKG', 'ETFC', 'FITB', 'KEY', 'MTB', 'HBAN', 'CMA', 'ZION', 'WBS', 'EWBC']\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Access a universe directly\n",
|
|
"sp500 = Universe.SP500\n",
|
|
"print(f\"\\nS&P 500: {len(sp500)} symbols\")\n",
|
|
"print(f\"First 10: {sp500[:10]}\")\n",
|
|
"print(f\"Last 10: {sp500[-10:]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "308ef34a",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:24.989001Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:24.988948Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.179572Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.178937Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.179136,
|
|
"end_time": "2026-06-13T02:34:25.993561+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.814425+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"S&P 500 sample: 735 rows, 5 symbols\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Use with DataManager.batch_load_universe for one-line loading\n",
|
|
"# (fetches all 503 S&P 500 symbols — use a smaller slice for demo)\n",
|
|
"sp500_sample = dm.batch_load(\n",
|
|
" symbols=sp500[:5],\n",
|
|
" start=\"2024-06-01\",\n",
|
|
" end=\"2024-12-31\",\n",
|
|
" provider=\"yahoo\",\n",
|
|
")\n",
|
|
"print(\n",
|
|
" f\"S&P 500 sample: {sp500_sample.shape[0]:,} rows, {sp500_sample['symbol'].n_unique()} symbols\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "80c615ec",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.181622Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.181453Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.185811Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.185006Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004836,
|
|
"end_time": "2026-06-13T02:34:26.001076+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:25.996240+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Custom universes registered:\n",
|
|
" etf_momentum: ['SPY', 'QQQ', 'IWM', 'EFA', 'EEM', 'TLT', 'GLD']\n",
|
|
" crypto_arb: ['BTCUSDT', 'ETHUSDT', 'SOLUSDT', 'BNBUSDT']\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Custom universes for your strategy\n",
|
|
"Universe.add_custom(\"etf_momentum\", [\"SPY\", \"QQQ\", \"IWM\", \"EFA\", \"EEM\", \"TLT\", \"GLD\"])\n",
|
|
"Universe.add_custom(\"crypto_arb\", [\"BTCUSDT\", \"ETHUSDT\", \"SOLUSDT\", \"BNBUSDT\"])\n",
|
|
"\n",
|
|
"print(\"\\nCustom universes registered:\")\n",
|
|
"for name in [\"etf_momentum\", \"crypto_arb\"]:\n",
|
|
" print(f\" {name}: {Universe.get(name)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "06283011",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001934,
|
|
"end_time": "2026-06-13T02:34:26.005436+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.003502+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## 3. HiveStorage: Partitioned Parquet\n",
|
|
"\n",
|
|
"For data you'll query repeatedly, Hive-partitioned Parquet is the storage\n",
|
|
"layer used throughout ml4t-data. The HiveStorage backend collapses the\n",
|
|
"logical key `equities/daily/AAPL` to a filesystem-safe directory name and\n",
|
|
"nests Hive-style year/month partitions underneath:\n",
|
|
"\n",
|
|
"```\n",
|
|
"hive_demo/\n",
|
|
"├── .metadata/\n",
|
|
"│ └── equities_daily_AAPL.json\n",
|
|
"└── equities_daily_AAPL/\n",
|
|
" ├── year=2024/month=1/data.parquet\n",
|
|
" ├── year=2024/month=2/data.parquet\n",
|
|
" └── ...\n",
|
|
"```\n",
|
|
"\n",
|
|
"**Benefits over flat files**:\n",
|
|
"- **Partition pruning**: Query \"last 30 days\" reads 1 file, not all of history\n",
|
|
"- **Incremental writes**: New data appends without rewriting existing partitions\n",
|
|
"- **Metadata tracking**: Know when each symbol was last updated"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bf6dca76",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.002541,
|
|
"end_time": "2026-06-13T02:34:26.009834+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.007293+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### DataManager with Storage"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "e93d9dad",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.189092Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.188786Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.194957Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.193729Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.00602,
|
|
"end_time": "2026-06-13T02:34:26.018949+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.012929+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Initialize storage\n",
|
|
"storage_config = StorageConfig(\n",
|
|
" base_path=STORAGE_DIR / \"hive_demo\",\n",
|
|
" compression=\"zstd\",\n",
|
|
" partition_granularity=\"month\",\n",
|
|
")\n",
|
|
"storage = HiveStorage(config=storage_config)\n",
|
|
"\n",
|
|
"# DataManager with storage — enables load/update/metadata operations\n",
|
|
"dm_stored = DataManager(storage=storage)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fd2a5a69",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.00158,
|
|
"end_time": "2026-06-13T02:34:26.023280+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.021700+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Load and Store\n",
|
|
"\n",
|
|
"`DataManager.load()` fetches from the provider and writes to Hive\n",
|
|
"partitions in one call. The storage key encodes the asset class, frequency,\n",
|
|
"and symbol."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "6ee15e3c",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.198837Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.198479Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.856213Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.855780Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.456203,
|
|
"end_time": "2026-06-13T02:34:26.481240+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.025037+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Stored AAPL → key: equities/daily/AAPL\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Stored MSFT → key: equities/daily/MSFT\n",
|
|
" Stored GOOGL → key: equities/daily/GOOGL\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"symbols = [\"AAPL\", \"MSFT\", \"GOOGL\"]\n",
|
|
"stored_keys = {}\n",
|
|
"for symbol in symbols:\n",
|
|
" key = dm_stored.load(symbol, \"2023-01-01\", \"2024-12-31\", provider=\"yahoo\")\n",
|
|
" stored_keys[symbol] = key\n",
|
|
" print(f\" Stored {symbol} → key: {key}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d350a502",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.0016,
|
|
"end_time": "2026-06-13T02:34:26.484634+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.483034+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Query Stored Data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "cde35387",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.857369Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.857178Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.864541Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.864255Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.009241,
|
|
"end_time": "2026-06-13T02:34:26.495477+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.486236+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Symbols in storage: ['equities/daily/AAPL', 'equities/daily/GOOGL', 'equities/daily/MSFT']\n",
|
|
"\n",
|
|
"AAPL 2024 only: 251 rows (partition-pruned)\n",
|
|
"Date range: 2024-01-02 to 2024-12-30\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# List what's in storage. `storage.list_keys()` walks the on-disk layout, so it\n",
|
|
"# reports every symbol regardless of metadata-file contents.\n",
|
|
"stored_symbols = sorted(storage.list_keys())\n",
|
|
"print(f\"Symbols in storage: {stored_symbols}\")\n",
|
|
"\n",
|
|
"# Read back with a date-range filter — only the matching month=k partitions\n",
|
|
"# touch disk, so reading 2024 from a 2-year archive halves the I/O.\n",
|
|
"aapl_2024 = storage.read(\n",
|
|
" stored_keys[\"AAPL\"],\n",
|
|
" start_date=datetime(2024, 1, 1),\n",
|
|
" end_date=datetime(2024, 12, 31),\n",
|
|
").collect()\n",
|
|
"print(f\"\\nAAPL 2024 only: {len(aapl_2024)} rows (partition-pruned)\")\n",
|
|
"print(f\"Date range: {aapl_2024['timestamp'].min().date()} to {aapl_2024['timestamp'].max().date()}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc900a23",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001605,
|
|
"end_time": "2026-06-13T02:34:26.498866+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.497261+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Metadata"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "d1efdde2",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.865563Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.865495Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.867542Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.867346Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004572,
|
|
"end_time": "2026-06-13T02:34:26.505034+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.500462+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"AAPL:\n",
|
|
" last_updated: 2026-06-15T22:21:25.451720\n",
|
|
" partitions: ['equities_daily_AAPL/year=2023/month=1', 'equities_daily_AAPL/year=2023/month=2', 'equities_daily_AAPL/year=2023/month=3', 'equities_daily_AAPL/year=2023/month=4', 'equities_daily_AAPL/year=2023/month=5', 'equities_daily_AAPL/year=2023/month=6', 'equities_daily_AAPL/year=2023/month=7', 'equities_daily_AAPL/year=2023/month=8', 'equities_daily_AAPL/year=2023/month=9', 'equities_daily_AAPL/year=2023/month=10', 'equities_daily_AAPL/year=2023/month=11', 'equities_daily_AAPL/year=2023/month=12', 'equities_daily_AAPL/year=2024/month=1', 'equities_daily_AAPL/year=2024/month=2', 'equities_daily_AAPL/year=2024/month=3', 'equities_daily_AAPL/year=2024/month=4', 'equities_daily_AAPL/year=2024/month=5', 'equities_daily_AAPL/year=2024/month=6', 'equities_daily_AAPL/year=2024/month=7', 'equities_daily_AAPL/year=2024/month=8', 'equities_daily_AAPL/year=2024/month=9', 'equities_daily_AAPL/year=2024/month=10', 'equities_daily_AAPL/year=2024/month=11', 'equities_daily_AAPL/year=2024/month=12']\n",
|
|
" row_count: 502\n",
|
|
" schema: ['timestamp', 'symbol', 'open', 'high', 'low', 'close', 'volume', 'year', 'month']\n",
|
|
" custom: {'provider': 'yahoo', 'symbol': 'AAPL', 'asset_class': 'equities', 'bar_type': 'time', 'bar_params': {'frequency': 'daily'}, 'exchange': 'UNKNOWN', 'calendar': None, 'start_date': '2023-01-03 00:00:00', 'end_date': '2024-12-31 00:00:00', 'last_updated': '2026-06-16 02:21:25.423773+00:00', 'schema_version': '1.0', 'download_utc_timestamp': '2026-06-16 02:21:25.423810+00:00', 'data_range': {'start': '2023-01-03 00:00:00', 'end': '2024-12-31 00:00:00'}, 'provider_params': {}}\n",
|
|
"\n",
|
|
"MSFT:\n",
|
|
" last_updated: 2026-06-15T22:21:25.680586\n",
|
|
" partitions: ['equities_daily_MSFT/year=2023/month=1', 'equities_daily_MSFT/year=2023/month=2', 'equities_daily_MSFT/year=2023/month=3', 'equities_daily_MSFT/year=2023/month=4', 'equities_daily_MSFT/year=2023/month=5', 'equities_daily_MSFT/year=2023/month=6', 'equities_daily_MSFT/year=2023/month=7', 'equities_daily_MSFT/year=2023/month=8', 'equities_daily_MSFT/year=2023/month=9', 'equities_daily_MSFT/year=2023/month=10', 'equities_daily_MSFT/year=2023/month=11', 'equities_daily_MSFT/year=2023/month=12', 'equities_daily_MSFT/year=2024/month=1', 'equities_daily_MSFT/year=2024/month=2', 'equities_daily_MSFT/year=2024/month=3', 'equities_daily_MSFT/year=2024/month=4', 'equities_daily_MSFT/year=2024/month=5', 'equities_daily_MSFT/year=2024/month=6', 'equities_daily_MSFT/year=2024/month=7', 'equities_daily_MSFT/year=2024/month=8', 'equities_daily_MSFT/year=2024/month=9', 'equities_daily_MSFT/year=2024/month=10', 'equities_daily_MSFT/year=2024/month=11', 'equities_daily_MSFT/year=2024/month=12']\n",
|
|
" row_count: 502\n",
|
|
" schema: ['timestamp', 'symbol', 'open', 'high', 'low', 'close', 'volume', 'year', 'month']\n",
|
|
" custom: {'provider': 'yahoo', 'symbol': 'MSFT', 'asset_class': 'equities', 'bar_type': 'time', 'bar_params': {'frequency': 'daily'}, 'exchange': 'UNKNOWN', 'calendar': None, 'start_date': '2023-01-03 00:00:00', 'end_date': '2024-12-31 00:00:00', 'last_updated': '2026-06-16 02:21:25.598102+00:00', 'schema_version': '1.0', 'download_utc_timestamp': '2026-06-16 02:21:25.598141+00:00', 'data_range': {'start': '2023-01-03 00:00:00', 'end': '2024-12-31 00:00:00'}, 'provider_params': {}}\n",
|
|
"\n",
|
|
"GOOGL:\n",
|
|
" last_updated: 2026-06-15T22:21:25.854207\n",
|
|
" partitions: ['equities_daily_GOOGL/year=2023/month=1', 'equities_daily_GOOGL/year=2023/month=2', 'equities_daily_GOOGL/year=2023/month=3', 'equities_daily_GOOGL/year=2023/month=4', 'equities_daily_GOOGL/year=2023/month=5', 'equities_daily_GOOGL/year=2023/month=6', 'equities_daily_GOOGL/year=2023/month=7', 'equities_daily_GOOGL/year=2023/month=8', 'equities_daily_GOOGL/year=2023/month=9', 'equities_daily_GOOGL/year=2023/month=10', 'equities_daily_GOOGL/year=2023/month=11', 'equities_daily_GOOGL/year=2023/month=12', 'equities_daily_GOOGL/year=2024/month=1', 'equities_daily_GOOGL/year=2024/month=2', 'equities_daily_GOOGL/year=2024/month=3', 'equities_daily_GOOGL/year=2024/month=4', 'equities_daily_GOOGL/year=2024/month=5', 'equities_daily_GOOGL/year=2024/month=6', 'equities_daily_GOOGL/year=2024/month=7', 'equities_daily_GOOGL/year=2024/month=8', 'equities_daily_GOOGL/year=2024/month=9', 'equities_daily_GOOGL/year=2024/month=10', 'equities_daily_GOOGL/year=2024/month=11', 'equities_daily_GOOGL/year=2024/month=12']\n",
|
|
" row_count: 502\n",
|
|
" schema: ['timestamp', 'symbol', 'open', 'high', 'low', 'close', 'volume', 'year', 'month']\n",
|
|
" custom: {'provider': 'yahoo', 'symbol': 'GOOGL', 'asset_class': 'equities', 'bar_type': 'time', 'bar_params': {'frequency': 'daily'}, 'exchange': 'UNKNOWN', 'calendar': None, 'start_date': '2023-01-03 00:00:00', 'end_date': '2024-12-31 00:00:00', 'last_updated': '2026-06-16 02:21:25.830270+00:00', 'schema_version': '1.0', 'download_utc_timestamp': '2026-06-16 02:21:25.830295+00:00', 'data_range': {'start': '2023-01-03 00:00:00', 'end': '2024-12-31 00:00:00'}, 'provider_params': {}}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Check metadata for stored symbols\n",
|
|
"for symbol in symbols:\n",
|
|
" meta = dm_stored.get_metadata(symbol)\n",
|
|
" if meta:\n",
|
|
" print(f\"\\n{symbol}:\")\n",
|
|
" for k, v in list(meta.items())[:5]:\n",
|
|
" print(f\" {k}: {v}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "487c6384",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001681,
|
|
"end_time": "2026-06-13T02:34:26.508336+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.506655+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Inspect Partition Structure"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "5a8ee493",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.868480Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.868416Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:25.871397Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:25.871165Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.005669,
|
|
"end_time": "2026-06-13T02:34:26.515643+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.509974+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total Parquet files: 72\n",
|
|
"\n",
|
|
"Example partition paths (first 8):\n",
|
|
" equities_daily_AAPL/year=2023/month=1/data.parquet (3.1 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=10/data.parquet (3.2 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=11/data.parquet (3.1 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=12/data.parquet (3.1 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=2/data.parquet (3.1 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=3/data.parquet (3.2 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=4/data.parquet (3.0 KB)\n",
|
|
" equities_daily_AAPL/year=2023/month=5/data.parquet (3.2 KB)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# See the actual file layout\n",
|
|
"hive_root = STORAGE_DIR / \"hive_demo\"\n",
|
|
"parquet_files = sorted(hive_root.rglob(\"*.parquet\"))\n",
|
|
"print(f\"Total Parquet files: {len(parquet_files)}\")\n",
|
|
"print(\"\\nExample partition paths (first 8):\")\n",
|
|
"for f in parquet_files[:8]:\n",
|
|
" rel = f.relative_to(hive_root)\n",
|
|
" size_kb = f.stat().st_size / 1024\n",
|
|
" print(f\" {rel} ({size_kb:.1f} KB)\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7c6c8af1",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001987,
|
|
"end_time": "2026-06-13T02:34:26.519530+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.517543+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## 4. Incremental Updates\n",
|
|
"\n",
|
|
"The key workflow: download history once, then **update daily** with only new data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dca7103d",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001651,
|
|
"end_time": "2026-06-13T02:34:26.523139+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.521488+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Update a Symbol"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "8f41b1ef",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:25.872215Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:25.872140Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:26.495629Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:26.495299Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.452575,
|
|
"end_time": "2026-06-13T02:34:26.977336+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.524761+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[2m2026-06-15 22:21:26\u001b[0m [\u001b[33m\u001b[1mwarning \u001b[0m] \u001b[1mGaps detected in data \u001b[0m \u001b[36mgap_count\u001b[0m=\u001b[35m192\u001b[0m \u001b[36mtotal_missing\u001b[0m=\u001b[35m395\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Updated AAPL → equities/daily/AAPL\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[2m2026-06-15 22:21:26\u001b[0m [\u001b[33m\u001b[1mwarning \u001b[0m] \u001b[1mGaps detected in data \u001b[0m \u001b[36mgap_count\u001b[0m=\u001b[35m192\u001b[0m \u001b[36mtotal_missing\u001b[0m=\u001b[35m395\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Updated MSFT → equities/daily/MSFT\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[2m2026-06-15 22:21:26\u001b[0m [\u001b[33m\u001b[1mwarning \u001b[0m] \u001b[1mGaps detected in data \u001b[0m \u001b[36mgap_count\u001b[0m=\u001b[35m192\u001b[0m \u001b[36mtotal_missing\u001b[0m=\u001b[35m395\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Updated GOOGL → equities/daily/GOOGL\n",
|
|
" AAPL last updated: 2026-06-15T22:21:26.048479\n",
|
|
" MSFT last updated: 2026-06-15T22:21:26.285237\n",
|
|
" GOOGL last updated: 2026-06-15T22:21:26.493498\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# update() checks what's already stored and only fetches new data\n",
|
|
"for symbol in symbols:\n",
|
|
" key = dm_stored.update(symbol, lookback_days=7, provider=\"yahoo\")\n",
|
|
" print(f\" Updated {symbol} → {key}\")\n",
|
|
"\n",
|
|
"# Verify data is current\n",
|
|
"for symbol in symbols:\n",
|
|
" meta = dm_stored.get_metadata(symbol)\n",
|
|
" if meta and \"last_updated\" in meta:\n",
|
|
" print(f\" {symbol} last updated: {meta['last_updated']}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b0ad63c4",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.00177,
|
|
"end_time": "2026-06-13T02:34:26.981029+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.979259+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Understanding Update Strategies\n",
|
|
"\n",
|
|
"ml4t-data supports four update strategies:\n",
|
|
"\n",
|
|
"| Strategy | Behavior | Use Case |\n",
|
|
"|----------|----------|----------|\n",
|
|
"| `INCREMENTAL` | Only fetch data after last stored timestamp | Daily updates (default) |\n",
|
|
"| `APPEND_ONLY` | Never modify existing rows | Audit-safe archives |\n",
|
|
"| `FULL_REFRESH` | Replace all data for the symbol | Recovery after corruption |\n",
|
|
"| `BACKFILL` | Fill gaps in historical data | Fix missing periods |\n",
|
|
"\n",
|
|
"The default `INCREMENTAL` strategy is correct for most workflows.\n",
|
|
"`DataManager.update()` uses it automatically."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "770915a5",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001671,
|
|
"end_time": "2026-06-13T02:34:26.984418+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.982747+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Gap Detection\n",
|
|
"\n",
|
|
"Before backtesting, verify data completeness. The IncrementalUpdater\n",
|
|
"can detect missing trading days."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "4d92c5ae",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:26.496628Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:26.496558Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:26.515719Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:26.515409Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.024571,
|
|
"end_time": "2026-06-13T02:34:27.010704+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:26.986133+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"AAPL: No gaps (complete)\n",
|
|
"MSFT: No gaps (complete)\n",
|
|
"GOOGL: No gaps (complete)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from ml4t.data.update_manager import GapDetector\n",
|
|
"\n",
|
|
"# Pass `exclude_weekends=True` so Saturdays and Sundays don't count as gaps.\n",
|
|
"# The cached series here is calendar-dense (each non-trading day carries the\n",
|
|
"# prior close forward), so the detector reports no gaps. For a sparse,\n",
|
|
"# trading-days-only feed it would instead flag every missing session, including\n",
|
|
"# holidays — without an exchange calendar it cannot tell a holiday from a true\n",
|
|
"# gap, so pair it with a calendar-aware check for end-of-day pipelines.\n",
|
|
"gap_detector = GapDetector(exclude_weekends=True)\n",
|
|
"\n",
|
|
"for symbol, key in stored_keys.items():\n",
|
|
" df = storage.read(key).collect()\n",
|
|
" gaps = gap_detector.detect_gaps(df, frequency=\"daily\")\n",
|
|
" if gaps:\n",
|
|
" print(f\"{symbol}: {len(gaps)} gap(s) detected\")\n",
|
|
" for gap in gaps[:3]:\n",
|
|
" print(f\" {gap['start'].date()} to {gap['end'].date()} ({gap['size_days']} days)\")\n",
|
|
" else:\n",
|
|
" print(f\"{symbol}: No gaps (complete)\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "74fc8acd",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001789,
|
|
"end_time": "2026-06-13T02:34:27.014422+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:27.012633+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## 5. Command-Line Interface\n",
|
|
"\n",
|
|
"ml4t-data includes a CLI for scripted workflows and cron jobs.\n",
|
|
"Here are the key commands:\n",
|
|
"\n",
|
|
"### Fetch Data\n",
|
|
"```bash\n",
|
|
"# Single symbol\n",
|
|
"ml4t-data fetch AAPL --start 2024-01-01 --end 2024-12-31\n",
|
|
"\n",
|
|
"# Multiple symbols\n",
|
|
"ml4t-data fetch SPY QQQ IWM TLT --provider yahoo --output data/etfs.parquet\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Update Stored Data\n",
|
|
"```bash\n",
|
|
"# Update a symbol (incremental — only fetches new data)\n",
|
|
"ml4t-data update AAPL --storage-path ./data\n",
|
|
"\n",
|
|
"# Update all stored symbols\n",
|
|
"ml4t-data update --all --storage-path ./data\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Validate Data Quality\n",
|
|
"```bash\n",
|
|
"# Run OHLCV validation on stored data\n",
|
|
"ml4t-data validate ./data/etfs.parquet\n",
|
|
"```\n",
|
|
"\n",
|
|
"### List Available Data\n",
|
|
"```bash\n",
|
|
"# List symbols in storage\n",
|
|
"ml4t-data list --storage-path ./data\n",
|
|
"\n",
|
|
"# List available providers\n",
|
|
"ml4t-data info --providers\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Automated Daily Updates (Cron)\n",
|
|
"```bash\n",
|
|
"# Daily at 6 PM EST (after US market close), Monday-Friday\n",
|
|
"0 18 * * 1-5 cd ~/ml4t && ml4t-data update --all --storage-path ./data >> logs/update.log 2>&1\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5d517c6e",
|
|
"metadata": {
|
|
"lines_to_next_cell": 2,
|
|
"papermill": {
|
|
"duration": 0.001681,
|
|
"end_time": "2026-06-13T02:34:27.017839+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:27.016158+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## 6. Putting It Together: Production Data Pipeline\n",
|
|
"\n",
|
|
"Here's the complete workflow combining everything above — the pattern\n",
|
|
"used by the book's `data/download_all.py` orchestrator."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "32f2eae6",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:26.516835Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:26.516765Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:26.519749Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:26.519508Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.00602,
|
|
"end_time": "2026-06-13T02:34:27.025634+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:27.019614+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def production_pipeline(\n",
|
|
" universe_name: str,\n",
|
|
" start: str,\n",
|
|
" end: str,\n",
|
|
" storage_path: Path,\n",
|
|
") -> pl.DataFrame:\n",
|
|
" \"\"\"Fetch, store, validate, and assemble a stacked DataFrame for a universe.\n",
|
|
"\n",
|
|
" The same pattern drives `data/download_all.py` for every asset class —\n",
|
|
" only the universe and provider differ.\n",
|
|
" \"\"\"\n",
|
|
" from ml4t.data.validation import OHLCVValidator\n",
|
|
"\n",
|
|
" symbols = Universe.get(universe_name)\n",
|
|
" print(f\"Universe '{universe_name}': {len(symbols)} symbols\")\n",
|
|
"\n",
|
|
" config = StorageConfig(base_path=storage_path, compression=\"zstd\")\n",
|
|
" store = HiveStorage(config=config)\n",
|
|
" manager = DataManager(storage=store, enable_validation=True)\n",
|
|
"\n",
|
|
" stored = {}\n",
|
|
" for symbol in symbols:\n",
|
|
" stored[symbol] = manager.load(symbol, start, end, provider=\"yahoo\")\n",
|
|
" print(f\"Fetched: {len(stored)} symbols\")\n",
|
|
"\n",
|
|
" validator = OHLCVValidator(max_return_threshold=0.5)\n",
|
|
" issues = 0\n",
|
|
" for symbol, key in stored.items():\n",
|
|
" df = store.read(key).collect()\n",
|
|
" result = validator.validate(df)\n",
|
|
" if not result.passed:\n",
|
|
" issues += result.error_count\n",
|
|
" print(f\" {symbol}: {result.error_count} validation issues\")\n",
|
|
" print(f\"Validated: {issues} total issue(s) across {len(stored)} symbols\")\n",
|
|
"\n",
|
|
" frames = [\n",
|
|
" store.read(key).collect().with_columns(pl.lit(symbol).alias(\"symbol\"))\n",
|
|
" for symbol, key in stored.items()\n",
|
|
" ]\n",
|
|
" combined = pl.concat(frames)\n",
|
|
" print(f\"Result: {combined.shape[0]:,} rows, {combined['symbol'].n_unique()} symbols\")\n",
|
|
" return combined"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "5ea5d2ba",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-16T02:21:26.520528Z",
|
|
"iopub.status.busy": "2026-06-16T02:21:26.520468Z",
|
|
"iopub.status.idle": "2026-06-16T02:21:27.343077Z",
|
|
"shell.execute_reply": "2026-06-16T02:21:27.342605Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.453422,
|
|
"end_time": "2026-06-13T02:34:27.480881+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:27.027459+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Universe 'etf_momentum': 7 symbols\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Fetched: 7 symbols\n",
|
|
" SPY: 1 validation issues\n",
|
|
" QQQ: 1 validation issues\n",
|
|
" IWM: 1 validation issues\n",
|
|
" EFA: 1 validation issues\n",
|
|
" EEM: 1 validation issues\n",
|
|
" TLT: 1 validation issues\n",
|
|
" GLD: 1 validation issues\n",
|
|
"Validated: 7 total issue(s) across 7 symbols\n",
|
|
"Result: 1,764 rows, 7 symbols\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div><style>\n",
|
|
".dataframe > thead > tr,\n",
|
|
".dataframe > tbody > tr {\n",
|
|
" text-align: right;\n",
|
|
" white-space: pre-wrap;\n",
|
|
"}\n",
|
|
"</style>\n",
|
|
"<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>symbol</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>datetime[μs]</td><td>str</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>2024-10-01 00:00:00</td><td>"SPY"</td><td>563.30519</td><td>563.953544</td><td>556.035444</td><td>558.609314</td><td>7.26688e7</td></tr><tr><td>2024-10-02 00:00:00</td><td>"SPY"</td><td>557.715436</td><td>559.866883</td><td>555.31839</td><td>558.845154</td><td>3.80978e7</td></tr><tr><td>2024-10-03 00:00:00</td><td>"SPY"</td><td>557.371502</td><td>559.768548</td><td>555.534429</td><td>557.823425</td><td>4.08465e7</td></tr><tr><td>2024-10-04 00:00:00</td><td>"SPY"</td><td>562.273665</td><td>563.265893</td><td>558.098486</td><td>562.892578</td><td>4.29391e7</td></tr><tr><td>2024-10-07 00:00:00</td><td>"SPY"</td><td>561.242154</td><td>561.890568</td><td>556.654387</td><td>557.803772</td><td>4.99647e7</td></tr></tbody></table></div>"
|
|
],
|
|
"text/plain": [
|
|
"shape: (5, 7)\n",
|
|
"┌─────────────────────┬────────┬────────────┬────────────┬────────────┬────────────┬───────────┐\n",
|
|
"│ timestamp ┆ symbol ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
|
|
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
|
"│ datetime[μs] ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │\n",
|
|
"╞═════════════════════╪════════╪════════════╪════════════╪════════════╪════════════╪═══════════╡\n",
|
|
"│ 2024-10-01 00:00:00 ┆ SPY ┆ 563.30519 ┆ 563.953544 ┆ 556.035444 ┆ 558.609314 ┆ 7.26688e7 │\n",
|
|
"│ 2024-10-02 00:00:00 ┆ SPY ┆ 557.715436 ┆ 559.866883 ┆ 555.31839 ┆ 558.845154 ┆ 3.80978e7 │\n",
|
|
"│ 2024-10-03 00:00:00 ┆ SPY ┆ 557.371502 ┆ 559.768548 ┆ 555.534429 ┆ 557.823425 ┆ 4.08465e7 │\n",
|
|
"│ 2024-10-04 00:00:00 ┆ SPY ┆ 562.273665 ┆ 563.265893 ┆ 558.098486 ┆ 562.892578 ┆ 4.29391e7 │\n",
|
|
"│ 2024-10-07 00:00:00 ┆ SPY ┆ 561.242154 ┆ 561.890568 ┆ 556.654387 ┆ 557.803772 ┆ 4.99647e7 │\n",
|
|
"└─────────────────────┴────────┴────────────┴────────────┴────────────┴────────────┴───────────┘"
|
|
]
|
|
},
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Run pipeline on a small universe\n",
|
|
"pipeline_output = production_pipeline(\n",
|
|
" universe_name=\"etf_momentum\",\n",
|
|
" start=\"2024-01-01\",\n",
|
|
" end=\"2024-12-31\",\n",
|
|
" storage_path=STORAGE_DIR / \"pipeline_demo\",\n",
|
|
")\n",
|
|
"\n",
|
|
"pipeline_output.head()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b450b660",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001854,
|
|
"end_time": "2026-06-13T02:34:27.484751+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:27.482897+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"A single validation issue per symbol on this 2024 ETF panel comes from the\n",
|
|
"`OHLCVValidator(max_return_threshold=0.5)` flagging the largest 1-day move\n",
|
|
"in each series — a sanity check, not a data error. The validator surfaces\n",
|
|
"candidates; downstream code decides whether to drop, winsorize, or pass\n",
|
|
"through. Section 2.6 (data quality) covers the trade-offs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ea645e10",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001757,
|
|
"end_time": "2026-06-13T02:34:27.488321+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:34:27.486564+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"\n",
|
|
"## Summary\n",
|
|
"\n",
|
|
"| Component | Purpose | Key Method |\n",
|
|
"|-----------|---------|------------|\n",
|
|
"| **DataManager** | Unified entry point | `fetch()`, `batch_load()`, `load()`, `update()` |\n",
|
|
"| **Universe** | Predefined symbol lists | `Universe.SP500`, `Universe.get(\"nasdaq100\")` |\n",
|
|
"| **HiveStorage** | Partitioned Parquet | `read()`, `write()`, partition pruning |\n",
|
|
"| **GapDetector** | Gap detection in time series | `detect_gaps()`, `detect_gaps_in_storage()` |\n",
|
|
"| **CLI** | Scripted workflows & cron | `ml4t-data fetch`, `ml4t-data update` |\n",
|
|
"\n",
|
|
"### The ml4t-data Workflow\n",
|
|
"\n",
|
|
"```\n",
|
|
"1. Initial load: dm.load(\"AAPL\", \"2020-01-01\", \"2024-12-31\")\n",
|
|
"2. Daily update: dm.update(\"AAPL\", lookback_days=7)\n",
|
|
"3. Gap check: gap_detector.detect_gaps(df, frequency=\"daily\")\n",
|
|
"4. Batch load: dm.batch_load_universe(\"sp500\", start, end)\n",
|
|
"5. Automate: cron + ml4t-data update --all\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Key Takeaways\n",
|
|
"\n",
|
|
"- **One entry point, many providers.** `DataManager.fetch()` hides whether the\n",
|
|
" bytes come from Yahoo, Binance, AlgoSeek, or local Hive parquet; the user\n",
|
|
" code does not change when providers do.\n",
|
|
"- **`load()` is cache-first, `fetch()` is provider-first.** Use `load()` for\n",
|
|
" research / backtesting (fast, offline, deterministic); use `fetch()` only\n",
|
|
" when the cache must be refreshed.\n",
|
|
"- **Universes are first-class.** `Universe.SP500` and friends keep symbol\n",
|
|
" lists out of notebook code and version-controlled in the library.\n",
|
|
"- **Gap detection is a separate concern.** `GapDetector` runs against\n",
|
|
" already-stored data; missing trading days surface as findings, not silent\n",
|
|
" nulls.\n",
|
|
"- **The CLI is the production surface.** Cron-driven `ml4t-data update --all`\n",
|
|
" is the same code path the notebook exercises.\n",
|
|
"\n",
|
|
"### Further Reading\n",
|
|
"\n",
|
|
"- **Incremental updates**: `19_incremental_updates` walks the update strategies\n",
|
|
" from this notebook in detail and shows how to schedule them.\n",
|
|
"- **Storage formats**: `20_storage_benchmark_file` compares Parquet, CSV, and HDF5;\n",
|
|
" `21_storage_benchmark_database` benchmarks Postgres-backed alternatives.\n",
|
|
"- **Data quality**: `13_data_quality_framework` covers validation and anomaly detection.\n",
|
|
"- **Provider comparison**: `16_provider_comparison` demonstrates multi-source acquisition.\n",
|
|
"- **ml4t-data docs**: [ml4trading.io/docs/data/](https://ml4trading.io/docs/data/)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"cell_metadata_filter": "tags,-all"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.14.3"
|
|
},
|
|
"ml4t_provenance": {
|
|
"source_py_blob": "3de86b3120a1a4d7e6449197db5110192a71b401",
|
|
"executed_at": "2026-06-16T02:23:54.483242+00:00",
|
|
"executor": "cpu-local",
|
|
"production": true,
|
|
"parameters": {}
|
|
},
|
|
"papermill": {
|
|
"default_parameters": {},
|
|
"duration": 6.502209,
|
|
"end_time": "2026-06-13T02:34:28.404525+00:00",
|
|
"environment_variables": {},
|
|
"exception": null,
|
|
"input_path": "02_financial_data_universe/18_data_management.ipynb",
|
|
"output_path": "02_financial_data_universe/18_data_management.ipynb",
|
|
"parameters": {},
|
|
"start_time": "2026-06-13T02:34:21.902316+00:00",
|
|
"version": "2.7.0"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|