wehub-resource-sync
1194 lines
41 KiB
Plaintext
1194 lines
41 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "080fa768",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.006624,
|
||
"end_time": "2026-06-13T02:32:36.435830+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:36.429206+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"# ETFs — Exploratory Data Analysis\n",
|
||
"\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"\n",
|
||
"**Purpose**: Profile the 100-ETF candidate universe sourced from Yahoo Finance\n",
|
||
"and confirm the category coverage, history, and data-quality characteristics\n",
|
||
"that drive the ETF rotation case study.\n",
|
||
"\n",
|
||
"**Learning objectives**:\n",
|
||
"\n",
|
||
"- Load the ETF panel via `data.load_etfs` and inspect its canonical schema.\n",
|
||
"- Quantify per-symbol coverage and identify ETFs with shorter history.\n",
|
||
"- Check OHLC invariants and null rates across the full panel.\n",
|
||
"- Compare liquidity and price ranges across asset-class categories.\n",
|
||
"\n",
|
||
"**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — ETPs).\n",
|
||
"\n",
|
||
"**Prerequisites**: `data` package on `PYTHONPATH`; ETF parquet present at\n",
|
||
"`ML4T_DATA_PATH/etfs/market/`. Run `python data/etfs/market/download.py` if\n",
|
||
"missing."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "3cce8875",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:36.444357Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:36.444151Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.175510Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.174939Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 1.737052,
|
||
"end_time": "2026-06-13T02:32:38.176013+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:36.438961+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:32: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.default_format is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.default_format instead.\n",
|
||
"\n",
|
||
" self.default_format = \"png\"\n",
|
||
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:33: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.default_width is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.default_width instead.\n",
|
||
"\n",
|
||
" self.default_width = 700\n",
|
||
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:34: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.default_height is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.default_height instead.\n",
|
||
"\n",
|
||
" self.default_height = 500\n",
|
||
".venv/lib/python3.14/site-packages/kaleido/scopes/plotly.py:35: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.default_scale is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.default_scale instead.\n",
|
||
"\n",
|
||
" self.default_scale = 1\n",
|
||
".venv/lib/python3.14/site-packages/plotly/io/_kaleido.py:133: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.mathjax is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.mathjax instead.\n",
|
||
"\n",
|
||
" setattr(self._scope, name, value)\n",
|
||
".venv/lib/python3.14/site-packages/plotly/io/_kaleido.py:141: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.plotlyjs is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.plotlyjs instead.\n",
|
||
"\n",
|
||
" scope.plotlyjs = os.path.join(package_dir, \"plotly.min.js\")\n",
|
||
".venv/lib/python3.14/site-packages/plotly/io/_kaleido.py:147: DeprecationWarning: \n",
|
||
"Use of plotly.io.kaleido.scope.mathjax is deprecated and support will be removed after September 2025.\n",
|
||
"Please use plotly.io.defaults.mathjax instead.\n",
|
||
"\n",
|
||
" scope.mathjax = (\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"\"\"\"ETFs — Exploratory data analysis of the multi-asset ETF universe.\"\"\"\n",
|
||
"\n",
|
||
"import polars as pl\n",
|
||
"from ml4t.data.etfs import ETFDataManager\n",
|
||
"\n",
|
||
"from data import load_etfs\n",
|
||
"from utils.data_quality import check_ohlc_invariants, per_asset_stats\n",
|
||
"from utils.paths import REPO_ROOT"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "946766e1",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.179609Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.179475Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.181567Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.181128Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004469,
|
||
"end_time": "2026-06-13T02:32:38.181902+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.177433+00:00",
|
||
"status": "completed"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Production defaults — Papermill injects overrides for CI\n",
|
||
"MAX_SYMBOLS = 0 # 0 = all"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "46c06883",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001109,
|
||
"end_time": "2026-06-13T02:32:38.184203+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.183094+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 1. Load and Inspect\n",
|
||
"\n",
|
||
"The ETF universe is stored as a single Parquet file containing daily OHLCV data\n",
|
||
"for 50 ETFs spanning multiple asset classes."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "5bd261e9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.187059Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.186956Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.212017Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.211555Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.027248,
|
||
"end_time": "2026-06-13T02:32:38.212597+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.185349+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== ETF Dataset ===\n",
|
||
"Shape: (470662, 7)\n",
|
||
"Columns: ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'symbol']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"etfs = load_etfs()\n",
|
||
"\n",
|
||
"print(\"=== ETF Dataset ===\")\n",
|
||
"print(f\"Shape: {etfs.shape}\")\n",
|
||
"print(f\"Columns: {etfs.columns}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "3646ec12",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.216941Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.216848Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.218885Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.218595Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004169,
|
||
"end_time": "2026-06-13T02:32:38.219261+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.215092+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Schema:\n",
|
||
" timestamp: Date\n",
|
||
" open: Float64\n",
|
||
" high: Float64\n",
|
||
" low: Float64\n",
|
||
" close: Float64\n",
|
||
" volume: Float64\n",
|
||
" symbol: String\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Schema overview\n",
|
||
"print(\"\\nSchema:\")\n",
|
||
"for col, dtype in etfs.schema.items():\n",
|
||
" print(f\" {col}: {dtype}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fc9bbffd",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001161,
|
||
"end_time": "2026-06-13T02:32:38.221650+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.220489+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"### Adjusted Prices\n",
|
||
"\n",
|
||
"Yahoo Finance returns split- and dividend-adjusted OHLC. The `close` column is\n",
|
||
"the adjusted close, so returns can be computed directly without a separate\n",
|
||
"`adj_close` column."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "67c9002c",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.00112,
|
||
"end_time": "2026-06-13T02:32:38.223947+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.222827+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 2. Coverage Summary"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "6bcdcb20",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.227044Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.226967Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.232200Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.231802Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.007204,
|
||
"end_time": "2026-06-13T02:32:38.232409+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.225205+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Coverage ===\n",
|
||
"Number of ETFs: 100\n",
|
||
"\n",
|
||
"Symbols: ACWI, ACWX, AGG, BIL, BND, BNDX, DBA, DBC, DIA, DVY, EEM, EFA, EMB, EWA, EWC, EWG, EWH, EWI, EWJ, EWL, EWN, EWP, EWQ, EWT, EWU, EWW, EWY, EWZ, EZA, FXB, FXE, FXI, FXY, GLD, GOVT, GSG, HYG, IAU, IBB, IEF, IEFA, IEMG, IJR, INDA, ITA, ITB, IVE, IVW, IWM, IYR, JNK, KRE, LQD, MCHI, MDY, MTUM, MUB, OIH, PPLT, QQQ, QUAL, RSP, SCHD, SDY, SHY, SLV, SMH, SOXX, SPY, THD, TIP, TLT, UNG, USMV, USO, UUP, VCSH, VEA, VGK, VIG, VLUE, VNQ, VTI, VTV, VUG, VWO, XBI, XLB, XLC, XLE, XLF, XLI, XLK, XLP, XLRE, XLU, XLV, XLY, XME, XRT\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Unique symbols\n",
|
||
"symbols = etfs[\"symbol\"].unique().sort().to_list()\n",
|
||
"print(\"=== Coverage ===\")\n",
|
||
"print(f\"Number of ETFs: {len(symbols)}\")\n",
|
||
"print(f\"\\nSymbols: {', '.join(symbols)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "201047cb",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.235310Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.235233Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.241687Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.241353Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.008244,
|
||
"end_time": "2026-06-13T02:32:38.241919+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.233675+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Date range: 2006-01-03 to 2025-12-31\n",
|
||
"Trading days: 5031\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Overall date range\n",
|
||
"date_range = etfs.select(\n",
|
||
" [\n",
|
||
" pl.col(\"timestamp\").min().alias(\"start\"),\n",
|
||
" pl.col(\"timestamp\").max().alias(\"end\"),\n",
|
||
" pl.col(\"timestamp\").n_unique().alias(\"unique_dates\"),\n",
|
||
" ]\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"\\nDate range: {date_range['start'][0]} to {date_range['end'][0]}\")\n",
|
||
"print(f\"Trading days: {date_range['unique_dates'][0]}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "8087d7a4",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.245424Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.245345Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.254346Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.254018Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.011263,
|
||
"end_time": "2026-06-13T02:32:38.254788+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.243525+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Symbols with full coverage: 59\n",
|
||
"Symbols with partial coverage: 41\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"symbol_stats = per_asset_stats(\n",
|
||
" etfs,\n",
|
||
" time_col=\"timestamp\",\n",
|
||
" asset_col=\"symbol\",\n",
|
||
" price_col=\"close\",\n",
|
||
" volume_col=\"volume\",\n",
|
||
")\n",
|
||
"\n",
|
||
"full_start = date_range[\"start\"][0]\n",
|
||
"full_end = date_range[\"end\"][0]\n",
|
||
"\n",
|
||
"partial = symbol_stats.filter(\n",
|
||
" (pl.col(\"start\").cast(pl.Date) != pl.lit(full_start).cast(pl.Date))\n",
|
||
" | (pl.col(\"end\").cast(pl.Date) != pl.lit(full_end).cast(pl.Date))\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Symbols with full coverage: {len(symbol_stats) - len(partial)}\")\n",
|
||
"print(f\"Symbols with partial coverage: {len(partial)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c50e0d9e",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001688,
|
||
"end_time": "2026-06-13T02:32:38.258270+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.256582+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"Most ETFs predate the 2006 start of the dataset, but a sizeable minority were\n",
|
||
"launched later — visible below as ETFs whose first observation is after\n",
|
||
"2006-01-03."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "c76c71cd",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.261904Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.261811Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.265081Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.264814Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.005422,
|
||
"end_time": "2026-06-13T02:32:38.265471+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.260049+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (41, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>start</th><th>end</th><th>rows</th></tr><tr><td>str</td><td>date</td><td>date</td><td>u32</td></tr></thead><tbody><tr><td>"DBC"</td><td>2006-02-06</td><td>2025-12-31</td><td>5008</td></tr><tr><td>"XBI"</td><td>2006-02-06</td><td>2025-12-31</td><td>5008</td></tr><tr><td>"USO"</td><td>2006-04-10</td><td>2025-12-31</td><td>4964</td></tr><tr><td>"SLV"</td><td>2006-04-28</td><td>2025-12-31</td><td>4951</td></tr><tr><td>"VIG"</td><td>2006-05-02</td><td>2025-12-31</td><td>4949</td></tr><tr><td>…</td><td>…</td><td>…</td><td>…</td></tr><tr><td>"VLUE"</td><td>2013-04-18</td><td>2025-12-31</td><td>3197</td></tr><tr><td>"BNDX"</td><td>2013-06-04</td><td>2025-12-31</td><td>3165</td></tr><tr><td>"QUAL"</td><td>2013-07-18</td><td>2025-12-31</td><td>3134</td></tr><tr><td>"XLRE"</td><td>2015-10-08</td><td>2025-12-31</td><td>2573</td></tr><tr><td>"XLC"</td><td>2018-06-19</td><td>2025-12-31</td><td>1895</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (41, 4)\n",
|
||
"┌────────┬────────────┬────────────┬──────┐\n",
|
||
"│ symbol ┆ start ┆ end ┆ rows │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ date ┆ date ┆ u32 │\n",
|
||
"╞════════╪════════════╪════════════╪══════╡\n",
|
||
"│ DBC ┆ 2006-02-06 ┆ 2025-12-31 ┆ 5008 │\n",
|
||
"│ XBI ┆ 2006-02-06 ┆ 2025-12-31 ┆ 5008 │\n",
|
||
"│ USO ┆ 2006-04-10 ┆ 2025-12-31 ┆ 4964 │\n",
|
||
"│ SLV ┆ 2006-04-28 ┆ 2025-12-31 ┆ 4951 │\n",
|
||
"│ VIG ┆ 2006-05-02 ┆ 2025-12-31 ┆ 4949 │\n",
|
||
"│ … ┆ … ┆ … ┆ … │\n",
|
||
"│ VLUE ┆ 2013-04-18 ┆ 2025-12-31 ┆ 3197 │\n",
|
||
"│ BNDX ┆ 2013-06-04 ┆ 2025-12-31 ┆ 3165 │\n",
|
||
"│ QUAL ┆ 2013-07-18 ┆ 2025-12-31 ┆ 3134 │\n",
|
||
"│ XLRE ┆ 2015-10-08 ┆ 2025-12-31 ┆ 2573 │\n",
|
||
"│ XLC ┆ 2018-06-19 ┆ 2025-12-31 ┆ 1895 │\n",
|
||
"└────────┴────────────┴────────────┴──────┘"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"partial.select([\"symbol\", \"start\", \"end\", \"rows\"]).sort(\"start\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ed192a9f",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001226,
|
||
"end_time": "2026-06-13T02:32:38.268094+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.266868+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3. ETF Categories\n",
|
||
"\n",
|
||
"Six categories cover 50 ETFs that span the major asset-class buckets used by\n",
|
||
"the rotation case study. The full universe contains 100 ETFs; the remaining\n",
|
||
"50 are kept as candidates for the universe-construction work in Chapter 6."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "0d9a61d4",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.271091Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.271011Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.273401Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.273127Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004319,
|
||
"end_time": "2026-06-13T02:32:38.273675+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.269356+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== ETF Categories ===\n",
|
||
" US Equity Broad: 7/7 ETFs\n",
|
||
" US Sectors: 11/11 ETFs\n",
|
||
" International: 12/12 ETFs\n",
|
||
" Fixed Income: 10/10 ETFs\n",
|
||
" Commodities: 6/6 ETFs\n",
|
||
" Specialty: 4/4 ETFs\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"ETF_CATEGORIES = {\n",
|
||
" \"US Equity Broad\": [\"SPY\", \"QQQ\", \"IWM\", \"DIA\", \"VTI\", \"MDY\", \"IJR\"],\n",
|
||
" \"US Sectors\": [\"XLB\", \"XLE\", \"XLF\", \"XLI\", \"XLK\", \"XLP\", \"XLU\", \"XLV\", \"XLY\", \"VNQ\", \"IYR\"],\n",
|
||
" \"International\": [\n",
|
||
" \"EFA\",\n",
|
||
" \"EEM\",\n",
|
||
" \"VEA\",\n",
|
||
" \"VWO\",\n",
|
||
" \"EWJ\",\n",
|
||
" \"EWG\",\n",
|
||
" \"FXI\",\n",
|
||
" \"EWZ\",\n",
|
||
" \"EWY\",\n",
|
||
" \"EWC\",\n",
|
||
" \"EWA\",\n",
|
||
" \"ACWI\",\n",
|
||
" ],\n",
|
||
" \"Fixed Income\": [\"AGG\", \"BND\", \"TLT\", \"IEF\", \"SHY\", \"LQD\", \"HYG\", \"TIP\", \"EMB\", \"MUB\"],\n",
|
||
" \"Commodities\": [\"GLD\", \"SLV\", \"USO\", \"UNG\", \"DBC\", \"GSG\"],\n",
|
||
" \"Specialty\": [\"IBB\", \"SMH\", \"KRE\", \"OIH\"],\n",
|
||
"}\n",
|
||
"\n",
|
||
"print(\"=== ETF Categories ===\")\n",
|
||
"for category, tickers in ETF_CATEGORIES.items():\n",
|
||
" available = [t for t in tickers if t in symbols]\n",
|
||
" print(f\" {category}: {len(available)}/{len(tickers)} ETFs\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "165f377c",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001256,
|
||
"end_time": "2026-06-13T02:32:38.276236+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.274980+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 4. Data Quality"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "1e5d1e6c",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.279219Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.279146Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.281052Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.280808Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.003812,
|
||
"end_time": "2026-06-13T02:32:38.281332+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.277520+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Data Quality ===\n",
|
||
"Total null values: 0\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check for nulls\n",
|
||
"null_counts = etfs.null_count()\n",
|
||
"total_nulls = null_counts.sum_horizontal()[0]\n",
|
||
"print(\"=== Data Quality ===\")\n",
|
||
"print(f\"Total null values: {total_nulls}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "d236a7ab",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.284370Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.284308Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.286503Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.286256Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004056,
|
||
"end_time": "2026-06-13T02:32:38.286766+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.282710+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Zero volume rows: 158 (0.03%)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check for zero volume days\n",
|
||
"zero_volume = etfs.filter(pl.col(\"volume\") == 0)\n",
|
||
"print(f\"Zero volume rows: {len(zero_volume)} ({100 * len(zero_volume) / len(etfs):.2f}%)\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "ce9749f5",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.289813Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.289752Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.296554Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.296192Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.008758,
|
||
"end_time": "2026-06-13T02:32:38.296873+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.288115+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"OHLC Invariants:\n",
|
||
" [OK] high_gte_low: 100.00%\n",
|
||
" [OK] high_gte_open: 100.00%\n",
|
||
" [WARN] high_gte_close: 99.90%\n",
|
||
" [OK] low_lte_open: 100.00%\n",
|
||
" [WARN] low_lte_close: 99.94%\n",
|
||
" [OK] volume_non_negative: 100.00%\n",
|
||
"\n",
|
||
"Total OHLC violations: 473\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# OHLC invariants\n",
|
||
"invariants = check_ohlc_invariants(etfs)\n",
|
||
"print(\"\\nOHLC Invariants:\")\n",
|
||
"for row in invariants.iter_rows(named=True):\n",
|
||
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
|
||
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")\n",
|
||
"\n",
|
||
"# Count violations\n",
|
||
"violations = etfs.filter(\n",
|
||
" (pl.col(\"high\") < pl.col(\"low\"))\n",
|
||
" | (pl.col(\"high\") < pl.col(\"open\"))\n",
|
||
" | (pl.col(\"high\") < pl.col(\"close\"))\n",
|
||
")\n",
|
||
"print(f\"\\nTotal OHLC violations: {len(violations)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "674db465",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001674,
|
||
"end_time": "2026-06-13T02:32:38.300172+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.298498+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"Violations occur where `high < close` or `low > close` after price adjustment.\n",
|
||
"These arise from floating-point/rounding precision in the stored adjusted OHLC\n",
|
||
"values (the same cumulative ratio is applied across all four fields, but small\n",
|
||
"per-field rounding can break the strict invariants the raw quotes satisfied).\n",
|
||
"At a tenth of a percent of rows they are immaterial for return and feature\n",
|
||
"calculations but worth being aware of when computing intraday range statistics."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ce26e633",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001622,
|
||
"end_time": "2026-06-13T02:32:38.303651+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.302029+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 5. Volume and Liquidity Distribution\n",
|
||
"\n",
|
||
"Understanding volume distributions helps identify liquidity constraints for trading."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "ef8b4863",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.307444Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.307351Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.325998Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.325481Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.021519,
|
||
"end_time": "2026-06-13T02:32:38.326726+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.305207+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Volume Distribution (Top 10 Most Liquid) ===\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (10, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>avg_volume</th><th>median_volume</th><th>volume_std</th></tr><tr><td>str</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>"SPY"</td><td>1.2621e8</td><td>9.55605e7</td><td>9.1133e7</td></tr><tr><td>"XLF"</td><td>7.3482e7</td><td>5.34418e7</td><td>6.7509e7</td></tr><tr><td>"QQQ"</td><td>6.3756e7</td><td>4.64661e7</td><td>5.1702e7</td></tr><tr><td>"EEM"</td><td>5.3074e7</td><td>4.80978e7</td><td>2.9103e7</td></tr><tr><td>"IWM"</td><td>4.3071e7</td><td>3.48317e7</td><td>2.8307e7</td></tr><tr><td>"XLE"</td><td>4.1028e7</td><td>3.5416e7</td><td>2.2452e7</td></tr><tr><td>"FXI"</td><td>2.4401e7</td><td>2.10447e7</td><td>1.7195e7</td></tr><tr><td>"XLU"</td><td>2.2636e7</td><td>2.05024e7</td><td>1.3144e7</td></tr><tr><td>"EWZ"</td><td>2.0082e7</td><td>1.85157e7</td><td>1.0813e7</td></tr><tr><td>"EFA"</td><td>1.9718e7</td><td>1.76512e7</td><td>1.1364e7</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (10, 4)\n",
|
||
"┌────────┬────────────┬───────────────┬────────────┐\n",
|
||
"│ symbol ┆ avg_volume ┆ median_volume ┆ volume_std │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ f64 ┆ f64 ┆ f64 │\n",
|
||
"╞════════╪════════════╪═══════════════╪════════════╡\n",
|
||
"│ SPY ┆ 1.2621e8 ┆ 9.55605e7 ┆ 9.1133e7 │\n",
|
||
"│ XLF ┆ 7.3482e7 ┆ 5.34418e7 ┆ 6.7509e7 │\n",
|
||
"│ QQQ ┆ 6.3756e7 ┆ 4.64661e7 ┆ 5.1702e7 │\n",
|
||
"│ EEM ┆ 5.3074e7 ┆ 4.80978e7 ┆ 2.9103e7 │\n",
|
||
"│ IWM ┆ 4.3071e7 ┆ 3.48317e7 ┆ 2.8307e7 │\n",
|
||
"│ XLE ┆ 4.1028e7 ┆ 3.5416e7 ┆ 2.2452e7 │\n",
|
||
"│ FXI ┆ 2.4401e7 ┆ 2.10447e7 ┆ 1.7195e7 │\n",
|
||
"│ XLU ┆ 2.2636e7 ┆ 2.05024e7 ┆ 1.3144e7 │\n",
|
||
"│ EWZ ┆ 2.0082e7 ┆ 1.85157e7 ┆ 1.0813e7 │\n",
|
||
"│ EFA ┆ 1.9718e7 ┆ 1.76512e7 ┆ 1.1364e7 │\n",
|
||
"└────────┴────────────┴───────────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Average daily volume by ETF\n",
|
||
"volume_stats = (\n",
|
||
" etfs.group_by(\"symbol\")\n",
|
||
" .agg(\n",
|
||
" [\n",
|
||
" pl.col(\"volume\").mean().alias(\"avg_volume\"),\n",
|
||
" pl.col(\"volume\").median().alias(\"median_volume\"),\n",
|
||
" pl.col(\"volume\").std().alias(\"volume_std\"),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
" .sort(\"avg_volume\", descending=True)\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"=== Volume Distribution (Top 10 Most Liquid) ===\")\n",
|
||
"volume_stats.head(10)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "b73ef7bb",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.333145Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.333036Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.343625Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.342855Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.014076,
|
||
"end_time": "2026-06-13T02:32:38.344116+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.330040+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"=== Average Daily Volume by Category ===\n",
|
||
" US Equity Broad: 35,557,363 shares/day\n",
|
||
" US Sectors: 20,205,419 shares/day\n",
|
||
" International: 13,274,392 shares/day\n",
|
||
" Fixed Income: 5,436,389 shares/day\n",
|
||
" Commodities: 5,550,398 shares/day\n",
|
||
" Specialty: 5,913,347 shares/day\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Volume by category\n",
|
||
"print(\"\\n=== Average Daily Volume by Category ===\")\n",
|
||
"for category, tickers in ETF_CATEGORIES.items():\n",
|
||
" category_vol = (\n",
|
||
" etfs.filter(pl.col(\"symbol\").is_in(tickers)).select(pl.col(\"volume\").mean()).item()\n",
|
||
" )\n",
|
||
" if category_vol is not None:\n",
|
||
" print(f\" {category}: {category_vol:,.0f} shares/day\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f841d90f",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.004144,
|
||
"end_time": "2026-06-13T02:32:38.351549+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.347405+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 6. Price Distribution\n",
|
||
"\n",
|
||
"Price levels across the ETF universe span a wide range."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "b24eb354",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.357777Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.357649Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.364057Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.363247Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.010441,
|
||
"end_time": "2026-06-13T02:32:38.365231+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.354790+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Price Distribution (Latest Date) ===\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (1, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>min_price</th><th>max_price</th><th>median_price</th><th>mean_price</th></tr><tr><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>12.26</td><td>680.062744</td><td>86.874882</td><td>127.252057</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (1, 4)\n",
|
||
"┌───────────┬────────────┬──────────────┬────────────┐\n",
|
||
"│ min_price ┆ max_price ┆ median_price ┆ mean_price │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ f64 ┆ f64 ┆ f64 ┆ f64 │\n",
|
||
"╞═══════════╪════════════╪══════════════╪════════════╡\n",
|
||
"│ 12.26 ┆ 680.062744 ┆ 86.874882 ┆ 127.252057 │\n",
|
||
"└───────────┴────────────┴──────────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Current price levels (most recent date)\n",
|
||
"latest = etfs.filter(pl.col(\"timestamp\") == etfs[\"timestamp\"].max())\n",
|
||
"price_dist = latest.select(\n",
|
||
" [\n",
|
||
" pl.col(\"close\").min().alias(\"min_price\"),\n",
|
||
" pl.col(\"close\").max().alias(\"max_price\"),\n",
|
||
" pl.col(\"close\").median().alias(\"median_price\"),\n",
|
||
" pl.col(\"close\").mean().alias(\"mean_price\"),\n",
|
||
" ]\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"=== Price Distribution (Latest Date) ===\")\n",
|
||
"price_dist"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "4ccd4664",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.372027Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.371872Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.377438Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.376927Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.009128,
|
||
"end_time": "2026-06-13T02:32:38.378030+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.368902+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Price Range by Category ===\n",
|
||
" US Equity Broad: $119.99 - $680.06\n",
|
||
" US Sectors: $42.39 - $154.69\n",
|
||
" International: $26.19 - $141.49\n",
|
||
" Fixed Income: $73.35 - $109.91\n",
|
||
" Commodities: $12.26 - $396.31\n",
|
||
" Specialty: $64.44 - $360.13\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Price range by category\n",
|
||
"print(\"=== Price Range by Category ===\")\n",
|
||
"for category, tickers in ETF_CATEGORIES.items():\n",
|
||
" category_prices = latest.filter(pl.col(\"symbol\").is_in(tickers))\n",
|
||
" min_p = category_prices[\"close\"].min()\n",
|
||
" max_p = category_prices[\"close\"].max()\n",
|
||
" print(f\" {category}: ${min_p:.2f} - ${max_p:.2f}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a794efcd",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001909,
|
||
"end_time": "2026-06-13T02:32:38.382562+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.380653+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 7. Loading by Symbol or via the ml4t-data Library\n",
|
||
"\n",
|
||
"`load_etfs(symbols=[...])` filters the panel to a subset; `ETFDataManager` is\n",
|
||
"the config-driven entry point used by the production download/refresh workflow\n",
|
||
"in `data/etfs/market/`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "85af2a36",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.386151Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.386078Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.391436Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.391099Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.007645,
|
||
"end_time": "2026-06-13T02:32:38.391796+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.384151+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"SPY via loader: (5031, 7)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"spy = load_etfs(symbols=[\"SPY\"])\n",
|
||
"print(f\"SPY via loader: {spy.shape}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "46f5bf86",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:38.395667Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:38.395577Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:38.401162Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:38.400802Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.008133,
|
||
"end_time": "2026-06-13T02:32:38.401604+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.393471+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"ETFDataManager loaded from data/etfs/market/config.yaml\n",
|
||
" Provider: yahoo\n",
|
||
" Date range: 2006-01-01 to 2025-12-31\n",
|
||
" Configured symbols: 100 across 9 categories\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"config_path = REPO_ROOT / \"data\" / \"etfs\" / \"market\" / \"config.yaml\"\n",
|
||
"etf_mgr = ETFDataManager.from_config(str(config_path))\n",
|
||
"configured = sum(len(group[\"symbols\"]) for group in etf_mgr.config.tickers.values())\n",
|
||
"print(f\"ETFDataManager loaded from {config_path}\")\n",
|
||
"print(f\" Provider: {etf_mgr.config.provider}\")\n",
|
||
"print(f\" Date range: {etf_mgr.config.start} to {etf_mgr.config.end}\")\n",
|
||
"print(f\" Configured symbols: {configured} across {len(etf_mgr.config.tickers)} categories\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e35b4471",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.00157,
|
||
"end_time": "2026-06-13T02:32:38.405205+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:38.403635+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"1. **Pre-adjusted prices**: Yahoo Finance returns split- and dividend-adjusted\n",
|
||
" OHLC. The `close` column is the adjusted close — return calculations need\n",
|
||
" no further adjustment.\n",
|
||
"2. **Coverage**: 100 ETFs across daily history from 2006-01-03 to 2025-12-31\n",
|
||
" (5,031 trading days), with 59 ETFs spanning the full window and 41 starting\n",
|
||
" later as new products were launched.\n",
|
||
"3. **Categorization**: Six categories (US Equity Broad, US Sectors,\n",
|
||
" International, Fixed Income, Commodities, Specialty) cover 50 of the 100\n",
|
||
" ETFs and form the candidate set for the rotation case study; the other 50\n",
|
||
" are reserved for the universe-construction work in Chapter 6.\n",
|
||
"4. **Mostly clean data**: Zero nulls and 473 OHLC violations\n",
|
||
" (`high < close` or `low > close`) — about 0.1% of rows, immaterial for\n",
|
||
" return and feature calculations.\n",
|
||
"5. **Liquidity variation**: SPY averages 126M shares/day; the Commodities\n",
|
||
" bucket averages 5.5M — a 23× difference that matters for transaction-cost\n",
|
||
" modeling in later chapters.\n",
|
||
"\n",
|
||
"### Next Steps\n",
|
||
"\n",
|
||
"- **§2.6 / `13_data_quality_framework`**: Systematic data-quality checks across\n",
|
||
" the seven datasets.\n",
|
||
"- **§2.7 / `15_survivorship_bias_detection`**: Survivorship and selection bias\n",
|
||
" in equity panels — a contrast to the ETF universe shown here.\n",
|
||
"- **Chapter 6**: Universe construction filters this 100-ETF candidate pool down\n",
|
||
" to the trading universe used by the ETF rotation case study."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"jupytext": {
|
||
"cell_metadata_filter": "tags,-all"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 3.442648,
|
||
"end_time": "2026-06-13T02:32:39.222594+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "02_financial_data_universe/03_etfs_eda.ipynb",
|
||
"output_path": "02_financial_data_universe/03_etfs_eda.ipynb",
|
||
"parameters": {},
|
||
"start_time": "2026-06-13T02:32:35.779946+00:00",
|
||
"version": "2.7.0"
|
||
},
|
||
"ml4t_provenance": {
|
||
"source_py_blob": "eedb1c2bb6f3a001ce2609340779b93fa62a2309",
|
||
"executed_at": "2026-06-13T02:32:39.431948+00:00",
|
||
"executor": "cpu-local",
|
||
"production": true,
|
||
"parameters": {},
|
||
"notes": "executed-state finalization batch (2026-06-13 run)"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|