wehub-resource-sync
790 lines
26 KiB
Plaintext
790 lines
26 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5ad079dc",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.005414,
|
|
"end_time": "2026-06-13T02:32:24.544112+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:24.538698+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"# US Equities — Exploratory Data Analysis\n",
|
|
"\n",
|
|
"**Docker image**: `ml4t`\n",
|
|
"\n",
|
|
"**Purpose**: Profile the Wiki Prices dataset of US equity OHLCV history and confirm\n",
|
|
"the inactive-symbol coverage that makes it usable for survivorship-bias-free\n",
|
|
"backtests.\n",
|
|
"\n",
|
|
"**Learning objectives**:\n",
|
|
"\n",
|
|
"- Load the equity panel via `data.load_us_equities` and inspect its canonical schema.\n",
|
|
"- Distinguish raw and split/dividend-adjusted price columns.\n",
|
|
"- Quantify the share of symbols that stop trading before the dataset end date.\n",
|
|
"- Check OHLC invariants and null rates across the full panel.\n",
|
|
"\n",
|
|
"**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — Equities).\n",
|
|
"\n",
|
|
"**Prerequisites**: `data` package on `PYTHONPATH`; Wiki Prices parquet present at\n",
|
|
"`ML4T_DATA_PATH/equities/market/us_equities/`. Run\n",
|
|
"`python data/equities/market/us_equities/download.py` if missing."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "fcba0234",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:24.551754Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:24.551580Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:25.784522Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:25.784148Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 1.237163,
|
|
"end_time": "2026-06-13T02:32:25.785024+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:24.547861+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\"\"\"US Equities — Exploratory data analysis of the Wiki Prices dataset.\"\"\"\n",
|
|
"\n",
|
|
"import polars as pl\n",
|
|
"\n",
|
|
"from data import load_us_equities\n",
|
|
"from utils.data_quality import check_ohlc_invariants"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "e2e6b56b",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:25.788403Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:25.788307Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:25.789984Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:25.789599Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.004284,
|
|
"end_time": "2026-06-13T02:32:25.790350+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:25.786066+00:00",
|
|
"status": "completed"
|
|
},
|
|
"tags": [
|
|
"parameters"
|
|
]
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Production defaults — Papermill injects overrides for CI\n",
|
|
"MAX_SYMBOLS = 0 # 0 = all"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a05e8378",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000762,
|
|
"end_time": "2026-06-13T02:32:25.791966+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:25.791204+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## 1. Load and Inspect"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "18d4c304",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:25.793964Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:25.793895Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.253321Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.252871Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.460993,
|
|
"end_time": "2026-06-13T02:32:26.253751+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:25.792758+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== Wiki Prices Dataset ===\n",
|
|
"Shape: (15389314, 14)\n",
|
|
"Columns: ['symbol', 'timestamp', 'open', 'high', 'low', 'close', 'volume', 'ex-dividend', 'split_ratio', 'adj_open', 'adj_high', 'adj_low', 'adj_close', 'adj_volume']\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"wiki = load_us_equities()\n",
|
|
"\n",
|
|
"print(\"=== Wiki Prices Dataset ===\")\n",
|
|
"print(f\"Shape: {wiki.shape}\")\n",
|
|
"print(f\"Columns: {wiki.columns}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "f3ce7b2d",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.256226Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.256110Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.257967Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.257731Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.003422,
|
|
"end_time": "2026-06-13T02:32:26.258212+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.254790+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Schema:\n",
|
|
" symbol: String\n",
|
|
" timestamp: Date\n",
|
|
" open: Float64\n",
|
|
" high: Float64\n",
|
|
" low: Float64\n",
|
|
" close: Float64\n",
|
|
" volume: Float64\n",
|
|
" ex-dividend: Float64\n",
|
|
" split_ratio: Float64\n",
|
|
" adj_open: Float64\n",
|
|
" adj_high: Float64\n",
|
|
" adj_low: Float64\n",
|
|
" adj_close: Float64\n",
|
|
" adj_volume: Float64\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Schema overview\n",
|
|
"print(\"\\nSchema:\")\n",
|
|
"for col, dtype in wiki.schema.items():\n",
|
|
" print(f\" {col}: {dtype}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7cf8fb2a",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000835,
|
|
"end_time": "2026-06-13T02:32:26.259980+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.259145+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"### Adjusted vs Raw Prices\n",
|
|
"\n",
|
|
"**Important**: This dataset contains both raw and adjusted prices.\n",
|
|
"\n",
|
|
"| Column Type | Examples | Use Case |\n",
|
|
"|-------------|----------|----------|\n",
|
|
"| Raw | `open`, `high`, `low`, `close`, `volume` | Historical analysis at actual prices |\n",
|
|
"| Adjusted | `adj_open`, `adj_high`, `adj_low`, `adj_close`, `adj_volume` | **Backtesting** (handles splits/dividends) |\n",
|
|
"\n",
|
|
"Always use `adj_*` columns for return calculations and strategy backtesting."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d550e3b",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000832,
|
|
"end_time": "2026-06-13T02:32:26.261676+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.260844+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## 2. Coverage Summary"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "3007d3af",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.264176Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.264095Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.348271Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.347931Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.086007,
|
|
"end_time": "2026-06-13T02:32:26.348675+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.262668+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== Coverage ===\n",
|
|
"Unique tickers: 3,199\n",
|
|
"Date range: 1962-01-02 to 2018-03-27\n",
|
|
"Total rows: 15,389,314\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Overall coverage\n",
|
|
"print(\"=== Coverage ===\")\n",
|
|
"print(f\"Unique tickers: {wiki['symbol'].n_unique():,}\")\n",
|
|
"print(f\"Date range: {wiki['timestamp'].min()} to {wiki['timestamp'].max()}\")\n",
|
|
"print(f\"Total rows: {len(wiki):,}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "07f7febc",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.351317Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.351234Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.723550Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.723172Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.374552,
|
|
"end_time": "2026-06-13T02:32:26.724359+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.349807+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Per-stock statistics\n",
|
|
"stock_stats = wiki.group_by(\"symbol\").agg(\n",
|
|
" [\n",
|
|
" pl.len().alias(\"days\"),\n",
|
|
" pl.col(\"timestamp\").min().alias(\"start\"),\n",
|
|
" pl.col(\"timestamp\").max().alias(\"end\"),\n",
|
|
" pl.col(\"adj_close\").mean().alias(\"avg_price\"),\n",
|
|
" ]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc6a8d8e",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001661,
|
|
"end_time": "2026-06-13T02:32:26.728002+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.726341+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"Per-symbol coverage distribution — number of trading days and mean adjusted price."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "eca30494",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.731023Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.730937Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.735753Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.735507Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.006674,
|
|
"end_time": "2026-06-13T02:32:26.736156+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.729482+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div><style>\n",
|
|
".dataframe > thead > tr,\n",
|
|
".dataframe > tbody > tr {\n",
|
|
" text-align: right;\n",
|
|
" white-space: pre-wrap;\n",
|
|
"}\n",
|
|
"</style>\n",
|
|
"<small>shape: (9, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>statistic</th><th>days</th><th>avg_price</th></tr><tr><td>str</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>"count"</td><td>3199.0</td><td>3199.0</td></tr><tr><td>"null_count"</td><td>0.0</td><td>0.0</td></tr><tr><td>"mean"</td><td>4810.663957</td><td>118.245432</td></tr><tr><td>"std"</td><td>2621.170846</td><td>2893.682465</td></tr><tr><td>"min"</td><td>4.0</td><td>0.669129</td></tr><tr><td>"25%"</td><td>2611.0</td><td>11.539385</td></tr><tr><td>"50%"</td><td>4974.0</td><td>18.519854</td></tr><tr><td>"75%"</td><td>6666.0</td><td>29.777315</td></tr><tr><td>"max"</td><td>14155.0</td><td>132897.234865</td></tr></tbody></table></div>"
|
|
],
|
|
"text/plain": [
|
|
"shape: (9, 3)\n",
|
|
"┌────────────┬─────────────┬───────────────┐\n",
|
|
"│ statistic ┆ days ┆ avg_price │\n",
|
|
"│ --- ┆ --- ┆ --- │\n",
|
|
"│ str ┆ f64 ┆ f64 │\n",
|
|
"╞════════════╪═════════════╪═══════════════╡\n",
|
|
"│ count ┆ 3199.0 ┆ 3199.0 │\n",
|
|
"│ null_count ┆ 0.0 ┆ 0.0 │\n",
|
|
"│ mean ┆ 4810.663957 ┆ 118.245432 │\n",
|
|
"│ std ┆ 2621.170846 ┆ 2893.682465 │\n",
|
|
"│ min ┆ 4.0 ┆ 0.669129 │\n",
|
|
"│ 25% ┆ 2611.0 ┆ 11.539385 │\n",
|
|
"│ 50% ┆ 4974.0 ┆ 18.519854 │\n",
|
|
"│ 75% ┆ 6666.0 ┆ 29.777315 │\n",
|
|
"│ max ┆ 14155.0 ┆ 132897.234865 │\n",
|
|
"└────────────┴─────────────┴───────────────┘"
|
|
]
|
|
},
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"stock_stats.select([\"days\", \"avg_price\"]).describe()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e3726365",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000927,
|
|
"end_time": "2026-06-13T02:32:26.738073+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.737146+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## 3. Survivorship Analysis\n",
|
|
"\n",
|
|
"A key feature of this dataset is that it includes stocks that ceased trading\n",
|
|
"before the dataset end date. This is critical for avoiding survivorship bias.\n",
|
|
"\n",
|
|
"**Note**: Stocks marked as \"inactive before end\" include both:\n",
|
|
"- Actually delisted companies (bankruptcy, acquisition, etc.)\n",
|
|
"- Stocks with incomplete coverage in this dataset\n",
|
|
"\n",
|
|
"The important point: these stocks are included, preventing survivorship bias."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "9dfb3c79",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.740278Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.740210Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.747184Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.746804Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.008672,
|
|
"end_time": "2026-06-13T02:32:26.747650+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.738978+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== Survivorship Analysis ===\n",
|
|
"Dataset end: 2018-03-27\n",
|
|
"Active at dataset end: 2,422 (75.7%)\n",
|
|
"Inactive before end: 777 (24.3%)\n",
|
|
"\n",
|
|
"This 24% inactive rate helps mitigate survivorship bias in backtests.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"dataset_end = wiki.select(pl.col(\"timestamp\").max()).item()\n",
|
|
"\n",
|
|
"# Identify stocks that stopped trading before dataset end\n",
|
|
"stock_stats = stock_stats.with_columns((pl.col(\"end\") < dataset_end).alias(\"inactive_before_end\"))\n",
|
|
"\n",
|
|
"n_active = stock_stats.filter(~pl.col(\"inactive_before_end\")).height\n",
|
|
"n_inactive = stock_stats.filter(pl.col(\"inactive_before_end\")).height\n",
|
|
"total = n_active + n_inactive\n",
|
|
"inactive_pct = n_inactive / total * 100\n",
|
|
"\n",
|
|
"print(\"=== Survivorship Analysis ===\")\n",
|
|
"print(f\"Dataset end: {dataset_end}\")\n",
|
|
"print(f\"Active at dataset end: {n_active:,} ({n_active / total * 100:.1f}%)\")\n",
|
|
"print(f\"Inactive before end: {n_inactive:,} ({inactive_pct:.1f}%)\")\n",
|
|
"print(f\"\\nThis {inactive_pct:.0f}% inactive rate helps mitigate survivorship bias in backtests.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "333e64d2",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.000978,
|
|
"end_time": "2026-06-13T02:32:26.749790+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.748812+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## 4. Data Quality"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "8041dec8",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.752249Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.752172Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.755615Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.755147Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.00514,
|
|
"end_time": "2026-06-13T02:32:26.755903+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.750763+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== Data Quality ===\n",
|
|
"Total null values: 1,299\n",
|
|
" open: 538 (0.0035%)\n",
|
|
" high: 55 (0.0004%)\n",
|
|
" low: 55 (0.0004%)\n",
|
|
" close: 1 (0.0000%)\n",
|
|
" split_ratio: 1 (0.0000%)\n",
|
|
" adj_open: 538 (0.0035%)\n",
|
|
" adj_high: 55 (0.0004%)\n",
|
|
" adj_low: 55 (0.0004%)\n",
|
|
" adj_close: 1 (0.0000%)\n",
|
|
"\n",
|
|
"Null rate: 0.0006%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Check for nulls across columns\n",
|
|
"null_counts = wiki.null_count()\n",
|
|
"total_nulls = null_counts.sum_horizontal()[0]\n",
|
|
"print(\"=== Data Quality ===\")\n",
|
|
"print(f\"Total null values: {total_nulls:,}\")\n",
|
|
"\n",
|
|
"# Show per-column breakdown (only columns with nulls)\n",
|
|
"for col in null_counts.columns:\n",
|
|
" val = null_counts[col][0]\n",
|
|
" if val > 0:\n",
|
|
" print(f\" {col}: {val:,} ({val / len(wiki) * 100:.4f}%)\")\n",
|
|
"\n",
|
|
"print(f\"\\nNull rate: {total_nulls / (len(wiki) * len(wiki.columns)) * 100:.4f}%\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "ac521b11",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.759001Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.758834Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:26.986021Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:26.985683Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.229428,
|
|
"end_time": "2026-06-13T02:32:26.986481+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.757053+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"OHLC Invariants (adjusted prices):\n",
|
|
" [OK] high_gte_low: 100.00%\n",
|
|
" [OK] high_gte_open: 100.00%\n",
|
|
" [OK] high_gte_close: 100.00%\n",
|
|
" [OK] low_lte_open: 100.00%\n",
|
|
" [OK] low_lte_close: 100.00%\n",
|
|
" [OK] volume_non_negative: 100.00%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# OHLC invariants on adjusted prices\n",
|
|
"invariants = check_ohlc_invariants(\n",
|
|
" wiki,\n",
|
|
" open_col=\"adj_open\",\n",
|
|
" high_col=\"adj_high\",\n",
|
|
" low_col=\"adj_low\",\n",
|
|
" close_col=\"adj_close\",\n",
|
|
" volume_col=\"adj_volume\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\"\\nOHLC Invariants (adjusted prices):\")\n",
|
|
"for row in invariants.iter_rows(named=True):\n",
|
|
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
|
|
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f11b91fe",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001525,
|
|
"end_time": "2026-06-13T02:32:26.989389+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.987864+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## 5. Example: Single Stock"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "911af454",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:26.992105Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:26.992013Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:27.006055Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:27.005651Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.015918,
|
|
"end_time": "2026-06-13T02:32:27.006360+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:26.990442+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=== AAPL Example ===\n",
|
|
"Trading days: 9,400\n",
|
|
"Date range: 1980-12-12 to 2018-03-27\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# AAPL as example\n",
|
|
"aapl = wiki.filter(pl.col(\"symbol\") == \"AAPL\").sort(\"timestamp\")\n",
|
|
"\n",
|
|
"print(\"=== AAPL Example ===\")\n",
|
|
"print(f\"Trading days: {len(aapl):,}\")\n",
|
|
"print(f\"Date range: {aapl['timestamp'].min()} to {aapl['timestamp'].max()}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "84290f70",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001127,
|
|
"end_time": "2026-06-13T02:32:27.008827+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:27.007700+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"Five most recent trading days for AAPL on adjusted prices."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "83c34cd1",
|
|
"metadata": {
|
|
"execution": {
|
|
"iopub.execute_input": "2026-06-13T02:32:27.011649Z",
|
|
"iopub.status.busy": "2026-06-13T02:32:27.011526Z",
|
|
"iopub.status.idle": "2026-06-13T02:32:27.014436Z",
|
|
"shell.execute_reply": "2026-06-13T02:32:27.014207Z"
|
|
},
|
|
"papermill": {
|
|
"duration": 0.00492,
|
|
"end_time": "2026-06-13T02:32:27.014850+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:27.009930+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div><style>\n",
|
|
".dataframe > thead > tr,\n",
|
|
".dataframe > tbody > tr {\n",
|
|
" text-align: right;\n",
|
|
" white-space: pre-wrap;\n",
|
|
"}\n",
|
|
"</style>\n",
|
|
"<small>shape: (5, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>adj_open</th><th>adj_high</th><th>adj_low</th><th>adj_close</th><th>adj_volume</th></tr><tr><td>date</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>2018-03-21</td><td>175.04</td><td>175.09</td><td>171.26</td><td>171.27</td><td>3.5247358e7</td></tr><tr><td>2018-03-22</td><td>170.0</td><td>172.68</td><td>168.6</td><td>168.845</td><td>4.1051076e7</td></tr><tr><td>2018-03-23</td><td>168.39</td><td>169.92</td><td>164.94</td><td>164.94</td><td>4.0248954e7</td></tr><tr><td>2018-03-26</td><td>168.07</td><td>173.1</td><td>166.44</td><td>172.77</td><td>3.6272617e7</td></tr><tr><td>2018-03-27</td><td>173.68</td><td>175.15</td><td>166.92</td><td>168.34</td><td>3.8962839e7</td></tr></tbody></table></div>"
|
|
],
|
|
"text/plain": [
|
|
"shape: (5, 6)\n",
|
|
"┌────────────┬──────────┬──────────┬─────────┬───────────┬─────────────┐\n",
|
|
"│ timestamp ┆ adj_open ┆ adj_high ┆ adj_low ┆ adj_close ┆ adj_volume │\n",
|
|
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
|
"│ date ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │\n",
|
|
"╞════════════╪══════════╪══════════╪═════════╪═══════════╪═════════════╡\n",
|
|
"│ 2018-03-21 ┆ 175.04 ┆ 175.09 ┆ 171.26 ┆ 171.27 ┆ 3.5247358e7 │\n",
|
|
"│ 2018-03-22 ┆ 170.0 ┆ 172.68 ┆ 168.6 ┆ 168.845 ┆ 4.1051076e7 │\n",
|
|
"│ 2018-03-23 ┆ 168.39 ┆ 169.92 ┆ 164.94 ┆ 164.94 ┆ 4.0248954e7 │\n",
|
|
"│ 2018-03-26 ┆ 168.07 ┆ 173.1 ┆ 166.44 ┆ 172.77 ┆ 3.6272617e7 │\n",
|
|
"│ 2018-03-27 ┆ 173.68 ┆ 175.15 ┆ 166.92 ┆ 168.34 ┆ 3.8962839e7 │\n",
|
|
"└────────────┴──────────┴──────────┴─────────┴───────────┴─────────────┘"
|
|
]
|
|
},
|
|
"execution_count": 12,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"aapl.select([\"timestamp\", \"adj_open\", \"adj_high\", \"adj_low\", \"adj_close\", \"adj_volume\"]).tail(5)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cc275e7f",
|
|
"metadata": {
|
|
"papermill": {
|
|
"duration": 0.001054,
|
|
"end_time": "2026-06-13T02:32:27.017022+00:00",
|
|
"exception": false,
|
|
"start_time": "2026-06-13T02:32:27.015968+00:00",
|
|
"status": "completed"
|
|
}
|
|
},
|
|
"source": [
|
|
"## Key Takeaways\n",
|
|
"\n",
|
|
"1. **Mitigates survivorship bias**: 24% of symbols stop trading before the\n",
|
|
" dataset end — including these inactive tickers is what makes the panel\n",
|
|
" usable for unbiased backtests.\n",
|
|
"2. **Always use adjusted prices for returns**: the `adj_*` columns absorb\n",
|
|
" splits and dividends; the raw `open/high/low/close/volume` columns remain\n",
|
|
" available for analyses that need actual traded levels.\n",
|
|
"3. **Long history, broad cross-section**: 3,199 symbols with a max single-symbol\n",
|
|
" span of ~56 years (14,155 trading days), covering multiple market regimes.\n",
|
|
"4. **Clean panel**: null rate is 0.0006% of values and the six adjusted-price\n",
|
|
" OHLC invariants hold on 100% of rows.\n",
|
|
"\n",
|
|
"**Next**: `02_corporate_actions` validates the adjustment factors that\n",
|
|
"produce the `adj_*` columns. **Book reference**: §2.2 (Equities)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"cell_metadata_filter": "tags,-all"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.14.3"
|
|
},
|
|
"papermill": {
|
|
"default_parameters": {},
|
|
"duration": 3.93752,
|
|
"end_time": "2026-06-13T02:32:27.835362+00:00",
|
|
"environment_variables": {},
|
|
"exception": null,
|
|
"input_path": "02_financial_data_universe/01_us_equities_eda.ipynb",
|
|
"output_path": "02_financial_data_universe/01_us_equities_eda.ipynb",
|
|
"parameters": {},
|
|
"start_time": "2026-06-13T02:32:23.897842+00:00",
|
|
"version": "2.7.0"
|
|
},
|
|
"ml4t_provenance": {
|
|
"source_py_blob": "f34a592b987b8854cd8e3015baf91ad8f580fc4e",
|
|
"executed_at": "2026-06-13T02:32:28.047302+00:00",
|
|
"executor": "cpu-local",
|
|
"production": true,
|
|
"parameters": {},
|
|
"notes": "executed-state finalization batch (2026-06-13 run)"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|