wehub-resource-sync
866 lines
31 KiB
Plaintext
866 lines
31 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fb07336e",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.006471,
|
||
"end_time": "2026-06-13T02:32:41.536804+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:41.530333+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"# CME Futures — Exploratory Data Analysis\n",
|
||
"\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"\n",
|
||
"**Purpose**: Profile the 30-product CME futures dataset (Databento, hourly,\n",
|
||
"2011–2025) and surface the contract / continuous structure that downstream\n",
|
||
"notebooks rely on.\n",
|
||
"\n",
|
||
"**Learning objectives**:\n",
|
||
"\n",
|
||
"- Understand the futures data hierarchy: product → contract → continuous series.\n",
|
||
"- Load individual contracts and continuous (rolled) series via\n",
|
||
" `load_cme_futures` and inspect the canonical `timestamp` / `product`\n",
|
||
" schema.\n",
|
||
"- Summarize per-product coverage and group products by asset class.\n",
|
||
"- Verify OHLC invariants on a representative continuous series.\n",
|
||
"\n",
|
||
"**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — Futures).\n",
|
||
"\n",
|
||
"**Prerequisites**: `data` package on `PYTHONPATH`; CME parquet present at\n",
|
||
"`ML4T_DATA_PATH/futures/`. Run `python data/futures/market/download.py` if\n",
|
||
"missing (Databento API key required)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "1830247e",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:41.541085Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:41.540762Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.760904Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.760453Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 1.222807,
|
||
"end_time": "2026-06-13T02:32:42.761451+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:41.538644+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\"\"\"CME Futures — Exploratory data analysis of the futures universe.\"\"\"\n",
|
||
"\n",
|
||
"import polars as pl\n",
|
||
"\n",
|
||
"from data import list_cme_products, load_cme_futures\n",
|
||
"from utils.data_quality import check_ohlc_invariants"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "bdf60381",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.765255Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.765168Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.767341Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.766636Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.005182,
|
||
"end_time": "2026-06-13T02:32:42.767745+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.762563+00:00",
|
||
"status": "completed"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Production defaults — Papermill injects overrides for CI\n",
|
||
"MAX_SYMBOLS = 0 # 0 = all"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "70f4bc2e",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000844,
|
||
"end_time": "2026-06-13T02:32:42.769552+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.768708+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 1. Configuration and Data Discovery\n",
|
||
"\n",
|
||
"The futures data uses a Hive-partitioned structure for efficient queries:\n",
|
||
"- `futures/continuous/product={PRODUCT}/`: Volume-rolled continuous contracts (hourly)\n",
|
||
"- `futures/individual/{PRODUCT}/data.parquet`: Individual contract price data\n",
|
||
"\n",
|
||
"We use `load_cme_futures()` for proper data loading with partition pruning."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "246bdafd",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.771820Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.771760Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.774699Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.774121Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004585,
|
||
"end_time": "2026-06-13T02:32:42.775020+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.770435+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Futures Universe ===\n",
|
||
"Available products: 30\n",
|
||
"\n",
|
||
"Products: 6A, 6B, 6C, 6E, 6J, 6S, CL, ES, GC, GF, HE, HG, HO, LE, NG, NQ, PL, RB, RTY, SI, YM, ZB, ZC, ZF, ZL, ZM, ZN, ZS, ZT, ZW\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Discover available products via the CME loader\n",
|
||
"products = list_cme_products()\n",
|
||
"\n",
|
||
"print(\"=== Futures Universe ===\")\n",
|
||
"print(f\"Available products: {len(products)}\")\n",
|
||
"print(f\"\\nProducts: {', '.join(products)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "893ba4a8",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000878,
|
||
"end_time": "2026-06-13T02:32:42.776865+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.775987+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"Map each product to a coarse asset-class bucket. The mapping covers every\n",
|
||
"product in the dataset; downstream chapters use the same bucket labels for\n",
|
||
"universe-construction and risk reporting."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "19ed20cd",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.779264Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.779153Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.781984Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.781419Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004493,
|
||
"end_time": "2026-06-13T02:32:42.782271+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.777778+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"ASSET_CLASS_MAP = {\n",
|
||
" \"ES\": \"Equity Index\",\n",
|
||
" \"NQ\": \"Equity Index\",\n",
|
||
" \"YM\": \"Equity Index\",\n",
|
||
" \"RTY\": \"Equity Index\",\n",
|
||
" \"ZN\": \"Rates\",\n",
|
||
" \"ZB\": \"Rates\",\n",
|
||
" \"ZF\": \"Rates\",\n",
|
||
" \"ZT\": \"Rates\",\n",
|
||
" \"CL\": \"Energy\",\n",
|
||
" \"NG\": \"Energy\",\n",
|
||
" \"HO\": \"Energy\",\n",
|
||
" \"RB\": \"Energy\",\n",
|
||
" \"GC\": \"Metals\",\n",
|
||
" \"SI\": \"Metals\",\n",
|
||
" \"HG\": \"Metals\",\n",
|
||
" \"PL\": \"Metals\",\n",
|
||
" \"6E\": \"FX\",\n",
|
||
" \"6J\": \"FX\",\n",
|
||
" \"6B\": \"FX\",\n",
|
||
" \"6A\": \"FX\",\n",
|
||
" \"6C\": \"FX\",\n",
|
||
" \"6S\": \"FX\",\n",
|
||
" \"ZC\": \"Grains\",\n",
|
||
" \"ZS\": \"Grains\",\n",
|
||
" \"ZW\": \"Grains\",\n",
|
||
" \"ZM\": \"Grains\",\n",
|
||
" \"ZL\": \"Grains\",\n",
|
||
" \"LE\": \"Livestock\",\n",
|
||
" \"HE\": \"Livestock\",\n",
|
||
" \"GF\": \"Livestock\",\n",
|
||
"}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "7dcc7f2b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.784842Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.784729Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.815497Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.814607Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.032845,
|
||
"end_time": "2026-06-13T02:32:42.816135+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.783290+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Products by Asset Class:\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (7, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>asset_class</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>"FX"</td><td>6</td></tr><tr><td>"Grains"</td><td>5</td></tr><tr><td>"Energy"</td><td>4</td></tr><tr><td>"Equity Index"</td><td>4</td></tr><tr><td>"Metals"</td><td>4</td></tr><tr><td>"Rates"</td><td>4</td></tr><tr><td>"Livestock"</td><td>3</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (7, 2)\n",
|
||
"┌──────────────┬─────┐\n",
|
||
"│ asset_class ┆ len │\n",
|
||
"│ --- ┆ --- │\n",
|
||
"│ str ┆ u32 │\n",
|
||
"╞══════════════╪═════╡\n",
|
||
"│ FX ┆ 6 │\n",
|
||
"│ Grains ┆ 5 │\n",
|
||
"│ Energy ┆ 4 │\n",
|
||
"│ Equity Index ┆ 4 │\n",
|
||
"│ Metals ┆ 4 │\n",
|
||
"│ Rates ┆ 4 │\n",
|
||
"│ Livestock ┆ 3 │\n",
|
||
"└──────────────┴─────┘"
|
||
]
|
||
},
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Count products by asset class\n",
|
||
"class_counts = (\n",
|
||
" pl.DataFrame({\"product\": products})\n",
|
||
" .with_columns(asset_class=pl.col(\"product\").replace(ASSET_CLASS_MAP))\n",
|
||
" .group_by(\"asset_class\")\n",
|
||
" .len()\n",
|
||
" .sort([\"len\", \"asset_class\"], descending=[True, False])\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"Products by Asset Class:\")\n",
|
||
"class_counts"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "07175acd",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001659,
|
||
"end_time": "2026-06-13T02:32:42.820076+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.818417+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 2. Data Structure Example: E-mini S&P 500 (ES)\n",
|
||
"\n",
|
||
"### Futures Key Hierarchy\n",
|
||
"\n",
|
||
"| Level | Example | Description |\n",
|
||
"|-------|---------|-------------|\n",
|
||
"| **Product** | ES | The underlying (E-mini S&P 500) |\n",
|
||
"| **Contract** | ESH4 | Specific expiration (March 2024) |\n",
|
||
"| **Continuous** | c0, c1 | Front month, first deferred |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "d61969a4",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.823125Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.823038Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.831229Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.830764Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.010089,
|
||
"end_time": "2026-06-13T02:32:42.831567+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.821478+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== ES Individual Contracts ===\n",
|
||
"Shape: (19361, 10)\n",
|
||
"Columns: ['timestamp', 'rtype', 'publisher_id', 'instrument_id', 'open', 'high', 'low', 'close', 'volume', 'product']\n",
|
||
"Date range: 2016-01-03 00:00:00+00:00 to 2025-12-30 00:00:00+00:00\n",
|
||
"Unique contracts: 194\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"es_individual = load_cme_futures(products=[\"ES\"], continuous=False, frequency=\"hourly\")\n",
|
||
"\n",
|
||
"print(\"=== ES Individual Contracts ===\")\n",
|
||
"print(f\"Shape: {es_individual.shape}\")\n",
|
||
"print(f\"Columns: {es_individual.columns}\")\n",
|
||
"print(f\"Date range: {es_individual['timestamp'].min()} to {es_individual['timestamp'].max()}\")\n",
|
||
"print(f\"Unique contracts: {es_individual['instrument_id'].n_unique()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "a409f932",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.834329Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.834216Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.843754Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.843399Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.011434,
|
||
"end_time": "2026-06-13T02:32:42.844170+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.832736+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== ES Continuous Series (front month) ===\n",
|
||
"Shape: (89031, 13)\n",
|
||
"Date range: 2011-01-02 23:00:00+00:00 to 2025-12-30 23:00:00+00:00\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"es_continuous = load_cme_futures(products=[\"ES\"], tenors=[0], continuous=True, frequency=\"hourly\")\n",
|
||
"\n",
|
||
"print(\"=== ES Continuous Series (front month) ===\")\n",
|
||
"print(f\"Shape: {es_continuous.shape}\")\n",
|
||
"print(f\"Date range: {es_continuous['timestamp'].min()} to {es_continuous['timestamp'].max()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1c8de1ff",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001035,
|
||
"end_time": "2026-06-13T02:32:42.846393+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.845358+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"Each individual contract trades for a finite window before expiry. Aggregating\n",
|
||
"by `instrument_id` shows the rollover pattern — quarterly contracts overlap\n",
|
||
"during the roll period."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "3be70a7d",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.848957Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.848871Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.853358Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.852983Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.006217,
|
||
"end_time": "2026-06-13T02:32:42.853630+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.847413+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Total ES contracts: 194\n",
|
||
"Most recent 5 contracts:\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (5, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>instrument_id</th><th>first_trade</th><th>last_trade</th><th>total_volume</th><th>observations</th></tr><tr><td>u32</td><td>datetime[ns, UTC]</td><td>datetime[ns, UTC]</td><td>u64</td><td>u32</td></tr></thead><tbody><tr><td>42004904</td><td>2025-10-02 00:00:00 UTC</td><td>2025-12-29 00:00:00 UTC</td><td>104</td><td>7</td></tr><tr><td>42140860</td><td>2025-10-03 00:00:00 UTC</td><td>2025-10-03 00:00:00 UTC</td><td>1</td><td>1</td></tr><tr><td>42018017</td><td>2025-10-10 00:00:00 UTC</td><td>2025-12-30 00:00:00 UTC</td><td>3831</td><td>52</td></tr><tr><td>42008091</td><td>2025-10-23 00:00:00 UTC</td><td>2025-12-16 00:00:00 UTC</td><td>7</td><td>5</td></tr><tr><td>42016422</td><td>2025-10-23 00:00:00 UTC</td><td>2025-12-17 00:00:00 UTC</td><td>10</td><td>4</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (5, 5)\n",
|
||
"┌───────────────┬─────────────────────────┬─────────────────────────┬──────────────┬──────────────┐\n",
|
||
"│ instrument_id ┆ first_trade ┆ last_trade ┆ total_volume ┆ observations │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ u32 ┆ datetime[ns, UTC] ┆ datetime[ns, UTC] ┆ u64 ┆ u32 │\n",
|
||
"╞═══════════════╪═════════════════════════╪═════════════════════════╪══════════════╪══════════════╡\n",
|
||
"│ 42004904 ┆ 2025-10-02 00:00:00 UTC ┆ 2025-12-29 00:00:00 UTC ┆ 104 ┆ 7 │\n",
|
||
"│ 42140860 ┆ 2025-10-03 00:00:00 UTC ┆ 2025-10-03 00:00:00 UTC ┆ 1 ┆ 1 │\n",
|
||
"│ 42018017 ┆ 2025-10-10 00:00:00 UTC ┆ 2025-12-30 00:00:00 UTC ┆ 3831 ┆ 52 │\n",
|
||
"│ 42008091 ┆ 2025-10-23 00:00:00 UTC ┆ 2025-12-16 00:00:00 UTC ┆ 7 ┆ 5 │\n",
|
||
"│ 42016422 ┆ 2025-10-23 00:00:00 UTC ┆ 2025-12-17 00:00:00 UTC ┆ 10 ┆ 4 │\n",
|
||
"└───────────────┴─────────────────────────┴─────────────────────────┴──────────────┴──────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"contract_stats = (\n",
|
||
" es_individual.group_by(\"instrument_id\")\n",
|
||
" .agg(\n",
|
||
" pl.col(\"timestamp\").min().alias(\"first_trade\"),\n",
|
||
" pl.col(\"timestamp\").max().alias(\"last_trade\"),\n",
|
||
" pl.col(\"volume\").sum().alias(\"total_volume\"),\n",
|
||
" pl.len().alias(\"observations\"),\n",
|
||
" )\n",
|
||
" .sort(\"first_trade\")\n",
|
||
")\n",
|
||
"print(f\"Total ES contracts: {len(contract_stats)}\")\n",
|
||
"print(\"Most recent 5 contracts:\")\n",
|
||
"contract_stats.tail(5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "030744ec",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.001054,
|
||
"end_time": "2026-06-13T02:32:42.855801+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.854747+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3. Coverage Summary\n",
|
||
"\n",
|
||
"Check data availability across all products."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0be32d87",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.001041,
|
||
"end_time": "2026-06-13T02:32:42.857925+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.856884+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"Summarize per-product coverage by loading the front-month continuous series\n",
|
||
"(`tenor=0`) for every product and recording its row count and date range."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "5fdcffb8",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.860611Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.860534Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:42.863095Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:42.862662Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004347,
|
||
"end_time": "2026-06-13T02:32:42.863352+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.859005+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def get_product_coverage(product_list: list[str]) -> pl.DataFrame:\n",
|
||
" \"\"\"Summarize continuous series coverage for each product (front month).\"\"\"\n",
|
||
" summaries = []\n",
|
||
" for product in product_list:\n",
|
||
" df = load_cme_futures(products=[product], tenors=[0], continuous=True, frequency=\"hourly\")\n",
|
||
" summaries.append(\n",
|
||
" {\n",
|
||
" \"product\": product,\n",
|
||
" \"asset_class\": ASSET_CLASS_MAP[product],\n",
|
||
" \"rows\": len(df),\n",
|
||
" \"start_date\": str(df[\"timestamp\"].min())[:10],\n",
|
||
" \"end_date\": str(df[\"timestamp\"].max())[:10],\n",
|
||
" }\n",
|
||
" )\n",
|
||
" return pl.DataFrame(summaries)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "f8fa8a21",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:42.866298Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:42.866227Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:43.007188Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:43.006836Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.142974,
|
||
"end_time": "2026-06-13T02:32:43.007534+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:42.864560+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Products with data: 30 / 30\n",
|
||
"Coverage by asset class:\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (7, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>asset_class</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>"FX"</td><td>6</td></tr><tr><td>"Grains"</td><td>5</td></tr><tr><td>"Energy"</td><td>4</td></tr><tr><td>"Equity Index"</td><td>4</td></tr><tr><td>"Metals"</td><td>4</td></tr><tr><td>"Rates"</td><td>4</td></tr><tr><td>"Livestock"</td><td>3</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (7, 2)\n",
|
||
"┌──────────────┬─────┐\n",
|
||
"│ asset_class ┆ len │\n",
|
||
"│ --- ┆ --- │\n",
|
||
"│ str ┆ u32 │\n",
|
||
"╞══════════════╪═════╡\n",
|
||
"│ FX ┆ 6 │\n",
|
||
"│ Grains ┆ 5 │\n",
|
||
"│ Energy ┆ 4 │\n",
|
||
"│ Equity Index ┆ 4 │\n",
|
||
"│ Metals ┆ 4 │\n",
|
||
"│ Rates ┆ 4 │\n",
|
||
"│ Livestock ┆ 3 │\n",
|
||
"└──────────────┴─────┘"
|
||
]
|
||
},
|
||
"execution_count": 10,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"coverage = get_product_coverage(products)\n",
|
||
"print(f\"Products with data: {len(coverage)} / {len(products)}\")\n",
|
||
"print(\"Coverage by asset class:\")\n",
|
||
"coverage.group_by(\"asset_class\").len().sort([\"len\", \"asset_class\"], descending=[True, False])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "54663965",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001516,
|
||
"end_time": "2026-06-13T02:32:43.010480+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:43.008964+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"A handful of representative products from each asset-class bucket:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "c4bc6406",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:43.013592Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:43.013498Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:43.016213Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:43.015939Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004893,
|
||
"end_time": "2026-06-13T02:32:43.016650+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:43.011757+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (7, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>product</th><th>asset_class</th><th>rows</th><th>start_date</th><th>end_date</th></tr><tr><td>str</td><td>str</td><td>i64</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>"6E"</td><td>"FX"</td><td>88326</td><td>"2011-01-02"</td><td>"2025-12-30"</td></tr><tr><td>"CL"</td><td>"Energy"</td><td>89383</td><td>"2011-01-02"</td><td>"2025-12-30"</td></tr><tr><td>"ES"</td><td>"Equity Index"</td><td>89031</td><td>"2011-01-02"</td><td>"2025-12-30"</td></tr><tr><td>"GC"</td><td>"Metals"</td><td>89424</td><td>"2011-01-02"</td><td>"2025-12-30"</td></tr><tr><td>"NQ"</td><td>"Equity Index"</td><td>89049</td><td>"2011-01-02"</td><td>"2025-12-30"</td></tr><tr><td>"ZC"</td><td>"Grains"</td><td>71426</td><td>"2011-01-03"</td><td>"2025-12-30"</td></tr><tr><td>"ZN"</td><td>"Rates"</td><td>88235</td><td>"2011-01-02"</td><td>"2025-12-30"</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (7, 5)\n",
|
||
"┌─────────┬──────────────┬───────┬────────────┬────────────┐\n",
|
||
"│ product ┆ asset_class ┆ rows ┆ start_date ┆ end_date │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ str ┆ i64 ┆ str ┆ str │\n",
|
||
"╞═════════╪══════════════╪═══════╪════════════╪════════════╡\n",
|
||
"│ 6E ┆ FX ┆ 88326 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
|
||
"│ CL ┆ Energy ┆ 89383 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
|
||
"│ ES ┆ Equity Index ┆ 89031 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
|
||
"│ GC ┆ Metals ┆ 89424 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
|
||
"│ NQ ┆ Equity Index ┆ 89049 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
|
||
"│ ZC ┆ Grains ┆ 71426 ┆ 2011-01-03 ┆ 2025-12-30 │\n",
|
||
"│ ZN ┆ Rates ┆ 88235 ┆ 2011-01-02 ┆ 2025-12-30 │\n",
|
||
"└─────────┴──────────────┴───────┴────────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"key_products = [\"ES\", \"NQ\", \"CL\", \"GC\", \"ZN\", \"6E\", \"ZC\"]\n",
|
||
"coverage.filter(pl.col(\"product\").is_in(key_products))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "880232f1",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001243,
|
||
"end_time": "2026-06-13T02:32:43.019253+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:43.018010+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 4. Data Quality"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "9c22a3a9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:43.022480Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:43.022386Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:43.028338Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:43.028002Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.008384,
|
||
"end_time": "2026-06-13T02:32:43.028857+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:43.020473+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== OHLC Invariants (ES Continuous) ===\n",
|
||
" [OK] high_gte_low: 100.00%\n",
|
||
" [OK] high_gte_open: 100.00%\n",
|
||
" [OK] high_gte_close: 100.00%\n",
|
||
" [OK] low_lte_open: 100.00%\n",
|
||
" [OK] low_lte_close: 100.00%\n",
|
||
" [OK] volume_non_negative: 100.00%\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"invariants = check_ohlc_invariants(es_continuous)\n",
|
||
"print(\"=== OHLC Invariants (ES Continuous) ===\")\n",
|
||
"for row in invariants.iter_rows(named=True):\n",
|
||
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
|
||
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "baee6bf4",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001275,
|
||
"end_time": "2026-06-13T02:32:43.031631+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:43.030356+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"1. **30 products across 7 asset-class buckets**: FX (6), Grains (5),\n",
|
||
" Energy / Equity Index / Metals / Rates (4 each), Livestock (3).\n",
|
||
"2. **Hierarchy**: each product has 100+ individual contracts (194 for ES) and\n",
|
||
" one or more continuous series; downstream notebooks operate on the\n",
|
||
" continuous front month unless they specifically need contract-level data.\n",
|
||
"3. **Hourly granularity**, full coverage 2011-01-02 through 2025-12-30 for\n",
|
||
" products with the longest history. ES individual contract data starts\n",
|
||
" later (2016) because earlier contracts have already rolled off.\n",
|
||
"4. **Canonical schema**: `timestamp` for time and `product` for entity (CME's\n",
|
||
" contract identity is non-trivial — see also `instrument_id` for individual\n",
|
||
" contracts).\n",
|
||
"5. **OHLC invariants hold at 100% for ES continuous** — all six checks pass\n",
|
||
" on every observation.\n",
|
||
"\n",
|
||
"### Next Steps\n",
|
||
"\n",
|
||
"- **`05_futures_session_aggregation`**: Aligning hourly bars to CME trading\n",
|
||
" sessions.\n",
|
||
"- **`06_futures_continuous`**: Roll detection and the three adjustment\n",
|
||
" methods (ratio, difference, calendar).\n",
|
||
"- **Chapter 8**: Feature engineering on term structure and roll yield."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"jupytext": {
|
||
"cell_metadata_filter": "tags,-all"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 5.126337,
|
||
"end_time": "2026-06-13T02:32:46.012766+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "02_financial_data_universe/04_cme_futures_eda.ipynb",
|
||
"output_path": "02_financial_data_universe/04_cme_futures_eda.ipynb",
|
||
"parameters": {},
|
||
"start_time": "2026-06-13T02:32:40.886429+00:00",
|
||
"version": "2.7.0"
|
||
},
|
||
"ml4t_provenance": {
|
||
"source_py_blob": "526df0bb2e3ca5cf445b7e70dafafc50f7895c6b",
|
||
"executed_at": "2026-06-13T02:32:46.227406+00:00",
|
||
"executor": "cpu-local",
|
||
"production": true,
|
||
"parameters": {},
|
||
"notes": "executed-state finalization batch (2026-06-13 run)"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|