{ "cells": [ { "cell_type": "markdown", "id": "fb07336e", "metadata": { "papermill": { "duration": 0.006471, "end_time": "2026-06-13T02:32:41.536804+00:00", "exception": false, "start_time": "2026-06-13T02:32:41.530333+00:00", "status": "completed" } }, "source": [ "# CME Futures — Exploratory Data Analysis\n", "\n", "**Docker image**: `ml4t`\n", "\n", "**Purpose**: Profile the 30-product CME futures dataset (Databento, hourly,\n", "2011–2025) and surface the contract / continuous structure that downstream\n", "notebooks rely on.\n", "\n", "**Learning objectives**:\n", "\n", "- Understand the futures data hierarchy: product → contract → continuous series.\n", "- Load individual contracts and continuous (rolled) series via\n", " `load_cme_futures` and inspect the canonical `timestamp` / `product`\n", " schema.\n", "- Summarize per-product coverage and group products by asset class.\n", "- Verify OHLC invariants on a representative continuous series.\n", "\n", "**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — Futures).\n", "\n", "**Prerequisites**: `data` package on `PYTHONPATH`; CME parquet present at\n", "`ML4T_DATA_PATH/futures/`. Run `python data/futures/market/download.py` if\n", "missing (Databento API key required)." ] }, { "cell_type": "code", "execution_count": 1, "id": "1830247e", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:41.541085Z", "iopub.status.busy": "2026-06-13T02:32:41.540762Z", "iopub.status.idle": "2026-06-13T02:32:42.760904Z", "shell.execute_reply": "2026-06-13T02:32:42.760453Z" }, "papermill": { "duration": 1.222807, "end_time": "2026-06-13T02:32:42.761451+00:00", "exception": false, "start_time": "2026-06-13T02:32:41.538644+00:00", "status": "completed" } }, "outputs": [], "source": [ "\"\"\"CME Futures — Exploratory data analysis of the futures universe.\"\"\"\n", "\n", "import polars as pl\n", "\n", "from data import list_cme_products, load_cme_futures\n", "from utils.data_quality import check_ohlc_invariants" ] }, { "cell_type": "code", "execution_count": 2, "id": "bdf60381", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.765255Z", "iopub.status.busy": "2026-06-13T02:32:42.765168Z", "iopub.status.idle": "2026-06-13T02:32:42.767341Z", "shell.execute_reply": "2026-06-13T02:32:42.766636Z" }, "papermill": { "duration": 0.005182, "end_time": "2026-06-13T02:32:42.767745+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.762563+00:00", "status": "completed" }, "tags": [ "parameters" ] }, "outputs": [], "source": [ "# Production defaults — Papermill injects overrides for CI\n", "MAX_SYMBOLS = 0 # 0 = all" ] }, { "cell_type": "markdown", "id": "70f4bc2e", "metadata": { "papermill": { "duration": 0.000844, "end_time": "2026-06-13T02:32:42.769552+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.768708+00:00", "status": "completed" } }, "source": [ "## 1. Configuration and Data Discovery\n", "\n", "The futures data uses a Hive-partitioned structure for efficient queries:\n", "- `futures/continuous/product={PRODUCT}/`: Volume-rolled continuous contracts (hourly)\n", "- `futures/individual/{PRODUCT}/data.parquet`: Individual contract price data\n", "\n", "We use `load_cme_futures()` for proper data loading with partition pruning." ] }, { "cell_type": "code", "execution_count": 3, "id": "246bdafd", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.771820Z", "iopub.status.busy": "2026-06-13T02:32:42.771760Z", "iopub.status.idle": "2026-06-13T02:32:42.774699Z", "shell.execute_reply": "2026-06-13T02:32:42.774121Z" }, "papermill": { "duration": 0.004585, "end_time": "2026-06-13T02:32:42.775020+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.770435+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=== Futures Universe ===\n", "Available products: 30\n", "\n", "Products: 6A, 6B, 6C, 6E, 6J, 6S, CL, ES, GC, GF, HE, HG, HO, LE, NG, NQ, PL, RB, RTY, SI, YM, ZB, ZC, ZF, ZL, ZM, ZN, ZS, ZT, ZW\n" ] } ], "source": [ "# Discover available products via the CME loader\n", "products = list_cme_products()\n", "\n", "print(\"=== Futures Universe ===\")\n", "print(f\"Available products: {len(products)}\")\n", "print(f\"\\nProducts: {', '.join(products)}\")" ] }, { "cell_type": "markdown", "id": "893ba4a8", "metadata": { "papermill": { "duration": 0.000878, "end_time": "2026-06-13T02:32:42.776865+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.775987+00:00", "status": "completed" } }, "source": [ "Map each product to a coarse asset-class bucket. The mapping covers every\n", "product in the dataset; downstream chapters use the same bucket labels for\n", "universe-construction and risk reporting." ] }, { "cell_type": "code", "execution_count": 4, "id": "19ed20cd", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.779264Z", "iopub.status.busy": "2026-06-13T02:32:42.779153Z", "iopub.status.idle": "2026-06-13T02:32:42.781984Z", "shell.execute_reply": "2026-06-13T02:32:42.781419Z" }, "papermill": { "duration": 0.004493, "end_time": "2026-06-13T02:32:42.782271+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.777778+00:00", "status": "completed" } }, "outputs": [], "source": [ "ASSET_CLASS_MAP = {\n", " \"ES\": \"Equity Index\",\n", " \"NQ\": \"Equity Index\",\n", " \"YM\": \"Equity Index\",\n", " \"RTY\": \"Equity Index\",\n", " \"ZN\": \"Rates\",\n", " \"ZB\": \"Rates\",\n", " \"ZF\": \"Rates\",\n", " \"ZT\": \"Rates\",\n", " \"CL\": \"Energy\",\n", " \"NG\": \"Energy\",\n", " \"HO\": \"Energy\",\n", " \"RB\": \"Energy\",\n", " \"GC\": \"Metals\",\n", " \"SI\": \"Metals\",\n", " \"HG\": \"Metals\",\n", " \"PL\": \"Metals\",\n", " \"6E\": \"FX\",\n", " \"6J\": \"FX\",\n", " \"6B\": \"FX\",\n", " \"6A\": \"FX\",\n", " \"6C\": \"FX\",\n", " \"6S\": \"FX\",\n", " \"ZC\": \"Grains\",\n", " \"ZS\": \"Grains\",\n", " \"ZW\": \"Grains\",\n", " \"ZM\": \"Grains\",\n", " \"ZL\": \"Grains\",\n", " \"LE\": \"Livestock\",\n", " \"HE\": \"Livestock\",\n", " \"GF\": \"Livestock\",\n", "}" ] }, { "cell_type": "code", "execution_count": 5, "id": "7dcc7f2b", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.784842Z", "iopub.status.busy": "2026-06-13T02:32:42.784729Z", "iopub.status.idle": "2026-06-13T02:32:42.815497Z", "shell.execute_reply": "2026-06-13T02:32:42.814607Z" }, "papermill": { "duration": 0.032845, "end_time": "2026-06-13T02:32:42.816135+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.783290+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Products by Asset Class:\n" ] }, { "data": { "text/html": [ "
\n", "shape: (7, 2)
asset_classlen
stru32
"FX"6
"Grains"5
"Energy"4
"Equity Index"4
"Metals"4
"Rates"4
"Livestock"3
" ], "text/plain": [ "shape: (7, 2)\n", "┌──────────────┬─────┐\n", "│ asset_class ┆ len │\n", "│ --- ┆ --- │\n", "│ str ┆ u32 │\n", "╞══════════════╪═════╡\n", "│ FX ┆ 6 │\n", "│ Grains ┆ 5 │\n", "│ Energy ┆ 4 │\n", "│ Equity Index ┆ 4 │\n", "│ Metals ┆ 4 │\n", "│ Rates ┆ 4 │\n", "│ Livestock ┆ 3 │\n", "└──────────────┴─────┘" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Count products by asset class\n", "class_counts = (\n", " pl.DataFrame({\"product\": products})\n", " .with_columns(asset_class=pl.col(\"product\").replace(ASSET_CLASS_MAP))\n", " .group_by(\"asset_class\")\n", " .len()\n", " .sort([\"len\", \"asset_class\"], descending=[True, False])\n", ")\n", "\n", "print(\"Products by Asset Class:\")\n", "class_counts" ] }, { "cell_type": "markdown", "id": "07175acd", "metadata": { "papermill": { "duration": 0.001659, "end_time": "2026-06-13T02:32:42.820076+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.818417+00:00", "status": "completed" } }, "source": [ "## 2. Data Structure Example: E-mini S&P 500 (ES)\n", "\n", "### Futures Key Hierarchy\n", "\n", "| Level | Example | Description |\n", "|-------|---------|-------------|\n", "| **Product** | ES | The underlying (E-mini S&P 500) |\n", "| **Contract** | ESH4 | Specific expiration (March 2024) |\n", "| **Continuous** | c0, c1 | Front month, first deferred |" ] }, { "cell_type": "code", "execution_count": 6, "id": "d61969a4", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.823125Z", "iopub.status.busy": "2026-06-13T02:32:42.823038Z", "iopub.status.idle": "2026-06-13T02:32:42.831229Z", "shell.execute_reply": "2026-06-13T02:32:42.830764Z" }, "papermill": { "duration": 0.010089, "end_time": "2026-06-13T02:32:42.831567+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.821478+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=== ES Individual Contracts ===\n", "Shape: (19361, 10)\n", "Columns: ['timestamp', 'rtype', 'publisher_id', 'instrument_id', 'open', 'high', 'low', 'close', 'volume', 'product']\n", "Date range: 2016-01-03 00:00:00+00:00 to 2025-12-30 00:00:00+00:00\n", "Unique contracts: 194\n" ] } ], "source": [ "es_individual = load_cme_futures(products=[\"ES\"], continuous=False, frequency=\"hourly\")\n", "\n", "print(\"=== ES Individual Contracts ===\")\n", "print(f\"Shape: {es_individual.shape}\")\n", "print(f\"Columns: {es_individual.columns}\")\n", "print(f\"Date range: {es_individual['timestamp'].min()} to {es_individual['timestamp'].max()}\")\n", "print(f\"Unique contracts: {es_individual['instrument_id'].n_unique()}\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "a409f932", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.834329Z", "iopub.status.busy": "2026-06-13T02:32:42.834216Z", "iopub.status.idle": "2026-06-13T02:32:42.843754Z", "shell.execute_reply": "2026-06-13T02:32:42.843399Z" }, "papermill": { "duration": 0.011434, "end_time": "2026-06-13T02:32:42.844170+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.832736+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=== ES Continuous Series (front month) ===\n", "Shape: (89031, 13)\n", "Date range: 2011-01-02 23:00:00+00:00 to 2025-12-30 23:00:00+00:00\n" ] } ], "source": [ "es_continuous = load_cme_futures(products=[\"ES\"], tenors=[0], continuous=True, frequency=\"hourly\")\n", "\n", "print(\"=== ES Continuous Series (front month) ===\")\n", "print(f\"Shape: {es_continuous.shape}\")\n", "print(f\"Date range: {es_continuous['timestamp'].min()} to {es_continuous['timestamp'].max()}\")" ] }, { "cell_type": "markdown", "id": "1c8de1ff", "metadata": { "papermill": { "duration": 0.001035, "end_time": "2026-06-13T02:32:42.846393+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.845358+00:00", "status": "completed" } }, "source": [ "Each individual contract trades for a finite window before expiry. Aggregating\n", "by `instrument_id` shows the rollover pattern — quarterly contracts overlap\n", "during the roll period." ] }, { "cell_type": "code", "execution_count": 8, "id": "3be70a7d", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.848957Z", "iopub.status.busy": "2026-06-13T02:32:42.848871Z", "iopub.status.idle": "2026-06-13T02:32:42.853358Z", "shell.execute_reply": "2026-06-13T02:32:42.852983Z" }, "papermill": { "duration": 0.006217, "end_time": "2026-06-13T02:32:42.853630+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.847413+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total ES contracts: 194\n", "Most recent 5 contracts:\n" ] }, { "data": { "text/html": [ "
\n", "shape: (5, 5)
instrument_idfirst_tradelast_tradetotal_volumeobservations
u32datetime[ns, UTC]datetime[ns, UTC]u64u32
420049042025-10-02 00:00:00 UTC2025-12-29 00:00:00 UTC1047
421408602025-10-03 00:00:00 UTC2025-10-03 00:00:00 UTC11
420180172025-10-10 00:00:00 UTC2025-12-30 00:00:00 UTC383152
420080912025-10-23 00:00:00 UTC2025-12-16 00:00:00 UTC75
420164222025-10-23 00:00:00 UTC2025-12-17 00:00:00 UTC104
" ], "text/plain": [ "shape: (5, 5)\n", "┌───────────────┬─────────────────────────┬─────────────────────────┬──────────────┬──────────────┐\n", "│ instrument_id ┆ first_trade ┆ last_trade ┆ total_volume ┆ observations │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ u32 ┆ datetime[ns, UTC] ┆ datetime[ns, UTC] ┆ u64 ┆ u32 │\n", "╞═══════════════╪═════════════════════════╪═════════════════════════╪══════════════╪══════════════╡\n", "│ 42004904 ┆ 2025-10-02 00:00:00 UTC ┆ 2025-12-29 00:00:00 UTC ┆ 104 ┆ 7 │\n", "│ 42140860 ┆ 2025-10-03 00:00:00 UTC ┆ 2025-10-03 00:00:00 UTC ┆ 1 ┆ 1 │\n", "│ 42018017 ┆ 2025-10-10 00:00:00 UTC ┆ 2025-12-30 00:00:00 UTC ┆ 3831 ┆ 52 │\n", "│ 42008091 ┆ 2025-10-23 00:00:00 UTC ┆ 2025-12-16 00:00:00 UTC ┆ 7 ┆ 5 │\n", "│ 42016422 ┆ 2025-10-23 00:00:00 UTC ┆ 2025-12-17 00:00:00 UTC ┆ 10 ┆ 4 │\n", "└───────────────┴─────────────────────────┴─────────────────────────┴──────────────┴──────────────┘" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "contract_stats = (\n", " es_individual.group_by(\"instrument_id\")\n", " .agg(\n", " pl.col(\"timestamp\").min().alias(\"first_trade\"),\n", " pl.col(\"timestamp\").max().alias(\"last_trade\"),\n", " pl.col(\"volume\").sum().alias(\"total_volume\"),\n", " pl.len().alias(\"observations\"),\n", " )\n", " .sort(\"first_trade\")\n", ")\n", "print(f\"Total ES contracts: {len(contract_stats)}\")\n", "print(\"Most recent 5 contracts:\")\n", "contract_stats.tail(5)" ] }, { "cell_type": "markdown", "id": "030744ec", "metadata": { "lines_to_next_cell": 2, "papermill": { "duration": 0.001054, "end_time": "2026-06-13T02:32:42.855801+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.854747+00:00", "status": "completed" } }, "source": [ "## 3. Coverage Summary\n", "\n", "Check data availability across all products." ] }, { "cell_type": "markdown", "id": "0be32d87", "metadata": { "lines_to_next_cell": 2, "papermill": { "duration": 0.001041, "end_time": "2026-06-13T02:32:42.857925+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.856884+00:00", "status": "completed" } }, "source": [ "Summarize per-product coverage by loading the front-month continuous series\n", "(`tenor=0`) for every product and recording its row count and date range." ] }, { "cell_type": "code", "execution_count": 9, "id": "5fdcffb8", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.860611Z", "iopub.status.busy": "2026-06-13T02:32:42.860534Z", "iopub.status.idle": "2026-06-13T02:32:42.863095Z", "shell.execute_reply": "2026-06-13T02:32:42.862662Z" }, "papermill": { "duration": 0.004347, "end_time": "2026-06-13T02:32:42.863352+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.859005+00:00", "status": "completed" } }, "outputs": [], "source": [ "def get_product_coverage(product_list: list[str]) -> pl.DataFrame:\n", " \"\"\"Summarize continuous series coverage for each product (front month).\"\"\"\n", " summaries = []\n", " for product in product_list:\n", " df = load_cme_futures(products=[product], tenors=[0], continuous=True, frequency=\"hourly\")\n", " summaries.append(\n", " {\n", " \"product\": product,\n", " \"asset_class\": ASSET_CLASS_MAP[product],\n", " \"rows\": len(df),\n", " \"start_date\": str(df[\"timestamp\"].min())[:10],\n", " \"end_date\": str(df[\"timestamp\"].max())[:10],\n", " }\n", " )\n", " return pl.DataFrame(summaries)" ] }, { "cell_type": "code", "execution_count": 10, "id": "f8fa8a21", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:42.866298Z", "iopub.status.busy": "2026-06-13T02:32:42.866227Z", "iopub.status.idle": "2026-06-13T02:32:43.007188Z", "shell.execute_reply": "2026-06-13T02:32:43.006836Z" }, "papermill": { "duration": 0.142974, "end_time": "2026-06-13T02:32:43.007534+00:00", "exception": false, "start_time": "2026-06-13T02:32:42.864560+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Products with data: 30 / 30\n", "Coverage by asset class:\n" ] }, { "data": { "text/html": [ "
\n", "shape: (7, 2)
asset_classlen
stru32
"FX"6
"Grains"5
"Energy"4
"Equity Index"4
"Metals"4
"Rates"4
"Livestock"3
" ], "text/plain": [ "shape: (7, 2)\n", "┌──────────────┬─────┐\n", "│ asset_class ┆ len │\n", "│ --- ┆ --- │\n", "│ str ┆ u32 │\n", "╞══════════════╪═════╡\n", "│ FX ┆ 6 │\n", "│ Grains ┆ 5 │\n", "│ Energy ┆ 4 │\n", "│ Equity Index ┆ 4 │\n", "│ Metals ┆ 4 │\n", "│ Rates ┆ 4 │\n", "│ Livestock ┆ 3 │\n", "└──────────────┴─────┘" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "coverage = get_product_coverage(products)\n", "print(f\"Products with data: {len(coverage)} / {len(products)}\")\n", "print(\"Coverage by asset class:\")\n", "coverage.group_by(\"asset_class\").len().sort([\"len\", \"asset_class\"], descending=[True, False])" ] }, { "cell_type": "markdown", "id": "54663965", "metadata": { "papermill": { "duration": 0.001516, "end_time": "2026-06-13T02:32:43.010480+00:00", "exception": false, "start_time": "2026-06-13T02:32:43.008964+00:00", "status": "completed" } }, "source": [ "A handful of representative products from each asset-class bucket:" ] }, { "cell_type": "code", "execution_count": 11, "id": "c4bc6406", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:43.013592Z", "iopub.status.busy": "2026-06-13T02:32:43.013498Z", "iopub.status.idle": "2026-06-13T02:32:43.016213Z", "shell.execute_reply": "2026-06-13T02:32:43.015939Z" }, "papermill": { "duration": 0.004893, "end_time": "2026-06-13T02:32:43.016650+00:00", "exception": false, "start_time": "2026-06-13T02:32:43.011757+00:00", "status": "completed" } }, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (7, 5)
productasset_classrowsstart_dateend_date
strstri64strstr
"6E""FX"88326"2011-01-02""2025-12-30"
"CL""Energy"89383"2011-01-02""2025-12-30"
"ES""Equity Index"89031"2011-01-02""2025-12-30"
"GC""Metals"89424"2011-01-02""2025-12-30"
"NQ""Equity Index"89049"2011-01-02""2025-12-30"
"ZC""Grains"71426"2011-01-03""2025-12-30"
"ZN""Rates"88235"2011-01-02""2025-12-30"
" ], "text/plain": [ "shape: (7, 5)\n", "┌─────────┬──────────────┬───────┬────────────┬────────────┐\n", "│ product ┆ asset_class ┆ rows ┆ start_date ┆ end_date │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ str ┆ str ┆ i64 ┆ str ┆ str │\n", "╞═════════╪══════════════╪═══════╪════════════╪════════════╡\n", "│ 6E ┆ FX ┆ 88326 ┆ 2011-01-02 ┆ 2025-12-30 │\n", "│ CL ┆ Energy ┆ 89383 ┆ 2011-01-02 ┆ 2025-12-30 │\n", "│ ES ┆ Equity Index ┆ 89031 ┆ 2011-01-02 ┆ 2025-12-30 │\n", "│ GC ┆ Metals ┆ 89424 ┆ 2011-01-02 ┆ 2025-12-30 │\n", "│ NQ ┆ Equity Index ┆ 89049 ┆ 2011-01-02 ┆ 2025-12-30 │\n", "│ ZC ┆ Grains ┆ 71426 ┆ 2011-01-03 ┆ 2025-12-30 │\n", "│ ZN ┆ Rates ┆ 88235 ┆ 2011-01-02 ┆ 2025-12-30 │\n", "└─────────┴──────────────┴───────┴────────────┴────────────┘" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "key_products = [\"ES\", \"NQ\", \"CL\", \"GC\", \"ZN\", \"6E\", \"ZC\"]\n", "coverage.filter(pl.col(\"product\").is_in(key_products))" ] }, { "cell_type": "markdown", "id": "880232f1", "metadata": { "papermill": { "duration": 0.001243, "end_time": "2026-06-13T02:32:43.019253+00:00", "exception": false, "start_time": "2026-06-13T02:32:43.018010+00:00", "status": "completed" } }, "source": [ "## 4. Data Quality" ] }, { "cell_type": "code", "execution_count": 12, "id": "9c22a3a9", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:32:43.022480Z", "iopub.status.busy": "2026-06-13T02:32:43.022386Z", "iopub.status.idle": "2026-06-13T02:32:43.028338Z", "shell.execute_reply": "2026-06-13T02:32:43.028002Z" }, "papermill": { "duration": 0.008384, "end_time": "2026-06-13T02:32:43.028857+00:00", "exception": false, "start_time": "2026-06-13T02:32:43.020473+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=== OHLC Invariants (ES Continuous) ===\n", " [OK] high_gte_low: 100.00%\n", " [OK] high_gte_open: 100.00%\n", " [OK] high_gte_close: 100.00%\n", " [OK] low_lte_open: 100.00%\n", " [OK] low_lte_close: 100.00%\n", " [OK] volume_non_negative: 100.00%\n" ] } ], "source": [ "invariants = check_ohlc_invariants(es_continuous)\n", "print(\"=== OHLC Invariants (ES Continuous) ===\")\n", "for row in invariants.iter_rows(named=True):\n", " status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n", " print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")" ] }, { "cell_type": "markdown", "id": "baee6bf4", "metadata": { "papermill": { "duration": 0.001275, "end_time": "2026-06-13T02:32:43.031631+00:00", "exception": false, "start_time": "2026-06-13T02:32:43.030356+00:00", "status": "completed" } }, "source": [ "## Key Takeaways\n", "\n", "1. **30 products across 7 asset-class buckets**: FX (6), Grains (5),\n", " Energy / Equity Index / Metals / Rates (4 each), Livestock (3).\n", "2. **Hierarchy**: each product has 100+ individual contracts (194 for ES) and\n", " one or more continuous series; downstream notebooks operate on the\n", " continuous front month unless they specifically need contract-level data.\n", "3. **Hourly granularity**, full coverage 2011-01-02 through 2025-12-30 for\n", " products with the longest history. ES individual contract data starts\n", " later (2016) because earlier contracts have already rolled off.\n", "4. **Canonical schema**: `timestamp` for time and `product` for entity (CME's\n", " contract identity is non-trivial — see also `instrument_id` for individual\n", " contracts).\n", "5. **OHLC invariants hold at 100% for ES continuous** — all six checks pass\n", " on every observation.\n", "\n", "### Next Steps\n", "\n", "- **`05_futures_session_aggregation`**: Aligning hourly bars to CME trading\n", " sessions.\n", "- **`06_futures_continuous`**: Roll detection and the three adjustment\n", " methods (ratio, difference, calendar).\n", "- **Chapter 8**: Feature engineering on term structure and roll yield." ] } ], "metadata": { "jupytext": { "cell_metadata_filter": "tags,-all" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.3" }, "papermill": { "default_parameters": {}, "duration": 5.126337, "end_time": "2026-06-13T02:32:46.012766+00:00", "environment_variables": {}, "exception": null, "input_path": "02_financial_data_universe/04_cme_futures_eda.ipynb", "output_path": "02_financial_data_universe/04_cme_futures_eda.ipynb", "parameters": {}, "start_time": "2026-06-13T02:32:40.886429+00:00", "version": "2.7.0" }, "ml4t_provenance": { "source_py_blob": "526df0bb2e3ca5cf445b7e70dafafc50f7895c6b", "executed_at": "2026-06-13T02:32:46.227406+00:00", "executor": "cpu-local", "production": true, "parameters": {}, "notes": "executed-state finalization batch (2026-06-13 run)" } }, "nbformat": 4, "nbformat_minor": 5 }